mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 20:46:28 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain reports and project guest transcripts into intake tasks. > - Chat maintenance repeatedly scans recent calls even when no report is due. > - Empty scans keep the tenant database active. > - Existing session and report rows already contain the work and its deadlines. > - This pull request moves report refresh to the shared coordinator and removes the final voice maintenance lane. > - Idle voice configuration and old call history no longer cause report scans. ## Linked Issues or Issue Description Refs #17583, #17580, #17578, #17576. This is slice 25 of the recurring-server sleep stack. It now follows merged #17583 and targets master. **What existing behavior does this improve?** Voice call-report refresh and guest transcript projection. **Current behavior** The chat interval scans terminal calls for reports. It repeats the scan even when there are no eligible calls or all saved deadlines are in the future. **Proposed behavior** Committed session transitions and signed callbacks wake a scoped worker. Startup reconstructs existing rows. Later refreshes use saved deadlines. The worker becomes quiet after all eligible windows expire. **Reason and benefit** Stop empty database scans without losing late transcript and billing updates. Keep the existing 24-hour refresh policy, provider response ordering, and guest-task trust restrictions. ## What Changed - Add a voice-owned report worker using the existing scoped deadline helper. No queue table or field is added; the existing session table receives a report-window index. - Save each attempt and its retry before provider I/O. Compare that ownership on settlement so an older response cannot overwrite a later callback or claim. - Preserve available reports when a response is incomplete or stale. Preserve their finite refresh window. - Route voice-change and reply call-ended writes through transaction-aware report signals. - Preserve guest transcript locks and use the observed document revision when updating an existing transcript. - Register the worker in app startup. Remove voice reconciliation from the chat interval. Keep explicit one-shot diagnostics. - Document producer scope, lifecycle behavior, tests, and external wake limits. ## Verification Published head: `758cfe15c3e6283d38ba512c8416ccce4d200862`. The follow-up after `c28de398` changes only two documentation files; the verified production, migration and test source is unchanged. It normally merges landed retention/master `980775e0e956eecdc28c73c9b4aca0407ad45e1a`. The 20 owned paths remain the complete slice, including officially regenerated migration 0397, its snapshot and required pruning of snapshot 0392. The non-database reviewed patch was preserved during ancestry integration; the only subsequent source edit makes one existing report test’s provider responses call-specific. All production report behavior and unrelated master changes are preserved. No shared worker, workflow or lockfile change belongs to reports. All 58 current-head checks are now clean, including the previously failing approval shard; exact-head Greptile is 5/5 with no new actionable issues. Ponytail’s current turn stopped without a verdict, an authorized nonblocking state. The prior conditional documentation requirement was fixed; old CI failures remain recorded below. - **Prior-head evidence at `9bf20467df`: 303 distinct targeted cases passed**, not rerun wholesale for this integration: all 132 voice integration cases; 170 cases across 11 complete report, retention, cleanup, recovery, company-removal, shared-worker and startup/hostname suites; and one complete snapshot/schema drift case. This includes 15 report, 13 retention and 14 cleanup lifecycle cases. Voice integration and the four other database owners use separate initially empty databases and migrate themselves. Report cases use their dedicated database after the setup repair. - Five real connection saves over 205 sessions with saved future report deadlines selected 1,025 session rows before the notification repair. They now select zero and preserve the original deadline. The report notification helper accepts only exact session scopes. Configuration, authority and connection hints retain their existing live recovery/reply/cleanup behavior but do not rebuild report history. Real session terminal writes, callback hints, report claims and receipts still reactivate reports. - Database/server typechecks and builds, migration numbering/safety, and module-boundary checks pass. Final server types/build also pass after integrating the retention review repairs. The parent 205-admission, five-configuration-save test still observes zero retained-history rows and preserves its deadline. - The old eager reader selected all 105 saved-report headers before yielding and failed the 100-row budget. Registered reconstruction now uses the existing paged contract and yields at 100 rows per scope refresh. A partial scan can restart on the same owner and reconstruct every saved deadline. At due time all 105 report IDs are fetched exactly once. Future deadlines remain quiet between actual retries. - The company-fairness case places a small company deterministically on page two and requires it in the next 25-work batch after discovery, with all 105 reports processed before the failed provider's retry. Guarded mutations catch a missing cursor reset and bypassed company rotation. Original source was restored before final acceptance. - At prior head `9bf20467df`, migration 0396 was generated with `pnpm db:generate` on the parent 0395 lineage. It creates the partial `(ended_at, id)` report-window index concurrently and replayably through the existing migration runner. All 394 historical SQL files and the journal prefix are unchanged; snapshots 0392 through 0396 remain. Fresh application migration and two replay invocations succeed with a valid index. The real plan case retains 20,001 old sessions, 205 recent sessions, precision-safe keysets, and fewer than 1,000 rows removed by filtering on each page. - Existing real hangup/callback/reply-error producers, callback ownership fencing, monotonic report results, saved retry before provider contact, guest transcript projection after a document lock, transaction rollback/nesting, drain and shutdown remain covered. No table, field or provider protocol is added. Available reports retain the existing finite 24-hour refresh policy. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28) records the failed fresh fixture, eager read, fairness-clock attempts, guards, commands, counts and limits. Older 174/138-case and workspace-wide results belong to their stated heads. No new complete local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Report configuration-scope repair](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28) records the current 303 cases and measured rescan repair. The prior `80207ba6bb` head had Ponytail 5/5; that review is stale after this change. Fresh exact-head Greptile 5/5 and all CI remain required. Actual failing Ponytail findings must be fixed; a stopped or paused turn without a verdict is an authorized nonblocker. Active no-verdict review waits are capped at 30 minutes from the original request/start (25-minute monitor cutoff). Older scores and parent checks are not inferred as current passes. - **Current-master verification: 185 distinct cases pass**: final complete voice integration 132, complete report/cleanup/retention 15+24+13, and snapshot/schema drift 1. Each database owner used its own empty fixture database. The first combined run had 183 passing cases and one report-test failure; it is preserved in the audit. A deliberately earlier-ended competing call reproduces the old one-shot mock consuming the intended response. The corrected test selects responses by provider session ID, verifies independent costs for both calls, then rejects stale target data. The isolated repair and full 132-case suite pass; production behavior and timeouts are unchanged. These 185 cases are not added to the older 303 count. - **Current migration lineage:** official `pnpm db:generate` after actual master creates `0397_known_madrox.sql`; the reviewed concurrent/idempotent report-index SQL is retained byte-for-byte. All 395 existing SQL files and journal entries are unchanged; snapshots 0393–0397 remain and the new snapshot links parent 0396. The application migrator boots a fresh database, repairs a deliberately invalid index during two concurrent replay invocations, records one history row, and replays again successfully. Database build, server build/typecheck, migration checks and module boundaries pass. [Full current-master acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/c28de398fef074f650eff532952ab6f5563739a8/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28). - **Review follow-up and preserved CI timeout:** Ponytail’s prior `c28de398` review was 4/5 advisory, with no must-fix production defect and a request to fix the stale design-doc migration reference before shipping. That document now names migration 0397 and links current 185-case acceptance; older 302/303-case evidence is explicitly historical. The optional suggestions about one-shot diagnostic inventory cost and inherited test-clock helpers remain disclosed. CI run `40703490340` failed the unchanged approval-queue backoff test on its existing 15-second timeout and then an activity-log FK in teardown (2,097 pass/one fail/one skip in that shard). The complete unchanged 28-case approval suite passes on a new isolated database; the exact CI timing cause remains unproven. No timeout, workflow or production logic was changed, and no old-head rerun or failure waiver is claimed. This documentation head requires fresh CI and exact-head Greptile. [Review and timeout ledger](https://github.com/paperclipai/paperclip/blob/758cfe15c3e6283d38ba512c8416ccce4d200862/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#review-documentation-and-preserved-remote-timeout-october-28). ## Risks - Process-local commit signals do not observe independent SQL writers or another server process. - Available reports still refresh for 24 hours. The provider's complete flag does not promise immutable final data. - Transcript projection retries remain within the existing report window. This change does not add indefinite projection repair. - One-shot diagnostic `reconcile(limit)` inventories the eligible window before slicing; it remains a manual diagnostic cost, not recurring idle work. Some inherited test helpers advance fake time during assertion retries, an optional reliability follow-up noted by review. - A local deadline cannot wake a stopped host. Cloud alarms and sleep eligibility remain unchanged. No deployed sleep or cost saving is claimed. - This PR depends on the preceding stack. Final master reconciliation and remote review gates remain required. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1297 lines
47 KiB
TypeScript
1297 lines
47 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { drizzle as drizzlePg } from "drizzle-orm/postgres-js";
|
|
import { readFile, readdir } from "node:fs/promises";
|
|
import { fileURLToPath } from "node:url";
|
|
import postgres from "postgres";
|
|
import { installDatabaseWorkSignals } from "./work-signals.js";
|
|
import { installDatabaseResponseCapture } from "./response-capture.js";
|
|
import { installPreparedQueries } from "./prepared-queries.js";
|
|
import * as schema from "./schema/index.js";
|
|
|
|
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
|
|
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
|
|
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
|
|
|
|
function createUtilitySql(url: string) {
|
|
return postgres(url, { max: 1, onnotice: () => {} });
|
|
}
|
|
|
|
type RegisteredPostgresClient = ReturnType<typeof postgres>;
|
|
|
|
/**
|
|
* Derives a registry key from a connection URL's host and port only. We must
|
|
* not retain or log the full URL, because it carries credentials.
|
|
*/
|
|
function hostPortKey(url: string): string {
|
|
const parsed = new URL(url);
|
|
return `${parsed.hostname}:${parsed.port || "5432"}`;
|
|
}
|
|
|
|
/**
|
|
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
|
|
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
|
|
* empty string falls back to the `PG*` environment variables), so `createDb`
|
|
* must tolerate it too: skip the registry entry and let the driver decide
|
|
* the outcome, instead of throwing an error the driver itself would not.
|
|
*/
|
|
function hostPortKeyOrNull(url: string): string | null {
|
|
try {
|
|
return hostPortKey(url);
|
|
} catch (error) {
|
|
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Tracks every client `createDb` hands out, keyed by host and port, so a test
|
|
// fixture can end them before it stops the Postgres cluster they point at. A
|
|
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
|
|
// server) retains nothing extra: an unreferenced client is pruned on its own.
|
|
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
|
|
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
|
|
({ hostPortKey, ref }) => {
|
|
const refs = clientsByHostPort.get(hostPortKey);
|
|
if (!refs) return;
|
|
refs.delete(ref);
|
|
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
|
|
},
|
|
);
|
|
|
|
function registerClient(key: string, client: RegisteredPostgresClient): void {
|
|
const ref = new WeakRef(client);
|
|
let refs = clientsByHostPort.get(key);
|
|
if (!refs) {
|
|
refs = new Set();
|
|
clientsByHostPort.set(key, refs);
|
|
}
|
|
refs.add(ref);
|
|
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
|
|
}
|
|
|
|
/**
|
|
* Ends every live client `createDb` handed out for the given URL's host and
|
|
* port, then forgets them. Call this before stopping a Postgres cluster: a
|
|
* client that outlives the cluster it points at can crash the process (a
|
|
* reserved connection's deferred write firing after the socket is gone).
|
|
* Swallows individual `end()` errors so one bad client cannot block the rest.
|
|
* An owned database on a shared test host must select database scope instead;
|
|
* its teardown cannot close another fixture's clients on that host.
|
|
*/
|
|
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
|
|
const key = hostPortKey(url);
|
|
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
|
|
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
|
|
const refs = clientsByHostPort.get(key);
|
|
if (!refs) return;
|
|
|
|
const clients: RegisteredPostgresClient[] = [];
|
|
for (const ref of refs) {
|
|
const client = ref.deref();
|
|
if (client && database !== undefined && client.options.database !== database) continue;
|
|
refs.delete(ref);
|
|
clientFinalizer.unregister(ref);
|
|
if (client) clients.push(client);
|
|
}
|
|
if (refs.size === 0) clientsByHostPort.delete(key);
|
|
|
|
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
|
|
}
|
|
|
|
function isSafeIdentifier(value: string): boolean {
|
|
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
|
}
|
|
|
|
function quoteIdentifier(value: string): string {
|
|
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
|
|
return `"${value.replaceAll("\"", "\"\"")}"`;
|
|
}
|
|
|
|
function quoteLiteral(value: string): string {
|
|
return `'${value.replaceAll("'", "''")}'`;
|
|
}
|
|
|
|
function splitMigrationStatements(content: string): string[] {
|
|
return content
|
|
.split("--> statement-breakpoint")
|
|
.map((statement) => statement.trim())
|
|
.filter((statement) => statement.length > 0);
|
|
}
|
|
|
|
export type MigrationState =
|
|
| {
|
|
status: "upToDate";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
journalEntryCount: number;
|
|
}
|
|
| {
|
|
status: "needsMigrations";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
pendingMigrations: string[];
|
|
journalEntryCount: number;
|
|
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
|
|
};
|
|
|
|
export interface DatabaseClientOptions {
|
|
prepare?: boolean;
|
|
/** postgres.js `max` — connection pool size (driver default: 10). */
|
|
maxConnections?: number;
|
|
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
|
|
idleTimeoutSeconds?: number;
|
|
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
|
|
connectTimeoutSeconds?: number;
|
|
/**
|
|
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
|
|
* connection is reused before the client replaces it (driver default: a
|
|
* random value between 30 and 60 minutes).
|
|
*/
|
|
maxLifetimeSeconds?: number;
|
|
/**
|
|
* postgres.js `connection.application_name`, shown in
|
|
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
|
|
* pool apart from other clients of the same database (driver default:
|
|
* `postgres.js`).
|
|
*/
|
|
applicationName?: string;
|
|
}
|
|
|
|
/**
|
|
* Idle pooled connections close after this many seconds unless
|
|
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
|
|
* idle connection open forever, so a process that stops issuing queries still
|
|
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
|
|
* to restore the driver default.
|
|
*/
|
|
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
|
|
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
|
|
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
|
|
|
|
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
|
|
const value = env[name]?.trim().toLowerCase();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (value === "true" || value === "1") return true;
|
|
if (value === "false" || value === "0") return false;
|
|
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
|
|
}
|
|
|
|
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^[1-9]\d*$/.test(value)) {
|
|
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
|
|
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Database client tuning from the environment, so hosted deployments can
|
|
* adapt to their connection topology (pooled endpoints, network latency)
|
|
* without editing source. Every variable is optional. This function returns
|
|
* only the values the environment sets; `resolveDatabaseClientOptions` adds
|
|
* Paperclip's own defaults on top, and the driver defaults apply to the rest
|
|
* — self-hosted setups need none of these.
|
|
*/
|
|
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
const options: DatabaseClientOptions = {};
|
|
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
|
|
if (prepare !== undefined) options.prepare = prepare;
|
|
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
|
|
if (maxConnections !== undefined) options.maxConnections = maxConnections;
|
|
// `0` is allowed here: it disables idle reaping (the driver default).
|
|
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
|
|
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
|
|
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
|
|
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
|
|
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
|
|
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
|
|
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
|
|
if (applicationName !== undefined) options.applicationName = applicationName;
|
|
return options;
|
|
}
|
|
|
|
/**
|
|
* Fills in Paperclip's defaults for the options the caller left unset: idle
|
|
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
|
|
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
|
|
* else stays at the driver default. An explicit value (including
|
|
* `idleTimeoutSeconds: 0`) always wins over the default.
|
|
*/
|
|
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
|
|
return {
|
|
...options,
|
|
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
|
|
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
|
|
};
|
|
}
|
|
|
|
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
|
|
const driverOptions: Record<string, unknown> = {};
|
|
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
|
|
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
|
|
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
|
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
|
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
|
if (options.applicationName !== undefined) {
|
|
driverOptions.connection = { application_name: options.applicationName };
|
|
}
|
|
return driverOptions;
|
|
}
|
|
|
|
// A long advisory-lock transaction must not borrow the normal pool that its
|
|
// work needs for progress writes. Keep connection configuration private to the
|
|
// originating Db lifetime; callers receive neither URLs nor credentials.
|
|
const dedicatedDbFactories = new WeakMap<object, () => Db>();
|
|
|
|
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>): Promise<T> {
|
|
const factory = dedicatedDbFactories.get(db);
|
|
if (!factory) throw new Error("dedicated_connection_requires_create_db");
|
|
const dedicated = factory();
|
|
try { return await action(dedicated); }
|
|
finally { await dedicated.$client.end({ timeout: 1 }); }
|
|
}
|
|
|
|
// postgres.js pipelines a query onto a busy connection when none is idle, so
|
|
// a short liveness write can wait behind an unrelated slow statement with no
|
|
// client-side bound. Liveness writes (controller-lease renewal) get a small
|
|
// private pool, created on first use and shared for the owner's lifetime.
|
|
const livenessDbFactories = new WeakMap<object, () => Db>();
|
|
const livenessDbs = new WeakMap<object, Db>();
|
|
const LIVENESS_POOL_MAX_CONNECTIONS = 2;
|
|
|
|
/** The private liveness pool for a `createDb` client; other clients (test
|
|
* doubles, transactions) are returned unchanged. */
|
|
export function livenessDb(db: Db): Db {
|
|
const existing = livenessDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = livenessDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
livenessDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
|
* a late renewal fails like a query on the ended primary pool. */
|
|
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
await livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
// Control checkpoints must not occupy the request or lease-renewal pools
|
|
// during an aligned burst. Share at most two lazy connections per owner.
|
|
const checkpointDbFactories = new WeakMap<object, () => Db>();
|
|
const checkpointDbs = new WeakMap<object, Db>();
|
|
const closedCheckpointOwners = new WeakSet<object>();
|
|
|
|
/** The bounded controller checkpoint pool; transactions and test doubles keep
|
|
* their existing database when they have no owning client factory. */
|
|
export function checkpointDb(db: Db): Db {
|
|
const existing = checkpointDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = checkpointDbFactories.get(db);
|
|
if (!factory) return db;
|
|
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
|
|
const created = factory();
|
|
checkpointDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Fence lazy creation immediately, and keep any ended pool cached. */
|
|
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedCheckpointOwners.add(db);
|
|
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
export function createDb(url: string, options?: DatabaseClientOptions) {
|
|
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
|
|
return createDbClient(url, resolved);
|
|
}
|
|
|
|
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) {
|
|
const sql = postgres(url, postgresJsOptions(resolved));
|
|
if (resolved.prepare === true) installPreparedQueries(sql);
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key) registerClient(key, sql);
|
|
// A disconnect can lose the response after a statement has committed.
|
|
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
|
|
// message cannot establish that replay is safe. Leave retries to callers
|
|
// that know the complete operation is idempotent.
|
|
const db = installDatabaseWorkSignals(drizzlePg(sql, { schema }), owner);
|
|
installDatabaseResponseCapture(db, schema);
|
|
dedicatedDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
|
|
}, db));
|
|
livenessDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease",
|
|
}, db));
|
|
checkpointDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 2, applicationName: "paperclip-runner-checkpoint",
|
|
}, db));
|
|
return db;
|
|
}
|
|
|
|
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const rows = await sql<{ data_directory: string | null }[]>`
|
|
SELECT current_setting('data_directory', true) AS data_directory
|
|
`;
|
|
const actual = rows[0]?.data_directory;
|
|
return typeof actual === "string" && actual.length > 0 ? actual : null;
|
|
} catch {
|
|
return null;
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
async function listMigrationFiles(): Promise<string[]> {
|
|
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
|
|
.map((entry) => entry.name)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
}
|
|
|
|
type MigrationJournalFile = {
|
|
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
|
|
};
|
|
|
|
type JournalMigrationEntry = {
|
|
fileName: string;
|
|
folderMillis: number;
|
|
order: number;
|
|
};
|
|
|
|
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
|
|
try {
|
|
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
|
|
const parsed = JSON.parse(raw) as MigrationJournalFile;
|
|
if (!Array.isArray(parsed.entries)) return [];
|
|
return parsed.entries
|
|
.map((entry, entryIndex) => {
|
|
if (typeof entry?.tag !== "string") return null;
|
|
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
|
|
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
|
|
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
|
|
})
|
|
.filter((entry): entry is JournalMigrationEntry => entry !== null);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
async function listJournalMigrationFiles(): Promise<string[]> {
|
|
const entries = await listJournalMigrationEntries();
|
|
return entries.map((entry) => entry.fileName);
|
|
}
|
|
|
|
async function readMigrationFileContent(migrationFile: string): Promise<string> {
|
|
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
|
}
|
|
|
|
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
|
|
return [...migrationFiles].sort((left, right) => {
|
|
const leftOrder = orderByFileName.get(left);
|
|
const rightOrder = orderByFileName.get(right);
|
|
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
|
|
if (leftOrder === undefined) return 1;
|
|
if (rightOrder === undefined) return -1;
|
|
if (leftOrder === rightOrder) return left.localeCompare(right);
|
|
return leftOrder - rightOrder;
|
|
});
|
|
}
|
|
|
|
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
|
|
|
|
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
|
|
await sql.unsafe("BEGIN");
|
|
try {
|
|
await action();
|
|
await sql.unsafe("COMMIT");
|
|
} catch (error) {
|
|
try {
|
|
await sql.unsafe("ROLLBACK");
|
|
} catch {
|
|
// Ignore rollback failures and surface the original error.
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async function latestMigrationCreatedAt(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
): Promise<number | null> {
|
|
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
|
|
);
|
|
const value = Number(rows[0]?.created_at ?? Number.NaN);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function normalizeFolderMillis(value: number | null | undefined): number {
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
|
|
return Math.trunc(value);
|
|
}
|
|
return Date.now();
|
|
}
|
|
|
|
async function ensureMigrationJournalTable(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
|
|
let migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
const drizzleSchema = quoteIdentifier("drizzle");
|
|
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
|
|
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
|
|
await sql.unsafe(
|
|
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
|
|
);
|
|
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
return { migrationTableSchema, columnNames };
|
|
}
|
|
|
|
async function migrationHistoryEntryExists(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
): Promise<boolean> {
|
|
const predicates: string[] = [];
|
|
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
|
|
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
|
|
if (predicates.length === 0) return false;
|
|
|
|
const rows = await sql.unsafe<{ one: number }[]>(
|
|
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
|
|
);
|
|
return rows.length > 0;
|
|
}
|
|
|
|
async function recordMigrationHistoryEntry(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
folderMillis: number,
|
|
): Promise<void> {
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
|
|
const createdAt = latestCreatedAt === null
|
|
? normalizeFolderMillis(folderMillis)
|
|
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(createdAt)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) return;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
// These idempotent migrations require top-level statements for batch commits
|
|
// or concurrent index builds. History is recorded only after completion.
|
|
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
|
"0313_private_task_access.sql",
|
|
"0314_private_task_draft_assets.sql",
|
|
"0332_little_night_nurse.sql",
|
|
"0345_luxuriant_carmella_unuscione.sql",
|
|
"0369_late_logan.sql",
|
|
"0381_mean_scream.sql",
|
|
"0382_narrow_the_executioner.sql",
|
|
"0385_open_madame_web.sql",
|
|
"0386_steady_epoch.sql",
|
|
"0387_boring_professor_monster.sql",
|
|
"0388_ambitious_micromax.sql",
|
|
"0389_brave_hulk.sql",
|
|
"0390_salty_killraven.sql",
|
|
"0391_third_roulette.sql",
|
|
"0392_small_frightful_four.sql",
|
|
"0393_open_grandmaster.sql",
|
|
"0395_tranquil_firestar.sql",
|
|
"0396_tan_paladin.sql",
|
|
"0397_known_madrox.sql",
|
|
|
|
|
|
"0384_funny_adam_warlock.sql",
|
|
"0383_tool_review_company_expiry.sql",
|
|
]);
|
|
|
|
async function applyPendingMigrationsManually(
|
|
url: string,
|
|
pendingMigrations: string[],
|
|
): Promise<void> {
|
|
if (pendingMigrations.length === 0) return;
|
|
|
|
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFileName = new Map(
|
|
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
|
|
);
|
|
|
|
const pool = createUtilitySql(url);
|
|
try {
|
|
const sql = await pool.reserve();
|
|
try {
|
|
// A session lock survives each batch COMMIT. Reserve the connection so
|
|
// pool rotation cannot release it while another migrator is waiting.
|
|
// A blocking lock SELECT retains its statement snapshot while waiting.
|
|
// A partial concurrent index build can wait for that snapshot, forming a
|
|
// deadlock with this lock. Each failed try ends its snapshot before sleep.
|
|
while (!(await sql<{ locked: boolean }[]>`
|
|
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
|
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
|
}
|
|
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of orderedPendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const existingEntry = await migrationHistoryEntryExists(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
);
|
|
if (existingEntry) continue;
|
|
|
|
const applyMigration = async () => {
|
|
for (const statement of splitMigrationStatements(migrationContent)) {
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
// A cancelled concurrent build leaves an invalid index. IF NOT
|
|
// EXISTS alone would skip it on retry and journal an incomplete index.
|
|
const index = statement.replace(/^\s*--.*$/gm, "").trim()
|
|
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
|
|
if (index) {
|
|
const invalid = await sql<{ invalid: boolean }[]>`
|
|
SELECT NOT i.indisvalid AS invalid FROM pg_index i
|
|
JOIN pg_class c ON c.oid = i.indexrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
|
|
if (invalid[0]?.invalid) {
|
|
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
|
|
}
|
|
}
|
|
}
|
|
// Older dev schemas can lack a replaced constraint. Preserve the
|
|
// published SQL/hash while allowing the harmless drop to proceed.
|
|
const executable = statement.replace(
|
|
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
|
|
"$1 IF EXISTS ",
|
|
);
|
|
await sql.unsafe(executable);
|
|
}
|
|
|
|
await recordMigrationHistoryEntry(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
folderMillisByFileName.get(migrationFile) ?? Date.now(),
|
|
);
|
|
};
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
await applyMigration();
|
|
} else {
|
|
await runInTransaction(sql, applyMigration);
|
|
}
|
|
}
|
|
} finally {
|
|
try {
|
|
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
|
|
} catch {
|
|
// Preserve the migration error if the session died; PostgreSQL releases
|
|
// its lock on disconnect, and pool.end() closes any remaining connection.
|
|
}
|
|
sql.release();
|
|
}
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
|
|
const mapped = new Map<string, string>();
|
|
|
|
await Promise.all(
|
|
migrationFiles.map(async (migrationFile) => {
|
|
const content = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(content).digest("hex");
|
|
mapped.set(hash, migrationFile);
|
|
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
|
|
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
|
|
}
|
|
}),
|
|
);
|
|
|
|
return mapped;
|
|
}
|
|
|
|
async function getMigrationTableColumnNames(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
): Promise<Set<string>> {
|
|
const columns = await sql.unsafe<{ column_name: string }[]>(
|
|
`
|
|
SELECT column_name
|
|
FROM information_schema.columns
|
|
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
|
|
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
|
|
`,
|
|
);
|
|
return new Set(columns.map((column) => column.column_name));
|
|
}
|
|
|
|
async function tableExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.tables
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnHasDataType(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
dataType: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ dataType: string; udtName: string }[]>`
|
|
SELECT data_type AS "dataType", udt_name AS "udtName"
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
`;
|
|
const expected = dataType.toLowerCase();
|
|
return rows.some((row) => (
|
|
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
|
|
));
|
|
}
|
|
|
|
async function indexExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
indexName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relkind = 'i'
|
|
AND c.relname = ${indexName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function constraintExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
constraintName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_constraint c
|
|
JOIN pg_namespace n ON n.oid = c.connamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.conname = ${constraintName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function functionExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
functionName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_proc p
|
|
JOIN pg_namespace n ON n.oid = p.pronamespace
|
|
WHERE n.nspname = 'public'
|
|
AND p.proname = ${functionName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function triggerExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
triggerName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_trigger t
|
|
JOIN pg_class c ON c.oid = t.tgrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND t.tgname = ${triggerName}
|
|
AND NOT t.tgisinternal
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function heartbeatEventSequencesAreUnique(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ unique: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_run_events
|
|
GROUP BY run_id, seq
|
|
HAVING count(*) > 1
|
|
) AS unique
|
|
`;
|
|
return rows[0]?.unique ?? false;
|
|
}
|
|
|
|
async function heartbeatNextEventSequencesAreCurrent(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ current: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_runs run
|
|
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
|
|
SELECT max(event.seq) + 1
|
|
FROM heartbeat_run_events event
|
|
WHERE event.run_id = run.id
|
|
), 1)
|
|
) AS current
|
|
`;
|
|
return rows[0]?.current ?? false;
|
|
}
|
|
|
|
async function migrationStatementAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
statement: string,
|
|
): Promise<boolean> {
|
|
const normalized = statement
|
|
.replace(/^\s*--.*$/gm, "")
|
|
.replace(/\s+/g, " ")
|
|
.trim();
|
|
|
|
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createTableMatch) {
|
|
return tableExists(sql, createTableMatch[1]);
|
|
}
|
|
|
|
const addColumnMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
|
|
);
|
|
if (addColumnMatch) {
|
|
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
|
|
}
|
|
|
|
const alterColumnTypeMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
|
|
);
|
|
if (alterColumnTypeMatch) {
|
|
return columnHasDataType(
|
|
sql,
|
|
alterColumnTypeMatch[1],
|
|
alterColumnTypeMatch[2],
|
|
alterColumnTypeMatch[3],
|
|
);
|
|
}
|
|
|
|
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createIndexMatch) {
|
|
return indexExists(sql, createIndexMatch[1]);
|
|
}
|
|
|
|
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
|
|
if (addConstraintMatch) {
|
|
return constraintExists(sql, addConstraintMatch[2]);
|
|
}
|
|
|
|
const createFunctionMatch = normalized.match(
|
|
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
|
|
);
|
|
if (createFunctionMatch) {
|
|
return functionExists(sql, createFunctionMatch[1]);
|
|
}
|
|
|
|
const createTriggerMatch = normalized.match(
|
|
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
|
|
);
|
|
if (createTriggerMatch) {
|
|
return triggerExists(sql, createTriggerMatch[1]);
|
|
}
|
|
|
|
// These native-runner repairs have persistent postconditions. Verify them
|
|
// instead of replaying them when a restored database is missing only the
|
|
// migration-history row.
|
|
if (
|
|
normalized.startsWith("WITH ranked AS (")
|
|
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
|
|
) {
|
|
return heartbeatEventSequencesAreUnique(sql);
|
|
}
|
|
if (
|
|
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
|
|
&& normalized.includes('SET "next_event_seq" = COALESCE')
|
|
) {
|
|
return heartbeatNextEventSequencesAreCurrent(sql);
|
|
}
|
|
|
|
// If we cannot reason about a statement safely, require manual migration.
|
|
return false;
|
|
}
|
|
|
|
async function migrationContentAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationContent: string,
|
|
): Promise<boolean> {
|
|
const statements = splitMigrationStatements(migrationContent);
|
|
if (statements.length === 0) return false;
|
|
|
|
for (const statement of statements) {
|
|
const applied = await migrationStatementAlreadyApplied(sql, statement);
|
|
if (!applied) return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async function loadAppliedMigrations(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
availableMigrations: string[],
|
|
): Promise<string[]> {
|
|
const quotedSchema = quoteIdentifier(migrationTableSchema);
|
|
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
|
|
if (columnNames.has("name")) {
|
|
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
|
|
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
|
|
}
|
|
|
|
if (columnNames.has("hash")) {
|
|
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
|
|
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
|
|
const appliedFromHashes = rows
|
|
.map((row) => hashesToMigrationFiles.get(row.hash))
|
|
.filter((name): name is string => Boolean(name));
|
|
|
|
if (appliedFromHashes.length > 0) {
|
|
// Best-effort: when all hashes resolve, this is authoritative.
|
|
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
|
|
|
|
// Partial hash resolution can happen when files have changed; return what we can trust.
|
|
return appliedFromHashes;
|
|
}
|
|
|
|
// Fallback only when hashes are unavailable/unresolved.
|
|
if (columnNames.has("created_at")) {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
if (journalEntries.length > 0) {
|
|
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
|
|
);
|
|
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
|
|
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
|
|
return journalEntries
|
|
.filter((entry) => availableMigrations.includes(entry.fileName))
|
|
.filter((entry) => entry.folderMillis <= lastCreatedAt)
|
|
.map((entry) => entry.fileName)
|
|
.slice(0, rows.length);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
|
|
const journalMigrationFiles = await listJournalMigrationFiles();
|
|
const appliedFromIds = rows
|
|
.map((row) => journalMigrationFiles[row.id - 1])
|
|
.filter((name): name is string => Boolean(name));
|
|
if (appliedFromIds.length > 0) return appliedFromIds;
|
|
|
|
return availableMigrations.slice(0, Math.max(0, rows.length));
|
|
}
|
|
|
|
export type MigrationHistoryReconcileResult = {
|
|
repairedMigrations: string[];
|
|
remainingMigrations: string[];
|
|
};
|
|
|
|
export async function reconcilePendingMigrationHistory(
|
|
url: string,
|
|
): Promise<MigrationHistoryReconcileResult> {
|
|
const state = await inspectMigrations(url);
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
return { repairedMigrations: [], remainingMigrations: [] };
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
const repairedMigrations: string[] = [];
|
|
|
|
try {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of state.pendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
|
|
if (!alreadyApplied) break;
|
|
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
|
|
const existingByHash = columnNames.has("hash")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
const existingByName = columnNames.has("name")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
if (existingByHash.length > 0 || existingByName.length > 0) {
|
|
if (columnNames.has("created_at")) {
|
|
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
|
|
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
|
|
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
|
|
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
repairedMigrations.push(migrationFile);
|
|
continue;
|
|
}
|
|
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(folderMillis)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) break;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
repairedMigrations.push(migrationFile);
|
|
}
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
|
|
const refreshed = await inspectMigrations(url);
|
|
return {
|
|
repairedMigrations,
|
|
remainingMigrations:
|
|
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
|
|
};
|
|
}
|
|
|
|
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
|
|
const rows = await sql<{ schemaName: string }[]>`
|
|
SELECT n.nspname AS "schemaName"
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
|
|
`;
|
|
|
|
if (rows.length === 0) return null;
|
|
|
|
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
|
|
if (drizzleSchema) return drizzleSchema.schemaName;
|
|
|
|
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
|
|
if (publicSchema) return publicSchema.schemaName;
|
|
|
|
return rows[0]?.schemaName ?? null;
|
|
}
|
|
|
|
export async function inspectMigrations(url: string): Promise<MigrationState> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const availableMigrations = await listMigrationFiles();
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
if (tableCount > 0) {
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-non-empty-db",
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-empty-db",
|
|
};
|
|
}
|
|
|
|
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const journalCountRows = await sql.unsafe<{ count: number }[]>(
|
|
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
|
|
);
|
|
const journalEntryCount = journalCountRows[0]?.count ?? 0;
|
|
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
|
|
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
|
|
if (pendingMigrations.length === 0) {
|
|
return {
|
|
status: "upToDate",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
journalEntryCount,
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
pendingMigrations,
|
|
journalEntryCount,
|
|
reason: "pending-migrations",
|
|
};
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function applyPendingMigrations(url: string): Promise<void> {
|
|
const initialState = await inspectMigrations(url);
|
|
if (initialState.status === "upToDate") return;
|
|
|
|
if (initialState.reason === "no-migration-journal-empty-db") {
|
|
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
|
|
// the whole journal in one transaction and cannot run committing backfills.
|
|
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
|
|
const bootstrappedState = await inspectMigrations(url);
|
|
if (bootstrappedState.status === "upToDate") return;
|
|
throw new Error(
|
|
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
|
|
if (initialState.reason === "no-migration-journal-non-empty-db") {
|
|
throw new Error(
|
|
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
|
|
);
|
|
}
|
|
|
|
let state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
|
|
const repair = await reconcilePendingMigrationHistory(url);
|
|
if (repair.repairedMigrations.length > 0) {
|
|
state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
}
|
|
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, state.pendingMigrations);
|
|
|
|
const finalState = await inspectMigrations(url);
|
|
if (finalState.status !== "upToDate") {
|
|
throw new Error(
|
|
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export type MigrationBootstrapResult =
|
|
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
|
|
| { migrated: false; reason: "already-migrated"; tableCount: number }
|
|
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
|
|
|
|
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
if (migrationTableSchema) {
|
|
return { migrated: false, reason: "already-migrated", tableCount };
|
|
}
|
|
|
|
if (tableCount > 0) {
|
|
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, await listMigrationFiles());
|
|
|
|
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function ensurePostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"created" | "exists"> {
|
|
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
|
|
throw new Error(`Unsafe database name: ${databaseName}`);
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const existing = await sql<{ one: number }[]>`
|
|
select 1 as one from pg_database where datname = ${databaseName} limit 1
|
|
`;
|
|
if (existing.length > 0) return "exists";
|
|
|
|
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "created";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function resetPostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"reset"> {
|
|
const quotedDatabaseName = quoteIdentifier(databaseName);
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
await sql`
|
|
select pg_terminate_backend(pid)
|
|
from pg_stat_activity
|
|
where datname = ${databaseName}
|
|
and pid <> pg_backend_pid()
|
|
`;
|
|
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
|
|
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "reset";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export type Db = ReturnType<typeof createDb>;
|