mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:19:23 +01:00
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
fc14f1ade2
commit
e8e548e856
9 files changed
+1621
-28
No files matched your search
@@ -0,0 +1,378 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
MIGRATION_COMPAT_REASON_CODES,
|
||||
checkNewMigrationCompatibility,
|
||||
classifyMigrationSequence,
|
||||
classifyMigrationSql,
|
||||
collectCreatedTables,
|
||||
parseMigrationCompatOverride,
|
||||
parseNonTransactionalMigrations,
|
||||
summarizeMigrationCompatibility,
|
||||
type MigrationCompatibility,
|
||||
type MigrationCompatReasonCode,
|
||||
} from "./migration-compatibility.js";
|
||||
|
||||
const BP = "--> statement-breakpoint\n";
|
||||
const NEW_TABLE = `CREATE TABLE "widgets" ("id" uuid PRIMARY KEY NOT NULL, "issue_id" uuid);${BP}`;
|
||||
|
||||
function codes(sql: string): MigrationCompatReasonCode[] {
|
||||
return classifyMigrationSql(sql).reasons.map((reason) => reason.code);
|
||||
}
|
||||
|
||||
type Case = readonly [name: string, sql: string, compatibility: MigrationCompatibility, code: MigrationCompatReasonCode];
|
||||
|
||||
const cases: readonly Case[] = [
|
||||
// Compatible: additive changes the previous release never touches.
|
||||
["create table", `CREATE TABLE "widgets" ("id" uuid PRIMARY KEY, "issue_id" uuid REFERENCES "issues"("id"));`, "online_compatible", "create_table"],
|
||||
["create table if not exists", `CREATE TABLE IF NOT EXISTS "widgets" ("id" uuid PRIMARY KEY);`, "online_compatible", "create_table_if_not_exists"],
|
||||
["create temp table", `CREATE TEMP TABLE scratch (old_id uuid) ON COMMIT DROP;`, "online_compatible", "create_temp_table"],
|
||||
["plain index on a table created here", `${NEW_TABLE}CREATE UNIQUE INDEX "widgets_uq" ON "widgets" USING btree ("issue_id");`, "online_compatible", "create_index_new_table"],
|
||||
["constraints on a table created here", `${NEW_TABLE}ALTER TABLE "widgets" ADD CONSTRAINT "widgets_issue_fk" FOREIGN KEY ("issue_id") REFERENCES "public"."issues"("id") ON DELETE cascade ON UPDATE no action;`, "online_compatible", "alter_new_table"],
|
||||
["trigger on a table created here", `${NEW_TABLE}CREATE TRIGGER widgets_touch BEFORE UPDATE OF issue_id ON "widgets" FOR EACH ROW EXECUTE FUNCTION touch();`, "online_compatible", "create_trigger_new_table"],
|
||||
["constant rows in a table created here", `${NEW_TABLE}INSERT INTO "widgets" ("id") VALUES ('00000000-0000-0000-0000-000000000001');`, "online_compatible", "seed_new_table"],
|
||||
["concurrent index", `CREATE INDEX CONCURRENTLY IF NOT EXISTS "issues_x_idx" ON "issues" USING btree ("company_id","id") WHERE "issues"."status" = 'done';`, "online_compatible", "create_index_concurrently"],
|
||||
["nullable column", `ALTER TABLE "issues" ADD COLUMN "time_zone" text;`, "online_compatible", "add_column"],
|
||||
["column with a constant default", `ALTER TABLE "issues" ADD COLUMN IF NOT EXISTS "attempts" integer DEFAULT 0 NOT NULL;`, "online_compatible", "add_column"],
|
||||
["column with a cast literal default", `ALTER TABLE "issues" ADD COLUMN "meta" jsonb NOT NULL DEFAULT '{"a": "b, c"}'::jsonb;`, "online_compatible", "add_column"],
|
||||
["column with a stable timestamp default", `ALTER TABLE "issues" ADD COLUMN "seen_at" timestamp with time zone DEFAULT now() NOT NULL;`, "online_compatible", "add_column"],
|
||||
["several additive actions", `ALTER TABLE "issues" ADD COLUMN "a" text, ADD COLUMN "b" numeric(24, 7) DEFAULT '0' NOT NULL;`, "online_compatible", "add_column"],
|
||||
["foreign key added NOT VALID", `ALTER TABLE "issues" ADD CONSTRAINT "issues_owner_fk" FOREIGN KEY ("owner_id") REFERENCES "users"("id") NOT VALID;`, "online_compatible", "add_constraint_not_valid"],
|
||||
["check added NOT VALID", `ALTER TABLE "issues" ADD CONSTRAINT "issues_kind_check" CHECK ("kind" IN ('a', 'b')) NOT VALID;`, "online_compatible", "add_constraint_not_valid"],
|
||||
["drop not null", `ALTER TABLE "issues" ALTER COLUMN "title" DROP NOT NULL;`, "online_compatible", "drop_not_null"],
|
||||
["create enum", `CREATE TYPE "public"."mood" AS ENUM('sad', 'ok');`, "online_compatible", "create_type"],
|
||||
["add enum value", `ALTER TYPE "public"."mood" ADD VALUE IF NOT EXISTS 'happy';`, "online_compatible", "add_enum_value"],
|
||||
["create sequence", `CREATE SEQUENCE "issue_numbers";`, "online_compatible", "create_sequence"],
|
||||
["create extension", `CREATE EXTENSION IF NOT EXISTS "pg_trgm";`, "online_compatible", "create_extension"],
|
||||
["new function", `CREATE FUNCTION touch() RETURNS trigger AS $fn$ BEGIN DELETE FROM issues; RETURN NEW; END; $fn$ LANGUAGE plpgsql;`, "online_compatible", "create_function"],
|
||||
["new view", `CREATE VIEW "open_issues" AS SELECT * FROM "issues";`, "online_compatible", "create_view"],
|
||||
["comment", `COMMENT ON COLUMN "issues"."title" IS 'DROP TABLE issues; is only text here';`, "online_compatible", "comment"],
|
||||
["no-op select", `SELECT 1;`, "online_compatible", "no_op"],
|
||||
["analyze", `ANALYZE "issues";`, "online_compatible", "analyze"],
|
||||
["extended statistics", `CREATE STATISTICS IF NOT EXISTS "s" ON ("a"), ("b") FROM "issues";`, "online_compatible", "create_statistics"],
|
||||
["guarded DO block around an additive statement", `DO $$ BEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'x' AND conrelid = 'issues'::regclass) THEN\n ALTER TABLE "issues" ADD CONSTRAINT "x" FOREIGN KEY ("a") REFERENCES "b"("id") NOT VALID;\n END IF;\nEND $$;`, "online_compatible", "add_constraint_not_valid"],
|
||||
["duplicate-tolerant DO block on a table created here", `${NEW_TABLE}DO $$ BEGIN\n ALTER TABLE "widgets" ADD CONSTRAINT "w_fk" FOREIGN KEY ("issue_id") REFERENCES "public"."issues"("id") ON DELETE cascade;\nEXCEPTION\n WHEN duplicate_object THEN null;\nEND $$;`, "online_compatible", "alter_new_table"],
|
||||
|
||||
// Requires staging: the previous release breaks or its writes are rejected.
|
||||
["drop table", `DROP TABLE IF EXISTS "widgets" CASCADE;`, "requires_staging", "drop_table"],
|
||||
["drop column", `ALTER TABLE "issues" DROP COLUMN IF EXISTS "legacy";`, "requires_staging", "drop_column"],
|
||||
["rename table", `ALTER TABLE "issues" RENAME TO "tasks";`, "requires_staging", "rename_table"],
|
||||
["rename column", `ALTER TABLE "issues" RENAME COLUMN "title" TO "name";`, "requires_staging", "rename_column"],
|
||||
["alter column type", `ALTER TABLE "issues" ALTER COLUMN "number" SET DATA TYPE bigint USING "number"::bigint;`, "requires_staging", "alter_column_type"],
|
||||
["set not null on an existing column", `ALTER TABLE "issues" ALTER COLUMN "title" SET NOT NULL;`, "requires_staging", "set_not_null"],
|
||||
["not null column without a default", `ALTER TABLE "issues" ADD COLUMN "public_id" varchar(128) NOT NULL;`, "requires_staging", "add_not_null_column_without_default"],
|
||||
["drop default", `ALTER TABLE "issues" ALTER COLUMN "status" DROP DEFAULT;`, "requires_staging", "drop_default"],
|
||||
["validated check", `ALTER TABLE "issues" ADD CONSTRAINT "issues_kind_check" CHECK ("kind" IN ('a', 'b'));`, "requires_staging", "add_validated_check"],
|
||||
["unique constraint", `ALTER TABLE "issues" ADD CONSTRAINT "issues_uq" UNIQUE("company_id","number");`, "requires_staging", "add_unique_constraint"],
|
||||
["unique index on an existing table", `CREATE UNIQUE INDEX CONCURRENTLY "issues_uq" ON "issues" ("company_id","number");`, "requires_staging", "create_unique_index_existing_table"],
|
||||
["drop type", `DROP TYPE "public"."mood";`, "requires_staging", "drop_type"],
|
||||
["rename enum value", `ALTER TYPE "mood" RENAME VALUE 'ok' TO 'fine';`, "requires_staging", "rename_type"],
|
||||
["truncate", `TRUNCATE TABLE "issues";`, "requires_staging", "truncate"],
|
||||
["delete", `DELETE FROM "issues" WHERE "status" = 'cancelled';`, "requires_staging", "delete_rows"],
|
||||
["delete inside a CTE", `WITH ranked AS (SELECT id FROM issues) DELETE FROM "issues" i USING ranked r WHERE i.id = r.id;`, "requires_staging", "delete_rows"],
|
||||
["guarded DO block around a validated check", `DO $$ BEGIN\n ALTER TABLE "issues" ADD CONSTRAINT "c" CHECK ("a" > 0);\nEXCEPTION WHEN duplicate_object THEN null;\nEND $$;`, "requires_staging", "add_validated_check"],
|
||||
|
||||
// Lock and duration hazards: unknown, reported under lock_hazard_ codes.
|
||||
["plain index on an existing table", `CREATE INDEX IF NOT EXISTS "issues_x_idx" ON "issues" USING btree ("company_id");`, "unknown", "lock_hazard_create_index"],
|
||||
["validated foreign key on an existing table", `ALTER TABLE "issues" ADD CONSTRAINT "issues_owner_fk" FOREIGN KEY ("owner_id") REFERENCES "public"."users"("id") ON DELETE set null ON UPDATE no action;`, "unknown", "lock_hazard_validated_foreign_key"],
|
||||
["validated foreign key in a guarded DO block", `DO $$ BEGIN\n IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'x') THEN\n ALTER TABLE "issues" ADD CONSTRAINT "x" FOREIGN KEY ("a") REFERENCES "b"("id");\n END IF;\nEND $$;`, "unknown", "lock_hazard_validated_foreign_key"],
|
||||
["validate constraint", `ALTER TABLE "issues" VALIDATE CONSTRAINT "issues_owner_fk";`, "unknown", "lock_hazard_validate_constraint"],
|
||||
["set not null on a column added here", `ALTER TABLE "issues" ADD COLUMN "kind" text DEFAULT 'task';${BP}ALTER TABLE "issues" ALTER COLUMN "kind" SET NOT NULL;`, "unknown", "lock_hazard_set_not_null"],
|
||||
["column with a volatile default", `ALTER TABLE "issues" ADD COLUMN "token" uuid DEFAULT gen_random_uuid() NOT NULL;`, "unknown", "lock_hazard_table_rewrite"],
|
||||
["generated column", `ALTER TABLE "issues" ADD COLUMN "n" bigint GENERATED ALWAYS AS IDENTITY;`, "unknown", "lock_hazard_table_rewrite"],
|
||||
["column with an inline foreign key", `ALTER TABLE "issues" ADD COLUMN "owner_id" uuid REFERENCES "users"("id");`, "unknown", "lock_hazard_add_column_constraint"],
|
||||
["update backfill", `UPDATE "issues" SET "kind" = 'task' WHERE "kind" IS NULL;`, "unknown", "lock_hazard_data_backfill"],
|
||||
["CTE backfill", `WITH ranked AS (SELECT id FROM issues) UPDATE "issues" i SET n = 1 FROM ranked r WHERE i.id = r.id;`, "unknown", "lock_hazard_data_backfill"],
|
||||
["batched backfill loop", `DO $$ DECLARE n integer; BEGIN LOOP UPDATE issues SET a = 1 WHERE id IN (SELECT id FROM issues LIMIT 1000); GET DIAGNOSTICS n = ROW_COUNT; EXIT WHEN n = 0; END LOOP; END $$;`, "unknown", "lock_hazard_data_backfill"],
|
||||
["copy from live tables into a new table", `${NEW_TABLE}INSERT INTO "widgets" ("id") SELECT "id" FROM "issues";`, "unknown", "lock_hazard_copy_into_new_table"],
|
||||
["lock table", `LOCK TABLE "issues" IN ACCESS EXCLUSIVE MODE;`, "unknown", "lock_hazard_lock_table"],
|
||||
|
||||
// Unknown: not positively recognized, or it changes what old code experiences.
|
||||
["insert into an existing table", `INSERT INTO "issues" ("id") VALUES ('1');`, "unknown", "data_change_existing_table"],
|
||||
["set default", `ALTER TABLE "issues" ALTER COLUMN "status" SET DEFAULT 'todo';`, "unknown", "set_default"],
|
||||
["drop constraint", `ALTER TABLE "issues" DROP CONSTRAINT "issues_owner_fk";`, "unknown", "drop_constraint"],
|
||||
["drop index", `DROP INDEX IF EXISTS "issues_x_idx";`, "unknown", "drop_index"],
|
||||
["drop trigger", `DROP TRIGGER IF EXISTS t ON "issues";`, "unknown", "drop_object"],
|
||||
["constraint from an existing index", `ALTER TABLE "issues" ADD CONSTRAINT "issues_uq" UNIQUE USING INDEX "issues_uq_idx";`, "unknown", "add_constraint_other"],
|
||||
["replace function", `CREATE OR REPLACE FUNCTION touch() RETURNS trigger AS $$ BEGIN RETURN NEW; END; $$ LANGUAGE plpgsql;`, "unknown", "replace_function"],
|
||||
["replace view", `CREATE OR REPLACE VIEW "open_issues" AS SELECT * FROM "issues";`, "unknown", "replace_view"],
|
||||
["trigger on an existing table", `CREATE TRIGGER t BEFORE INSERT OR UPDATE OF a, b ON "issues" FOR EACH ROW EXECUTE FUNCTION touch();`, "unknown", "create_trigger_existing_table"],
|
||||
["create table as", `CREATE TABLE "issues_copy" AS SELECT * FROM "issues";`, "unknown", "create_table_as"],
|
||||
["procedural DO block", `DO $$ BEGIN EXECUTE format('ALTER TABLE %I ADD COLUMN x text', 'issues'); END $$;`, "unknown", "do_block_procedural"],
|
||||
["DO block with an unrecognized guard", `DO $$ BEGIN IF current_setting('x') = 'y' THEN ALTER TABLE "issues" ADD COLUMN "a" text; END IF; END $$;`, "unknown", "do_block_procedural"],
|
||||
["DO block that raises", `DO $$ BEGIN IF EXISTS (SELECT 1 FROM issues) THEN RAISE EXCEPTION 'no'; END IF; END $$;`, "unknown", "do_block_procedural"],
|
||||
["unrecognized alter table action", `ALTER TABLE "issues" ENABLE ROW LEVEL SECURITY;`, "unknown", "unrecognized_alter_table_action"],
|
||||
["unrecognized statement", `GRANT SELECT ON "issues" TO reader;`, "unknown", "unrecognized_statement"],
|
||||
["vacuum", `VACUUM FULL "issues";`, "unknown", "unrecognized_statement"],
|
||||
["empty file", `-- nothing here\n`, "unknown", "no_statements"],
|
||||
];
|
||||
|
||||
describe("classifyMigrationSql", () => {
|
||||
it.each(cases)("%s", (_name, sql, compatibility, code) => {
|
||||
const result = classifyMigrationSql(sql);
|
||||
expect(result.reasons.map((reason) => reason.code)).toContain(code);
|
||||
expect(result.compatibility).toBe(compatibility);
|
||||
expect(result.classified).toBe(compatibility);
|
||||
expect(result.override).toBeNull();
|
||||
});
|
||||
|
||||
it("has a test case for every reason code", () => {
|
||||
const covered = new Set<string>([...cases.map((entry) => entry[3]), "invalid_override"]);
|
||||
expect(MIGRATION_COMPAT_REASON_CODES.filter((code) => !covered.has(code))).toEqual([]);
|
||||
});
|
||||
|
||||
it("takes the worst statement: staging over unknown over compatible", () => {
|
||||
const additive = `ALTER TABLE "issues" ADD COLUMN "a" text;`;
|
||||
const hazard = `CREATE INDEX "i" ON "issues" ("a");`;
|
||||
const breaking = `ALTER TABLE "issues" DROP COLUMN "b";`;
|
||||
expect(classifyMigrationSql([additive, hazard].join(BP)).compatibility).toBe("unknown");
|
||||
expect(classifyMigrationSql([additive, hazard, breaking].join(BP)).compatibility).toBe("requires_staging");
|
||||
expect(classifyMigrationSql([additive, hazard, breaking].join(BP)).reasons).toEqual([
|
||||
expect.objectContaining({ code: "add_column", compatibility: "online_compatible", statements: [1] }),
|
||||
expect.objectContaining({ code: "lock_hazard_create_index", compatibility: "unknown", statements: [2] }),
|
||||
expect.objectContaining({ code: "drop_column", compatibility: "requires_staging", statements: [3] }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("splits statements that share a chunk without a breakpoint", () => {
|
||||
expect(codes(`ALTER TABLE "issues" ADD COLUMN "a" text;\nDROP TABLE "issues";`)).toEqual(["add_column", "drop_table"]);
|
||||
});
|
||||
|
||||
it("classifies each action of a multi-action ALTER TABLE", () => {
|
||||
expect(codes(`ALTER TABLE "issues" ADD COLUMN "a" text, DROP COLUMN "b", ALTER COLUMN "c" TYPE bigint;`))
|
||||
.toEqual(["add_column", "drop_column", "alter_column_type"]);
|
||||
});
|
||||
|
||||
it("does not let comments or string contents hide or invent statements", () => {
|
||||
expect(classifyMigrationSql(`-- DROP TABLE "issues";\n/* TRUNCATE issues; */\nALTER TABLE "issues" ADD COLUMN "a" text DEFAULT 'x; DROP TABLE issues;';`).compatibility)
|
||||
.toBe("online_compatible");
|
||||
expect(classifyMigrationSql(`COMMENT ON TABLE "issues" IS E'it\\'s';${BP}DROP TABLE "issues";`).compatibility).toBe("requires_staging");
|
||||
expect(classifyMigrationSql(`COMMENT ON TABLE "issues" IS 'it''s';\nDROP TABLE "issues";`).compatibility).toBe("requires_staging");
|
||||
});
|
||||
|
||||
it("keeps a quoted identifier with an escaped quote distinct from other names", () => {
|
||||
// "iss""ues" is a table named iss"ues, not issues.
|
||||
const sql = `CREATE TABLE "iss""ues" (id uuid);${BP}DROP TABLE "issues";${BP}ALTER TABLE issues DROP COLUMN title;`;
|
||||
expect(codes(sql)).toEqual(["create_table", "drop_table", "drop_column"]);
|
||||
expect(classifyMigrationSql(sql).compatibility).toBe("requires_staging");
|
||||
expect(collectCreatedTables(sql)).toEqual(['"iss%22ues"']);
|
||||
expect(classifyMigrationSql(`CREATE TABLE "iss""ues" (id uuid);${BP}CREATE INDEX i ON "iss""ues" (id);`).compatibility).toBe("online_compatible");
|
||||
expect(classifyMigrationSql(`CREATE TABLE "Issues" (id uuid);${BP}DROP TABLE issues;`).compatibility).toBe("requires_staging");
|
||||
});
|
||||
|
||||
it("treats quoted, unquoted and schema-qualified names of a new table alike", () => {
|
||||
expect(classifyMigrationSql(`CREATE TABLE public.widgets (id uuid);${BP}CREATE INDEX "w" ON "widgets" (id);${BP}ALTER TABLE "public"."widgets" ADD COLUMN a text NOT NULL;`).compatibility)
|
||||
.toBe("online_compatible");
|
||||
});
|
||||
|
||||
it("does not take CREATE TABLE IF NOT EXISTS as proof that a table is new", () => {
|
||||
const sql = `CREATE TABLE IF NOT EXISTS issues (id uuid);${BP}ALTER TABLE issues DROP COLUMN title;${BP}CREATE INDEX "i" ON issues (id);`;
|
||||
expect(classifyMigrationSql(sql)).toMatchObject({ compatibility: "requires_staging" });
|
||||
expect(codes(sql)).toEqual(["create_table_if_not_exists", "drop_column", "lock_hazard_create_index"]);
|
||||
// An earlier migration created the table: it exists.
|
||||
expect(classifyMigrationSql(sql, { existingTables: new Set(["issues"]) }).compatibility).toBe("requires_staging");
|
||||
// No earlier migration created it: the table is new, whatever the spelling.
|
||||
expect(classifyMigrationSql(sql, { existingTables: new Set(["projects"]) }).compatibility).toBe("online_compatible");
|
||||
expect(classifyMigrationSql(sql.replace("issues (id uuid)", `"public"."issues" (id uuid)`), { existingTables: new Set() }).compatibility)
|
||||
.toBe("online_compatible");
|
||||
// A temporary table always belongs to the migration session.
|
||||
expect(classifyMigrationSql(`CREATE TEMP TABLE IF NOT EXISTS scratch (id uuid);${BP}DROP TABLE scratch;`).compatibility).toBe("online_compatible");
|
||||
});
|
||||
|
||||
it("classifies a sequence knowing the tables earlier migrations created", () => {
|
||||
const create = `DO $$ BEGIN IF to_regclass('public.widgets') IS NULL THEN CREATE TABLE "widgets" ("id" uuid); END IF; END $$;`;
|
||||
const repair = `CREATE TABLE IF NOT EXISTS "widgets" ("id" uuid);${BP}ALTER TABLE "widgets" ADD COLUMN "n" integer NOT NULL;`;
|
||||
const rename = `ALTER TABLE "widgets" RENAME TO "gadgets";`;
|
||||
const again = `CREATE TABLE IF NOT EXISTS "gadgets" ("id" uuid);${BP}CREATE INDEX "g" ON "gadgets" ("id");`;
|
||||
expect(collectCreatedTables(create)).toEqual(["widgets"]);
|
||||
expect(collectCreatedTables(rename)).toEqual(["gadgets"]);
|
||||
expect(classifyMigrationSequence([repair, create, repair, rename, again]).map((result) => result.compatibility))
|
||||
.toEqual(["online_compatible", "online_compatible", "requires_staging", "requires_staging", "unknown"]);
|
||||
expect(classifyMigrationSequence([repair], [create])[0].compatibility).toBe("requires_staging");
|
||||
});
|
||||
|
||||
it("reports whether a migration needs to run outside a transaction", () => {
|
||||
expect(classifyMigrationSql(`CREATE INDEX CONCURRENTLY "i" ON "issues" ("a");`).usesConcurrently).toBe(true);
|
||||
expect(classifyMigrationSql(`DROP INDEX CONCURRENTLY IF EXISTS "i";`).usesConcurrently).toBe(true);
|
||||
expect(classifyMigrationSql(`COMMENT ON TABLE "issues" IS 'built CONCURRENTLY';${BP}CREATE INDEX "i" ON "issues" ("a");`).usesConcurrently).toBe(false);
|
||||
});
|
||||
|
||||
it("does not treat an existing table as new because another table was created", () => {
|
||||
expect(classifyMigrationSql(`${NEW_TABLE}CREATE INDEX "i" ON "widgets_archive" ("id");`).compatibility).toBe("unknown");
|
||||
});
|
||||
|
||||
it("never classifies an unrecognized ADD CONSTRAINT form as a column", () => {
|
||||
expect(codes(`ALTER TABLE "issues" ADD CONSTRAINT "c" NOT NULL "title";`)).toEqual(["add_constraint_other"]);
|
||||
});
|
||||
|
||||
it("classifies every statement inside a guarded DO block", () => {
|
||||
const sql = `DO $$ BEGIN\n IF EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'x') THEN\n ALTER TABLE "issues" DROP CONSTRAINT "x";\n END IF;\n ALTER TABLE "issues" ADD CONSTRAINT "x" FOREIGN KEY ("a") REFERENCES "b"("id") NOT VALID;\nEND $$;`;
|
||||
expect(codes(sql)).toEqual(["drop_constraint", "add_constraint_not_valid"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("migration-compat override", () => {
|
||||
const hazard = `CREATE INDEX "i" ON "issues" ("a");`;
|
||||
|
||||
it("replaces the verdict and keeps the classifier's own result", () => {
|
||||
const result = classifyMigrationSql(`-- paperclip:migration-compat online_compatible: issues has under 1k rows on every stack\n${hazard}`);
|
||||
expect(result).toMatchObject({
|
||||
compatibility: "online_compatible",
|
||||
classified: "unknown",
|
||||
override: { compatibility: "online_compatible", justification: "issues has under 1k rows on every stack" },
|
||||
overrideError: null,
|
||||
});
|
||||
expect(result.reasons.map((reason) => reason.code)).toEqual(["lock_hazard_create_index"]);
|
||||
});
|
||||
|
||||
it("can mark a contract migration compatible and an additive one staged", () => {
|
||||
expect(classifyMigrationSql(`-- Contract step.\n\n-- paperclip:migration-compat online_compatible: no release since v0.3 reads issues.legacy\nALTER TABLE "issues" DROP COLUMN "legacy";`))
|
||||
.toMatchObject({ compatibility: "online_compatible", classified: "requires_staging" });
|
||||
expect(classifyMigrationSql(`-- paperclip:migration-compat requires_staging: old servers crash on the new enum value\nALTER TYPE "mood" ADD VALUE 'x';`))
|
||||
.toMatchObject({ compatibility: "requires_staging", classified: "online_compatible" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["missing justification", `-- paperclip:migration-compat online_compatible\n${hazard}`],
|
||||
["empty justification", `-- paperclip:migration-compat online_compatible: \n${hazard}`],
|
||||
["token justification", `-- paperclip:migration-compat online_compatible: ok\n${hazard}`],
|
||||
["unknown value", `-- paperclip:migration-compat unknown: we could not decide today\n${hazard}`],
|
||||
["unsupported value", `-- paperclip:migration-compat safe: this one is certainly fine\n${hazard}`],
|
||||
["missing colon", `-- paperclip:migration-compat online_compatible because it is fine\n${hazard}`],
|
||||
["two directives", `-- paperclip:migration-compat online_compatible: the table is tiny everywhere\n-- paperclip:migration-compat requires_staging: or maybe it is not\n${hazard}`],
|
||||
["after the first statement", `${hazard}\n-- paperclip:migration-compat online_compatible: the table is tiny everywhere\n`],
|
||||
["inside a block comment", `/* paperclip:migration-compat online_compatible: the table is tiny everywhere */\n${hazard}`],
|
||||
["on a breakpoint line", `--> statement-breakpoint paperclip:migration-compat online_compatible: the table is tiny\n${hazard}`],
|
||||
])("rejects an override with %s", (_name, sql) => {
|
||||
const result = classifyMigrationSql(sql);
|
||||
expect(result.override).toBeNull();
|
||||
expect(result.overrideError).toMatch(/paperclip:migration-compat/);
|
||||
expect(result.compatibility).toBe("unknown");
|
||||
expect(result.reasons.map((reason) => reason.code)).toContain("invalid_override");
|
||||
});
|
||||
|
||||
it("never lets a rejected override improve the verdict", () => {
|
||||
expect(classifyMigrationSql(`-- paperclip:migration-compat online_compatible\nDROP TABLE "issues";`).compatibility).toBe("requires_staging");
|
||||
expect(classifyMigrationSql(`-- paperclip:migration-compat online_compatible\nALTER TABLE "issues" ADD COLUMN "a" text;`).compatibility).toBe("unknown");
|
||||
});
|
||||
|
||||
it("reads nothing when the file has no directive", () => {
|
||||
expect(parseMigrationCompatOverride(`-- An ordinary comment about compat.\n${hazard}`)).toEqual({ override: null, error: null });
|
||||
});
|
||||
});
|
||||
|
||||
describe("checkNewMigrationCompatibility", () => {
|
||||
const additive = { fileName: "0500_additive.sql", sql: `ALTER TABLE "issues" ADD COLUMN "a" text;` };
|
||||
const unknown = { fileName: "0501_unknown.sql", sql: `CREATE INDEX "i" ON "issues" ("a");${BP}UPDATE "issues" SET a = 'x';` };
|
||||
const staged = { fileName: "0502_staged.sql", sql: `ALTER TABLE "issues" DROP COLUMN "a";` };
|
||||
|
||||
it("passes with no new migrations and with compatible ones", () => {
|
||||
expect(checkNewMigrationCompatibility([])).toMatchObject({ passed: true, errors: [], notices: [] });
|
||||
expect(checkNewMigrationCompatibility([additive])).toMatchObject({ passed: true, errors: [], notices: [] });
|
||||
});
|
||||
|
||||
it("fails an unknown migration and tells the author what to do", () => {
|
||||
const result = checkNewMigrationCompatibility([additive, unknown]);
|
||||
expect(result.passed).toBe(false);
|
||||
expect(result.errors).toHaveLength(1);
|
||||
const [error] = result.errors;
|
||||
expect(error).toContain("0501_unknown.sql");
|
||||
expect(error).toContain("lock_hazard_create_index (statement 1)");
|
||||
expect(error).toContain("lock_hazard_data_backfill (statement 2)");
|
||||
expect(error).toContain("CREATE INDEX CONCURRENTLY");
|
||||
expect(error).toContain("expand migration now and a contract migration in a later release");
|
||||
expect(error).toContain("-- paperclip:migration-compat online_compatible: <why");
|
||||
expect(error).toContain("-- paperclip:migration-compat requires_staging: <what");
|
||||
expect(error).toContain("doc/DATABASE.md");
|
||||
});
|
||||
|
||||
it("passes an unknown migration that carries a justified override, with a notice", () => {
|
||||
const result = checkNewMigrationCompatibility([{ ...unknown, sql: `-- paperclip:migration-compat online_compatible: issues is small and the update is idempotent\n${unknown.sql}` }]);
|
||||
expect(result).toMatchObject({ passed: true, errors: [] });
|
||||
expect(result.notices).toEqual([expect.stringMatching(/0501_unknown\.sql: online_compatible by override \(classifier said unknown\): issues is small/)]);
|
||||
});
|
||||
|
||||
it("fails an override without a justification, even on a compatible migration", () => {
|
||||
const result = checkNewMigrationCompatibility([{ ...additive, sql: `-- paperclip:migration-compat online_compatible:\n${additive.sql}` }]);
|
||||
expect(result.passed).toBe(false);
|
||||
expect(result.errors).toEqual([expect.stringMatching(/0500_additive\.sql: .*needs a justification/)]);
|
||||
});
|
||||
|
||||
it("fails a CONCURRENTLY migration that the migrator would run inside a transaction", () => {
|
||||
const concurrent = { fileName: "packages/db/src/migrations/0503_index.sql", sql: `CREATE INDEX CONCURRENTLY IF NOT EXISTS "i" ON "issues" ("a");` };
|
||||
const result = checkNewMigrationCompatibility([concurrent], { nonTransactionalMigrations: new Set(["0313_private_task_access.sql"]) });
|
||||
expect(result.passed).toBe(false);
|
||||
expect(result.errors).toEqual([expect.stringMatching(/0503_index\.sql: uses CONCURRENTLY[\s\S]*Add "0503_index\.sql" to MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS in packages\/db\/src\/client\.ts/)]);
|
||||
expect(checkNewMigrationCompatibility([concurrent], { nonTransactionalMigrations: new Set(["0503_index.sql"]) }).passed).toBe(true);
|
||||
// Without the list (a caller that cannot read the migrator) the check is skipped.
|
||||
expect(checkNewMigrationCompatibility([concurrent]).passed).toBe(true);
|
||||
});
|
||||
|
||||
it("judges new migrations after the ones already on the target branch", () => {
|
||||
const repair = { fileName: "0504_repair.sql", sql: `CREATE TABLE IF NOT EXISTS "widgets" ("id" uuid);${BP}CREATE INDEX "w" ON "widgets" ("id");` };
|
||||
expect(checkNewMigrationCompatibility([repair], { priorMigrationSql: [] }).passed).toBe(true);
|
||||
expect(checkNewMigrationCompatibility([repair], { priorMigrationSql: [`CREATE TABLE "widgets" ("id" uuid);`] }).passed).toBe(false);
|
||||
expect(checkNewMigrationCompatibility([repair]).passed).toBe(false);
|
||||
// The second new migration sees the table the first one created.
|
||||
expect(checkNewMigrationCompatibility([repair, { ...repair, fileName: "0505_again.sql" }], { priorMigrationSql: [] }).errors)
|
||||
.toEqual([expect.stringContaining("0505_again.sql")]);
|
||||
});
|
||||
|
||||
it("does not fail requires_staging, and prints a notice instead", () => {
|
||||
const result = checkNewMigrationCompatibility([staged]);
|
||||
expect(result).toMatchObject({ passed: true, errors: [] });
|
||||
expect(result.notices).toEqual([expect.stringMatching(/0502_staged\.sql: requires_staging[\s\S]*drop_column \(statement 1\)[\s\S]*does not fail the check yet/)]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the shipped migrations", () => {
|
||||
const dbDir = fileURLToPath(new URL("..", import.meta.url));
|
||||
const migrationsDir = `${dbDir}src/migrations`;
|
||||
const journal = JSON.parse(readFileSync(`${migrationsDir}/meta/_journal.json`, "utf8")) as { entries: { idx: number; tag: string }[] };
|
||||
const files = [...journal.entries]
|
||||
.sort((left, right) => left.idx - right.idx || left.tag.localeCompare(right.tag))
|
||||
.map((entry) => `${entry.tag}.sql`);
|
||||
const classified = classifyMigrationSequence(files.map((file) => readFileSync(`${migrationsDir}/${file}`, "utf8")));
|
||||
const results = files.map((file, index) => ({ file, result: classified[index] }));
|
||||
|
||||
it("classifies every file without an invalid override and with a consistent summary", () => {
|
||||
expect(results.filter(({ result }) => result.overrideError).map(({ file }) => file)).toEqual([]);
|
||||
const summary = summarizeMigrationCompatibility(results.map(({ result }) => result));
|
||||
expect(summary.total).toBe(results.length);
|
||||
expect(Object.values(summary.byCompatibility).reduce((sum, count) => sum + count, 0)).toBe(results.length);
|
||||
// The rules must keep recognizing the common additive shapes.
|
||||
expect(summary.byCompatibility.online_compatible).toBeGreaterThan(results.length * 0.4);
|
||||
});
|
||||
|
||||
it("keeps known hazardous migrations out of online_compatible", () => {
|
||||
const verdict = (prefix: string) => results.find(({ file }) => file.startsWith(prefix))?.result.compatibility;
|
||||
// 0224 adds a NOT NULL column without a default to a live table.
|
||||
expect(verdict("0224_")).toBe("requires_staging");
|
||||
// 0313 backfills privacy columns in batched loops.
|
||||
expect(verdict("0313_")).not.toBe("online_compatible");
|
||||
// 0466 drops a column the previous release wrote.
|
||||
expect(verdict("0466_")).toBe("requires_staging");
|
||||
});
|
||||
|
||||
it("finds the migrator's non-transactional list, which holds every CONCURRENTLY migration", () => {
|
||||
const listed = parseNonTransactionalMigrations(readFileSync(`${dbDir}src/client.ts`, "utf8"));
|
||||
expect(listed?.has("0313_private_task_access.sql")).toBe(true);
|
||||
expect(results.filter(({ file, result }) => result.usesConcurrently && !listed?.has(file)).map(({ file }) => file)).toEqual([]);
|
||||
expect(parseNonTransactionalMigrations("const other = new Set([]);")).toBeNull();
|
||||
});
|
||||
|
||||
it("classifies the files named on the command line, after the migrations that precede them", () => {
|
||||
const run = (...args: string[]) => execFileSync(process.execPath, ["--import", "tsx", "scripts/migration-compatibility.ts", ...args], { cwd: dbDir, encoding: "utf8" });
|
||||
const one = run("src/migrations/0224_unified_adapter_auth_sessions.sql");
|
||||
expect(one).toMatch(/^0224_unified_adapter_auth_sessions\.sql: requires_staging\n/);
|
||||
expect(one).toContain("add_not_null_column_without_default");
|
||||
expect(one.match(/^\S+\.sql: /gm)).toHaveLength(1);
|
||||
const two = JSON.parse(run("--json", "src/migrations/0468_busy_wendell_rand.sql", "src/migrations/0466_drop_project_environment.sql")) as { fileName: string; compatibility: string }[];
|
||||
expect(two.map(({ fileName, compatibility }) => [fileName, compatibility])).toEqual([
|
||||
["0468_busy_wendell_rand.sql", "online_compatible"],
|
||||
["0466_drop_project_environment.sql", "requires_staging"],
|
||||
]);
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -0,0 +1,679 @@
|
||||
// Classifies whether a migration can be applied while the previous release's
|
||||
// application code is still reading and writing the same database.
|
||||
//
|
||||
// This file is loaded with Node's built-in type stripping by the release and
|
||||
// policy scripts, which run before dependencies are installed. Keep it free of
|
||||
// imports and of TypeScript syntax that needs transpiling (enums, namespaces,
|
||||
// parameter properties).
|
||||
|
||||
export type MigrationCompatibility = "online_compatible" | "requires_staging" | "unknown";
|
||||
|
||||
export const MIGRATION_COMPATIBILITY_VALUES: readonly MigrationCompatibility[] = [
|
||||
"online_compatible",
|
||||
"requires_staging",
|
||||
"unknown",
|
||||
];
|
||||
|
||||
export const MIGRATION_COMPAT_DIRECTIVE = "paperclip:migration-compat";
|
||||
export const MIGRATION_COMPAT_MIN_JUSTIFICATION_LENGTH = 10;
|
||||
|
||||
type ReasonDefinition = {
|
||||
readonly compatibility: MigrationCompatibility;
|
||||
readonly message: string;
|
||||
};
|
||||
|
||||
// Codes that start with `lock_hazard_` mark statements whose only known problem
|
||||
// is how long they hold a lock that blocks the old release. They classify as
|
||||
// `unknown`, not as compatible with a warning: a deployer that may not pause
|
||||
// must not start a statement whose duration grows with table size.
|
||||
const REASONS = {
|
||||
create_table: { compatibility: "online_compatible", message: "Creates a table the previous release does not use." },
|
||||
create_table_if_not_exists: { compatibility: "online_compatible", message: "Creates a table only when it is missing. It may already exist, so later statements on it are judged as changes to an existing table." },
|
||||
create_temp_table: { compatibility: "online_compatible", message: "Creates a temporary table visible only to the migration session." },
|
||||
alter_new_table: { compatibility: "online_compatible", message: "Changes a table created earlier in this migration." },
|
||||
create_index_new_table: { compatibility: "online_compatible", message: "Builds an index on a table created earlier in this migration." },
|
||||
create_index_concurrently: { compatibility: "online_compatible", message: "Builds a non-unique index without blocking writes." },
|
||||
add_column: { compatibility: "online_compatible", message: "Adds a column that is nullable or has a constant default." },
|
||||
add_constraint_not_valid: { compatibility: "online_compatible", message: "Adds a NOT VALID constraint, which does not scan existing rows." },
|
||||
drop_not_null: { compatibility: "online_compatible", message: "Relaxes a NOT NULL column." },
|
||||
create_type: { compatibility: "online_compatible", message: "Creates a type the previous release does not use." },
|
||||
add_enum_value: { compatibility: "online_compatible", message: "Adds a value to an enum type." },
|
||||
create_sequence: { compatibility: "online_compatible", message: "Creates a sequence the previous release does not use." },
|
||||
create_extension: { compatibility: "online_compatible", message: "Installs an extension." },
|
||||
create_function: { compatibility: "online_compatible", message: "Creates a function that did not exist before." },
|
||||
create_view: { compatibility: "online_compatible", message: "Creates a view that did not exist before." },
|
||||
create_trigger_new_table: { compatibility: "online_compatible", message: "Creates a trigger on a table created earlier in this migration." },
|
||||
comment: { compatibility: "online_compatible", message: "Sets a catalog comment." },
|
||||
create_statistics: { compatibility: "online_compatible", message: "Creates extended planner statistics without blocking reads or writes." },
|
||||
analyze: { compatibility: "online_compatible", message: "Refreshes planner statistics without blocking reads or writes." },
|
||||
no_op: { compatibility: "online_compatible", message: "Has no effect on the schema or data." },
|
||||
seed_new_table: { compatibility: "online_compatible", message: "Writes rows to a table created earlier in this migration." },
|
||||
|
||||
drop_table: { compatibility: "requires_staging", message: "Drops a table the previous release may still use." },
|
||||
drop_column: { compatibility: "requires_staging", message: "Drops a column the previous release may still read or write." },
|
||||
rename_table: { compatibility: "requires_staging", message: "Renames a table the previous release still addresses by its old name." },
|
||||
rename_column: { compatibility: "requires_staging", message: "Renames a column the previous release still addresses by its old name." },
|
||||
alter_column_type: { compatibility: "requires_staging", message: "Changes a column type, which rewrites the table and can reject the previous release's values." },
|
||||
set_not_null: { compatibility: "requires_staging", message: "Sets NOT NULL on a column the previous release may still leave empty." },
|
||||
add_not_null_column_without_default: { compatibility: "requires_staging", message: "Adds a NOT NULL column without a default; inserts from the previous release fail." },
|
||||
drop_default: { compatibility: "requires_staging", message: "Drops a default the previous release may rely on when it inserts." },
|
||||
add_validated_check: { compatibility: "requires_staging", message: "Adds a validated CHECK constraint that the previous release's writes could violate." },
|
||||
add_unique_constraint: { compatibility: "requires_staging", message: "Adds a UNIQUE or PRIMARY KEY constraint to an existing table that the previous release's writes could violate." },
|
||||
create_unique_index_existing_table: { compatibility: "requires_staging", message: "Adds a unique index to an existing table that the previous release's writes could violate." },
|
||||
drop_type: { compatibility: "requires_staging", message: "Drops a type the previous release may still use." },
|
||||
rename_type: { compatibility: "requires_staging", message: "Renames a type or enum value the previous release still uses." },
|
||||
truncate: { compatibility: "requires_staging", message: "Truncates a table the previous release is using." },
|
||||
delete_rows: { compatibility: "requires_staging", message: "Deletes rows from a table the previous release is using." },
|
||||
|
||||
lock_hazard_create_index: { compatibility: "unknown", message: "Builds an index on an existing table without CONCURRENTLY, blocking writes for the whole build." },
|
||||
lock_hazard_validated_foreign_key: { compatibility: "unknown", message: "Adds a validated foreign key to an existing table, scanning it while writes are blocked. Add it NOT VALID instead." },
|
||||
lock_hazard_validate_constraint: { compatibility: "unknown", message: "Validates a constraint by scanning the whole table." },
|
||||
lock_hazard_set_not_null: { compatibility: "unknown", message: "Sets NOT NULL on a column added in this migration, scanning the table under an exclusive lock." },
|
||||
lock_hazard_table_rewrite: { compatibility: "unknown", message: "Adds a column whose default, identity, or generated expression rewrites the whole table under an exclusive lock." },
|
||||
lock_hazard_add_column_constraint: { compatibility: "unknown", message: "Adds a column with an inline constraint, which scans or indexes the table under an exclusive lock." },
|
||||
lock_hazard_data_backfill: { compatibility: "unknown", message: "Rewrites rows of an existing table; duration and row locks grow with table size, and the previous release keeps writing the old shape." },
|
||||
lock_hazard_copy_into_new_table: { compatibility: "unknown", message: "Copies rows from live tables into a new table; duration grows with the source while this migration's locks are held." },
|
||||
lock_hazard_lock_table: { compatibility: "unknown", message: "Takes an explicit table lock." },
|
||||
|
||||
data_change_existing_table: { compatibility: "unknown", message: "Inserts rows into a table the previous release is using." },
|
||||
set_default: { compatibility: "unknown", message: "Changes the default the previous release receives when it inserts." },
|
||||
drop_constraint: { compatibility: "unknown", message: "Drops a constraint; the previous release may rely on it for upserts or cascades." },
|
||||
drop_index: { compatibility: "unknown", message: "Drops an index; the previous release may rely on it for upserts or for query speed." },
|
||||
drop_object: { compatibility: "unknown", message: "Drops a database object the previous release may depend on." },
|
||||
add_constraint_other: { compatibility: "unknown", message: "Adds a constraint the classifier cannot prove safe." },
|
||||
replace_function: { compatibility: "unknown", message: "Replaces a function, which can change behaviour the previous release depends on." },
|
||||
replace_view: { compatibility: "unknown", message: "Replaces or materializes a view." },
|
||||
create_trigger_existing_table: { compatibility: "unknown", message: "Adds a trigger to an existing table, changing what the previous release's writes do." },
|
||||
create_table_as: { compatibility: "unknown", message: "Creates a table from a query; its duration grows with the source tables." },
|
||||
do_block_procedural: { compatibility: "unknown", message: "Runs a procedural DO block the classifier does not interpret." },
|
||||
unrecognized_alter_table_action: { compatibility: "unknown", message: "Uses an ALTER TABLE action the classifier does not recognize." },
|
||||
unrecognized_statement: { compatibility: "unknown", message: "Uses a statement the classifier does not recognize." },
|
||||
no_statements: { compatibility: "unknown", message: "Contains no SQL statements." },
|
||||
invalid_override: { compatibility: "unknown", message: "Has a malformed migration-compat override." },
|
||||
} as const satisfies Record<string, ReasonDefinition>;
|
||||
|
||||
export type MigrationCompatReasonCode = keyof typeof REASONS;
|
||||
|
||||
export const MIGRATION_COMPAT_REASON_CODES = Object.keys(REASONS) as MigrationCompatReasonCode[];
|
||||
|
||||
export type MigrationCompatReason = {
|
||||
readonly code: MigrationCompatReasonCode;
|
||||
readonly compatibility: MigrationCompatibility;
|
||||
readonly message: string;
|
||||
/** 1-based positions of the top-level statements that produced this reason. */
|
||||
readonly statements: readonly number[];
|
||||
};
|
||||
|
||||
export type MigrationCompatOverride = {
|
||||
readonly compatibility: Exclude<MigrationCompatibility, "unknown">;
|
||||
readonly justification: string;
|
||||
};
|
||||
|
||||
export type MigrationCompatResult = {
|
||||
/** The verdict a deployer acts on: the override when one is present and valid. */
|
||||
readonly compatibility: MigrationCompatibility;
|
||||
/** The classifier's own verdict, before any override. */
|
||||
readonly classified: MigrationCompatibility;
|
||||
readonly reasons: readonly MigrationCompatReason[];
|
||||
readonly override: MigrationCompatOverride | null;
|
||||
/** Set when the file carries an override directive that cannot be honoured. */
|
||||
readonly overrideError: string | null;
|
||||
/** True when a statement uses CONCURRENTLY, which cannot run in a transaction. */
|
||||
readonly usesConcurrently: boolean;
|
||||
};
|
||||
|
||||
export type MigrationCompatOptions = {
|
||||
/**
|
||||
* Tables created by the migrations that apply before this one. Without it,
|
||||
* CREATE TABLE IF NOT EXISTS is never proof that a table is new.
|
||||
*/
|
||||
readonly existingTables?: ReadonlySet<string>;
|
||||
};
|
||||
|
||||
type Statement = {
|
||||
/** Lowercase, comments removed, literals masked as '?', dollar bodies as $?$. */
|
||||
readonly text: string;
|
||||
readonly dollarBodies: readonly string[];
|
||||
};
|
||||
|
||||
type Context = {
|
||||
readonly existingTables: ReadonlySet<string> | null;
|
||||
readonly createdTables: Set<string>;
|
||||
readonly addedColumns: Set<string>;
|
||||
};
|
||||
|
||||
const IDENT = String.raw`(?:"[^"]+"|[a-z_][a-z0-9_$]*)`;
|
||||
const QUALIFIED = String.raw`(?:${IDENT}\.)?${IDENT}`;
|
||||
|
||||
function dollarTagAt(sql: string, index: number): string | null {
|
||||
const match = /^\$(?:[A-Za-z_][A-Za-z0-9_]*)?\$/.exec(sql.slice(index, index + 130));
|
||||
return match ? match[0] : null;
|
||||
}
|
||||
|
||||
/** Splits SQL into statements and normalizes each for keyword matching. */
|
||||
function parseStatements(sql: string): Statement[] {
|
||||
const statements: Statement[] = [];
|
||||
let text = "";
|
||||
let dollarBodies: string[] = [];
|
||||
const flush = () => {
|
||||
const normalized = text.replace(/\s+/g, " ").trim();
|
||||
if (normalized) statements.push({ text: normalized, dollarBodies });
|
||||
text = "";
|
||||
dollarBodies = [];
|
||||
};
|
||||
for (let i = 0; i < sql.length;) {
|
||||
const char = sql[i];
|
||||
const next = sql[i + 1];
|
||||
if (char === "-" && next === "-") {
|
||||
const end = sql.indexOf("\n", i);
|
||||
const comment = sql.slice(i, end === -1 ? sql.length : end);
|
||||
// Drizzle's separator is a comment; it always ends a statement.
|
||||
if (/^-->\s*statement-breakpoint/.test(comment)) flush();
|
||||
else text += " ";
|
||||
i = end === -1 ? sql.length : end;
|
||||
} else if (char === "/" && next === "*") {
|
||||
let depth = 1;
|
||||
i += 2;
|
||||
while (i < sql.length && depth > 0) {
|
||||
if (sql[i] === "/" && sql[i + 1] === "*") { depth += 1; i += 2; }
|
||||
else if (sql[i] === "*" && sql[i + 1] === "/") { depth -= 1; i += 2; }
|
||||
else i += 1;
|
||||
}
|
||||
text += " ";
|
||||
} else if (char === "'") {
|
||||
// E'...' strings escape quotes with a backslash.
|
||||
const escapes = /(?:^|[^a-z0-9_$])e$/.test(text);
|
||||
i += 1;
|
||||
while (i < sql.length) {
|
||||
if (escapes && sql[i] === "\\") i += 2;
|
||||
else if (sql[i] === "'" && sql[i + 1] === "'") i += 2;
|
||||
else if (sql[i] === "'") break;
|
||||
else i += 1;
|
||||
}
|
||||
i += 1;
|
||||
text += "'?'";
|
||||
} else if (char === '"') {
|
||||
// A doubled quote is a literal quote inside the identifier.
|
||||
let name = "";
|
||||
i += 1;
|
||||
while (i < sql.length) {
|
||||
if (sql[i] === '"' && sql[i + 1] === '"') { name += '"'; i += 2; }
|
||||
else if (sql[i] === '"') break;
|
||||
else { name += sql[i]; i += 1; }
|
||||
}
|
||||
i += 1;
|
||||
// Plain lowercase names are equivalent quoted or not. Any other name
|
||||
// stays quoted, with its own quotes encoded so it remains one token.
|
||||
text += /^[a-z_][a-z0-9_$]*$/.test(name) ? name : `"${name.replaceAll("%", "%25").replaceAll('"', "%22")}"`;
|
||||
} else if (char === "$" && dollarTagAt(sql, i)) {
|
||||
const tag = dollarTagAt(sql, i)!;
|
||||
const end = sql.indexOf(tag, i + tag.length);
|
||||
dollarBodies.push(sql.slice(i + tag.length, end === -1 ? sql.length : end));
|
||||
text += " $?$ ";
|
||||
i = end === -1 ? sql.length : end + tag.length;
|
||||
} else if (char === ";") {
|
||||
flush();
|
||||
i += 1;
|
||||
} else {
|
||||
text += char.toLowerCase();
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
flush();
|
||||
return statements;
|
||||
}
|
||||
|
||||
function tableName(name: string): string {
|
||||
return name.replace(/^only /, "").replace(/^public\./, "");
|
||||
}
|
||||
|
||||
/** Returns the index after the parenthesis group that opens at `start`, or -1. */
|
||||
function closingParen(text: string, start: number): number {
|
||||
let depth = 0;
|
||||
for (let i = start; i < text.length; i += 1) {
|
||||
if (text[i] === "(") depth += 1;
|
||||
else if (text[i] === ")") {
|
||||
depth -= 1;
|
||||
if (depth === 0) return i + 1;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
function splitTopLevelCommas(text: string): string[] {
|
||||
const parts: string[] = [];
|
||||
let depth = 0;
|
||||
let current = "";
|
||||
for (const char of text) {
|
||||
if (char === "(" || char === "[") depth += 1;
|
||||
else if (char === ")" || char === "]") depth -= 1;
|
||||
if (char === "," && depth === 0) {
|
||||
parts.push(current.trim());
|
||||
current = "";
|
||||
} else current += char;
|
||||
}
|
||||
if (current.trim()) parts.push(current.trim());
|
||||
return parts;
|
||||
}
|
||||
|
||||
// A default that PostgreSQL stores in the catalog without rewriting the table:
|
||||
// a literal, or a non-volatile timestamp function.
|
||||
const CONSTANT_DEFAULT = new RegExp(
|
||||
String.raw`^\(?(?:'\?'|-?\d+(?:\.\d+)?|true|false|null|now\(\)|current_timestamp|array\[\])\)?` +
|
||||
String.raw`(?:::[a-z_][a-z0-9_ ]*(?:\(\d+(?:, ?\d+)?\))?(?:\[\])?)*$`,
|
||||
);
|
||||
|
||||
function classifyAddColumn(table: string, column: string, definition: string, context: Context): MigrationCompatReasonCode {
|
||||
if (/\bgenerated\b|\b(?:small|big)?serial\b/.test(definition)) return "lock_hazard_table_rewrite";
|
||||
if (/\b(?:references|unique|primary key|check|exclude)\b/.test(definition)) return "lock_hazard_add_column_constraint";
|
||||
const notNull = /\bnot null\b/.test(definition);
|
||||
const defaultMatch = /\bdefault (.+?)(?: not null| null)?$/.exec(definition);
|
||||
if (defaultMatch && !CONSTANT_DEFAULT.test(defaultMatch[1].trim())) return "lock_hazard_table_rewrite";
|
||||
if (notNull && !defaultMatch) return "add_not_null_column_without_default";
|
||||
context.addedColumns.add(`${table}.${column}`);
|
||||
return "add_column";
|
||||
}
|
||||
|
||||
function classifyAlterTableAction(table: string, action: string, context: Context): MigrationCompatReasonCode {
|
||||
let match: RegExpExecArray | null;
|
||||
if ((match = new RegExp(String.raw`^add (?:constraint ${IDENT} )?(foreign key|check|unique|primary key|exclude)\b(.*)$`).exec(action))) {
|
||||
if (/\bnot valid$/.test(match[2])) return "add_constraint_not_valid";
|
||||
if (match[1] === "foreign key") return "lock_hazard_validated_foreign_key";
|
||||
if (match[1] === "check") return "add_validated_check";
|
||||
if (match[1] === "exclude" || /\busing index\b/.test(match[2])) return "add_constraint_other";
|
||||
return "add_unique_constraint";
|
||||
}
|
||||
if (/^add constraint\b/.test(action)) return "add_constraint_other";
|
||||
if ((match = new RegExp(String.raw`^add (?:column )?(?:if not exists )?(${IDENT}) (.+)$`).exec(action))) {
|
||||
return classifyAddColumn(table, match[1], match[2], context);
|
||||
}
|
||||
if (/^drop constraint\b/.test(action)) return "drop_constraint";
|
||||
if (/^drop\b/.test(action)) return "drop_column";
|
||||
if (/^rename constraint\b/.test(action)) return "unrecognized_alter_table_action";
|
||||
if (/^rename to\b/.test(action)) return "rename_table";
|
||||
if (/^rename\b/.test(action)) return "rename_column";
|
||||
if (/^validate constraint\b/.test(action)) return "lock_hazard_validate_constraint";
|
||||
if ((match = new RegExp(String.raw`^alter (?:column )?(${IDENT}) (.+)$`).exec(action))) {
|
||||
const change = match[2];
|
||||
if (/^(?:set data )?type\b/.test(change)) return "alter_column_type";
|
||||
if (change === "set not null") {
|
||||
return context.addedColumns.has(`${table}.${match[1]}`) ? "lock_hazard_set_not_null" : "set_not_null";
|
||||
}
|
||||
if (change === "drop not null") return "drop_not_null";
|
||||
if (/^set default\b/.test(change)) return "set_default";
|
||||
if (change === "drop default") return "drop_default";
|
||||
}
|
||||
return "unrecognized_alter_table_action";
|
||||
}
|
||||
|
||||
function classifyDoBlock(statement: Statement, context: Context, nested: boolean): MigrationCompatReasonCode[] {
|
||||
if (nested || statement.dollarBodies.length !== 1 || !/^do (?:language plpgsql )?\$\?\$(?: language plpgsql)?$/.test(statement.text)) {
|
||||
return ["do_block_procedural"];
|
||||
}
|
||||
const pieces = parseStatements(statement.dollarBodies[0]);
|
||||
const procedural = pieces.length === 0 || !/^begin\b/.test(pieces[0].text) ||
|
||||
pieces.some((piece) => /^(?:begin )?(?:declare|loop|for|foreach|while|execute|perform|raise|return|else|elsif)\b/.test(piece.text) ||
|
||||
/(?:^| )then (?:execute|perform|raise|return|loop|for|foreach|while)\b/.test(piece.text));
|
||||
if (procedural) {
|
||||
const codes: MigrationCompatReasonCode[] = ["do_block_procedural"];
|
||||
if (pieces.some((piece) => /\b(?:update|insert into|delete from)\b/.test(piece.text))) codes.push("lock_hazard_data_backfill");
|
||||
return codes;
|
||||
}
|
||||
|
||||
// What remains is a guarded sequence: BEGIN <sql>; ... [EXCEPTION WHEN ...
|
||||
// THEN NULL;] END, where <sql> may sit inside IF [NOT] EXISTS (...) THEN ...
|
||||
// END IF. A guard only makes a statement conditional, so the statement is
|
||||
// classified as if it always runs.
|
||||
const codes: MigrationCompatReasonCode[] = [];
|
||||
let depth = 0;
|
||||
let closed = false;
|
||||
for (let index = 0; index < pieces.length; index += 1) {
|
||||
let text = pieces[index].text;
|
||||
if (index === 0) text = text.replace(/^begin\b ?/, "");
|
||||
if (closed) return ["do_block_procedural"];
|
||||
while (/^if\b/.test(text)) {
|
||||
let conditionEnd = -1;
|
||||
const exists = /^if (?:not )?exists ?\(/.exec(text);
|
||||
const regclass = /^if to_regclass\('\?'\) is (?:not )?null/.exec(text);
|
||||
if (exists) conditionEnd = closingParen(text, exists[0].length - 1);
|
||||
else if (regclass) conditionEnd = regclass[0].length;
|
||||
if (conditionEnd === -1 || !text.slice(conditionEnd).startsWith(" then ")) return ["do_block_procedural"];
|
||||
text = text.slice(conditionEnd + " then ".length);
|
||||
depth += 1;
|
||||
}
|
||||
if (text === "" || text === "null") continue;
|
||||
if (text === "end if") {
|
||||
depth -= 1;
|
||||
if (depth < 0) return ["do_block_procedural"];
|
||||
} else if (text === "end") {
|
||||
if (depth !== 0 || index !== pieces.length - 1) return ["do_block_procedural"];
|
||||
closed = true;
|
||||
} else if (/^exception when [a-z_ ]+ then null$/.test(text)) {
|
||||
if (depth !== 0) return ["do_block_procedural"];
|
||||
} else {
|
||||
codes.push(...classifyStatement({ text, dollarBodies: pieces[index].dollarBodies }, context, true));
|
||||
}
|
||||
}
|
||||
if (!closed || codes.length === 0) return ["do_block_procedural"];
|
||||
return codes;
|
||||
}
|
||||
|
||||
function classifyStatement(statement: Statement, context: Context, nested = false): MigrationCompatReasonCode[] {
|
||||
const text = statement.text;
|
||||
const isNew = (name: string) => context.createdTables.has(tableName(name));
|
||||
let match: RegExpExecArray | null;
|
||||
|
||||
if (/^do\b/.test(text)) return classifyDoBlock(statement, context, nested);
|
||||
|
||||
if ((match = new RegExp(String.raw`^create (?:(?:global |local )?(temp|temporary) |unlogged )?table (if not exists )?(${QUALIFIED})(.*)$`).exec(text))) {
|
||||
const temporary = Boolean(match[1]);
|
||||
const name = tableName(match[3]);
|
||||
// Plain CREATE TABLE fails when the table exists, and a temporary table
|
||||
// belongs to this session. IF NOT EXISTS proves nothing by itself: the
|
||||
// table is new only when no earlier migration created it.
|
||||
const provenNew = temporary || !match[2] || (context.existingTables !== null && !context.existingTables.has(name));
|
||||
let rest = match[4].trim();
|
||||
if (rest.startsWith("(")) {
|
||||
const end = closingParen(rest, 0);
|
||||
rest = end === -1 ? rest : rest.slice(end);
|
||||
}
|
||||
if (/\b(?:partition of|inherits|like)\b/.test(rest)) return ["unrecognized_statement"];
|
||||
if (/\bas (?:select|with|values|table)\b/.test(rest)) {
|
||||
// A temporary copy only reads; it is still registered as migration-owned.
|
||||
if (temporary) context.createdTables.add(name);
|
||||
return ["create_table_as"];
|
||||
}
|
||||
if (!provenNew) return ["create_table_if_not_exists"];
|
||||
context.createdTables.add(name);
|
||||
return [temporary ? "create_temp_table" : "create_table"];
|
||||
}
|
||||
|
||||
if ((match = new RegExp(String.raw`^create (unique )?index (concurrently )?(?:if not exists )?(?:${IDENT} )?on (?:only )?(${QUALIFIED})[ (]`).exec(text))) {
|
||||
if (isNew(match[3])) return ["create_index_new_table"];
|
||||
if (match[1]) return ["create_unique_index_existing_table"];
|
||||
return [match[2] ? "create_index_concurrently" : "lock_hazard_create_index"];
|
||||
}
|
||||
|
||||
if ((match = new RegExp(String.raw`^alter table (?:if exists )?(?:only )?(${QUALIFIED}) (.+)$`).exec(text))) {
|
||||
if (isNew(match[1])) return ["alter_new_table"];
|
||||
const table = tableName(match[1]);
|
||||
return splitTopLevelCommas(match[2]).map((action) => classifyAlterTableAction(table, action, context));
|
||||
}
|
||||
|
||||
if ((match = new RegExp(String.raw`^drop table (?:if exists )?(.+?)(?: cascade| restrict)?$`).exec(text))) {
|
||||
return splitTopLevelCommas(match[1]).every(isNew) ? ["alter_new_table"] : ["drop_table"];
|
||||
}
|
||||
if (/^drop index\b/.test(text)) return ["drop_index"];
|
||||
if (/^drop (?:type|domain)\b/.test(text)) return ["drop_type"];
|
||||
if (/^drop\b/.test(text)) return ["drop_object"];
|
||||
if (/^truncate\b/.test(text)) return ["truncate"];
|
||||
|
||||
if ((match = new RegExp(String.raw`^delete from (?:only )?(${QUALIFIED})\b`).exec(text))) {
|
||||
return [isNew(match[1]) ? "seed_new_table" : "delete_rows"];
|
||||
}
|
||||
if ((match = new RegExp(String.raw`^update (?:only )?(${QUALIFIED})\b`).exec(text))) {
|
||||
return [isNew(match[1]) ? "seed_new_table" : "lock_hazard_data_backfill"];
|
||||
}
|
||||
if ((match = new RegExp(String.raw`^insert into (${QUALIFIED})\b(.*)$`).exec(text))) {
|
||||
if (!isNew(match[1])) return ["data_change_existing_table"];
|
||||
// Copying from live tables can run long while this migration's locks are held.
|
||||
return [/\b(?:select|with|table)\b/.test(match[2]) ? "lock_hazard_copy_into_new_table" : "seed_new_table"];
|
||||
}
|
||||
if (/^with\b/.test(text)) {
|
||||
if (/\bdelete from\b/.test(text)) return ["delete_rows"];
|
||||
if (/\b(?:update|insert into)\b/.test(text)) return ["lock_hazard_data_backfill"];
|
||||
return ["unrecognized_statement"];
|
||||
}
|
||||
|
||||
if (/^create type\b/.test(text)) return ["create_type"];
|
||||
if (new RegExp(String.raw`^alter type ${QUALIFIED} add value\b`).test(text)) return ["add_enum_value"];
|
||||
if (new RegExp(String.raw`^alter type ${QUALIFIED} rename\b`).test(text)) return ["rename_type"];
|
||||
if (/^create (?:temp |temporary )?sequence\b/.test(text)) return ["create_sequence"];
|
||||
if (/^create extension\b/.test(text)) return ["create_extension"];
|
||||
if (/^create (?:function|procedure)\b/.test(text)) return ["create_function"];
|
||||
if (/^create or replace (?:function|procedure)\b/.test(text)) return ["replace_function"];
|
||||
if (/^create (?:temp |temporary )?view\b/.test(text)) return ["create_view"];
|
||||
if (/^create (?:or replace |materialized )(?:temp |temporary )?view\b/.test(text)) return ["replace_view"];
|
||||
if ((match = new RegExp(String.raw`^create (?:or replace )?(?:constraint )?trigger ${IDENT} .*? on (?:only )?(${QUALIFIED})\b`).exec(text))) {
|
||||
return [isNew(match[1]) && !/^create or replace\b/.test(text) ? "create_trigger_new_table" : "create_trigger_existing_table"];
|
||||
}
|
||||
if (/^comment on\b/.test(text)) return ["comment"];
|
||||
if (/^create statistics\b/.test(text)) return ["create_statistics"];
|
||||
if (/^analyze\b/.test(text)) return ["analyze"];
|
||||
if (text === "select 1") return ["no_op"];
|
||||
if (/^lock (?:table )?/.test(text)) return ["lock_hazard_lock_table"];
|
||||
return ["unrecognized_statement"];
|
||||
}
|
||||
|
||||
function worst(values: readonly MigrationCompatibility[]): MigrationCompatibility {
|
||||
if (values.includes("requires_staging")) return "requires_staging";
|
||||
if (values.includes("unknown")) return "unknown";
|
||||
return "online_compatible";
|
||||
}
|
||||
|
||||
type OverrideParse = { readonly override: MigrationCompatOverride | null; readonly error: string | null };
|
||||
|
||||
/**
|
||||
* Reads the override directive from the migration file's leading comment block.
|
||||
* Nothing outside the file can supply or change an override.
|
||||
*/
|
||||
export function parseMigrationCompatOverride(sql: string): OverrideParse {
|
||||
const lines = sql.split(/\r?\n/);
|
||||
const directive = new RegExp(String.raw`^\s*--\s*${MIGRATION_COMPAT_DIRECTIVE}(?![\w-])(.*)$`);
|
||||
const found: string[] = [];
|
||||
let leading = true;
|
||||
let misplaced = false;
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (leading && trimmed !== "" && (!trimmed.startsWith("--") || trimmed.startsWith("-->"))) leading = false;
|
||||
if (!line.includes(MIGRATION_COMPAT_DIRECTIVE)) continue;
|
||||
const match = leading ? directive.exec(line) : null;
|
||||
if (match) found.push(match[1]);
|
||||
else misplaced = true;
|
||||
}
|
||||
const usage = `Write it as the file's first lines: "-- ${MIGRATION_COMPAT_DIRECTIVE} online_compatible: <why this is safe>" (or requires_staging).`;
|
||||
if (misplaced) return { override: null, error: `The ${MIGRATION_COMPAT_DIRECTIVE} directive must be a "--" comment in the leading comment block, before any SQL. ${usage}` };
|
||||
if (found.length === 0) return { override: null, error: null };
|
||||
if (found.length > 1) return { override: null, error: `The file has more than one ${MIGRATION_COMPAT_DIRECTIVE} directive. ${usage}` };
|
||||
const match = /^\s+([a-z_]+)\s*(?::\s*(.*))?$/.exec(found[0]);
|
||||
const compatibility = match?.[1];
|
||||
if (!match || (compatibility !== "online_compatible" && compatibility !== "requires_staging")) {
|
||||
return { override: null, error: `The ${MIGRATION_COMPAT_DIRECTIVE} directive must name online_compatible or requires_staging. ${usage}` };
|
||||
}
|
||||
const justification = (match[2] ?? "").trim();
|
||||
if (justification.length < MIGRATION_COMPAT_MIN_JUSTIFICATION_LENGTH) {
|
||||
return { override: null, error: `The ${MIGRATION_COMPAT_DIRECTIVE} directive needs a justification of at least ${MIGRATION_COMPAT_MIN_JUSTIFICATION_LENGTH} characters after the colon. ${usage}` };
|
||||
}
|
||||
return { override: { compatibility, justification }, error: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Decides whether a migration can be applied while the previous release's app
|
||||
* code is still reading and writing the database, without breaking that code
|
||||
* and without long blocking locks. Anything the classifier does not positively
|
||||
* recognize is `unknown`.
|
||||
*/
|
||||
export function classifyMigrationSql(sql: string, options: MigrationCompatOptions = {}): MigrationCompatResult {
|
||||
const context: Context = { existingTables: options.existingTables ?? null, createdTables: new Set(), addedColumns: new Set() };
|
||||
const statementsByCode = new Map<MigrationCompatReasonCode, number[]>();
|
||||
const record = (code: MigrationCompatReasonCode, position: number) => {
|
||||
const positions = statementsByCode.get(code) ?? [];
|
||||
if (!positions.includes(position)) positions.push(position);
|
||||
statementsByCode.set(code, positions);
|
||||
};
|
||||
|
||||
const statements = parseStatements(sql);
|
||||
if (statements.length === 0) record("no_statements", 0);
|
||||
statements.forEach((statement, index) => {
|
||||
for (const code of classifyStatement(statement, context)) record(code, index + 1);
|
||||
});
|
||||
const { override, error } = parseMigrationCompatOverride(sql);
|
||||
if (error) record("invalid_override", 0);
|
||||
|
||||
const reasons: MigrationCompatReason[] = [...statementsByCode].map(([code, positions]) => ({
|
||||
code,
|
||||
compatibility: REASONS[code].compatibility,
|
||||
message: code === "invalid_override" && error ? error : REASONS[code].message,
|
||||
statements: positions.filter((position) => position > 0),
|
||||
}));
|
||||
const classified = worst(reasons.filter((reason) => reason.code !== "invalid_override").map((reason) => reason.compatibility));
|
||||
return {
|
||||
compatibility: override ? override.compatibility : worst(reasons.map((reason) => reason.compatibility)),
|
||||
classified,
|
||||
reasons,
|
||||
override,
|
||||
overrideError: error,
|
||||
usesConcurrently: statements.some((statement) => /\bconcurrently\b/.test(statement.text)),
|
||||
};
|
||||
}
|
||||
|
||||
/** Names of the tables a migration creates or renames to, including inside DO blocks. */
|
||||
export function collectCreatedTables(sql: string): string[] {
|
||||
const names: string[] = [];
|
||||
const created = new RegExp(String.raw`\bcreate (?:unlogged )?table (?:if not exists )?(${QUALIFIED})`, "g");
|
||||
const renamed = new RegExp(String.raw`\brename to (${IDENT})`, "g");
|
||||
const visit = (statements: Statement[]) => {
|
||||
for (const statement of statements) {
|
||||
for (const match of statement.text.matchAll(created)) names.push(tableName(match[1]));
|
||||
for (const match of statement.text.matchAll(renamed)) names.push(tableName(match[1]));
|
||||
for (const body of statement.dollarBodies) visit(parseStatements(body));
|
||||
}
|
||||
};
|
||||
visit(parseStatements(sql));
|
||||
return names;
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies migrations in apply order. Each one is judged knowing the tables
|
||||
* that `priorSql` and the earlier entries of `sqls` created.
|
||||
*/
|
||||
export function classifyMigrationSequence(sqls: readonly string[], priorSql: readonly string[] = []): MigrationCompatResult[] {
|
||||
const existingTables = new Set<string>(priorSql.flatMap(collectCreatedTables));
|
||||
return sqls.map((sql) => {
|
||||
const result = classifyMigrationSql(sql, { existingTables });
|
||||
for (const table of collectCreatedTables(sql)) existingTables.add(table);
|
||||
return result;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads, from the source of packages/db/src/client.ts, the migration files the
|
||||
* migrator runs outside a transaction. Returns null when the list is not found.
|
||||
*/
|
||||
export function parseNonTransactionalMigrations(executorSource: string): Set<string> | null {
|
||||
const match = /MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS\s*=\s*new Set(?:<[^>]*>)?\(\[([\s\S]*?)\]\)/.exec(executorSource);
|
||||
if (!match) return null;
|
||||
return new Set([...match[1].matchAll(/"([^"]+\.sql)"/g)].map((entry) => entry[1]));
|
||||
}
|
||||
|
||||
export type MigrationCompatInput = { readonly fileName: string; readonly sql: string };
|
||||
|
||||
export type MigrationCompatGuardResult = {
|
||||
readonly passed: boolean;
|
||||
/** One entry per migration that blocks the pull request. */
|
||||
readonly errors: readonly string[];
|
||||
/** Informational lines: staged migrations and accepted overrides. */
|
||||
readonly notices: readonly string[];
|
||||
readonly results: readonly { readonly fileName: string; readonly result: MigrationCompatResult }[];
|
||||
};
|
||||
|
||||
function describeReasons(result: MigrationCompatResult, compatibility: MigrationCompatibility): string[] {
|
||||
return result.reasons
|
||||
.filter((reason) => reason.compatibility === compatibility)
|
||||
.map((reason) => ` - ${reason.code}${reason.statements.length ? ` (statement ${reason.statements.join(", ")})` : ""}: ${reason.message}`);
|
||||
}
|
||||
|
||||
export type MigrationCompatGuardOptions = {
|
||||
/**
|
||||
* SQL of every migration that applies before the new ones. Without it,
|
||||
* CREATE TABLE IF NOT EXISTS never counts as creating a new table.
|
||||
*/
|
||||
readonly priorMigrationSql?: readonly string[];
|
||||
/**
|
||||
* File names the migrator runs outside a transaction (see
|
||||
* parseNonTransactionalMigrations). When given, a new migration that uses
|
||||
* CONCURRENTLY must be in it.
|
||||
*/
|
||||
readonly nonTransactionalMigrations?: ReadonlySet<string>;
|
||||
};
|
||||
|
||||
/**
|
||||
* The pull-request guard for migrations a change adds, in apply order.
|
||||
* `unknown` without a valid override fails. `requires_staging` passes with a
|
||||
* notice until deployers enforce staging.
|
||||
*/
|
||||
export function checkNewMigrationCompatibility(
|
||||
migrations: readonly MigrationCompatInput[],
|
||||
options: MigrationCompatGuardOptions = {},
|
||||
): MigrationCompatGuardResult {
|
||||
const errors: string[] = [];
|
||||
const notices: string[] = [];
|
||||
const sqls = migrations.map((migration) => migration.sql);
|
||||
const classified = options.priorMigrationSql
|
||||
? classifyMigrationSequence(sqls, options.priorMigrationSql)
|
||||
: sqls.map((sql) => classifyMigrationSql(sql));
|
||||
const results = migrations.map(({ fileName }, index) => ({ fileName, result: classified[index] }));
|
||||
for (const { fileName, result } of results) {
|
||||
const baseName = fileName.split("/").pop() ?? fileName;
|
||||
if (result.usesConcurrently && options.nonTransactionalMigrations && !options.nonTransactionalMigrations.has(baseName)) {
|
||||
errors.push([
|
||||
`${fileName}: uses CONCURRENTLY, which PostgreSQL rejects inside a transaction, and the migrator runs each file in one transaction.`,
|
||||
` Add "${baseName}" to MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS in packages/db/src/client.ts.`,
|
||||
" Make every statement in the file idempotent (IF NOT EXISTS), because a failed run starts again from the first statement.",
|
||||
].join("\n"));
|
||||
}
|
||||
if (result.overrideError) {
|
||||
errors.push(`${fileName}: ${result.overrideError}`);
|
||||
} else if (result.compatibility === "unknown") {
|
||||
errors.push([
|
||||
`${fileName}: cannot tell whether this migration is safe to apply while the previous release is still running.`,
|
||||
...describeReasons(result, "unknown"),
|
||||
" Do one of the following:",
|
||||
" 1. Rewrite it so every statement is recognized as compatible, for example CREATE INDEX CONCURRENTLY,",
|
||||
" ADD CONSTRAINT ... NOT VALID, or a nullable column or constant default.",
|
||||
" 2. Split it into an expand migration now and a contract migration in a later release.",
|
||||
" 3. If you have verified it yourself, make these the first lines of the file:",
|
||||
` -- ${MIGRATION_COMPAT_DIRECTIVE} online_compatible: <why the previous release keeps working and no lock is held long>`,
|
||||
` or, if the previous release cannot run against it:`,
|
||||
` -- ${MIGRATION_COMPAT_DIRECTIVE} requires_staging: <what has to ship first>`,
|
||||
" See \"Migration compatibility with the running release\" in doc/DATABASE.md.",
|
||||
].join("\n"));
|
||||
} else if (result.compatibility === "requires_staging") {
|
||||
notices.push([
|
||||
`${fileName}: requires_staging. The previous release cannot keep running against this migration, so a deployer that may not pause will hold it back.`,
|
||||
...(result.override ? [` - override: ${result.override.justification}`] : describeReasons(result, "requires_staging")),
|
||||
" This does not fail the check yet. Prefer an expand migration now and the contract migration in a later release.",
|
||||
].join("\n"));
|
||||
} else if (result.override) {
|
||||
notices.push(`${fileName}: online_compatible by override (classifier said ${result.classified}): ${result.override.justification}`);
|
||||
}
|
||||
}
|
||||
return { passed: errors.length === 0, errors, notices, results };
|
||||
}
|
||||
|
||||
export type MigrationCompatSummary = {
|
||||
readonly total: number;
|
||||
readonly byCompatibility: Record<MigrationCompatibility, number>;
|
||||
readonly overrides: number;
|
||||
/** Migrations per blocking reason code, most frequent first. */
|
||||
readonly blockingReasons: readonly { readonly code: MigrationCompatReasonCode; readonly compatibility: MigrationCompatibility; readonly migrations: number }[];
|
||||
};
|
||||
|
||||
export function summarizeMigrationCompatibility(results: readonly MigrationCompatResult[]): MigrationCompatSummary {
|
||||
const byCompatibility: Record<MigrationCompatibility, number> = { online_compatible: 0, requires_staging: 0, unknown: 0 };
|
||||
const counts = new Map<MigrationCompatReasonCode, number>();
|
||||
for (const result of results) {
|
||||
byCompatibility[result.compatibility] += 1;
|
||||
for (const reason of result.reasons) {
|
||||
if (reason.compatibility !== "online_compatible") counts.set(reason.code, (counts.get(reason.code) ?? 0) + 1);
|
||||
}
|
||||
}
|
||||
return {
|
||||
total: results.length,
|
||||
byCompatibility,
|
||||
overrides: results.filter((result) => result.override).length,
|
||||
blockingReasons: [...counts]
|
||||
.map(([code, migrations]) => ({ code, compatibility: REASONS[code].compatibility, migrations }))
|
||||
.sort((left, right) => right.migrations - left.migrations || left.code.localeCompare(right.code)),
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user