mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
fc14f1ade2
commit
e8e548e856
9 files changed
+1621
-28
No files matched your search
@@ -7,6 +7,7 @@ import { copyFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { gunzipSync } from "node:zlib";
|
||||
import { assertMetadata, tarManifest, versionFor } from "./preview-artifacts.mjs";
|
||||
|
||||
export const artifactBase = "https://d1p6rlowie26tp.cloudfront.net/cloud-migrators/v1";
|
||||
@@ -30,10 +31,111 @@ function assertDescriptor(pin, extension) {
|
||||
}
|
||||
}
|
||||
|
||||
const compatibilityValues = ["online_compatible", "requires_staging", "unknown"];
|
||||
const migrationsPrefix = "package/dist/migrations/";
|
||||
|
||||
/** Regular files under `prefix` in a package archive, by archive path. */
|
||||
function tarFiles(bytes, prefix) {
|
||||
const tar = gunzipSync(bytes, { maxOutputLength: 128 * 1024 * 1024 });
|
||||
const files = new Map();
|
||||
for (let offset = 0; offset + 512 <= tar.length;) {
|
||||
const h = tar.subarray(offset, offset + 512);
|
||||
if (h.every((v) => v === 0)) break;
|
||||
const field = (start, size) => h.subarray(start, start + size).toString("utf8").split("\0")[0].trim();
|
||||
const sizeText = field(124, 12);
|
||||
if (!/^[0-7]+$/.test(sizeText)) throw new Error("Invalid package archive.");
|
||||
const size = Number.parseInt(sizeText, 8);
|
||||
if (offset + 512 + size > tar.length) throw new Error("Truncated package archive.");
|
||||
const name = `${field(345, 155) ? field(345, 155) + "/" : ""}${field(0, 100)}`;
|
||||
if ([0, 48].includes(h[156]) && name.startsWith(prefix)) {
|
||||
if (files.has(name)) throw new Error("Duplicate package archive entry.");
|
||||
files.set(name, tar.subarray(offset + 512, offset + 512 + size));
|
||||
}
|
||||
offset += 512 + Math.ceil(size / 512) * 512;
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
/** The journal of the packed migrator, in apply order, with each file's bytes. */
|
||||
function packedMigrations(dbBytes) {
|
||||
const files = tarFiles(dbBytes, migrationsPrefix);
|
||||
const journal = files.get(`${migrationsPrefix}meta/_journal.json`);
|
||||
if (!journal) throw new Error("Migrator package has no migration journal.");
|
||||
const entries = JSON.parse(journal.toString("utf8")).entries;
|
||||
if (!Array.isArray(entries) || entries.length === 0) throw new Error("Migrator package has an empty migration journal.");
|
||||
// The migrator's apply order: journal idx, then file name. The shipped
|
||||
// journal repeats an idx, so idx alone does not identify a migration.
|
||||
return [...entries].sort((left, right) => left.idx - right.idx || (String(left.tag) < String(right.tag) ? -1 : 1)).map(({ idx, tag }) => {
|
||||
const sql = files.get(`${migrationsPrefix}${tag}.sql`);
|
||||
if (!sql) throw new Error(`Migrator package is missing migration ${tag}.`);
|
||||
// The same digest the migrator records in the database's migration history.
|
||||
return { idx, tag, sql, sqlSha256: createHash("sha256").update(sql).digest("hex") };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether each packed migration can be applied while the previous release is
|
||||
* still running. `classify` is classifyMigrationSequence from @paperclipai/db:
|
||||
* it takes the SQL of every migration in apply order. The overrides it honours
|
||||
* come only from the packed SQL itself.
|
||||
*/
|
||||
export function migrationCompatibility(dbBytes, classify) {
|
||||
const packed = packedMigrations(dbBytes);
|
||||
const results = classify(packed.map(({ sql }) => sql.toString("utf8")));
|
||||
if (!Array.isArray(results) || results.length !== packed.length) throw new Error("The classifier did not return one result per migration.");
|
||||
return {
|
||||
version: 1,
|
||||
migrations: packed.map(({ idx, tag, sqlSha256 }, position) => {
|
||||
const result = results[position];
|
||||
return {
|
||||
idx, tag, sqlSha256,
|
||||
compatibility: result.compatibility,
|
||||
classified: result.classified,
|
||||
override: Boolean(result.override),
|
||||
...(result.override ? { overrideJustification: result.override.justification } : {}),
|
||||
reasons: result.reasons.map(({ code, compatibility, message, statements }) => ({ code, compatibility, message, statements: [...statements] })),
|
||||
};
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function assertMigrationCompatibility(value) {
|
||||
const fail = () => { throw new Error("Invalid migration compatibility data."); };
|
||||
if (value?.version !== 1 || !Array.isArray(value.migrations) || value.migrations.length === 0) fail();
|
||||
const text = (candidate) => typeof candidate === "string" && candidate.length > 0 && candidate.length <= 2000;
|
||||
value.migrations.forEach((migration, position) => {
|
||||
const previous = value.migrations[position - 1];
|
||||
if (!Number.isSafeInteger(migration?.idx) || migration.idx < 0 ||
|
||||
(previous && (migration.idx < previous.idx || (migration.idx === previous.idx && !(migration.tag > previous.tag)))) ||
|
||||
typeof migration.tag !== "string" || !/^\d{4}_[A-Za-z0-9_]+$/.test(migration.tag) ||
|
||||
typeof migration.sqlSha256 !== "string" || !/^[0-9a-f]{64}$/.test(migration.sqlSha256) ||
|
||||
!compatibilityValues.includes(migration.compatibility) || !compatibilityValues.includes(migration.classified) ||
|
||||
typeof migration.override !== "boolean" || !Array.isArray(migration.reasons)) fail();
|
||||
// An override always carries its justification and never yields `unknown`.
|
||||
if (migration.override ? !text(migration.overrideJustification) || migration.compatibility === "unknown" : migration.overrideJustification !== undefined) fail();
|
||||
for (const reason of migration.reasons) {
|
||||
if (typeof reason?.code !== "string" || !/^[a-z_]+$/.test(reason.code) || !compatibilityValues.includes(reason.compatibility) || !text(reason.message) ||
|
||||
!Array.isArray(reason.statements) || !reason.statements.every((statement) => Number.isSafeInteger(statement) && statement > 0)) fail();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
export function assertManifest(manifest, sha) {
|
||||
if (manifest?.version !== 1 || manifest.sourceSha !== sha || manifest.packageVersion !== versionFor(sha)) throw new Error("Artifact source identity mismatch.");
|
||||
for (const name of names) assertDescriptor(manifest.packages?.[name], "tgz");
|
||||
assertDescriptor(manifest.lockfile, "json");
|
||||
// Additive since the classifier shipped; manifests published earlier lack it.
|
||||
if (manifest.migrationCompatibility !== undefined) assertMigrationCompatibility(manifest.migrationCompatibility);
|
||||
}
|
||||
|
||||
/** The published classification must describe exactly the migrations in db.tgz. */
|
||||
function assertCompatibilityMatchesPackage(manifest, dbBytes) {
|
||||
const published = manifest.migrationCompatibility?.migrations;
|
||||
const packed = packedMigrations(dbBytes);
|
||||
if (!published || published.length !== packed.length || packed.some(({ idx, tag, sqlSha256 }, position) =>
|
||||
published[position].idx !== idx || published[position].tag !== tag || published[position].sqlSha256 !== sqlSha256)) {
|
||||
throw new Error("Migration compatibility data does not match the packed migrations.");
|
||||
}
|
||||
}
|
||||
|
||||
export function assertLockfile(lock, manifest) {
|
||||
@@ -61,8 +163,9 @@ export function assertLockfile(lock, manifest) {
|
||||
}
|
||||
}
|
||||
|
||||
export function buildBundle(directory, sha, { exec = execFileSync } = {}) {
|
||||
export function buildBundle(directory, sha, { exec = execFileSync, classify } = {}) {
|
||||
versionFor(sha);
|
||||
if (typeof classify !== "function") throw new Error("buildBundle needs the migration compatibility classifier.");
|
||||
directory = path.resolve(directory);
|
||||
const packages = {};
|
||||
for (const name of names) {
|
||||
@@ -83,8 +186,11 @@ export function buildBundle(directory, sha, { exec = execFileSync } = {}) {
|
||||
lock.packages[""].dependencies = { "@paperclipai/db": versionFor(sha) };
|
||||
for (const name of names) lock.packages[`node_modules/@paperclipai/${name}`].resolved = packages[name].url;
|
||||
const lockBytes = Buffer.from(JSON.stringify(lock) + "\n");
|
||||
const manifest = { version: 1, sourceSha: sha, packageVersion: versionFor(sha), packages, lockfile: descriptor(lockBytes, "json") };
|
||||
const dbBytes = readFileSync(path.join(directory, "db.tgz"));
|
||||
const manifest = { version: 1, sourceSha: sha, packageVersion: versionFor(sha), packages, lockfile: descriptor(lockBytes, "json"),
|
||||
migrationCompatibility: migrationCompatibility(dbBytes, classify) };
|
||||
assertManifest(manifest, sha);
|
||||
assertCompatibilityMatchesPackage(manifest, dbBytes);
|
||||
assertLockfile(lock, manifest);
|
||||
writeFileSync(path.join(directory, "package-lock.json"), lockBytes);
|
||||
writeFileSync(path.join(directory, "manifest.json"), JSON.stringify(manifest) + "\n");
|
||||
@@ -103,6 +209,7 @@ export function validateBundle(directory, sha) {
|
||||
const bytes = readFileSync(path.join(directory, `${name}.tgz`));
|
||||
verifyBytes(bytes, manifest.packages[name]);
|
||||
assertMetadata(tarManifest(bytes), `@paperclipai/${name}`, sha);
|
||||
if (name === "db") assertCompatibilityMatchesPackage(manifest, bytes);
|
||||
}
|
||||
const bytes = readFileSync(path.join(directory, "package-lock.json"));
|
||||
verifyBytes(bytes, manifest.lockfile);
|
||||
@@ -160,6 +267,7 @@ export async function verifyPublished(sha, fetchImpl = fetch, { verifyProvenance
|
||||
const bytes = await download(manifest.packages[name].url, fetchImpl);
|
||||
verifyBytes(bytes, manifest.packages[name]);
|
||||
assertMetadata(tarManifest(bytes), `@paperclipai/${name}`, sha);
|
||||
if (name === "db" && manifest.migrationCompatibility !== undefined) assertCompatibilityMatchesPackage(manifest, bytes);
|
||||
}));
|
||||
const lock = await download(manifest.lockfile.url, fetchImpl);
|
||||
verifyBytes(lock, manifest.lockfile);
|
||||
@@ -200,7 +308,12 @@ export async function publishBundle(directory, sha, { exec = execFileSync, fetch
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
const [command, directory, sha] = process.argv.slice(2);
|
||||
try {
|
||||
if (command === "build") buildBundle(directory, sha);
|
||||
if (command === "build") {
|
||||
// Type-stripped by Node; the module has no imports, so the build job
|
||||
// needs nothing beyond the checkout.
|
||||
const { classifyMigrationSequence } = await import(new URL("../packages/db/src/migration-compatibility.ts", import.meta.url).href);
|
||||
buildBundle(directory, sha, { classify: classifyMigrationSequence });
|
||||
}
|
||||
else if (command === "validate") validateBundle(directory, sha);
|
||||
else if (command === "verify-install") verifyInstall(directory, sha);
|
||||
else if (command === "publish") await publishBundle(directory, sha);
|
||||
|
||||
@@ -7,25 +7,45 @@ import { mkdtempSync, readFileSync, writeFileSync, rmSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { gzipSync } from "node:zlib";
|
||||
import { artifactBase, assertManifest, assertLockfile, buildBundle, descriptor, validateBundle, verifyPublished, publishBundle } from "./cloud-migrator-artifacts.mjs";
|
||||
import { createHash } from "node:crypto";
|
||||
import { artifactBase, assertManifest, assertLockfile, buildBundle, descriptor, migrationCompatibility, validateBundle, verifyPublished, publishBundle } from "./cloud-migrator-artifacts.mjs";
|
||||
import { previewManifest, versionFor } from "./preview-artifacts.mjs";
|
||||
import { classifyMigrationSequence as classify } from "../packages/db/src/migration-compatibility.ts";
|
||||
|
||||
const sha = "a".repeat(40);
|
||||
const migrations = {
|
||||
"0000_first": 'CREATE TABLE IF NOT EXISTS "widgets" ("id" uuid PRIMARY KEY NOT NULL);\n--> statement-breakpoint\nCREATE INDEX "widgets_idx" ON "widgets" ("id");\n',
|
||||
// IF NOT EXISTS does not make a table from an earlier migration new again.
|
||||
"0001_second": 'CREATE TABLE IF NOT EXISTS "widgets" ("id" uuid);\n--> statement-breakpoint\nCREATE INDEX "widgets_again_idx" ON "widgets" ("id");\n',
|
||||
"0002_third": '-- paperclip:migration-compat online_compatible: no release since 0.2 reads issues.legacy\nALTER TABLE "issues" DROP COLUMN "legacy";\n',
|
||||
};
|
||||
function tarEntry(name, content) {
|
||||
const bytes = Buffer.from(content);
|
||||
const header = Buffer.alloc(512);
|
||||
header.write(name); header.write(bytes.length.toString(8).padStart(11, "0"), 124, 11); header[156] = 48;
|
||||
// A real tar header, so npm can install this fixture as well as inspect it.
|
||||
header.fill(32, 148, 156);
|
||||
const sum = header.reduce((a, b) => a + b, 0);
|
||||
header.write(sum.toString(8).padStart(6, "0") + "\0 ", 148, 8);
|
||||
const padded = Buffer.alloc(Math.ceil(bytes.length / 512) * 512); bytes.copy(padded);
|
||||
return Buffer.concat([header, padded]);
|
||||
}
|
||||
function dbArchive(files = migrations) {
|
||||
const journal = { version: "7", dialect: "postgresql", entries: Object.keys(files).map((tag, idx) => ({ idx, version: "7", when: idx, tag, breakpoints: true })) };
|
||||
return gzipSync(Buffer.concat([
|
||||
tarEntry("package/package.json", JSON.stringify(previewManifest({ name: "@paperclipai/db", dependencies: {} }, sha))),
|
||||
tarEntry("package/dist/migrations/meta/_journal.json", JSON.stringify(journal)),
|
||||
...Object.entries(files).map(([tag, sql]) => tarEntry(`package/dist/migrations/${tag}.sql`, sql)),
|
||||
Buffer.alloc(1024),
|
||||
]));
|
||||
}
|
||||
function fixture(t) {
|
||||
const dir = mkdtempSync(path.join(os.tmpdir(), "migrator-artifact-test-"));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
for (const name of ["db", "shared"]) {
|
||||
const bytes = Buffer.from(JSON.stringify(previewManifest({ name: `@paperclipai/${name}`, dependencies: {} }, sha)));
|
||||
const header = Buffer.alloc(512);
|
||||
header.write("package/package.json"); header.write(bytes.length.toString(8).padStart(11, "0"), 124, 11); header[156] = 48;
|
||||
// A real tar header, so npm can install this fixture as well as inspect it.
|
||||
header.fill(32, 148, 156);
|
||||
const sum = header.reduce((a, b) => a + b, 0);
|
||||
header.write(sum.toString(8).padStart(6, "0") + "\0 ", 148, 8);
|
||||
const padded = Buffer.alloc(Math.ceil(bytes.length / 512) * 512); bytes.copy(padded);
|
||||
writeFileSync(path.join(dir, `${name}.tgz`), gzipSync(Buffer.concat([header, padded, Buffer.alloc(1024)])));
|
||||
}
|
||||
const manifest = buildBundle(dir, sha, { exec: (cmd, args, options) => {
|
||||
writeFileSync(path.join(dir, "db.tgz"), dbArchive());
|
||||
writeFileSync(path.join(dir, "shared.tgz"), gzipSync(Buffer.concat([
|
||||
tarEntry("package/package.json", JSON.stringify(previewManifest({ name: "@paperclipai/shared", dependencies: {} }, sha))), Buffer.alloc(1024)])));
|
||||
const manifest = buildBundle(dir, sha, { classify, exec: (cmd, args, options) => {
|
||||
assert.equal(cmd, "npm"); assert.ok(args.includes("--ignore-scripts"));
|
||||
const localRoot = JSON.parse(readFileSync(path.join(options.cwd, "package.json")));
|
||||
assert.deepEqual(localRoot.dependencies, { "@paperclipai/db": "file:db.tgz", "@paperclipai/shared": "file:shared.tgz" });
|
||||
@@ -53,6 +73,67 @@ test("bundle pins the exact source pair and complete lockfile without new npm lo
|
||||
assert.deepEqual(await verifyPublished(sha, fetchImpl), manifest);
|
||||
});
|
||||
|
||||
test("manifest publishes how each packed migration behaves under the previous release", (t) => {
|
||||
const { manifest } = fixture(t);
|
||||
const digest = (tag) => createHash("sha256").update(migrations[tag]).digest("hex");
|
||||
assert.deepEqual(manifest.migrationCompatibility.version, 1);
|
||||
assert.deepEqual(manifest.migrationCompatibility.migrations.map(({ reasons, ...entry }) => ({ ...entry, reasons: reasons.map((reason) => reason.code) })), [
|
||||
{ idx: 0, tag: "0000_first", sqlSha256: digest("0000_first"), compatibility: "online_compatible", classified: "online_compatible", override: false,
|
||||
reasons: ["create_table", "create_index_new_table"] },
|
||||
{ idx: 1, tag: "0001_second", sqlSha256: digest("0001_second"), compatibility: "unknown", classified: "unknown", override: false,
|
||||
reasons: ["create_table_if_not_exists", "lock_hazard_create_index"] },
|
||||
{ idx: 2, tag: "0002_third", sqlSha256: digest("0002_third"), compatibility: "online_compatible", classified: "requires_staging", override: true,
|
||||
overrideJustification: "no release since 0.2 reads issues.legacy", reasons: ["drop_column"] },
|
||||
]);
|
||||
const reason = manifest.migrationCompatibility.migrations[1].reasons[1];
|
||||
assert.deepEqual(Object.keys(reason), ["code", "compatibility", "message", "statements"]);
|
||||
assert.deepEqual([reason.compatibility, reason.statements], ["unknown", [2]]);
|
||||
// Everything a consumer pinned before this field existed is unchanged.
|
||||
assert.deepEqual(Object.keys(manifest), ["version", "sourceSha", "packageVersion", "packages", "lockfile", "migrationCompatibility"]);
|
||||
assert.equal(manifest.version, 1);
|
||||
});
|
||||
|
||||
test("migration compatibility is optional for earlier manifests and fails closed when malformed or stale", async (t) => {
|
||||
const { dir, manifest, files, fetchImpl } = fixture(t);
|
||||
const earlier = structuredClone(manifest); delete earlier.migrationCompatibility;
|
||||
assert.doesNotThrow(() => assertManifest(earlier, sha));
|
||||
for (const mutate of [
|
||||
(c) => { c.version = 2; },
|
||||
(c) => { c.migrations = []; },
|
||||
(c) => { c.migrations[0].compatibility = "safe"; },
|
||||
(c) => { c.migrations[0].classified = undefined; },
|
||||
(c) => { c.migrations[0].sqlSha256 = "abc"; },
|
||||
(c) => { c.migrations[2].idx = 0; },
|
||||
(c) => { c.migrations[1].tag = c.migrations[0].tag; c.migrations[1].idx = 0; },
|
||||
(c) => { c.migrations[0].override = true; },
|
||||
(c) => { c.migrations[2].overrideJustification = ""; },
|
||||
(c) => { c.migrations[2].compatibility = "unknown"; },
|
||||
(c) => { c.migrations[0].overrideJustification = "without an override"; },
|
||||
(c) => { c.migrations[1].reasons[0].statements = [0]; },
|
||||
(c) => { c.migrations[1].reasons[1].compatibility = "fine"; },
|
||||
(c) => { c.migrations[1].reasons[0].code = 7; },
|
||||
]) {
|
||||
const bad = structuredClone(manifest); mutate(bad.migrationCompatibility);
|
||||
assert.throws(() => assertManifest(bad, sha), /Invalid migration compatibility data/);
|
||||
}
|
||||
assert.throws(() => buildBundle(dir, sha), /classifier/);
|
||||
// A well-formed classification of different SQL must not be published or accepted.
|
||||
const stale = structuredClone(manifest);
|
||||
stale.migrationCompatibility = migrationCompatibility(dbArchive({ ...migrations, "0001_second": 'DROP TABLE "issues";\n' }), classify);
|
||||
assertManifest(stale, sha);
|
||||
files.set(`${artifactBase}/${sha}/manifest.json`, Buffer.from(JSON.stringify(stale)));
|
||||
await assert.rejects(verifyPublished(sha, fetchImpl), /does not match the packed migrations/);
|
||||
writeFileSync(path.join(dir, "manifest.json"), JSON.stringify(stale));
|
||||
assert.throws(() => validateBundle(dir, sha), /does not match the packed migrations/);
|
||||
writeFileSync(path.join(dir, "manifest.json"), JSON.stringify(earlier));
|
||||
assert.throws(() => validateBundle(dir, sha), /does not match the packed migrations/);
|
||||
assert.throws(() => migrationCompatibility(dbArchive({}), classify), /empty migration journal/);
|
||||
assert.throws(() => migrationCompatibility(dbArchive(), () => []), /one result per migration/);
|
||||
// The shipped journal repeats idx 178; order then follows the file name, as in the migrator.
|
||||
const repeated = structuredClone(manifest); repeated.migrationCompatibility.migrations[1].idx = 0;
|
||||
assert.doesNotThrow(() => assertManifest(repeated, sha));
|
||||
});
|
||||
|
||||
test("source identity, content hashes, size, and origin fail closed", async (t) => {
|
||||
const { dir, manifest, files, fetchImpl } = fixture(t);
|
||||
for (const mutate of [
|
||||
@@ -109,7 +190,7 @@ test("download failures and oversized objects never count as available", async (
|
||||
test("real npm ci installs the new pair from pinned archives with an empty cache", async (t) => {
|
||||
const { dir } = fixture(t);
|
||||
// Real npm resolution uses local archives; neither package version exists on npm.
|
||||
const manifest = buildBundle(dir, sha);
|
||||
const manifest = buildBundle(dir, sha, { classify });
|
||||
const lock = JSON.parse(readFileSync(path.join(dir, "package-lock.json")));
|
||||
const requests = [];
|
||||
const server = createServer((req, res) => {
|
||||
|
||||
Reference in new issue
Block a user