Files
PaperClipAI/packages/db/src/project-environment-overlap-migration.test.ts
T
DottaandPaperclip cda23f08ac fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path

> - Paperclip controls agent work while web servers change versions.
> - The old and new servers can share a database during that change.
> - Older servers still read and write the project environment column.
> - Migration 0466 removes that column before those servers stop
serving.
> - A later repair alone leaves a committed missing-column window and
loses values.
> - This pull request applies the exact drop and repair in one
transaction.
> - Old reads and saved values survive while current placement behavior
stays unchanged.

## Linked Issues or Issue Description

Refs #20835 and #20852.

**What happened?**

Migration 0466 drops `projects.environment_id`. The previous server
still selects that column and reads it during admission. Applying a
separate repair afterward leaves a window in which those requests fail.
The drop also removes saved values.

**Expected behavior**

A database that has not yet applied 0466 must keep the column and its
values visible to concurrent old servers throughout the upgrade. Both
published migration entries must be recorded truthfully. Current project
responses must keep the new workspace placement contract.

**Steps to reproduce**

1. Use a database before migration 0466 with nonempty project
environment values.
2. Keep an old project writer transaction open and start the pending
migrator.
3. Start an old project read, then commit the writer.
4. Without this repair, the saved values disappear. With this repair,
the reader sees the preserved values and both migrations commit
together.

## What Changed

- Add migration 0467 to restore the legacy nullable UUID column and its
original foreign key. Leave the published 0466 SQL unchanged.
- Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock
the project table, retain nonempty values in a temporary table, execute
both real SQL files, restore values, and record both real journal
entries before commit.
- Apply the additive 0467 repair normally when 0466 was already
committed.
- Keep a deprecated database-only field and omit it from current project
API responses. Current placement logic stays unchanged.
- Add PostgreSQL regressions for concurrent old reads and writes,
idempotence, rollback, and an already-applied drop. Document the overlap
contract and retain the five newest generated snapshots.

## Verification

- Node 24 cached agent preflight passed.
- `pnpm exec vitest run
packages/db/src/project-environment-overlap-migration.test.ts
server/src/__tests__/project-environments.test.ts -t 'project
environment rolling migration|saves storage, not'`: 4 passed in 32.57
seconds against real disposable PostgreSQL.
- Before the owner fix, the regression had two failures: saved values
disappeared and the restore-failure case committed. The fixed case
proves that both DDL and journal entries roll back together.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/db check:migrations`: passed.
- `git diff --check`: passed. Migration 0466 bytes are unchanged. The
generated snapshot set is 0463 through 0467.
- Review found a raw DELETE response that still exposed the
database-only field. The delete projection now omits it; a real
authorized HTTP DELETE passed with successful cleanup. The concurrency
test now observes the old reader waiting inside PostgreSQL before
releasing the old writer. All three migration cases passed with that
barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as
failed attempts; they were not product-regression passes.
- Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh
Greptile 5/5 with no open findings. [Required final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612)
passed, including build, typecheck, migration checks, all test shards
and the aggregate verification gate. The separate Superagent security
scan ended inconclusive/neutral; it did not produce a passed scan.
Deployed acceptance remains pending. No hosted database or app
deployment was changed by this branch.

## Risks

The pair takes a table lock for its transaction. Operators must retain
their normal lock and statement timeouts. Both SQL files and their exact
hashes must match before the pair runs. A failed restore rolls back the
drop and both journal entries.

If 0466 was already committed, 0467 restores the column and foreign key.
It cannot recover values that were already deleted by the earlier
release. This repair does not claim to recover those values.

## Model Used

OpenAI Codex (GPT-6), with tool-assisted implementation and real
PostgreSQL qualification. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 11:08:30 -06:00

131 lines
7.3 KiB
TypeScript

import { createHash, randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import postgres from "postgres";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { applyPendingMigrations, inspectMigrations } from "./client.js";
import {
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
const migrations = ["0466_drop_project_environment.sql", "0467_fair_human_torch.sql"];
const hashes = await Promise.all(migrations.map(async (file) => createHash("sha256")
.update(await readFile(new URL(`./migrations/${file}`, import.meta.url))).digest("hex")));
describePostgres("project environment rolling migration", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let sql: ReturnType<typeof postgres>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-project-environment-overlap-");
sql = postgres(database.connectionString, { max: 1, onnotice: () => {} });
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await sql?.end();
await database?.cleanup();
});
async function prepare() {
await sql`DELETE FROM projects`;
await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash IN ${sql(hashes)}`;
const environmentIds = [randomUUID(), randomUUID()];
await sql`INSERT INTO environments (id, name, driver) VALUES
(${environmentIds[0]}, ${randomUUID()}, 'ssh'), (${environmentIds[1]}, ${randomUUID()}, 'ssh')`;
const projectIds = [randomUUID(), randomUUID(), randomUUID()];
for (let index = 0; index < projectIds.length; index++) {
const companyId = randomUUID();
await sql`INSERT INTO companies (id, name, issue_prefix) VALUES (${companyId}, 'Legacy company', ${`P${index}${companyId.slice(0, 6)}`})`;
await sql`INSERT INTO projects (id, company_id, name, environment_id) VALUES
(${projectIds[index]}, ${companyId}, 'Legacy project', ${environmentIds[index] ?? null})`;
}
return { environmentIds, projectIds };
}
const rows = () => sql`SELECT id, environment_id FROM projects ORDER BY id`;
const receipts = () => sql`SELECT hash FROM drizzle.__drizzle_migrations WHERE hash IN ${sql(hashes)} ORDER BY hash`;
it("preserves committed old-writer values and gives concurrent old readers an unbroken column", async () => {
const { environmentIds, projectIds } = await prepare();
const writer = postgres(database.connectionString, { max: 1, onnotice: () => {} });
const readerUrl = new URL(database.connectionString);
readerUrl.searchParams.set("application_name", "project-column-overlap-reader");
const reader = postgres(readerUrl.toString(), { max: 1, onnotice: () => {} });
const migrationUrl = new URL(database.connectionString);
migrationUrl.searchParams.set("application_name", "project-column-overlap-migrator");
let migration: Promise<void> | undefined;
let read: Promise<unknown> | undefined;
try {
await writer`BEGIN`;
await writer`UPDATE projects SET environment_id = ${environmentIds[1]} WHERE id = ${projectIds[0]}`;
migration = applyPendingMigrations(migrationUrl.toString());
await expect.poll(async () => (await sql`SELECT EXISTS (
SELECT 1 FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.relation = 'projects'::regclass AND l.mode = 'AccessExclusiveLock' AND NOT l.granted
AND a.application_name = 'project-column-overlap-migrator') AS waiting`)[0]?.waiting,
{ timeout: 10_000 }).toBe(true);
read = reader`SELECT id, environment_id FROM projects ORDER BY id`.then((result) => Array.from(result));
await expect.poll(async () => (await sql`SELECT EXISTS (
SELECT 1 FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.relation = 'projects'::regclass AND l.mode = 'AccessShareLock' AND NOT l.granted
AND a.application_name = 'project-column-overlap-reader') AS waiting`)[0]?.waiting,
{ timeout: 10_000 }).toBe(true);
const before = Array.from(await writer`SELECT id, environment_id FROM projects ORDER BY id`);
await writer`COMMIT`;
await migration;
expect(await read).toEqual(before);
expect(Array.from(await rows())).toEqual(before);
expect(Array.from(await receipts())).toEqual(hashes.toSorted().map((hash) => ({ hash })));
const state = await inspectMigrations(database.connectionString);
expect(state.status).toBe("upToDate");
await applyPendingMigrations(database.connectionString);
expect(Array.from(await rows())).toEqual(before);
expect(await receipts()).toHaveLength(2);
await writer`UPDATE projects SET environment_id = ${environmentIds[0]} WHERE id = ${projectIds[0]}`;
await sql`DELETE FROM environments WHERE id = ${environmentIds[0]}`;
expect((await sql`SELECT environment_id FROM projects WHERE id = ${projectIds[0]}`)[0]?.environment_id).toBeNull();
} finally {
await writer`ROLLBACK`;
await Promise.allSettled([migration, read]);
await Promise.all([writer.end(), reader.end()]);
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("rolls back both DDL and genuine receipts if restoring the legacy values fails", async () => {
await prepare();
const before = Array.from(await rows());
await sql.unsafe(`CREATE FUNCTION reject_project_environment_restore() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN RAISE EXCEPTION 'project_environment_restore_fixture'; END $$`);
await sql.unsafe(`CREATE TRIGGER reject_project_environment_restore BEFORE UPDATE ON projects
FOR EACH ROW EXECUTE FUNCTION reject_project_environment_restore()`);
try {
await expect(applyPendingMigrations(database.connectionString)).rejects.toThrow("project_environment_restore_fixture");
expect(Array.from(await rows())).toEqual(before);
expect(await receipts()).toHaveLength(0);
} finally {
await sql.unsafe("DROP TRIGGER reject_project_environment_restore ON projects");
await sql.unsafe("DROP FUNCTION reject_project_environment_restore()");
}
await applyPendingMigrations(database.connectionString);
expect(Array.from(await rows())).toEqual(before);
expect(await receipts()).toHaveLength(2);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("applies only the additive repair when the published drop was already committed", async () => {
await sql`DELETE FROM projects`;
await sql.unsafe('ALTER TABLE projects DROP COLUMN environment_id');
await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash = ${hashes[1]}`;
await applyPendingMigrations(database.connectionString);
expect((await sql`SELECT data_type, is_nullable FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = 'projects' AND column_name = 'environment_id'`)[0])
.toEqual({ data_type: "uuid", is_nullable: "YES" });
expect((await sql`SELECT confdeltype FROM pg_constraint WHERE conrelid = 'projects'::regclass
AND conname = 'projects_environment_id_environments_id_fk'`)[0]?.confdeltype).toBe("n");
expect(await receipts()).toHaveLength(2);
expect((await inspectMigrations(database.connectionString)).status).toBe("upToDate");
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
});