Files
PaperClipAI/packages/db/src/backup-lib.test.ts
T
Devin FoleyandPaperclip 9b624a110e fix: checkpoint database backups before idle sleep (#15629)
## Thinking Path

> - Paperclip coordinates agent work and preserves company state.
> - Hosted instances can stop compute when admission and all work
sources are quiet.
> - The periodic database backup timer currently blocks every automatic
sleep.
> - Disabling backups would remove recovery points.
> - This change offers an explicit final-backup checkpoint before sleep.
> - A verified archive and restart marker preserve recovery while
compute is stopped.

## Linked Issues or Issue Description

**What happened?**

An otherwise idle instance with automatic backups enabled always reports
background work. Operators cannot reclaim idle compute without disabling
those backups. The work inventory also queries the database before
checking known local blockers.

**Expected behavior**

An operator can enable checkpoint mode. The server may authorize sleep
only after all other work is quiet and a fresh backup is verified under
the same owned hold. Backups resume on restart with the existing
retention policy.

**Steps to reproduce**

Acquire a bounded idle drain on an instance with no application work and
automatic backups enabled. Read its owned idle safety report. It reports
present even when all other work is clear. With this change and
`PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1`, a successful final
backup can permit sleep; backup failures still refuse it.

**Paperclip version or commit**

Reproduced from master at 4265cb3a2b.

**Deployment mode**

Hosted single-process instances with persistent backup storage and
serialized sleep/wake operations.

Searched open backup and sleep PRs and issues. Related: #15522 and
#15599 establish owned idle holds and plugin drains. #15358 proposes
general backup health/retention validation; this change verifies only
opted-in sleep checkpoints and their restart catch-up.

## What Changed

- Add an off-by-default checkpoint flag. Keep automatic backups enabled.
- Reject known local blockers before database work.
- Finish a new backup during each owned safety check, including final
revalidation.
- Verify the complete gzip archive and sync its file and directory
before pruning older recovery points.
- Use unique checkpoint filenames so repeated checks cannot replace an
earlier verified archive.
- Keep backup work counted through dump, validation and persistence.
Changed owners, expired holds, concurrent work and errors refuse sleep.
- Persist a restart marker before the dump. Attempt a fresh backup on
restart before idle eligibility, then resume the normal timer. Only a
verified fresh backup clears the marker.
- Restrict managed checkpoint reads to verified Cloud control actors.
Tenant instance-admin elevation does not grant backup authority.
- Keep archive verification outside engine fallback so failures retain
their original cause.
- Document that recurring duplicate archives pause while asleep. The
final checkpoint and historical archives remain on persistent storage;
retention does not prune while asleep.

## Verification

- Focused idle, spool, backup and route checks: 137 tests passed,
including restoration into a separate PostgreSQL database.
- Backup library and checkpoint tests: 15 passed, including rejection
before historical backups can be pruned.
- `pnpm -r typecheck` and `pnpm build` passed. Follow-up server
typecheck and database build cover the verification callback.
- Review follow-up: 51 backup/restore/startup tests, 122 route/safety
tests, 32 signed-control tests, and server typecheck passed. Startup
cases exercise both flag states and retry after failure. The final
test-cleanup adjustment passes all 35 startup tests.
- The full local run hit four ancestor-directory skill-fixture failures
and one tool-route failure. All five passed from an isolated checkout at
949210 (3 targeted chat/tool checks and the complete 39-test email
suite). The redundant local run was stopped after final-head hosted CI
passed. No full local pass is claimed.
- Added-line secret/private-reference review and `git diff --check`
passed.
- [Final-head hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/37853380365)
passed, including typecheck, build, all general and serialized test
shards, all eight E2E shards and the canary dry run. Apex is 5/5 on
5b1de388ef, with no new findings and all review threads resolved. The
branch is conflict-free. Live staging activation awaits merge and a
canonical image; no production configuration or deployment changed.

## Risks

- The flag defaults off. This is for a single process whose host
controls every writer and preserves its backup volume across sleep.
Independent database writers require a different backup owner.
- Safety reads perform a backup only after a valid owned drain and all
other checks pass. Slow backups can outlive the caller timeout; they
remain counted until their work settles and cannot authorize sleep after
hold expiry.
- Initial and final reads each take a fresh backup. Hosts must allow
sufficient request time while retaining shutdown headroom.
- Retention settings are unchanged. Sleeping instances stop producing
duplicate hourly archives and retain their last checkpoint. Filesystem
data, encryption keys and off-provider disaster recovery remain separate
responsibilities.
- No schema migration. Disable the checkpoint flag to restore the
blanket backup blocker; automatic backups and an existing restart marker
remain active.

## Model Used

OpenAI Codex, GPT-6. Used reasoning, repository inspection, code editing
and command execution. The runtime did not expose an exact model
revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
identifier
- [x] Focused and isolated local checks pass; final-head hosted CI is
green (local full-run limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 15:58:55 -07:00

666 lines
26 KiB
TypeScript

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { gunzipSync, gzipSync } from "node:zlib";
import { afterEach, describe, expect, it } from "vitest";
import postgres from "postgres";
import { createBufferedTextFileWriter, runDatabaseBackup, runDatabaseRestore } from "./backup-lib.js";
import { ensurePostgresDatabase } from "./client.js";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const cleanups: Array<() => Promise<void> | void> = [];
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
function createTempDir(prefix: string): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
cleanups.push(() => {
fs.rmSync(dir, { recursive: true, force: true });
});
return dir;
}
async function createTempDatabase(): Promise<string> {
const db = await startEmbeddedPostgresTestDatabase("paperclip-db-backup-");
cleanups.push(db.cleanup);
return db.connectionString;
}
async function createSiblingDatabase(connectionString: string, databaseName: string): Promise<string> {
const adminUrl = new URL(connectionString);
adminUrl.pathname = "/postgres";
await ensurePostgresDatabase(adminUrl.toString(), databaseName);
const targetUrl = new URL(connectionString);
targetUrl.pathname = `/${databaseName}`;
return targetUrl.toString();
}
afterEach(async () => {
while (cleanups.length > 0) {
const cleanup = cleanups.pop();
await cleanup?.();
}
}, 60_000);
if (!embeddedPostgresSupport.supported) {
console.warn(
`Skipping embedded Postgres backup tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
}
describe("createBufferedTextFileWriter", () => {
it("preserves line boundaries across buffered flushes", async () => {
const tempDir = createTempDir("paperclip-buffered-writer-");
const outputPath = path.join(tempDir, "backup.sql");
const writer = createBufferedTextFileWriter(outputPath, 16);
const lines = [
"-- header",
"BEGIN;",
"",
"INSERT INTO test VALUES (1);",
"-- footer",
];
for (const line of lines) {
writer.emit(line);
}
await writer.close();
expect(fs.readFileSync(outputPath, "utf8")).toBe(lines.join("\n"));
});
});
describeEmbeddedPostgres("runDatabaseBackup", () => {
it("preserves a post-pg_dump verification error without falling back or pruning", async () => {
const connectionString = await createTempDatabase();
const backupDir = createTempDir("checkpoint-pg-dump-");
const executable = path.join(backupDir, "pg-dump-fixture");
fs.writeFileSync(executable, "#!/bin/sh\nprintf 'SELECT 1;\\n'\n", { mode: 0o700 });
const history = path.join(backupDir, "paperclip-history.sql.gz");
const historyBytes = gzipSync("SELECT 'history';");
fs.writeFileSync(history, historyBytes);
fs.utimesSync(history, new Date(0), new Date(0));
const previous = process.env.PAPERCLIP_PG_DUMP_PATH;
process.env.PAPERCLIP_PG_DUMP_PATH = executable;
const failure = new Error("fixture archive fsync failed");
let verified = 0;
try {
await expect(runDatabaseBackup({
connectionString, backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
verifyBeforePrune: async archive => {
verified++;
expect(gunzipSync(fs.readFileSync(archive)).toString()).toBe("SELECT 1;\n");
throw failure;
},
})).rejects.toBe(failure);
expect(verified).toBe(1);
expect(fs.readFileSync(history)).toEqual(historyBytes);
expect(fs.readdirSync(backupDir).filter(name => name.endsWith(".sql.gz"))).toEqual(["paperclip-history.sql.gz"]);
} finally {
if (previous === undefined) delete process.env.PAPERCLIP_PG_DUMP_PATH;
else process.env.PAPERCLIP_PG_DUMP_PATH = previous;
}
});
it("preserves identity generation, sequence options and progress in JavaScript backups", async () => {
const source = await createTempDatabase();
const target = await createSiblingDatabase(source, "identity_restore_target");
const sourceSql = postgres(source, { max: 1, onnotice: () => {} });
const targetSql = postgres(target, { max: 1, onnotice: () => {} });
try {
await sourceSql.unsafe(`
CREATE SCHEMA identity_fixture;
CREATE TABLE identity_fixture.records (
id bigint GENERATED ALWAYS AS IDENTITY (START WITH 7 INCREMENT BY 2) PRIMARY KEY,
default_id integer GENERATED BY DEFAULT AS IDENTITY,
value text NOT NULL
);
INSERT INTO identity_fixture.records (value) VALUES ('one'), ('two');
`);
const backup = await runDatabaseBackup({ connectionString: source, backupDir: createTempDir("identity-backup-"), backupEngine: "javascript", retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 } });
await runDatabaseRestore({ connectionString: target, backupFile: backup.backupFile });
expect(await targetSql`SELECT id::text, default_id, value FROM identity_fixture.records ORDER BY id`)
.toEqual([{ id: "7", default_id: 1, value: "one" }, { id: "9", default_id: 2, value: "two" }]);
expect(await targetSql`INSERT INTO identity_fixture.records (value) VALUES ('three') RETURNING id::text, default_id`)
.toEqual([{ id: "11", default_id: 3 }]);
await expect(targetSql`INSERT INTO identity_fixture.records (id, value) VALUES (99, 'explicit')`).rejects.toThrow("non-DEFAULT");
} finally {
await sourceSql.end();
await targetSql.end();
}
}, 30_000);
it(
"keeps the newest backup for each retained calendar month",
async () => {
const sourceConnectionString = await createTempDatabase();
const backupDir = createTempDir("paperclip-db-backup-retention-");
const realDateNow = Date.now;
Date.now = () => Date.UTC(2026, 2, 31, 12, 0, 0);
const janNewest = path.join(backupDir, "paperclip-test-2026-01-28T12-00-00.sql.gz");
const janOlder = path.join(backupDir, "paperclip-test-2026-01-10T12-00-00.sql.gz");
const decOld = path.join(backupDir, "paperclip-test-2025-12-15T12-00-00.sql.gz");
try {
fs.writeFileSync(janNewest, "jan-newest");
fs.writeFileSync(janOlder, "jan-older");
fs.writeFileSync(decOld, "dec-old");
fs.utimesSync(janNewest, new Date("2026-01-28T12:00:00Z"), new Date("2026-01-28T12:00:00Z"));
fs.utimesSync(janOlder, new Date("2026-01-10T12:00:00Z"), new Date("2026-01-10T12:00:00Z"));
fs.utimesSync(decOld, new Date("2025-12-15T12:00:00Z"), new Date("2025-12-15T12:00:00Z"));
const result = await runDatabaseBackup({
connectionString: sourceConnectionString,
backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 2 },
filenamePrefix: "paperclip-test",
});
expect(result.prunedCount).toBe(2);
expect(fs.existsSync(janNewest)).toBe(true);
expect(fs.existsSync(janOlder)).toBe(false);
expect(fs.existsSync(decOld)).toBe(false);
} finally {
Date.now = realDateNow;
}
},
30_000,
);
it(
"backs up and restores large table payloads without materializing one giant string",
async () => {
const sourceConnectionString = await createTempDatabase();
const restoreConnectionString = await createSiblingDatabase(
sourceConnectionString,
"paperclip_restore_target",
);
const backupDir = createTempDir("paperclip-db-backup-output-");
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
try {
await sourceSql.unsafe(`
CREATE TYPE "public"."backup_test_state" AS ENUM ('pending', 'done');
`);
await sourceSql.unsafe(`
CREATE TABLE "public"."backup_test_records" (
"id" serial PRIMARY KEY,
"title" text NOT NULL,
"payload" text NOT NULL,
"state" "public"."backup_test_state" NOT NULL,
"metadata" jsonb,
"created_at" timestamptz NOT NULL DEFAULT now()
);
`);
await sourceSql.unsafe(`
CREATE FUNCTION "public"."backup_test_mark_done"()
RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW."state" := 'done';
RETURN NEW;
END;
$$;
CREATE TRIGGER "backup_test_mark_done_trigger"
BEFORE UPDATE OF "title" ON "public"."backup_test_records"
FOR EACH ROW
EXECUTE FUNCTION "public"."backup_test_mark_done"();
`);
const payload = "x".repeat(8192);
for (let index = 0; index < 160; index += 1) {
const createdAt = new Date(Date.UTC(2026, 0, 1, 0, 0, index));
await sourceSql`
INSERT INTO "public"."backup_test_records" (
"title",
"payload",
"state",
"metadata",
"created_at"
)
VALUES (
${`row-${index}`},
${payload},
${index % 2 === 0 ? "pending" : "done"}::"public"."backup_test_state",
${JSON.stringify({ index, even: index % 2 === 0 })}::jsonb,
${createdAt}
)
`;
}
const result = await runDatabaseBackup({
connectionString: sourceConnectionString,
backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
filenamePrefix: "paperclip-test",
backupEngine: "javascript",
});
expect(result.backupFile).toMatch(/paperclip-test-.*\.sql\.gz$/);
expect(result.sizeBytes).toBeGreaterThan(0);
expect(fs.existsSync(result.backupFile)).toBe(true);
await runDatabaseRestore({
connectionString: restoreConnectionString,
backupFile: result.backupFile,
});
const counts = await restoreSql.unsafe<{ count: number }[]>(`
SELECT count(*)::int AS count
FROM "public"."backup_test_records"
`);
expect(counts[0]?.count).toBe(160);
const sampleRows = await restoreSql.unsafe<{
title: string;
payload: string;
state: string;
metadata: { index: number; even: boolean } | string;
}[]>(`
SELECT "title", "payload", "state"::text AS "state", "metadata"
FROM "public"."backup_test_records"
WHERE "title" IN ('row-0', 'row-159')
ORDER BY "title"
`);
expect(sampleRows.map((row) => ({
...row,
metadata: typeof row.metadata === "string" ? JSON.parse(row.metadata) : row.metadata,
}))).toEqual([
{
title: "row-0",
payload,
state: "pending",
metadata: { index: 0, even: true },
},
{
title: "row-159",
payload,
state: "done",
metadata: { index: 159, even: false },
},
]);
await restoreSql.unsafe(`
UPDATE "public"."backup_test_records"
SET "title" = 'triggered'
WHERE "title" = 'row-0'
`);
const triggeredRows = await restoreSql.unsafe<{ state: string }[]>(`
SELECT "state"::text AS "state"
FROM "public"."backup_test_records"
WHERE "title" = 'triggered'
`);
expect(triggeredRows).toEqual([{ state: "done" }]);
} finally {
await sourceSql.end();
await restoreSql.end();
}
},
60_000,
);
it(
"backs up and restores non-public database schemas and migration history",
async () => {
const sourceConnectionString = await createTempDatabase();
const restoreConnectionString = await createSiblingDatabase(
sourceConnectionString,
"paperclip_full_logical_restore_target",
);
const backupDir = createTempDir("paperclip-db-full-logical-backup-");
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
try {
await sourceSql.unsafe(`
CREATE SCHEMA IF NOT EXISTS "drizzle";
CREATE TABLE IF NOT EXISTS "drizzle"."__drizzle_migrations" (
"id" serial PRIMARY KEY,
"hash" text NOT NULL,
"created_at" bigint
);
INSERT INTO "drizzle"."__drizzle_migrations" ("hash", "created_at")
VALUES ('paperclip-migration-history', 1770000000000);
`);
await sourceSql.unsafe(`
CREATE TABLE "public"."backup_parent_records" (
"id" uuid PRIMARY KEY,
"name" text NOT NULL
);
INSERT INTO "public"."backup_parent_records" ("id", "name")
VALUES ('11111111-1111-4111-8111-111111111111', 'parent');
`);
await sourceSql.unsafe(`
CREATE TABLE "public"."plugin_rows" (
"id" serial PRIMARY KEY,
"note" text NOT NULL
);
CREATE TABLE "public"."audit_rows" (
"id" serial PRIMARY KEY,
"secret_note" text
);
INSERT INTO "public"."plugin_rows" ("note")
VALUES ('public-collision');
INSERT INTO "public"."audit_rows" ("secret_note")
VALUES ('public-secret');
`);
await sourceSql.unsafe(`
CREATE SCHEMA "plugin_backup_scope";
CREATE TYPE "plugin_backup_scope"."plugin_status" AS ENUM ('ready', 'done');
CREATE TABLE "plugin_backup_scope"."plugin_rows" (
"id" serial PRIMARY KEY,
"parent_id" uuid NOT NULL REFERENCES "public"."backup_parent_records"("id") ON DELETE CASCADE,
"status" "plugin_backup_scope"."plugin_status" NOT NULL,
"note" text NOT NULL
);
CREATE TABLE "plugin_backup_scope"."audit_rows" (
"id" serial PRIMARY KEY,
"secret_note" text
);
CREATE UNIQUE INDEX "plugin_rows_note_uq" ON "plugin_backup_scope"."plugin_rows" ("note");
INSERT INTO "plugin_backup_scope"."plugin_rows" ("parent_id", "status", "note")
VALUES ('11111111-1111-4111-8111-111111111111', 'ready', 'first');
INSERT INTO "plugin_backup_scope"."audit_rows" ("secret_note")
VALUES ('plugin-secret');
`);
const result = await runDatabaseBackup({
connectionString: sourceConnectionString,
backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
filenamePrefix: "paperclip-full-logical-test",
backupEngine: "javascript",
excludeTables: ["plugin_rows"],
nullifyColumns: {
audit_rows: ["secret_note"],
},
});
await runDatabaseRestore({
connectionString: restoreConnectionString,
backupFile: result.backupFile,
});
const migrationRows = await restoreSql.unsafe<{ hash: string }[]>(`
SELECT "hash"
FROM "drizzle"."__drizzle_migrations"
WHERE "hash" = 'paperclip-migration-history'
`);
expect(migrationRows).toEqual([{ hash: "paperclip-migration-history" }]);
const pluginRows = await restoreSql.unsafe<{ note: string; status: string; parent_name: string }[]>(`
SELECT r."note", r."status"::text AS "status", p."name" AS "parent_name"
FROM "plugin_backup_scope"."plugin_rows" r
JOIN "public"."backup_parent_records" p ON p."id" = r."parent_id"
`);
expect(pluginRows).toEqual([{ note: "first", status: "ready", parent_name: "parent" }]);
const publicCollisionRows = await restoreSql.unsafe<{ count: number }[]>(`
SELECT count(*)::int AS count
FROM "public"."plugin_rows"
`);
expect(publicCollisionRows[0]?.count).toBe(0);
const publicAuditRows = await restoreSql.unsafe<{ secret_note: string | null }[]>(`
SELECT "secret_note"
FROM "public"."audit_rows"
`);
expect(publicAuditRows).toEqual([{ secret_note: null }]);
const pluginAuditRows = await restoreSql.unsafe<{ secret_note: string | null }[]>(`
SELECT "secret_note"
FROM "plugin_backup_scope"."audit_rows"
`);
expect(pluginAuditRows).toEqual([{ secret_note: "plugin-secret" }]);
await expect(
restoreSql.unsafe(`
INSERT INTO "plugin_backup_scope"."plugin_rows" ("parent_id", "status", "note")
VALUES ('11111111-1111-4111-8111-111111111111', 'done', 'first')
`),
).rejects.toThrow();
} finally {
await sourceSql.end();
await restoreSql.end();
}
},
60_000,
);
it(
"preserves composite foreign key column order without duplicate referenced columns",
async () => {
const sourceConnectionString = await createTempDatabase();
const restoreConnectionString = await createSiblingDatabase(
sourceConnectionString,
"paperclip_composite_fk_restore_target",
);
const backupDir = createTempDir("paperclip-db-composite-fk-backup-");
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
try {
await sourceSql.unsafe(`
CREATE SCHEMA "plugin_composite_fk";
CREATE TABLE "plugin_composite_fk"."content_cases" (
"id" uuid PRIMARY KEY,
"company_id" uuid NOT NULL,
"title" text NOT NULL,
CONSTRAINT "content_cases_company_case_unique" UNIQUE ("company_id", "id")
);
CREATE TABLE "plugin_composite_fk"."content_case_signals" (
"company_id" uuid NOT NULL,
"case_id" uuid NOT NULL,
"signal" text NOT NULL,
"scopes" text[] NOT NULL,
"warnings" jsonb DEFAULT '[]'::jsonb NOT NULL,
CONSTRAINT "content_case_signals_company_case"
FOREIGN KEY ("company_id", "case_id")
REFERENCES "plugin_composite_fk"."content_cases" ("company_id", "id")
ON DELETE CASCADE
);
INSERT INTO "plugin_composite_fk"."content_cases" ("company_id", "id", "title")
VALUES (
'11111111-1111-4111-8111-111111111111',
'22222222-2222-4222-8222-222222222222',
'case'
);
INSERT INTO "plugin_composite_fk"."content_case_signals" ("company_id", "case_id", "signal", "scopes", "warnings")
VALUES (
'11111111-1111-4111-8111-111111111111',
'22222222-2222-4222-8222-222222222222',
'signal',
ARRAY['upstream_import:preview', 'scope with space', 'quoted "scope"', 'NULL', 'null'],
jsonb_build_array('json warning', jsonb_build_object('code', 'quoted "value"'))
);
`);
const result = await runDatabaseBackup({
connectionString: sourceConnectionString,
backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
filenamePrefix: "paperclip-composite-fk-test",
backupEngine: "javascript",
});
await runDatabaseRestore({
connectionString: restoreConnectionString,
backupFile: result.backupFile,
});
const rows = await restoreSql.unsafe<{
signal: string;
title: string;
scopes: string[];
warnings: Array<string | { code: string }>;
}[]>(`
SELECT s."signal", c."title", s."scopes", s."warnings"
FROM "plugin_composite_fk"."content_case_signals" s
JOIN "plugin_composite_fk"."content_cases" c
ON c."company_id" = s."company_id"
AND c."id" = s."case_id"
`);
expect(rows).toEqual([
{
signal: "signal",
title: "case",
scopes: ["upstream_import:preview", "scope with space", 'quoted "scope"', "NULL", "null"],
warnings: ["json warning", { code: 'quoted "value"' }],
},
]);
await expect(
restoreSql.unsafe(`
INSERT INTO "plugin_composite_fk"."content_case_signals" ("company_id", "case_id", "signal", "scopes")
VALUES (
'11111111-1111-4111-8111-111111111111',
'33333333-3333-4333-8333-333333333333',
'orphan',
ARRAY[]::text[]
)
`),
).rejects.toThrow();
} finally {
await sourceSql.end();
await restoreSql.end();
}
},
60_000,
);
it(
"restores fallback COPY data when child tables are dumped before parent tables",
async () => {
const sourceConnectionString = await createTempDatabase();
const restoreConnectionString = await createSiblingDatabase(
sourceConnectionString,
"paperclip_copy_fk_restore_target",
);
const backupDir = createTempDir("paperclip-db-copy-fk-backup-");
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
const originalPgDumpPath = process.env.PAPERCLIP_PG_DUMP_PATH;
const originalPsqlPath = process.env.PAPERCLIP_PSQL_PATH;
process.env.PAPERCLIP_PG_DUMP_PATH = "/bin/false";
process.env.PAPERCLIP_PSQL_PATH = "/bin/false";
try {
await sourceSql.unsafe(`
CREATE TABLE "public"."zzz_parent_records" (
"id" uuid PRIMARY KEY,
"name" text NOT NULL
);
CREATE TABLE "public"."aaa_child_records" (
"id" uuid PRIMARY KEY,
"parent_id" uuid NOT NULL REFERENCES "public"."zzz_parent_records"("id") ON DELETE CASCADE,
"note" text NOT NULL
);
INSERT INTO "public"."zzz_parent_records" ("id", "name")
VALUES ('11111111-1111-4111-8111-111111111111', 'parent');
INSERT INTO "public"."aaa_child_records" ("id", "parent_id", "note")
VALUES (
'22222222-2222-4222-8222-222222222222',
'11111111-1111-4111-8111-111111111111',
'child emitted before parent'
);
`);
const result = await runDatabaseBackup({
connectionString: sourceConnectionString,
backupDir,
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
filenamePrefix: "paperclip-copy-fk-test",
backupEngine: "auto",
});
const backupSql = gunzipSync(await fs.promises.readFile(result.backupFile)).toString("utf8");
expect(backupSql.indexOf("-- Data for: public.aaa_child_records")).toBeGreaterThan(-1);
expect(backupSql.indexOf("-- Data for: public.aaa_child_records")).toBeLessThan(
backupSql.indexOf("-- Data for: public.zzz_parent_records"),
);
expect(backupSql).not.toContain(" FROM stdin;");
await runDatabaseRestore({
connectionString: restoreConnectionString,
backupFile: result.backupFile,
});
const rows = await restoreSql.unsafe<{ note: string; name: string }[]>(`
SELECT child."note", parent."name"
FROM "public"."aaa_child_records" child
JOIN "public"."zzz_parent_records" parent ON parent."id" = child."parent_id"
`);
expect(rows).toEqual([{ note: "child emitted before parent", name: "parent" }]);
} finally {
if (originalPgDumpPath === undefined) {
delete process.env.PAPERCLIP_PG_DUMP_PATH;
} else {
process.env.PAPERCLIP_PG_DUMP_PATH = originalPgDumpPath;
}
if (originalPsqlPath === undefined) {
delete process.env.PAPERCLIP_PSQL_PATH;
} else {
process.env.PAPERCLIP_PSQL_PATH = originalPsqlPath;
}
await sourceSql.end();
await restoreSql.end();
}
},
60_000,
);
it(
"restores legacy public-only backups without migration history",
async () => {
const restoreConnectionString = await createTempDatabase();
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
const backupDir = createTempDir("paperclip-db-restore-manual-");
const backupFile = path.join(backupDir, "manual.sql");
try {
await fs.promises.writeFile(
backupFile,
[
"-- Paperclip database backup",
"-- Created: 2026-04-06T00:00:00.000Z",
"",
"BEGIN;",
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
"CREATE TABLE public.restore_stream_test (id integer primary key, payload text not null);",
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
"INSERT INTO public.restore_stream_test (id, payload)",
"VALUES (1, 'hello');",
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
"COMMIT;",
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
].join("\n"),
"utf8",
);
await runDatabaseRestore({
connectionString: restoreConnectionString,
backupFile,
});
const rows = await restoreSql.unsafe<{ payload: string }[]>(`
SELECT payload
FROM public.restore_stream_test
`);
expect(rows).toEqual([{ payload: "hello" }]);
} finally {
await restoreSql.end();
}
},
20_000,
);
});