mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 09:07:46 +02:00
## Thinking Path > - Paperclip manages AI agents and their work. > - The experimental Runner owns provider processes and durable sessions. > - Pi needs working task execution and human controls. > - The five-PR stack must preserve changes already on master. > - The runtime implementation has merged in #14921; this layer carries setup documentation and a narrowly scoped CI fixture repair. > - This PR belongs to native GitHub stack #15602, ending at #14956. ## Linked Issues or Issue Description Refs #14436, #14631, #14743 and #14956. Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are #14921, #14922, #14923, #14924 and #14956. The user authorized the complete merge after checks pass. Existing `pi_local` execution is unchanged. Accounting and wider provider/platform qualification remain deferred. ## What Changed - Create the warm Pi fake sidecar and its directory with explicit private modes, independent of the runner umask. Preserve the qualified-asset permission guard and every warm-admission timing assertion. - Document explicit public Pi host setup and verified Linux companion import for Daytona. - Preserve historical installation and qualification checkpoints with their original source identities. - Use the repository's existing `EMBEDDED_POSTGRES_TEST_TIMEOUT_MS` for the backup error fixture, which starts a real PostgreSQL cluster. Preserve all error, archive, and retention assertions. ## Verification - Full local `pnpm -r typecheck` passes, including Runner Rust, server and UI. Focused integration checks pass: 194 Runner admission/environment tests, 63 profile/credential tests with one expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice tests. - Full local `pnpm build` passes on the final source. - Fresh final-source checks pass: all 698 Rust workspace tests (32 binaries), 156 credential/profile/controller tests with one expected skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests. - The profile-21 Pi materializer passes on the native host with the official pinned Node 24.21.0 and its npm. It verifies all 150 locked packages, the patched dependency and the exact closure. Setup/package bundle tests and UI token gates pass. - The old hashes were reproduced for all three supported targets before calculating the patched graph. New closure hashes are darwin-arm64 `282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`, darwin-x64 `64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and linux-x64 `713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`. Independent native platform execution is separate from graph identity reproduction. - Historical cloud qualification remains unchanged: all seven core cases pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`, harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image `sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`, and accepted Sonnet 4.6/low fixture. All 215 canonical files and all seven cleanup checks pass independent verification. These are profile-19 results and are not relabeled as fresh profile-22 runs. - Current Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [The readiness plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md) preserves campaign and failed-attempt provenance. - Merge only after every PR's current-head CI and fresh review pass. Linux CI covers the full suites, build and browser tests. The local embedded Postgres API-authority suite cannot start on this macOS/Node 26 host, so Linux CI must confirm that suite. ### Fresh profile-22 core qualification — 2026-10-08 All seven accepted core cases pass canonically on Pi profile 22, with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. This model is a fixture; production accepts the caller's explicit Pi provider/model. Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`. Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`. Immutable cloud image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`. Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [Hosted Linux image and clean-install verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023) passes, including all 20 source-bound archives, normal CLI/Pi setup, companion import and the production pack reader. This exact installation source includes the latest master integration and the corrected Pi warm instruction-root fence. Full local typecheck/build and current-head hosted CI verify the final stack. All 13 focused real-root regressions pass. The full local executor suite passed 662 tests; 15 database tests could not start the Mac embedded PostgreSQL service. Hosted Linux CI passes the full required verification and E2E checks. These fresh results keep their own source identity; profile-19 results remain historical. | Core path | Canonical campaign | Retained archive SHA-256 | | --- | --- | --- | | File edit, validation, download and Done | `pi-core22-replyfix-0-1791511228` | 23 files; `a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` | | Pending question and controller restart | `pi-core22-replyfix-1-1791511376` | 33 files; `6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` | | Three-turn session/process/workspace continuity | `pi-core22-replyfix-2-1791511587` | 23 files; `7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` | | Four typed questions and browser reconnects | `pi-core22-replyfix-3-1791511881` | 42 files; `9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` | | Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22 files; `a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` | | Same-turn steering and permission denial | `pi-core22-replyfix-5-1791512261` | 39 files; `c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` | | Stop during pending permission | `pi-core22-replyfix-6-1791512390` | 33 files; `7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` | All 215 canonical files (28714587 bytes) are independently hash-verified. All seven cleanup grades pass, with no owned runtime process or temporary root after each case. Automatic retries are zero. The owned cloud host stopped normally after retention. The prior profile-22 warm attempt remains failed and separately retained: archive SHA-256 `1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its original canonical classification is preserved. Diagnosis reproduced a product bug comparing an agent-files root against an unset checkpoint-only field. The fix stores the admitted physical root separately from the adopted per-run collection capability. The real-root regression fails before the fix and passes afterward, including rejection of a changed physical root. Fixture, grader, model and all seven accepted case IDs are unchanged; this fresh campaign tests final-reply publication after file registration first. The intermediate restart attempt also remains failed and retained: archive SHA-256 `5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its original canonical classification is preserved. Pi resumed, wrote the verified answer and completed its task; exact runner suspension was proven, but idle stop consumed about 5.2s and left under 3s for the drain acknowledgement. The Pi-only default shutdown grace is now 15s, preserving a full 5s drain round trip and a finite suspension reserve. Explicit caller deadlines, other provider defaults, literal drain receipts and exact suspension identity checks remain unchanged. The timing regression fails before this correction and passes afterward; all 18 focused settlement tests and Runner typecheck pass. The final-source file attempt is also preserved as failed (`candidate_failure`), archive SHA-256 `db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`. Native edit, validation, exact downloadable artifact and Done/succeeded all passed, and the exact final reply was durably recorded. A workspace recovery owner completed before the live heartbeat reached presentation, leaving that reply absent from task chat. Recovery now materializes only a completed final reply from the accepted turn of an ordinary internal Done task, preserving issue/run/contract binding, suppression, external-chat authorization and same-run deduplication. The database regression covers the generated file-preparation receipt, suppression, unapproved external continuation and replay. Server typecheck and all 49 response-selection tests pass; hosted Linux verifies the database regression because embedded PostgreSQL cannot start on this Mac. The delayed-final-answer database regression passes on [the final root-source Linux server shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152), alongside 1,108 passing tests. The first root Runner shard had one unchanged durable-resume test exceed its 5-second timeout; the identical top-source shard and the isolated exact test passed. One rerun of that failed job and its required aggregate passed without source or test changes. The original failed job log and the single-rerun receipt remain retained. ### October 9 merge verification Current merge head: `18744bf27490d6f064f9d5bac2c2af202c590a99`. All current-head checks pass, including `ci / verify` and `ci / e2e`; exact-head Greptile review is 5/5 with no unresolved threads. Current master conflicts are resolved. The user authorized the maintainer override of the code-owner review gate after these checks. The seven retained live core cases remain bound to source `3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image. ## Risks - The security correction changes the dependency closure and profile identity. Old sessions must reopen on the new profile. Exact identities and credential bindings fail closed. - The runner remains experimental and requires explicit selection. Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E model is a fixture. - Accounting and the broad platform/provider matrix remain deferred. This merge does not publish a release or deploy a service. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, editing and tool use. The exact serving ID and context window are not exposed in this session. Final live qualification uses Pi 1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
667 lines
26 KiB
TypeScript
667 lines
26 KiB
TypeScript
import fs from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { gunzipSync, gzipSync } from "node:zlib";
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
import postgres from "postgres";
|
|
import { createBufferedTextFileWriter, runDatabaseBackup, runDatabaseRestore } from "./backup-lib.js";
|
|
import { ensurePostgresDatabase } from "./client.js";
|
|
import {
|
|
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./test-embedded-postgres.js";
|
|
|
|
const cleanups: Array<() => Promise<void> | void> = [];
|
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
function createTempDir(prefix: string): string {
|
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
|
cleanups.push(() => {
|
|
fs.rmSync(dir, { recursive: true, force: true });
|
|
});
|
|
return dir;
|
|
}
|
|
|
|
async function createTempDatabase(): Promise<string> {
|
|
const db = await startEmbeddedPostgresTestDatabase("paperclip-db-backup-");
|
|
cleanups.push(db.cleanup);
|
|
return db.connectionString;
|
|
}
|
|
|
|
async function createSiblingDatabase(connectionString: string, databaseName: string): Promise<string> {
|
|
const adminUrl = new URL(connectionString);
|
|
adminUrl.pathname = "/postgres";
|
|
await ensurePostgresDatabase(adminUrl.toString(), databaseName);
|
|
const targetUrl = new URL(connectionString);
|
|
targetUrl.pathname = `/${databaseName}`;
|
|
return targetUrl.toString();
|
|
}
|
|
|
|
afterEach(async () => {
|
|
while (cleanups.length > 0) {
|
|
const cleanup = cleanups.pop();
|
|
await cleanup?.();
|
|
}
|
|
}, 60_000);
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres backup tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
describe("createBufferedTextFileWriter", () => {
|
|
it("preserves line boundaries across buffered flushes", async () => {
|
|
const tempDir = createTempDir("paperclip-buffered-writer-");
|
|
const outputPath = path.join(tempDir, "backup.sql");
|
|
const writer = createBufferedTextFileWriter(outputPath, 16);
|
|
const lines = [
|
|
"-- header",
|
|
"BEGIN;",
|
|
"",
|
|
"INSERT INTO test VALUES (1);",
|
|
"-- footer",
|
|
];
|
|
|
|
for (const line of lines) {
|
|
writer.emit(line);
|
|
}
|
|
|
|
await writer.close();
|
|
|
|
expect(fs.readFileSync(outputPath, "utf8")).toBe(lines.join("\n"));
|
|
});
|
|
});
|
|
|
|
describeEmbeddedPostgres("runDatabaseBackup", () => {
|
|
it("preserves a post-pg_dump verification error without falling back or pruning", async () => {
|
|
const connectionString = await createTempDatabase();
|
|
const backupDir = createTempDir("checkpoint-pg-dump-");
|
|
const executable = path.join(backupDir, "pg-dump-fixture");
|
|
fs.writeFileSync(executable, "#!/bin/sh\nprintf 'SELECT 1;\\n'\n", { mode: 0o700 });
|
|
const history = path.join(backupDir, "paperclip-history.sql.gz");
|
|
const historyBytes = gzipSync("SELECT 'history';");
|
|
fs.writeFileSync(history, historyBytes);
|
|
fs.utimesSync(history, new Date(0), new Date(0));
|
|
const previous = process.env.PAPERCLIP_PG_DUMP_PATH;
|
|
process.env.PAPERCLIP_PG_DUMP_PATH = executable;
|
|
const failure = new Error("fixture archive fsync failed");
|
|
let verified = 0;
|
|
try {
|
|
await expect(runDatabaseBackup({
|
|
connectionString, backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
|
|
verifyBeforePrune: async archive => {
|
|
verified++;
|
|
expect(gunzipSync(fs.readFileSync(archive)).toString()).toBe("SELECT 1;\n");
|
|
throw failure;
|
|
},
|
|
})).rejects.toBe(failure);
|
|
expect(verified).toBe(1);
|
|
expect(fs.readFileSync(history)).toEqual(historyBytes);
|
|
expect(fs.readdirSync(backupDir).filter(name => name.endsWith(".sql.gz"))).toEqual(["paperclip-history.sql.gz"]);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_PG_DUMP_PATH;
|
|
else process.env.PAPERCLIP_PG_DUMP_PATH = previous;
|
|
}
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
|
|
it("preserves identity generation, sequence options and progress in JavaScript backups", async () => {
|
|
const source = await createTempDatabase();
|
|
const target = await createSiblingDatabase(source, "identity_restore_target");
|
|
const sourceSql = postgres(source, { max: 1, onnotice: () => {} });
|
|
const targetSql = postgres(target, { max: 1, onnotice: () => {} });
|
|
try {
|
|
await sourceSql.unsafe(`
|
|
CREATE SCHEMA identity_fixture;
|
|
CREATE TABLE identity_fixture.records (
|
|
id bigint GENERATED ALWAYS AS IDENTITY (START WITH 7 INCREMENT BY 2) PRIMARY KEY,
|
|
default_id integer GENERATED BY DEFAULT AS IDENTITY,
|
|
value text NOT NULL
|
|
);
|
|
INSERT INTO identity_fixture.records (value) VALUES ('one'), ('two');
|
|
`);
|
|
const backup = await runDatabaseBackup({ connectionString: source, backupDir: createTempDir("identity-backup-"), backupEngine: "javascript", retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 } });
|
|
await runDatabaseRestore({ connectionString: target, backupFile: backup.backupFile });
|
|
expect(await targetSql`SELECT id::text, default_id, value FROM identity_fixture.records ORDER BY id`)
|
|
.toEqual([{ id: "7", default_id: 1, value: "one" }, { id: "9", default_id: 2, value: "two" }]);
|
|
expect(await targetSql`INSERT INTO identity_fixture.records (value) VALUES ('three') RETURNING id::text, default_id`)
|
|
.toEqual([{ id: "11", default_id: 3 }]);
|
|
await expect(targetSql`INSERT INTO identity_fixture.records (id, value) VALUES (99, 'explicit')`).rejects.toThrow("non-DEFAULT");
|
|
} finally {
|
|
await sourceSql.end();
|
|
await targetSql.end();
|
|
}
|
|
}, 30_000);
|
|
|
|
it(
|
|
"keeps the newest backup for each retained calendar month",
|
|
async () => {
|
|
const sourceConnectionString = await createTempDatabase();
|
|
const backupDir = createTempDir("paperclip-db-backup-retention-");
|
|
const realDateNow = Date.now;
|
|
Date.now = () => Date.UTC(2026, 2, 31, 12, 0, 0);
|
|
|
|
const janNewest = path.join(backupDir, "paperclip-test-2026-01-28T12-00-00.sql.gz");
|
|
const janOlder = path.join(backupDir, "paperclip-test-2026-01-10T12-00-00.sql.gz");
|
|
const decOld = path.join(backupDir, "paperclip-test-2025-12-15T12-00-00.sql.gz");
|
|
|
|
try {
|
|
fs.writeFileSync(janNewest, "jan-newest");
|
|
fs.writeFileSync(janOlder, "jan-older");
|
|
fs.writeFileSync(decOld, "dec-old");
|
|
|
|
fs.utimesSync(janNewest, new Date("2026-01-28T12:00:00Z"), new Date("2026-01-28T12:00:00Z"));
|
|
fs.utimesSync(janOlder, new Date("2026-01-10T12:00:00Z"), new Date("2026-01-10T12:00:00Z"));
|
|
fs.utimesSync(decOld, new Date("2025-12-15T12:00:00Z"), new Date("2025-12-15T12:00:00Z"));
|
|
|
|
const result = await runDatabaseBackup({
|
|
connectionString: sourceConnectionString,
|
|
backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 2 },
|
|
filenamePrefix: "paperclip-test",
|
|
});
|
|
|
|
expect(result.prunedCount).toBe(2);
|
|
expect(fs.existsSync(janNewest)).toBe(true);
|
|
expect(fs.existsSync(janOlder)).toBe(false);
|
|
expect(fs.existsSync(decOld)).toBe(false);
|
|
} finally {
|
|
Date.now = realDateNow;
|
|
}
|
|
},
|
|
30_000,
|
|
);
|
|
|
|
it(
|
|
"backs up and restores large table payloads without materializing one giant string",
|
|
async () => {
|
|
const sourceConnectionString = await createTempDatabase();
|
|
const restoreConnectionString = await createSiblingDatabase(
|
|
sourceConnectionString,
|
|
"paperclip_restore_target",
|
|
);
|
|
const backupDir = createTempDir("paperclip-db-backup-output-");
|
|
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
|
|
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
|
|
|
|
try {
|
|
await sourceSql.unsafe(`
|
|
CREATE TYPE "public"."backup_test_state" AS ENUM ('pending', 'done');
|
|
`);
|
|
await sourceSql.unsafe(`
|
|
CREATE TABLE "public"."backup_test_records" (
|
|
"id" serial PRIMARY KEY,
|
|
"title" text NOT NULL,
|
|
"payload" text NOT NULL,
|
|
"state" "public"."backup_test_state" NOT NULL,
|
|
"metadata" jsonb,
|
|
"created_at" timestamptz NOT NULL DEFAULT now()
|
|
);
|
|
`);
|
|
await sourceSql.unsafe(`
|
|
CREATE FUNCTION "public"."backup_test_mark_done"()
|
|
RETURNS trigger
|
|
LANGUAGE plpgsql
|
|
AS $$
|
|
BEGIN
|
|
NEW."state" := 'done';
|
|
RETURN NEW;
|
|
END;
|
|
$$;
|
|
CREATE TRIGGER "backup_test_mark_done_trigger"
|
|
BEFORE UPDATE OF "title" ON "public"."backup_test_records"
|
|
FOR EACH ROW
|
|
EXECUTE FUNCTION "public"."backup_test_mark_done"();
|
|
`);
|
|
|
|
const payload = "x".repeat(8192);
|
|
for (let index = 0; index < 160; index += 1) {
|
|
const createdAt = new Date(Date.UTC(2026, 0, 1, 0, 0, index));
|
|
await sourceSql`
|
|
INSERT INTO "public"."backup_test_records" (
|
|
"title",
|
|
"payload",
|
|
"state",
|
|
"metadata",
|
|
"created_at"
|
|
)
|
|
VALUES (
|
|
${`row-${index}`},
|
|
${payload},
|
|
${index % 2 === 0 ? "pending" : "done"}::"public"."backup_test_state",
|
|
${JSON.stringify({ index, even: index % 2 === 0 })}::jsonb,
|
|
${createdAt}
|
|
)
|
|
`;
|
|
}
|
|
|
|
const result = await runDatabaseBackup({
|
|
connectionString: sourceConnectionString,
|
|
backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
|
|
filenamePrefix: "paperclip-test",
|
|
backupEngine: "javascript",
|
|
});
|
|
|
|
expect(result.backupFile).toMatch(/paperclip-test-.*\.sql\.gz$/);
|
|
expect(result.sizeBytes).toBeGreaterThan(0);
|
|
expect(fs.existsSync(result.backupFile)).toBe(true);
|
|
|
|
await runDatabaseRestore({
|
|
connectionString: restoreConnectionString,
|
|
backupFile: result.backupFile,
|
|
});
|
|
|
|
const counts = await restoreSql.unsafe<{ count: number }[]>(`
|
|
SELECT count(*)::int AS count
|
|
FROM "public"."backup_test_records"
|
|
`);
|
|
expect(counts[0]?.count).toBe(160);
|
|
|
|
const sampleRows = await restoreSql.unsafe<{
|
|
title: string;
|
|
payload: string;
|
|
state: string;
|
|
metadata: { index: number; even: boolean } | string;
|
|
}[]>(`
|
|
SELECT "title", "payload", "state"::text AS "state", "metadata"
|
|
FROM "public"."backup_test_records"
|
|
WHERE "title" IN ('row-0', 'row-159')
|
|
ORDER BY "title"
|
|
`);
|
|
expect(sampleRows.map((row) => ({
|
|
...row,
|
|
metadata: typeof row.metadata === "string" ? JSON.parse(row.metadata) : row.metadata,
|
|
}))).toEqual([
|
|
{
|
|
title: "row-0",
|
|
payload,
|
|
state: "pending",
|
|
metadata: { index: 0, even: true },
|
|
},
|
|
{
|
|
title: "row-159",
|
|
payload,
|
|
state: "done",
|
|
metadata: { index: 159, even: false },
|
|
},
|
|
]);
|
|
|
|
await restoreSql.unsafe(`
|
|
UPDATE "public"."backup_test_records"
|
|
SET "title" = 'triggered'
|
|
WHERE "title" = 'row-0'
|
|
`);
|
|
const triggeredRows = await restoreSql.unsafe<{ state: string }[]>(`
|
|
SELECT "state"::text AS "state"
|
|
FROM "public"."backup_test_records"
|
|
WHERE "title" = 'triggered'
|
|
`);
|
|
expect(triggeredRows).toEqual([{ state: "done" }]);
|
|
} finally {
|
|
await sourceSql.end();
|
|
await restoreSql.end();
|
|
}
|
|
},
|
|
60_000,
|
|
);
|
|
|
|
it(
|
|
"backs up and restores non-public database schemas and migration history",
|
|
async () => {
|
|
const sourceConnectionString = await createTempDatabase();
|
|
const restoreConnectionString = await createSiblingDatabase(
|
|
sourceConnectionString,
|
|
"paperclip_full_logical_restore_target",
|
|
);
|
|
const backupDir = createTempDir("paperclip-db-full-logical-backup-");
|
|
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
|
|
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
|
|
|
|
try {
|
|
await sourceSql.unsafe(`
|
|
CREATE SCHEMA IF NOT EXISTS "drizzle";
|
|
CREATE TABLE IF NOT EXISTS "drizzle"."__drizzle_migrations" (
|
|
"id" serial PRIMARY KEY,
|
|
"hash" text NOT NULL,
|
|
"created_at" bigint
|
|
);
|
|
INSERT INTO "drizzle"."__drizzle_migrations" ("hash", "created_at")
|
|
VALUES ('paperclip-migration-history', 1770000000000);
|
|
`);
|
|
await sourceSql.unsafe(`
|
|
CREATE TABLE "public"."backup_parent_records" (
|
|
"id" uuid PRIMARY KEY,
|
|
"name" text NOT NULL
|
|
);
|
|
INSERT INTO "public"."backup_parent_records" ("id", "name")
|
|
VALUES ('11111111-1111-4111-8111-111111111111', 'parent');
|
|
`);
|
|
await sourceSql.unsafe(`
|
|
CREATE TABLE "public"."plugin_rows" (
|
|
"id" serial PRIMARY KEY,
|
|
"note" text NOT NULL
|
|
);
|
|
CREATE TABLE "public"."audit_rows" (
|
|
"id" serial PRIMARY KEY,
|
|
"secret_note" text
|
|
);
|
|
INSERT INTO "public"."plugin_rows" ("note")
|
|
VALUES ('public-collision');
|
|
INSERT INTO "public"."audit_rows" ("secret_note")
|
|
VALUES ('public-secret');
|
|
`);
|
|
await sourceSql.unsafe(`
|
|
CREATE SCHEMA "plugin_backup_scope";
|
|
CREATE TYPE "plugin_backup_scope"."plugin_status" AS ENUM ('ready', 'done');
|
|
CREATE TABLE "plugin_backup_scope"."plugin_rows" (
|
|
"id" serial PRIMARY KEY,
|
|
"parent_id" uuid NOT NULL REFERENCES "public"."backup_parent_records"("id") ON DELETE CASCADE,
|
|
"status" "plugin_backup_scope"."plugin_status" NOT NULL,
|
|
"note" text NOT NULL
|
|
);
|
|
CREATE TABLE "plugin_backup_scope"."audit_rows" (
|
|
"id" serial PRIMARY KEY,
|
|
"secret_note" text
|
|
);
|
|
CREATE UNIQUE INDEX "plugin_rows_note_uq" ON "plugin_backup_scope"."plugin_rows" ("note");
|
|
INSERT INTO "plugin_backup_scope"."plugin_rows" ("parent_id", "status", "note")
|
|
VALUES ('11111111-1111-4111-8111-111111111111', 'ready', 'first');
|
|
INSERT INTO "plugin_backup_scope"."audit_rows" ("secret_note")
|
|
VALUES ('plugin-secret');
|
|
`);
|
|
|
|
const result = await runDatabaseBackup({
|
|
connectionString: sourceConnectionString,
|
|
backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
|
|
filenamePrefix: "paperclip-full-logical-test",
|
|
backupEngine: "javascript",
|
|
excludeTables: ["plugin_rows"],
|
|
nullifyColumns: {
|
|
audit_rows: ["secret_note"],
|
|
},
|
|
});
|
|
|
|
await runDatabaseRestore({
|
|
connectionString: restoreConnectionString,
|
|
backupFile: result.backupFile,
|
|
});
|
|
|
|
const migrationRows = await restoreSql.unsafe<{ hash: string }[]>(`
|
|
SELECT "hash"
|
|
FROM "drizzle"."__drizzle_migrations"
|
|
WHERE "hash" = 'paperclip-migration-history'
|
|
`);
|
|
expect(migrationRows).toEqual([{ hash: "paperclip-migration-history" }]);
|
|
|
|
const pluginRows = await restoreSql.unsafe<{ note: string; status: string; parent_name: string }[]>(`
|
|
SELECT r."note", r."status"::text AS "status", p."name" AS "parent_name"
|
|
FROM "plugin_backup_scope"."plugin_rows" r
|
|
JOIN "public"."backup_parent_records" p ON p."id" = r."parent_id"
|
|
`);
|
|
expect(pluginRows).toEqual([{ note: "first", status: "ready", parent_name: "parent" }]);
|
|
|
|
const publicCollisionRows = await restoreSql.unsafe<{ count: number }[]>(`
|
|
SELECT count(*)::int AS count
|
|
FROM "public"."plugin_rows"
|
|
`);
|
|
expect(publicCollisionRows[0]?.count).toBe(0);
|
|
|
|
const publicAuditRows = await restoreSql.unsafe<{ secret_note: string | null }[]>(`
|
|
SELECT "secret_note"
|
|
FROM "public"."audit_rows"
|
|
`);
|
|
expect(publicAuditRows).toEqual([{ secret_note: null }]);
|
|
|
|
const pluginAuditRows = await restoreSql.unsafe<{ secret_note: string | null }[]>(`
|
|
SELECT "secret_note"
|
|
FROM "plugin_backup_scope"."audit_rows"
|
|
`);
|
|
expect(pluginAuditRows).toEqual([{ secret_note: "plugin-secret" }]);
|
|
|
|
await expect(
|
|
restoreSql.unsafe(`
|
|
INSERT INTO "plugin_backup_scope"."plugin_rows" ("parent_id", "status", "note")
|
|
VALUES ('11111111-1111-4111-8111-111111111111', 'done', 'first')
|
|
`),
|
|
).rejects.toThrow();
|
|
} finally {
|
|
await sourceSql.end();
|
|
await restoreSql.end();
|
|
}
|
|
},
|
|
60_000,
|
|
);
|
|
|
|
it(
|
|
"preserves composite foreign key column order without duplicate referenced columns",
|
|
async () => {
|
|
const sourceConnectionString = await createTempDatabase();
|
|
const restoreConnectionString = await createSiblingDatabase(
|
|
sourceConnectionString,
|
|
"paperclip_composite_fk_restore_target",
|
|
);
|
|
const backupDir = createTempDir("paperclip-db-composite-fk-backup-");
|
|
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
|
|
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
|
|
|
|
try {
|
|
await sourceSql.unsafe(`
|
|
CREATE SCHEMA "plugin_composite_fk";
|
|
CREATE TABLE "plugin_composite_fk"."content_cases" (
|
|
"id" uuid PRIMARY KEY,
|
|
"company_id" uuid NOT NULL,
|
|
"title" text NOT NULL,
|
|
CONSTRAINT "content_cases_company_case_unique" UNIQUE ("company_id", "id")
|
|
);
|
|
CREATE TABLE "plugin_composite_fk"."content_case_signals" (
|
|
"company_id" uuid NOT NULL,
|
|
"case_id" uuid NOT NULL,
|
|
"signal" text NOT NULL,
|
|
"scopes" text[] NOT NULL,
|
|
"warnings" jsonb DEFAULT '[]'::jsonb NOT NULL,
|
|
CONSTRAINT "content_case_signals_company_case"
|
|
FOREIGN KEY ("company_id", "case_id")
|
|
REFERENCES "plugin_composite_fk"."content_cases" ("company_id", "id")
|
|
ON DELETE CASCADE
|
|
);
|
|
INSERT INTO "plugin_composite_fk"."content_cases" ("company_id", "id", "title")
|
|
VALUES (
|
|
'11111111-1111-4111-8111-111111111111',
|
|
'22222222-2222-4222-8222-222222222222',
|
|
'case'
|
|
);
|
|
INSERT INTO "plugin_composite_fk"."content_case_signals" ("company_id", "case_id", "signal", "scopes", "warnings")
|
|
VALUES (
|
|
'11111111-1111-4111-8111-111111111111',
|
|
'22222222-2222-4222-8222-222222222222',
|
|
'signal',
|
|
ARRAY['upstream_import:preview', 'scope with space', 'quoted "scope"', 'NULL', 'null'],
|
|
jsonb_build_array('json warning', jsonb_build_object('code', 'quoted "value"'))
|
|
);
|
|
`);
|
|
|
|
const result = await runDatabaseBackup({
|
|
connectionString: sourceConnectionString,
|
|
backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
|
|
filenamePrefix: "paperclip-composite-fk-test",
|
|
backupEngine: "javascript",
|
|
});
|
|
|
|
await runDatabaseRestore({
|
|
connectionString: restoreConnectionString,
|
|
backupFile: result.backupFile,
|
|
});
|
|
|
|
const rows = await restoreSql.unsafe<{
|
|
signal: string;
|
|
title: string;
|
|
scopes: string[];
|
|
warnings: Array<string | { code: string }>;
|
|
}[]>(`
|
|
SELECT s."signal", c."title", s."scopes", s."warnings"
|
|
FROM "plugin_composite_fk"."content_case_signals" s
|
|
JOIN "plugin_composite_fk"."content_cases" c
|
|
ON c."company_id" = s."company_id"
|
|
AND c."id" = s."case_id"
|
|
`);
|
|
expect(rows).toEqual([
|
|
{
|
|
signal: "signal",
|
|
title: "case",
|
|
scopes: ["upstream_import:preview", "scope with space", 'quoted "scope"', "NULL", "null"],
|
|
warnings: ["json warning", { code: 'quoted "value"' }],
|
|
},
|
|
]);
|
|
|
|
await expect(
|
|
restoreSql.unsafe(`
|
|
INSERT INTO "plugin_composite_fk"."content_case_signals" ("company_id", "case_id", "signal", "scopes")
|
|
VALUES (
|
|
'11111111-1111-4111-8111-111111111111',
|
|
'33333333-3333-4333-8333-333333333333',
|
|
'orphan',
|
|
ARRAY[]::text[]
|
|
)
|
|
`),
|
|
).rejects.toThrow();
|
|
} finally {
|
|
await sourceSql.end();
|
|
await restoreSql.end();
|
|
}
|
|
},
|
|
60_000,
|
|
);
|
|
|
|
it(
|
|
"restores fallback COPY data when child tables are dumped before parent tables",
|
|
async () => {
|
|
const sourceConnectionString = await createTempDatabase();
|
|
const restoreConnectionString = await createSiblingDatabase(
|
|
sourceConnectionString,
|
|
"paperclip_copy_fk_restore_target",
|
|
);
|
|
const backupDir = createTempDir("paperclip-db-copy-fk-backup-");
|
|
const sourceSql = postgres(sourceConnectionString, { max: 1, onnotice: () => {} });
|
|
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
|
|
const originalPgDumpPath = process.env.PAPERCLIP_PG_DUMP_PATH;
|
|
const originalPsqlPath = process.env.PAPERCLIP_PSQL_PATH;
|
|
process.env.PAPERCLIP_PG_DUMP_PATH = "/bin/false";
|
|
process.env.PAPERCLIP_PSQL_PATH = "/bin/false";
|
|
|
|
try {
|
|
await sourceSql.unsafe(`
|
|
CREATE TABLE "public"."zzz_parent_records" (
|
|
"id" uuid PRIMARY KEY,
|
|
"name" text NOT NULL
|
|
);
|
|
CREATE TABLE "public"."aaa_child_records" (
|
|
"id" uuid PRIMARY KEY,
|
|
"parent_id" uuid NOT NULL REFERENCES "public"."zzz_parent_records"("id") ON DELETE CASCADE,
|
|
"note" text NOT NULL
|
|
);
|
|
INSERT INTO "public"."zzz_parent_records" ("id", "name")
|
|
VALUES ('11111111-1111-4111-8111-111111111111', 'parent');
|
|
INSERT INTO "public"."aaa_child_records" ("id", "parent_id", "note")
|
|
VALUES (
|
|
'22222222-2222-4222-8222-222222222222',
|
|
'11111111-1111-4111-8111-111111111111',
|
|
'child emitted before parent'
|
|
);
|
|
`);
|
|
|
|
const result = await runDatabaseBackup({
|
|
connectionString: sourceConnectionString,
|
|
backupDir,
|
|
retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 },
|
|
filenamePrefix: "paperclip-copy-fk-test",
|
|
backupEngine: "auto",
|
|
});
|
|
|
|
const backupSql = gunzipSync(await fs.promises.readFile(result.backupFile)).toString("utf8");
|
|
expect(backupSql.indexOf("-- Data for: public.aaa_child_records")).toBeGreaterThan(-1);
|
|
expect(backupSql.indexOf("-- Data for: public.aaa_child_records")).toBeLessThan(
|
|
backupSql.indexOf("-- Data for: public.zzz_parent_records"),
|
|
);
|
|
expect(backupSql).not.toContain(" FROM stdin;");
|
|
|
|
await runDatabaseRestore({
|
|
connectionString: restoreConnectionString,
|
|
backupFile: result.backupFile,
|
|
});
|
|
|
|
const rows = await restoreSql.unsafe<{ note: string; name: string }[]>(`
|
|
SELECT child."note", parent."name"
|
|
FROM "public"."aaa_child_records" child
|
|
JOIN "public"."zzz_parent_records" parent ON parent."id" = child."parent_id"
|
|
`);
|
|
expect(rows).toEqual([{ note: "child emitted before parent", name: "parent" }]);
|
|
} finally {
|
|
if (originalPgDumpPath === undefined) {
|
|
delete process.env.PAPERCLIP_PG_DUMP_PATH;
|
|
} else {
|
|
process.env.PAPERCLIP_PG_DUMP_PATH = originalPgDumpPath;
|
|
}
|
|
if (originalPsqlPath === undefined) {
|
|
delete process.env.PAPERCLIP_PSQL_PATH;
|
|
} else {
|
|
process.env.PAPERCLIP_PSQL_PATH = originalPsqlPath;
|
|
}
|
|
await sourceSql.end();
|
|
await restoreSql.end();
|
|
}
|
|
},
|
|
60_000,
|
|
);
|
|
|
|
it(
|
|
"restores legacy public-only backups without migration history",
|
|
async () => {
|
|
const restoreConnectionString = await createTempDatabase();
|
|
const restoreSql = postgres(restoreConnectionString, { max: 1, onnotice: () => {} });
|
|
const backupDir = createTempDir("paperclip-db-restore-manual-");
|
|
const backupFile = path.join(backupDir, "manual.sql");
|
|
|
|
try {
|
|
await fs.promises.writeFile(
|
|
backupFile,
|
|
[
|
|
"-- Paperclip database backup",
|
|
"-- Created: 2026-04-06T00:00:00.000Z",
|
|
"",
|
|
"BEGIN;",
|
|
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
|
|
"CREATE TABLE public.restore_stream_test (id integer primary key, payload text not null);",
|
|
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
|
|
"INSERT INTO public.restore_stream_test (id, payload)",
|
|
"VALUES (1, 'hello');",
|
|
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
|
|
"COMMIT;",
|
|
"-- paperclip statement breakpoint 69f6f3f1-42fd-46a6-bf17-d1d85f8f3900",
|
|
].join("\n"),
|
|
"utf8",
|
|
);
|
|
|
|
await runDatabaseRestore({
|
|
connectionString: restoreConnectionString,
|
|
backupFile,
|
|
});
|
|
|
|
const rows = await restoreSql.unsafe<{ payload: string }[]>(`
|
|
SELECT payload
|
|
FROM public.restore_stream_test
|
|
`);
|
|
expect(rows).toEqual([{ payload: "hello" }]);
|
|
} finally {
|
|
await restoreSql.end();
|
|
}
|
|
},
|
|
20_000,
|
|
);
|
|
});
|