Files
PaperClipAI/packages/db/src/test-embedded-postgres.test.ts
T
DottaandPaperclip 33fc7fcddc fix(db): clean up owned PostgreSQL test servers (#18627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Database, server, and CLI tests use real PostgreSQL fixtures.
> - Each fixture must own its server until PostgreSQL has stopped.
> - The old teardown returned after five seconds even when its server
remained alive.
> - A startup failure could then mark the host unsupported and skip
database coverage.
> - This pull request waits for owned shutdown and makes supported-host
startup failures fail collection.
> - A shared-server command reduces process creation while keeping each
fixture's database isolated.

## Linked Issues or Issue Description

Refs: #7756 for related development-process ownership. That
development-server issue remains outside this test-fixture fix.
Related PR: #16799 collects macOS startup diagnostics. This PR changes
test shutdown and failure behavior.

**What happened?**

Repeated test runs can exhaust macOS PostgreSQL semaphore resources. The
fixture abandoned slow shutdown and swallowed stop errors. The support
probe converted ordinary startup failures into skipped integration
coverage.

**Expected behavior**

Tests must stop their PostgreSQL servers, retain files if stop fails,
and report resource exhaustion as a failure. Fixtures on a shared server
must not close another database's clients.

**Steps to reproduce**

Start a fixture whose PostgreSQL stop exceeds five seconds, or make a
supported-host startup fail with semaphore exhaustion. The former helper
returned from teardown or reported unsupported coverage. The new
regression tests exercise both paths without exhausting host resources.

**Paperclip version or commit**

Source checkout based on current master.

**Deployment mode**

Local source development and CI tests.

## What Changed

- Add explicit ownership for test PostgreSQL children. Wait for fast
shutdown, then request immediate PostgreSQL shutdown if needed. Keep
cleanup idempotent and fail when exit cannot be confirmed.
- Publish reusable templates only after clean shutdown. Do not publish a
crashed or immediately stopped cluster.
- Stop owned children on POSIX worker signals and process exit. Preserve
test directories when shutdown fails. Remove conflicting dependency exit
hooks.
- Retry port collisions only. Fail collection for resource exhaustion,
unavailable configured servers, and other startup failures on supported
platforms. Preserve skips for explicitly unsupported platforms.
- Move administrative-server support into the common fixture used by
database, server, and CLI tests. Close clients at database scope and
migrate cloned templates before use.
- Add a directly invoked shared-server runner to own one test server and
run suites against separate disposable databases. Preserve child exit
status and support an existing disposable server.
- Unify cancellation: stop and wait for the test process group before
PostgreSQL teardown. Add nested-process coverage, real worker-exit
coverage, shutdown regressions, shared-database isolation checks, and
setup documentation. Replace ignored heartbeat completion polls with the
existing execution drain before asserting persisted outcomes.
- Finish owned cluster-directory cleanup after signal-driven
cancellation. Preserve prepared templates across shared-fixture tests.
Remove stopped directories asynchronously, using native removal on POSIX
with a Node fallback.
- Align database and CLI fixture hook budgets with the existing
30-second server budget. This lets the ten-second shutdown protocol
finish and report errors. The idle-sleep suite uses the existing
90-second fixture budget for its unusually large final directory
cleanup.

## Verification

- The semaphore-exhaustion regression failed against the old helper
because it resolved to `supported: false` instead of rejecting.
- All 37 focused tests pass on the final runtime code (`e4de00ab40`).
They include real worker exit, SIGTERM, nested-process SIGINT/SIGTERM
cancellation with PostgreSQL available during child cleanup, failed
template shutdown, unsupported platforms, stale SQL templates, and
directory removal. A child exiting with status 7 makes the shared-server
runner return status 7 after cleanup.
- Full local `pnpm -r typecheck` and `pnpm build` pass on the final
runtime code (`e4de00ab40`). Subsequent changes only adjust server-test
lifecycle waits, the idle-sleep cleanup budget, and documentation; the
server typecheck also passes with those final tests.
- A fresh shared-server run on `90b65fd8ad` passed all 68 database files
and 297 tests. Earlier preparation runs had two ten-second teardown
timeouts and a stale-template failure; the final helper migrates cloned
templates and permits the full shutdown budget. These earlier failures
remain accounted for here.
- The local full repository attempt stopped on a Discord receiver-lease
test after 338 passing chat tests. That exact case passed in isolation.
This full-suite attempt is not a pass.
- On `90b65fd8ad`, [server shard
6](https://github.com/paperclipai/paperclip/actions/runs/41496495253/job/125452949369)
and its one unchanged retry reported two 15-second cached-input
heartbeat timeouts and a 30-second idle-sleep teardown timeout. All ten
cached-input cases then passed locally with their original deadlines. A
local parallel run of the full files reported eight behavioral/hook
failures and two suite teardown timeouts (281 tests passed).
- Serial idle-sleep runs passed all 225 test assertions but overran the
30-second final hook. Tracing measured client closure in 4 ms,
PostgreSQL exit after 5 seconds, and synchronous directory removal
taking another 35 seconds. Asynchronous Node removal and native removal
also exceeded that hook under the full workload. Cleanup now remains
asynchronous and awaited, and only this heavy suite's final hook uses
the existing 90-second fixture budget. No PostgreSQL shutdown deadline
or test assertion was weakened. The serial run then passed the complete
idle-sleep file, including all 225 tests and teardown. Its heartbeat
file passed 63/64 cases, exposing an ignored three-second completion
poll. After replacing that poll with an execution drain, another timer
case exposed the same pattern (63/64 passed). All remaining ignored
completion polls now await execution drain without changing test
deadlines. The complete heartbeat file then passed all 64 tests. The
intentional asserted lock-contention poll is unchanged. The final server
typecheck passes, and a process audit found no PostgreSQL server left by
the completed local runs.
- On final head `2cf316d3f571e4516890cba743ecffeb9703cce3`, the
previously failing server shard 6, database lane, full typecheck, and
build passed. The [first CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/41506358136)
then lost self-hosted runners in the canary and serialized-server lanes,
with additional jobs cancelled. The server-2 lane had one 30-second
timeout in the unchanged workspace-recovery HTTP Retry test (2,282 other
tests passed). That exact test passed locally with its original deadline
(28.78 seconds including imports). One unchanged retry of only failed
and cancelled jobs completed successfully on the exact final commit
(workflow attempt 2). The workspace-recovery timeout case and every
interrupted lane passed; successful first-attempt jobs were retained.
The additional Paperclip Review check remains queued, waiting for its
assigned agent; it has no unresolved GitHub review threads. Greptile
completed review on this exact final head at 5/5, with zero unresolved
review threads. All four original review findings and their threads are
resolved; additional Windows shutdown, cancellation-directory, and
template-reuse findings are fixed.
- `pnpm check:module-boundaries` and `git diff --check` passed. Post-run
process checks found no PostgreSQL servers left by the completed
verification runs.

## Risks

- Immediate PostgreSQL shutdown can require crash recovery. This
fallback applies to disposable test clusters and runs only after fast
shutdown times out.
- No process can handle `SIGKILL` or machine power loss. Host resource
cleanup remains a separate backstop.
- Directory cleanup on cancellation applies to the shared-server
runner's own cluster. Directly terminated test workers stop their
PostgreSQL process but retain disposable files for inspection. The
process-exit guard cannot await file removal.
- The two-second `pg_ctl -W` deadline bounds signal dispatch. It is
separate from the five-second and thirty-second server-exit waits. A
control-command failure fails cleanup and retains its directory; it does
not report successful cleanup or force-delete live files.
- Shared fixtures require a disposable PostgreSQL server and a role that
can create databases. The directly invoked shared command is opt-in; the
existing `pnpm test` command remains available.
- These changes do not stop unrelated development servers or delete
host-wide IPC resources. Clean shutdown uses the owned `pg_ctl` binary
on every supported platform. Its real control path was tested on macOS.
A Windows host was not available for live verification; POSIX signal
tests skip on Windows.

## Model Used

- OpenAI Codex, based on GPT-6. The runtime does not expose the exact
serving variant or context-window size. The agent used reasoning, shell
execution, code editing, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 14:24:48 -06:00

562 lines
24 KiB
TypeScript

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import postgres from "postgres";
import { afterAll, afterEach, describe, expect, it } from "vitest";
import { inspectMigrations } from "./client.js";
import { drainTestPostgresOwners, TestPostgresShutdownError } from "./test-postgres-owner.js";
import {
__embeddedPostgresStartMaxAttemptsForTests as MAX_ATTEMPTS,
__embeddedPostgresTemplateRootDirForTests as templateRootDir,
__isUsableEmbeddedPostgresTemplateForTests as isUsableTemplate,
__lastEmbeddedPostgresStartModeForTests as lastStartMode,
__resetEmbeddedPostgresTemplateStateForTests as resetTemplateState,
__setEmbeddedPostgresCtorProviderForTests,
__startEmbeddedPostgresWithRetryForTests as startWithRetry,
computeEmbeddedPostgresTemplateKey,
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const originalAdminUrl = process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL;
delete process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL;
afterAll(() => {
if (originalAdminUrl !== undefined) process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL = originalAdminUrl;
});
// A fake embedded-postgres constructor. It records every constructed instance so
// the test can assert the retry uses a fresh port and a fresh data directory each
// attempt. `start()` emits the same output the real cluster writes for a port
// conflict, then rejects with an empty message (the real rejection shape). The
// option type matches the real constructor so no type cast is needed.
type FakeOptions = {
databaseDir: string;
user: string;
password: string;
port: number;
persistent: boolean;
initdbFlags?: string[];
onLog?: (message: unknown) => void;
onError?: (message: unknown) => void;
};
const BIND_CONFLICT_LOG = 'could not bind IPv4 address "127.0.0.1": Address already in use';
function makeFakeCtor(failFirst: number) {
const constructed: FakeOptions[] = [];
let started = 0;
class FakeEmbeddedPostgres {
private readonly options: FakeOptions;
constructor(options: FakeOptions) {
this.options = options;
constructed.push(options);
}
async initialise(): Promise<void> {}
async start(): Promise<void> {
started += 1;
if (started <= failFirst) {
// Mirror the real failure: Postgres logs the reason, then `start()`
// rejects with an Error whose message is empty.
this.options.onLog?.(BIND_CONFLICT_LOG);
throw new Error();
}
}
async stop(): Promise<void> {}
}
return { ctor: FakeEmbeddedPostgres, constructed };
}
describe("startEmbeddedPostgresWithRetry", () => {
afterEach(() => {
__setEmbeddedPostgresCtorProviderForTests(null);
});
it("recovers from a transient port conflict and returns on a later attempt", async () => {
const { ctor, constructed } = makeFakeCtor(2);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
const started = await startWithRetry("paperclip-retry-recover-");
// The first two attempts fail, the third succeeds.
expect(constructed).toHaveLength(3);
// Each attempt uses a fresh data directory. The two failed directories are
// removed. The returned directory still exists.
const dataDirs = constructed.map((options) => options.databaseDir);
expect(new Set(dataDirs).size).toBe(3);
expect(fs.existsSync(dataDirs[0])).toBe(false);
expect(fs.existsSync(dataDirs[1])).toBe(false);
expect(started.dataDir).toBe(dataDirs[2]);
expect(fs.existsSync(started.dataDir)).toBe(true);
// Each attempt allocates a port.
for (const options of constructed) {
expect(Number.isInteger(options.port)).toBe(true);
expect(options.port).toBeGreaterThan(0);
}
// Clean up the returned attempt.
await started.instance.stop();
fs.rmSync(started.dataDir, { recursive: true, force: true });
});
it("throws with the real Postgres output after the attempt bound", async () => {
const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
await expect(startWithRetry("paperclip-retry-fail-")).rejects.toThrow(/after \d+ attempts/);
// The retry stops at the bound and does not loop forever.
expect(constructed).toHaveLength(MAX_ATTEMPTS);
// Every failed attempt removes its data directory.
for (const options of constructed) {
expect(fs.existsSync(options.databaseDir)).toBe(false);
}
});
it("keeps the real failure reason instead of a generic fallback", async () => {
const { ctor } = makeFakeCtor(Number.POSITIVE_INFINITY);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
const error = await startWithRetry("paperclip-retry-reason-").catch((caught: unknown) => caught);
expect(error).toBeInstanceOf(Error);
// The thrown message carries the captured Postgres output, not only the
// generic "embedded Postgres startup failed" text.
expect((error as Error).message).toContain("Address already in use");
});
});
describe("computeEmbeddedPostgresTemplateKey", () => {
function writeMigrations(dir: string, files: Record<string, string>) {
fs.mkdirSync(path.join(dir, "meta"), { recursive: true });
fs.writeFileSync(path.join(dir, "meta", "_journal.json"), JSON.stringify({ entries: Object.keys(files) }));
for (const [name, content] of Object.entries(files)) {
fs.writeFileSync(path.join(dir, name), content);
}
}
it("is stable for identical inputs and changes with any migration, flag, or version", () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-key-"));
try {
const a = path.join(root, "a");
const b = path.join(root, "b");
writeMigrations(a, { "0000_one.sql": "create table one();", "0001_two.sql": "create table two();" });
writeMigrations(b, { "0000_one.sql": "create table one();", "0001_two.sql": "create table two();" });
const fixed = { packageVersion: "18.1.0", initdbFlags: ["--locale=C"], user: "paperclip" };
const keyA = computeEmbeddedPostgresTemplateKey(a, fixed);
expect(keyA).toMatch(/^[0-9a-f]{24}$/);
expect(computeEmbeddedPostgresTemplateKey(a, fixed)).toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(b, fixed)).toBe(keyA);
// Content change, new file, flags, version, and user each change the key.
fs.writeFileSync(path.join(b, "0001_two.sql"), "create table two(id int);");
const contentChanged = computeEmbeddedPostgresTemplateKey(b, fixed);
expect(contentChanged).not.toBe(keyA);
fs.writeFileSync(path.join(b, "0002_three.sql"), "create table three();");
expect(computeEmbeddedPostgresTemplateKey(b, fixed)).not.toBe(contentChanged);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, initdbFlags: ["--locale=C", "--encoding=UTF8"] })).not.toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, packageVersion: "18.2.0" })).not.toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, user: "other" })).not.toBe(keyA);
} finally {
fs.rmSync(root, { recursive: true, force: true });
}
});
it("hashes the real migrations directory of this package", () => {
const key = computeEmbeddedPostgresTemplateKey();
expect(key).toMatch(/^[0-9a-f]{24}$/);
expect(computeEmbeddedPostgresTemplateKey()).toBe(key);
});
});
const describeOnPosix = process.platform === "win32" ? describe.skip : describe;
describe("isUsableTemplate", () => {
let root: string;
function writeCluster(dir: string): void {
fs.mkdirSync(dir, { recursive: true, mode: 0o700 });
fs.writeFileSync(path.join(dir, "PG_VERSION"), "18\n");
}
afterEach(() => {
fs.rmSync(root, { recursive: true, force: true });
});
it("accepts a private directory that holds a cleanly stopped cluster", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
expect(isUsableTemplate(template)).toBe(true);
// A running or crashed cluster leaves `postmaster.pid` behind.
fs.writeFileSync(path.join(template, "postmaster.pid"), "1\n");
expect(isUsableTemplate(template)).toBe(false);
fs.rmSync(path.join(template, "postmaster.pid"));
expect(isUsableTemplate(template)).toBe(true);
// A directory without a cluster, or no directory at all.
fs.rmSync(path.join(template, "PG_VERSION"));
expect(isUsableTemplate(template)).toBe(false);
expect(isUsableTemplate(path.join(root, "missing"))).toBe(false);
});
it("rejects a symbolic link even when it points at a usable cluster", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
const link = path.join(root, "link");
fs.symlinkSync(template, link, "dir");
expect(isUsableTemplate(template)).toBe(true);
expect(isUsableTemplate(link)).toBe(false);
});
describeOnPosix("on POSIX", () => {
it("rejects a cluster directory that group or others can write", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
for (const mode of [0o720, 0o702, 0o777]) {
fs.chmodSync(template, mode);
expect(isUsableTemplate(template), `mode ${mode.toString(8)}`).toBe(false);
}
// Read bits for others are not a write path; Postgres narrows the copy
// to 0700 before it starts anyway.
fs.chmodSync(template, 0o755);
expect(isUsableTemplate(template)).toBe(true);
fs.chmodSync(template, 0o700);
expect(isUsableTemplate(template)).toBe(true);
});
});
});
// Real-cluster coverage of the template path. Skipped on hosts that cannot run
// embedded Postgres, like every other suite that uses the fixture.
const support = await getEmbeddedPostgresTestSupport();
const describeWithPostgres = support.supported ? describe : describe.skip;
// Points `os.tmpdir()` at a private scratch directory so a test can drive the
// cache root from an empty state. `os.tmpdir()` reads the environment on every
// call: TMPDIR on POSIX, TEMP and then TMP on Windows, so all three are set.
// Returns the scratch directory; `restore` puts the environment and the
// fixture's cached state back.
const TMPDIR_VARIABLES = ["TMPDIR", "TEMP", "TMP"] as const;
function useScratchTmpdir(prefix: string): { scratch: string; restore(): void } {
const scratch = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
const previous = new Map(TMPDIR_VARIABLES.map((name) => [name, process.env[name]] as const));
for (const name of TMPDIR_VARIABLES) process.env[name] = scratch;
return {
scratch,
restore() {
for (const [name, value] of previous) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
resetTemplateState();
fs.rmSync(scratch, { recursive: true, force: true });
},
};
}
describeWithPostgres("startEmbeddedPostgresTestDatabase template path", () => {
const originalTemplateSetting = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
afterEach(() => {
if (originalTemplateSetting === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = originalTemplateSetting;
});
it("does not start a replacement when a copied cluster cannot stop", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const ready = await startEmbeddedPostgresTestDatabase("paperclip-template-stop-ready-");
await ready.cleanup();
const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY);
let stops = 0;
class FailedStopPostgres extends ctor {
async stop(): Promise<void> {
if (++stops === 1) throw new Error("owned stop failed");
}
}
__setEmbeddedPostgresCtorProviderForTests(async () => FailedStopPostgres);
try {
await expect(startEmbeddedPostgresTestDatabase("paperclip-template-stop-fail-")).rejects.toBeInstanceOf(TestPostgresShutdownError);
expect(constructed).toHaveLength(1);
expect(fs.existsSync(constructed[0].databaseDir)).toBe(true);
} finally {
await drainTestPostgresOwners();
__setEmbeddedPostgresCtorProviderForTests(null);
for (const options of constructed) fs.rmSync(path.dirname(options.databaseDir), { recursive: true, force: true });
}
});
it("publishes a complete template and starts suites from a copy of it", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
// This test covers the template path, so it enables the cache even when
// the suite runs under PAPERCLIP_TEST_POSTGRES_TEMPLATE=0. The first
// start then builds the template if the support probe did not.
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-copy-");
try {
expect(lastStartMode()).toBe("template");
// The copy is fully migrated and independent of the template.
const state = await inspectMigrations(database.connectionString);
expect(state.status).toBe("upToDate");
const sql = postgres(database.connectionString, { max: 1 });
try {
await sql`create table paperclip_template_copy_probe (id int)`;
} finally {
await sql.end();
}
} finally {
await database.cleanup();
}
const rootDir = templateRootDir();
const templateDir = path.join(rootDir, computeEmbeddedPostgresTemplateKey());
expect(fs.existsSync(path.join(templateDir, "PG_VERSION"))).toBe(true);
expect(fs.existsSync(path.join(templateDir, "postmaster.pid"))).toBe(false);
expect(isUsableTemplate(templateDir)).toBe(true);
if (process.platform !== "win32") {
// The root is private to this user, and the template is a data
// directory Postgres accepts: both are 0700.
expect(path.basename(rootDir)).toBe(`paperclip-embedded-postgres-templates-${process.getuid?.()}`);
expect(fs.statSync(rootDir).mode & 0o777).toBe(0o700);
expect(fs.statSync(templateDir).mode & 0o777).toBe(0o700);
}
// Writing to the copy never touches the template, so a second copy does
// not see the table.
const second = await startEmbeddedPostgresTestDatabase("paperclip-template-copy-");
try {
expect(lastStartMode()).toBe("template");
const sql = postgres(second.connectionString, { max: 1 });
try {
const rows = await sql`select to_regclass('paperclip_template_copy_probe') as name`;
expect(rows[0]?.name).toBeNull();
} finally {
await sql.end();
}
} finally {
await second.cleanup();
}
});
it("refreshes the template's timestamp on every reuse so idle time drives pruning", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
// Make sure a template exists before ageing it: the support probe does
// not publish one under PAPERCLIP_TEST_POSTGRES_TEMPLATE=0, and a name
// filter can run this test on its own.
resetTemplateState();
const first = await startEmbeddedPostgresTestDatabase("paperclip-template-touch-");
await first.cleanup();
expect(lastStartMode()).toBe("template");
const templateDir = path.join(templateRootDir(), computeEmbeddedPostgresTemplateKey());
expect(isUsableTemplate(templateDir)).toBe(true);
// Age the template as if it had not been used for ten days.
const old = new Date(Date.now() - 10 * 24 * 60 * 60 * 1000);
fs.utimesSync(templateDir, old, old);
expect(fs.statSync(templateDir).mtimeMs).toBeLessThan(Date.now() - 9 * 24 * 60 * 60 * 1000);
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-touch-");
try {
expect(lastStartMode()).toBe("template");
} finally {
await database.cleanup();
}
expect(fs.statSync(templateDir).mtimeMs).toBeGreaterThan(Date.now() - 60 * 1000);
});
it("rebuilds the template when it was pruned after this process resolved it", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-pruned-");
try {
resetTemplateState();
const first = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await first.cleanup();
expect(lastStartMode()).toBe("template");
const templateDir = path.join(templateRootDir(), computeEmbeddedPostgresTemplateKey());
expect(isUsableTemplate(templateDir)).toBe(true);
// Another process prunes the template while this one still holds its path.
fs.rmSync(templateDir, { recursive: true, force: true });
const second = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await second.cleanup();
// This start fell back without retrying the missing path; the fallback
// forgot the stale path, so the next start rebuilt the template.
expect(lastStartMode()).toBe("fresh");
const third = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await third.cleanup();
expect(lastStartMode()).toBe("template");
expect(isUsableTemplate(templateDir)).toBe(true);
} finally {
scratch.restore();
}
});
it("runs the fresh initdb path when PAPERCLIP_TEST_POSTGRES_TEMPLATE=0", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = "0";
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-bypass-");
try {
expect(lastStartMode()).toBe("fresh");
const state = await inspectMigrations(database.connectionString);
expect(state.status).toBe("upToDate");
} finally {
await database.cleanup();
}
});
describeOnPosix("cache root safety", () => {
it("uses fresh clusters when the cache root is not a private directory of this user", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-root-");
const warnings: string[] = [];
const warn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(" "));
};
try {
// Someone pre-created the predictable root path, world-writable, the
// way a foreign user on a shared host would.
const rootDir = templateRootDir();
expect(rootDir.startsWith(scratch.scratch)).toBe(true);
fs.mkdirSync(rootDir);
fs.chmodSync(rootDir, 0o777);
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-root-copy-");
try {
expect(lastStartMode()).toBe("fresh");
} finally {
await database.cleanup();
}
// Nothing was published into the rejected root, and the reason was logged once.
expect(fs.readdirSync(rootDir)).toEqual([]);
expect(warnings.filter((line) => line.includes("template cache root"))).toHaveLength(1);
expect(warnings[0]).toContain("writable by group or others");
// The probe still reports support, from the throwaway cluster.
resetTemplateState();
expect(await getEmbeddedPostgresTestSupport()).toEqual({ supported: true });
} finally {
console.warn = warn;
scratch.restore();
}
});
it("uses fresh clusters when the cache root path is a symbolic link", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-root-");
const warn = console.warn;
console.warn = () => {};
try {
const target = path.join(scratch.scratch, "elsewhere");
fs.mkdirSync(target, { mode: 0o700 });
fs.symlinkSync(target, templateRootDir(), "dir");
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-root-copy-");
try {
expect(lastStartMode()).toBe("fresh");
} finally {
await database.cleanup();
}
expect(fs.readdirSync(target)).toEqual([]);
} finally {
console.warn = warn;
scratch.restore();
}
});
});
});
describe("support probe when the template cannot be built", () => {
afterEach(() => {
__setEmbeddedPostgresCtorProviderForTests(null);
resetTemplateState();
});
it("fails the suite when PostgreSQL cannot allocate semaphores", async () => {
const original = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = "0";
const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY);
class ExhaustedPostgres extends ctor {
async start(): Promise<void> {
throw new Error("could not create semaphores: No space left on device (semget)");
}
}
__setEmbeddedPostgresCtorProviderForTests(async () => ExhaustedPostgres);
resetTemplateState();
try {
await expect(getEmbeddedPostgresTestSupport()).rejects.toThrow("could not create semaphores");
expect(constructed).toHaveLength(1);
} finally {
if (original === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = original;
}
});
it("still reports an explicitly unsupported platform as unsupported", async () => {
const original = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = "0";
__setEmbeddedPostgresCtorProviderForTests(async () => {
throw new Error('Unsupported arch "test-cpu" for platform "test-os"');
});
resetTemplateState();
try {
expect(await getEmbeddedPostgresTestSupport()).toMatchObject({
supported: false,
reason: expect.stringContaining('Unsupported arch "test-cpu"'),
});
} finally {
if (original === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = original;
}
});
it("falls through to the fresh probe instead of reporting the host as unsupported", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
const originalTemplateSetting = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-build-fail-");
const warnings: string[] = [];
const warn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(" "));
};
try {
// This test runs on every platform, so check the redirect took effect
// before asserting on paths under it.
expect(os.tmpdir()).toBe(scratch.scratch);
// The fake cluster "starts" without listening, so the template build
// fails after the start, at the migration step, the way a broken
// migration would. A real broken migration then fails the suite on
// the fresh path instead of being hidden by `describe.skip`.
const { ctor, constructed } = makeFakeCtor(0);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
resetTemplateState();
expect(await getEmbeddedPostgresTestSupport()).toEqual({ supported: true });
// One build attempt, then the throwaway probe, both under the scratch root.
expect(constructed).toHaveLength(2);
for (const options of constructed) expect(options.databaseDir.startsWith(scratch.scratch)).toBe(true);
expect(constructed[0]?.databaseDir).toContain("paperclip-embedded-postgres-template-build-");
expect(constructed[1]?.databaseDir).toContain("paperclip-embedded-postgres-probe-");
expect(warnings.filter((line) => line.includes("template build failed"))).toHaveLength(1);
// Nothing was published.
const rootDir = templateRootDir();
expect(fs.existsSync(rootDir) ? fs.readdirSync(rootDir) : []).toEqual([]);
} finally {
console.warn = warn;
scratch.restore();
if (originalTemplateSetting === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = originalTemplateSetting;
}
});
});