mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-05 21:02:34 +01:00
b5b0dbb03fc6272e7f761d63d794f424a8920e26
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
33fc7fcddc |
fix(db): clean up owned PostgreSQL test servers (#18627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Database, server, and CLI tests use real PostgreSQL fixtures. > - Each fixture must own its server until PostgreSQL has stopped. > - The old teardown returned after five seconds even when its server remained alive. > - A startup failure could then mark the host unsupported and skip database coverage. > - This pull request waits for owned shutdown and makes supported-host startup failures fail collection. > - A shared-server command reduces process creation while keeping each fixture's database isolated. ## Linked Issues or Issue Description Refs: #7756 for related development-process ownership. That development-server issue remains outside this test-fixture fix. Related PR: #16799 collects macOS startup diagnostics. This PR changes test shutdown and failure behavior. **What happened?** Repeated test runs can exhaust macOS PostgreSQL semaphore resources. The fixture abandoned slow shutdown and swallowed stop errors. The support probe converted ordinary startup failures into skipped integration coverage. **Expected behavior** Tests must stop their PostgreSQL servers, retain files if stop fails, and report resource exhaustion as a failure. Fixtures on a shared server must not close another database's clients. **Steps to reproduce** Start a fixture whose PostgreSQL stop exceeds five seconds, or make a supported-host startup fail with semaphore exhaustion. The former helper returned from teardown or reported unsupported coverage. The new regression tests exercise both paths without exhausting host resources. **Paperclip version or commit** Source checkout based on current master. **Deployment mode** Local source development and CI tests. ## What Changed - Add explicit ownership for test PostgreSQL children. Wait for fast shutdown, then request immediate PostgreSQL shutdown if needed. Keep cleanup idempotent and fail when exit cannot be confirmed. - Publish reusable templates only after clean shutdown. Do not publish a crashed or immediately stopped cluster. - Stop owned children on POSIX worker signals and process exit. Preserve test directories when shutdown fails. Remove conflicting dependency exit hooks. - Retry port collisions only. Fail collection for resource exhaustion, unavailable configured servers, and other startup failures on supported platforms. Preserve skips for explicitly unsupported platforms. - Move administrative-server support into the common fixture used by database, server, and CLI tests. Close clients at database scope and migrate cloned templates before use. - Add a directly invoked shared-server runner to own one test server and run suites against separate disposable databases. Preserve child exit status and support an existing disposable server. - Unify cancellation: stop and wait for the test process group before PostgreSQL teardown. Add nested-process coverage, real worker-exit coverage, shutdown regressions, shared-database isolation checks, and setup documentation. Replace ignored heartbeat completion polls with the existing execution drain before asserting persisted outcomes. - Finish owned cluster-directory cleanup after signal-driven cancellation. Preserve prepared templates across shared-fixture tests. Remove stopped directories asynchronously, using native removal on POSIX with a Node fallback. - Align database and CLI fixture hook budgets with the existing 30-second server budget. This lets the ten-second shutdown protocol finish and report errors. The idle-sleep suite uses the existing 90-second fixture budget for its unusually large final directory cleanup. ## Verification - The semaphore-exhaustion regression failed against the old helper because it resolved to `supported: false` instead of rejecting. - All 37 focused tests pass on the final runtime code (`e4de00ab40`). They include real worker exit, SIGTERM, nested-process SIGINT/SIGTERM cancellation with PostgreSQL available during child cleanup, failed template shutdown, unsupported platforms, stale SQL templates, and directory removal. A child exiting with status 7 makes the shared-server runner return status 7 after cleanup. - Full local `pnpm -r typecheck` and `pnpm build` pass on the final runtime code (`e4de00ab40`). Subsequent changes only adjust server-test lifecycle waits, the idle-sleep cleanup budget, and documentation; the server typecheck also passes with those final tests. - A fresh shared-server run on `90b65fd8ad` passed all 68 database files and 297 tests. Earlier preparation runs had two ten-second teardown timeouts and a stale-template failure; the final helper migrates cloned templates and permits the full shutdown budget. These earlier failures remain accounted for here. - The local full repository attempt stopped on a Discord receiver-lease test after 338 passing chat tests. That exact case passed in isolation. This full-suite attempt is not a pass. - On `90b65fd8ad`, [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/41496495253/job/125452949369) and its one unchanged retry reported two 15-second cached-input heartbeat timeouts and a 30-second idle-sleep teardown timeout. All ten cached-input cases then passed locally with their original deadlines. A local parallel run of the full files reported eight behavioral/hook failures and two suite teardown timeouts (281 tests passed). - Serial idle-sleep runs passed all 225 test assertions but overran the 30-second final hook. Tracing measured client closure in 4 ms, PostgreSQL exit after 5 seconds, and synchronous directory removal taking another 35 seconds. Asynchronous Node removal and native removal also exceeded that hook under the full workload. Cleanup now remains asynchronous and awaited, and only this heavy suite's final hook uses the existing 90-second fixture budget. No PostgreSQL shutdown deadline or test assertion was weakened. The serial run then passed the complete idle-sleep file, including all 225 tests and teardown. Its heartbeat file passed 63/64 cases, exposing an ignored three-second completion poll. After replacing that poll with an execution drain, another timer case exposed the same pattern (63/64 passed). All remaining ignored completion polls now await execution drain without changing test deadlines. The complete heartbeat file then passed all 64 tests. The intentional asserted lock-contention poll is unchanged. The final server typecheck passes, and a process audit found no PostgreSQL server left by the completed local runs. - On final head `2cf316d3f571e4516890cba743ecffeb9703cce3`, the previously failing server shard 6, database lane, full typecheck, and build passed. The [first CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41506358136) then lost self-hosted runners in the canary and serialized-server lanes, with additional jobs cancelled. The server-2 lane had one 30-second timeout in the unchanged workspace-recovery HTTP Retry test (2,282 other tests passed). That exact test passed locally with its original deadline (28.78 seconds including imports). One unchanged retry of only failed and cancelled jobs completed successfully on the exact final commit (workflow attempt 2). The workspace-recovery timeout case and every interrupted lane passed; successful first-attempt jobs were retained. The additional Paperclip Review check remains queued, waiting for its assigned agent; it has no unresolved GitHub review threads. Greptile completed review on this exact final head at 5/5, with zero unresolved review threads. All four original review findings and their threads are resolved; additional Windows shutdown, cancellation-directory, and template-reuse findings are fixed. - `pnpm check:module-boundaries` and `git diff --check` passed. Post-run process checks found no PostgreSQL servers left by the completed verification runs. ## Risks - Immediate PostgreSQL shutdown can require crash recovery. This fallback applies to disposable test clusters and runs only after fast shutdown times out. - No process can handle `SIGKILL` or machine power loss. Host resource cleanup remains a separate backstop. - Directory cleanup on cancellation applies to the shared-server runner's own cluster. Directly terminated test workers stop their PostgreSQL process but retain disposable files for inspection. The process-exit guard cannot await file removal. - The two-second `pg_ctl -W` deadline bounds signal dispatch. It is separate from the five-second and thirty-second server-exit waits. A control-command failure fails cleanup and retains its directory; it does not report successful cleanup or force-delete live files. - Shared fixtures require a disposable PostgreSQL server and a role that can create databases. The directly invoked shared command is opt-in; the existing `pnpm test` command remains available. - These changes do not stop unrelated development servers or delete host-wide IPC resources. Clean shutdown uses the owned `pg_ctl` binary on every supported platform. Its real control path was tested on macOS. A Windows host was not available for live verification; POSIX signal tests skip on Windows. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving variant or context-window size. The agent used reasoning, shell execution, code editing, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
85f4205de3 |
perf(test): reuse a migrated embedded Postgres template across suites (#15861)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Pull requests to this repository run a large CI matrix. The slowest
check sets the time a contributor waits for a green run.
> - After #15853, the twelve `server (n/12)` shards are the slowest
check in every fully green PR run created since its merge (518s to 548s;
`Build` is 482s to 504s). Each shard is about 60s of setup and 454s to
490s of Vitest, with one worker, so its wall time is the sum of its
suites.
> - From two shard logs, 22 to 24 of the 71 to 74 suites in a shard
start an embedded Postgres cluster. Each one runs `initdb`, starts the
cluster, and applies all 331 migrations before its first test. The
smallest such suite takes 6s with one or two trivial tests. On top of
that, the `getEmbeddedPostgresTestSupport` probe starts and stops a
second throwaway cluster at import time in 19 of those suites, which is
why their per-file import overhead is 3.5s to 4.1s against 0.9s to 1.0s
for suites without Postgres.
> - A cluster data directory that was shut down cleanly can be copied
and started again as an independent cluster, and the migrated schema is
the same for every suite.
> - This pull request builds one migrated cluster per host, publishes
its data directory as a template under the OS temp directory, and starts
every later suite from a copy of it. The probe reuses the template
instead of starting a cluster.
> - The benefit is about 7s to 8s per Postgres suite on the CI runners
(measured 3.2s to 0.35s per start and 0.75s to 0.03s per probe locally),
about 120s to 170s per server shard, and a faster local test loop for
every suite that uses the fixture.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The `server (n/12)` PR checks are the slowest check in every fully green
PR run since 2026-10-11 05:27Z. Their Vitest step runs suites one at a
time, and about a third of each shard's time is embedded Postgres setup
that produces the same migrated cluster every time.
**Subsystem affected**
Test infrastructure: the embedded Postgres test fixture in
`@paperclipai/db` (`packages/db/src/test-embedded-postgres.ts`), used by
349 server suites, the db package suites, and the CLI suites.
**Current behavior**
`startEmbeddedPostgresTestDatabase` runs `initdb`, starts the cluster,
creates the database, and applies all 331 migrations for every suite.
`getEmbeddedPostgresTestSupport` starts and stops a second throwaway
cluster once per test file, because the module-level cache does not
survive Vitest's per-file isolation. Measured locally: 3.2s per suite
start plus 0.4s cleanup, and 0.75s per probe. On the CI runners the
smallest Postgres suite takes 6s and the probe adds 2.5s to 3s of import
time per file.
**Proposed behavior**
The first caller on a host builds one migrated cluster, stops it, and
renames its data directory to
`$TMPDIR/paperclip-embedded-postgres-templates-<uid>/<key>`. Every later
caller copies the template (about 90 MB) and starts Postgres on the
copy, then still runs `ensurePostgresDatabase` and
`applyPendingMigrations`, which are no-ops on a current template. The
probe treats a published template as proof that the host can run the
cluster. The key hashes the embedded-postgres version, the initdb flags,
the user, and the name and content of every migration file plus the
journal, so a changed migration builds a new template.
`PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` bypasses the cache. Any template
failure falls back to the fresh path.
A template is a Postgres data directory, and starting it executes its
configuration, so the cache is private to the current OS user: the root
name carries the uid, the root is created with mode 0700, and both the
root and each template are used only when `lstat` shows a real directory
(not a symlink) owned by the current user with no group or world write
bit. A root that fails those checks (for example one pre-created by
another user in a shared `/tmp`) is left alone and the process uses
fresh clusters.
**Reason and benefit**
The slowest check decides the wall-clock time of every PR run. Measured
locally, a template start takes 0.3s to 0.4s instead of 3.2s, cleanup
0.15s instead of 0.43s, and the probe 0.03s instead of 0.75s. Four
server suites that passed in both modes dropped by 3.4s to 6.0s each;
the db `backup-lib` suite dropped from 42.7s to 20.9s. The CI runners
are about twice as slow, so the expected saving per server shard is 120s
to 170s.
**Breaking changes**
None. The returned database is fully migrated in both paths, each suite
still gets its own cluster and data directory, and a failure on the
template path falls back to the existing fresh path.
## What Changed
- `packages/db/src/test-embedded-postgres.ts`: adds the template cache.
`computeEmbeddedPostgresTemplateKey` hashes the inputs that determine a
migrated cluster. `buildEmbeddedPostgresTemplate` starts a fresh cluster
through the existing bounded retry, creates the database, applies
migrations, waits for the real `stop()` (30s bound), and publishes the
data directory with an atomic rename; a concurrent builder that loses
the rename discards its copy.
`startEmbeddedPostgresFromTemplateWithRetry` copies the template into a
fresh temp directory, sets mode 0700, and starts the cluster with the
same per-attempt port and directory rotation as the fresh path.
`startEmbeddedPostgresTestDatabase` tries the template first and falls
back to the fresh path with one warning per process.
`probeEmbeddedPostgresSupport` returns supported when a template exists
or can be built, and keeps its throwaway-cluster path for the case where
the template could not be published. Templates for other keys are pruned
after seven days. The connection strings and initdb flags move into
shared constants; no other behavior changes.
- `packages/db/src/test-embedded-postgres.test.ts`: two unit tests for
the key (stable for identical inputs; changes with migration content, a
new file, flags, version, or user; hashes this package's real migrations
directory) and two real-cluster tests (the published template is
complete and private, a suite starts from it in `template` mode with
migrations up to date, a table written to one copy is not visible in the
next copy, and `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` runs the `fresh`
path). The real-cluster tests skip on hosts without embedded Postgres,
like every other fixture suite.
- `doc/DEVELOPING.md`: a short section under Test Commands describing
the template, its key, and the bypass variable.
**Review fixes (second commit), one per finding:**
- Private cache root (Greptile P1, Superagent P2). The root is
`$TMPDIR/paperclip-embedded-postgres-templates-<uid>` (user name on
Windows), created with mode 0700. `prepareTemplateRootDir` and
`isUsableTemplate` both reject, via `lstat`, anything that is a symlink,
not a directory, owned by another uid, or writable by group or others; a
rejected root logs one warning and the process uses fresh clusters. A
root created by the first commit with a wider mode is narrowed to 0700
if this user owns it.
- Template failures never skip suites (Greptile P1).
`getEmbeddedPostgresTemplateDir` no longer rejects. A build or root
failure resolves `null` with one warning, and
`probeEmbeddedPostgresSupport` then runs the throwaway-cluster probe, so
only a cluster that cannot start reports `supported: false`. A migration
that cannot apply fails the suite on the fresh path instead of being
hidden by `describe.skip`.
- Idle-based pruning (Greptile P2). Each reuse refreshes the template's
timestamp with `utimes`, so the seven-day prune measures idle time. A
template that disappears after a process resolved its path is not
retried five times: `startEmbeddedPostgresFromTemplateWithRetry` checks
the template before each copy and throws at once, the suite falls back
to a fresh cluster, and the fallback forgets the path so the next caller
in that process rebuilds the template.
- The template-path test enables the cache for itself and restores the
original `PAPERCLIP_TEST_POSTGRES_TEMPLATE` value afterwards (Greptile
P1), so the suite passes under `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` as
well.
- Eight new tests: `isUsableTemplate` accepts a private cluster
directory and rejects a missing `PG_VERSION`, a leftover
`postmaster.pid`, a symlink, and (POSIX) group- or world-writable modes;
a world-writable pre-created root and a symlinked root both fall back to
fresh clusters, publish nothing, log one warning, and still report
`supported: true` from the probe; a ten-day-old template is refreshed on
reuse; a pruned template is rebuilt on the next start; and a template
build that fails after the cluster started falls through to the fresh
probe. The tests that need an empty cache point `os.tmpdir()` at a
scratch directory (`TMPDIR`, `TEMP` and `TMP` together, so the redirect
holds on Windows as well) and restore it.
**Review fixes (third commit,
|
||
|
Nicky LeachandPaperclip
|
4813ed3f0c |
fix(db): harden embedded Postgres test start with bounded retry (#10540)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The database layer uses embedded Postgres for isolated test runs > - A port probe can fail when another process takes the same port before Postgres binds it > - That race can make a test fail even when the code under test is fine > - This pull request adds bounded retry and clearer error text to the embedded Postgres start path > - The benefit is more stable tests and faster diagnosis when startup still fails ## Linked Issues or Issue Description No public GitHub issue exists for this change. This PR fixes a flaky embedded Postgres test start path. The helper can lose a port between probe and bind. This PR retries the start with a fresh port and a fresh data directory. Related public context: - Refs #7259 - Refs #9769 ## What Changed - Add bounded retry around embedded Postgres initialization and start. - Stop each failed attempt and remove its data directory before the next attempt. - Capture Postgres output in the thrown error so the failure is easier to read. - Add unit coverage for retry success, retry exhaustion, and the improved error text. ## Verification - `pnpm --filter @paperclipai/db exec vitest run src/test-embedded-postgres.test.ts src/embedded-postgres-error.test.ts` - `pnpm --filter @paperclipai/db exec vitest run` - `pnpm --filter @paperclipai/db exec vitest run` passed in the worktree after the change. - `worktree.test.ts > quarantines copied live execution state in seeded worktree databases` passed. - A real cluster loop of 100 starts passed with 0 failures. ## Risks - The retry can hide a real startup fault until the fifth try. - The bound keeps the wait short, and the final error still shows the captured Postgres log. - This change only affects the embedded Postgres test start helper. ## Model Used OpenAI Codex, GPT-5, tool-use enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |