mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: [<holder pid>]}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:<boot id>` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:<boot id>`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
104 lines
5.5 KiB
Markdown
104 lines
5.5 KiB
Markdown
---
|
||
title: Database
|
||
summary: Embedded PGlite vs Docker Postgres vs hosted
|
||
---
|
||
|
||
Paperclip uses PostgreSQL via Drizzle ORM. There are three ways to run the database.
|
||
|
||
## 1. Embedded PostgreSQL (Default)
|
||
|
||
Zero config. If you don't set `DATABASE_URL`, the server starts an embedded PostgreSQL instance automatically.
|
||
|
||
```sh
|
||
pnpm dev
|
||
```
|
||
|
||
On first start, the server:
|
||
|
||
1. Creates `~/.paperclip/instances/default/db/` for storage
|
||
2. Ensures the `paperclip` database exists
|
||
3. Runs migrations automatically
|
||
4. Starts serving requests
|
||
|
||
Data persists across restarts. To reset: `rm -rf ~/.paperclip/instances/default/db`.
|
||
|
||
The Docker quickstart also uses embedded PostgreSQL by default.
|
||
|
||
## 2. Local PostgreSQL (Docker)
|
||
|
||
For a full PostgreSQL server locally:
|
||
|
||
```sh
|
||
docker compose up -d
|
||
```
|
||
|
||
This starts PostgreSQL 17 on `localhost:5432`. Set the connection string:
|
||
|
||
```sh
|
||
cp .env.example .env
|
||
# DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip
|
||
```
|
||
|
||
Push the schema:
|
||
|
||
```sh
|
||
DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip \
|
||
npx drizzle-kit push
|
||
```
|
||
|
||
## 3. Hosted PostgreSQL (Supabase)
|
||
|
||
For production, use a hosted provider like [Supabase](https://supabase.com/).
|
||
|
||
1. Create a project at [database.new](https://database.new)
|
||
2. Copy the connection string from Project Settings > Database
|
||
3. Set `DATABASE_URL` in your `.env`
|
||
|
||
Use the **direct connection** (port 5432) for migrations and the **pooled connection** (port 6543) for the application.
|
||
|
||
If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment:
|
||
|
||
```sh
|
||
DATABASE_PREPARED_STATEMENTS=false
|
||
```
|
||
|
||
For a direct connection or a pooler that supports named prepared statements,
|
||
`DATABASE_PREPARED_STATEMENTS=true` also prepares parameterized ORM queries and
|
||
queries inside transactions and reserved or dedicated connections. Check the
|
||
provider's configuration before you enable it. Preparation caches statement
|
||
metadata, not query results. Unset preserves the existing ORM behavior and the
|
||
driver default for tagged SQL. Explicit per-query options still apply.
|
||
Each database client prepares at most 1,000 distinct statements and skips
|
||
statements longer than 8 KB. If a prepared statement fails because the pooler's copy is
|
||
stale after a schema change or was lost, Paperclip stops preparing that
|
||
statement and retries it once unprepared.
|
||
|
||
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the server's main pool reports `application_name=paperclip:<boot id>` (other pools use names that start with `paperclip`). See [Connection pool settings](#connection-pool-settings).
|
||
|
||
## Connection Pool Settings
|
||
|
||
The server opens one postgres.js pool for its own queries (and a second one when `DATABASE_MIGRATION_URL` points at a different connection). Every setting is optional:
|
||
|
||
| Variable | Default | Effect |
|
||
|----------|---------|--------|
|
||
| `DATABASE_POOL_MAX` | `10` (driver) | Maximum pooled connections. |
|
||
| `DATABASE_IDLE_TIMEOUT_SECONDS` | `60` | Close a pooled connection after this much idle time. `0` keeps idle connections open forever (the driver default). |
|
||
| `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. |
|
||
| `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. |
|
||
| `DATABASE_APPLICATION_NAME` | `paperclip:<boot id>` (server main pool) | Value of `application_name` in `pg_stat_activity`. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with `SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%';` |
|
||
| `DATABASE_DEDICATED_CONNECTION_MAX` | `24` | Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait. |
|
||
| `DATABASE_DEDICATED_CONNECTION_WAIT_MS` | `30000` | After this wait, a queued caller gets a dedicated connection over the limit. |
|
||
| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. |
|
||
|
||
The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients. Each server process also opens private pools and dedicated lock connections. With the defaults, one process can open up to 73 connections: `DATABASE_POOL_MAX` (10), 9 in private pools (lease renewal, checkpoints, file storage), 2 notification listeners, and up to 52 dedicated lock connections (`application_name=paperclip-workspace-finalization-lock`). `DATABASE_DEDICATED_CONNECTION_MAX` (default 24) bounds the dedicated connections in use; the worst case is twice that value plus 4 idle. Count all of these per process when you size `max_connections`, or lower `DATABASE_DEDICATED_CONNECTION_MAX` to fit.
|
||
|
||
## Switching Between Modes
|
||
|
||
| `DATABASE_URL` | Mode |
|
||
|----------------|------|
|
||
| Not set | Embedded PostgreSQL |
|
||
| `postgres://...localhost...` | Local Docker PostgreSQL |
|
||
| `postgres://...supabase.com...` | Hosted Supabase |
|
||
|
||
The Drizzle schema (`packages/db/src/schema/`) is the same regardless of mode.
|