## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: [<holder pid>]}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:<boot id>` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:<boot id>`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5.5 KiB
title, summary
| title | summary |
|---|---|
| Database | Embedded PGlite vs Docker Postgres vs hosted |
Paperclip uses PostgreSQL via Drizzle ORM. There are three ways to run the database.
1. Embedded PostgreSQL (Default)
Zero config. If you don't set DATABASE_URL, the server starts an embedded PostgreSQL instance automatically.
pnpm dev
On first start, the server:
- Creates
~/.paperclip/instances/default/db/for storage - Ensures the
paperclipdatabase exists - Runs migrations automatically
- Starts serving requests
Data persists across restarts. To reset: rm -rf ~/.paperclip/instances/default/db.
The Docker quickstart also uses embedded PostgreSQL by default.
2. Local PostgreSQL (Docker)
For a full PostgreSQL server locally:
docker compose up -d
This starts PostgreSQL 17 on localhost:5432. Set the connection string:
cp .env.example .env
# DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip
Push the schema:
DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip \
npx drizzle-kit push
3. Hosted PostgreSQL (Supabase)
For production, use a hosted provider like Supabase.
- Create a project at database.new
- Copy the connection string from Project Settings > Database
- Set
DATABASE_URLin your.env
Use the direct connection (port 5432) for migrations and the pooled connection (port 6543) for the application.
If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment:
DATABASE_PREPARED_STATEMENTS=false
For a direct connection or a pooler that supports named prepared statements,
DATABASE_PREPARED_STATEMENTS=true also prepares parameterized ORM queries and
queries inside transactions and reserved or dedicated connections. Check the
provider's configuration before you enable it. Preparation caches statement
metadata, not query results. Unset preserves the existing ORM behavior and the
driver default for tagged SQL. Explicit per-query options still apply.
Each database client prepares at most 1,000 distinct statements and skips
statements longer than 8 KB. If a prepared statement fails because the pooler's copy is
stale after a schema change or was lost, Paperclip stops preparing that
statement and retries it once unprepared.
Related optional client tuning: DATABASE_POOL_MAX, DATABASE_IDLE_TIMEOUT_SECONDS, DATABASE_CONNECT_TIMEOUT_SECONDS, DATABASE_MAX_LIFETIME_SECONDS, DATABASE_APPLICATION_NAME. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (DATABASE_IDLE_TIMEOUT_SECONDS=0 keeps them open) and the server's main pool reports application_name=paperclip:<boot id> (other pools use names that start with paperclip). See Connection pool settings.
Connection Pool Settings
The server opens one postgres.js pool for its own queries (and a second one when DATABASE_MIGRATION_URL points at a different connection). Every setting is optional:
| Variable | Default | Effect |
|---|---|---|
DATABASE_POOL_MAX |
10 (driver) |
Maximum pooled connections. |
DATABASE_IDLE_TIMEOUT_SECONDS |
60 |
Close a pooled connection after this much idle time. 0 keeps idle connections open forever (the driver default). |
DATABASE_CONNECT_TIMEOUT_SECONDS |
30 (driver) |
Give up on a connection attempt after this long. |
DATABASE_MAX_LIFETIME_SECONDS |
30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. |
DATABASE_APPLICATION_NAME |
paperclip:<boot id> (server main pool) |
Value of application_name in pg_stat_activity. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%'; |
DATABASE_DEDICATED_CONNECTION_MAX |
24 |
Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait. |
DATABASE_DEDICATED_CONNECTION_WAIT_MS |
30000 |
After this wait, a queued caller gets a dedicated connection over the limit. |
DATABASE_PREPARED_STATEMENTS |
Unset | true opts parameterized ORM queries into preparation. false disables preparation. Use false when the pooler does not support named statements. Unset preserves existing behavior. |
The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size max_connections on the PostgreSQL side for at least DATABASE_POOL_MAX per server process plus your other clients. Each server process also opens private pools and dedicated lock connections. With the defaults, one process can open up to 73 connections: DATABASE_POOL_MAX (10), 9 in private pools (lease renewal, checkpoints, file storage), 2 notification listeners, and up to 52 dedicated lock connections (application_name=paperclip-workspace-finalization-lock). DATABASE_DEDICATED_CONNECTION_MAX (default 24) bounds the dedicated connections in use; the worst case is twice that value plus 4 idle. Count all of these per process when you size max_connections, or lower DATABASE_DEDICATED_CONNECTION_MAX to fit.
Switching Between Modes
DATABASE_URL |
Mode |
|---|---|
| Not set | Embedded PostgreSQL |
postgres://...localhost... |
Local Docker PostgreSQL |
postgres://...supabase.com... |
Hosted Supabase |
The Drizzle schema (packages/db/src/schema/) is the same regardless of mode.