Files
PaperClipAI/docs/deploy/database.md
T
DottaandClaude Opus 5.5 3afb2c215b fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores the events that a runner sends for each run, and
it records the identity of the runner process
> - Sometimes the server stops storing a run's events for minutes, and
the server log stays quiet
> - In two restart-chaos cases, the identity write of the run started
and then did not commit for about 197 s and 480 s
> - The existing database report only fires when the reattach phase ends
at its cap, so it did not fire for the cold-start case
> - This pull request logs the pending writes and the blocking database
sessions whenever any run write stays pending for 30 s
> - The benefit is that the next stall names the session and the query
that hold the run, so the fix can target one proven holder

## Linked Issues or Issue Description

Refs #21946

**What happened?**
Restart-chaos waves on master showed two stalls in event storage.
1. Cold start of an OpenCode run, with no restart. The runner emitted
`session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z,
about 197 s later. The second `native.process_identity_recorded` write
of the run started at 22:13:23.706Z, and `runner.turn.submit` took
197270 ms. A second wave showed the same shape (about 191 s). The server
log had nothing in that window.
2. After a hard kill, every API GET timed out from 22:35:24Z to
22:43:23Z, server CPU was 0%, and no run events were stored. The process
identity transaction of the run began at 22:35:22.6Z and committed at
about 22:43:23Z.

**Expected behavior**
When the server cannot store a run's events, the server log names the
cause: which run writes are pending, and which database sessions hold or
wait for locks.

**Steps to reproduce**
1. Start a transaction that locks a `heartbeat_runs` row (`for update`)
and stays idle.
2. Record a process identity for that run
(`persistHeartbeatRunProcessMetadata`).
3. Before this change, nothing is logged while the write waits. After
this change, a warning names the idle holder and the waiting statement
after 30 s. The new database test does this with a 300 ms threshold.

**Relevant logs or output**
Example warning from the new test:
`runner event intake stalled; pending run writes, database sessions and
lock waits` with `operations: [{stage: "process_identity", pendingMs:
301}]` and `waits: [{state: "idle in transaction", query: "select id
from heartbeat_runs where id = ? ... for update"}, {waitEventType:
"Lock", blockedBy: [<holder pid>]}]`.

## What Changed

- New `server/src/services/run-intake-stall-watch.ts`. It keeps an
in-memory list of pending run operations: event receipts
(`PaperclipControlPlanePort.appendEvent`), event writes
(`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process
identity writes (`persistHeartbeatRunProcessMetadata`).
- When an operation stays pending for 30 s, the watch logs one warning
per stalled run. The warning has: the pending operations and their age,
the pending count and oldest age per run, the pool size, the database
sessions per application name and state, the lock waits and holders
(`pg_blocking_pids`, query without literals, cut to 160 characters), and
the event-loop lag. Reports are at most one per 30 s. A second line logs
when a reported operation finishes.
- The watch arms a timer only while an operation is pending. It reads
the database only when it reports, through the private liveness pool
with a 5 s statement timeout.
- The server names its main pool `paperclip:<boot id>` unless
`DATABASE_APPLICATION_NAME` is set. This shows which server process owns
each session after a restart.
- `readDatabaseSessionWaits` moves from the reattach phase to
`server/src/services/database-session-waits.ts`. The reattach phase
re-exports it.
- `@paperclipai/db` exports `databaseClientOptionsFromEnv` and
`DatabaseClientOptions`.
- `doc/DATABASE.md` describes the new log lines and the pool name.
- I did not add `idle_in_transaction_session_timeout` to the main pool.
Some code holds a transaction open on purpose, and a blind timeout could
abort valid work. The new report names idle-in-transaction holders
instead.
- These are server log lines only. This change does not touch Telemetry
or OpenTelemetry.

## Verification

- `cd server && npx vitest run
src/services/run-intake-stall-watch.test.ts` (5 tests). The database
test holds a run row in an idle transaction, starts a process identity
write, and checks that the warning names the holder pid, its query, the
waiting statement, and the pool name. It is red without the tracking in
`run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a
length of 1`), and green with it.
- Unit tests with fake timers check one report per stalled run, the 30 s
throttle, the clear line, and that no timer stays armed when nothing is
pending.
- `cd server && npx vitest run
src/services/controller-process-identity.test.ts` checks the pool name
and the `DATABASE_APPLICATION_NAME` override.
- `npx tsc --noEmit` passes for `server` and `packages/db`.
- `pnpm verify` ran locally on the affected tests.

## Risks

- Low risk. Each run event adds one map insert and one map delete. No
database work happens while writes finish within 30 s.
- The main pool `application_name` changes from `paperclip` to
`paperclip:<boot id>`. A dashboard that filters on exactly `paperclip`
must use a prefix match, or the operator can set
`DATABASE_APPLICATION_NAME`.
- A report reads `pg_stat_activity` on the liveness pool. A report
during a full database stall can fail; the warning then has
`databaseReadError` and still lists the pending operations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:15:40 -06:00

5.5 KiB
Raw Blame History

title, summary
title summary
Database Embedded PGlite vs Docker Postgres vs hosted

Paperclip uses PostgreSQL via Drizzle ORM. There are three ways to run the database.

1. Embedded PostgreSQL (Default)

Zero config. If you don't set DATABASE_URL, the server starts an embedded PostgreSQL instance automatically.

pnpm dev

On first start, the server:

  1. Creates ~/.paperclip/instances/default/db/ for storage
  2. Ensures the paperclip database exists
  3. Runs migrations automatically
  4. Starts serving requests

Data persists across restarts. To reset: rm -rf ~/.paperclip/instances/default/db.

The Docker quickstart also uses embedded PostgreSQL by default.

2. Local PostgreSQL (Docker)

For a full PostgreSQL server locally:

docker compose up -d

This starts PostgreSQL 17 on localhost:5432. Set the connection string:

cp .env.example .env
# DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip

Push the schema:

DATABASE_URL=postgres://paperclip:paperclip@localhost:5432/paperclip \
  npx drizzle-kit push

3. Hosted PostgreSQL (Supabase)

For production, use a hosted provider like Supabase.

  1. Create a project at database.new
  2. Copy the connection string from Project Settings > Database
  3. Set DATABASE_URL in your .env

Use the direct connection (port 5432) for migrations and the pooled connection (port 6543) for the application.

If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment:

DATABASE_PREPARED_STATEMENTS=false

For a direct connection or a pooler that supports named prepared statements, DATABASE_PREPARED_STATEMENTS=true also prepares parameterized ORM queries and queries inside transactions and reserved or dedicated connections. Check the provider's configuration before you enable it. Preparation caches statement metadata, not query results. Unset preserves the existing ORM behavior and the driver default for tagged SQL. Explicit per-query options still apply. Each database client prepares at most 1,000 distinct statements and skips statements longer than 8 KB. If a prepared statement fails because the pooler's copy is stale after a schema change or was lost, Paperclip stops preparing that statement and retries it once unprepared.

Related optional client tuning: DATABASE_POOL_MAX, DATABASE_IDLE_TIMEOUT_SECONDS, DATABASE_CONNECT_TIMEOUT_SECONDS, DATABASE_MAX_LIFETIME_SECONDS, DATABASE_APPLICATION_NAME. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (DATABASE_IDLE_TIMEOUT_SECONDS=0 keeps them open) and the server's main pool reports application_name=paperclip:<boot id> (other pools use names that start with paperclip). See Connection pool settings.

Connection Pool Settings

The server opens one postgres.js pool for its own queries (and a second one when DATABASE_MIGRATION_URL points at a different connection). Every setting is optional:

Variable Default Effect
DATABASE_POOL_MAX 10 (driver) Maximum pooled connections.
DATABASE_IDLE_TIMEOUT_SECONDS 60 Close a pooled connection after this much idle time. 0 keeps idle connections open forever (the driver default).
DATABASE_CONNECT_TIMEOUT_SECONDS 30 (driver) Give up on a connection attempt after this long.
DATABASE_MAX_LIFETIME_SECONDS 30–60 min, randomized (driver) Recycle a pooled connection once it is this old.
DATABASE_APPLICATION_NAME paperclip:<boot id> (server main pool) Value of application_name in pg_stat_activity. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%';
DATABASE_DEDICATED_CONNECTION_MAX 24 Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait.
DATABASE_DEDICATED_CONNECTION_WAIT_MS 30000 After this wait, a queued caller gets a dedicated connection over the limit.
DATABASE_PREPARED_STATEMENTS Unset true opts parameterized ORM queries into preparation. false disables preparation. Use false when the pooler does not support named statements. Unset preserves existing behavior.

The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size max_connections on the PostgreSQL side for at least DATABASE_POOL_MAX per server process plus your other clients. Each server process also opens private pools and dedicated lock connections. With the defaults, one process can open up to 73 connections: DATABASE_POOL_MAX (10), 9 in private pools (lease renewal, checkpoints, file storage), 2 notification listeners, and up to 52 dedicated lock connections (application_name=paperclip-workspace-finalization-lock). DATABASE_DEDICATED_CONNECTION_MAX (default 24) bounds the dedicated connections in use; the worst case is twice that value plus 4 idle. Count all of these per process when you size max_connections, or lower DATABASE_DEDICATED_CONNECTION_MAX to fit.

Switching Between Modes

DATABASE_URL Mode
Not set Embedded PostgreSQL
postgres://...localhost... Local Docker PostgreSQL
postgres://...supabase.com... Hosted Supabase

The Drizzle schema (packages/db/src/schema/) is the same regardless of mode.