refactor: cut DATABASE.md to setup and schema-change rules (#21700)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DATABASE.md` before a schema or migration change
> - That file grew to 2,611 lines. About 1,950 of them were notes on the
tables of one feature, added one pull request at a time
> - The setup steps and the migration rules started at line 792, below
those notes
> - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and
the rules that apply to every schema change
> - The benefit is a guide that a person or an agent can read in full
before a schema change

## Linked Issues or Issue Description

Refs #21670 (the same cleanup for `doc/DEVELOPING.md`).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DATABASE.md`

**What's wrong?**

The file was 2,611 lines. It opened with a Better Auth upgrade note and
492 lines on task query performance. The schema change workflow started
at line 1367.

**Suggested fix**

Keep database setup and the general rules in `doc/DATABASE.md`. Remove
the notes on individual tables. The schema and its tests describe those.

## What Changed

- `doc/DATABASE.md` is now 377 lines. It keeps the three database modes,
client tuning, the schema change workflow, migration compatibility, the
migration authoring checklist, migration snapshots, the rule for retries
after a lost connection, transaction-aware work signals, plugin
namespaces, and backups. The wording of the kept text is unchanged.
- New `doc/database-client-internals.md` holds the response-size
diagnostic, the notes on the postgres.js patch, and the transaction
memory note. The text is moved without wording changes.
- "Experimental remote workspace snapshots" moved to
`doc/workspace-storage.md`, which links to it.
- "Execution admission accounting" moved to
`server/src/modules/README.md`, which links to it.
- Deleted about 1,950 lines of notes on one feature each. No other file
linked to them. The deleted sections are: Better Auth 1.7.16 upgrade,
Durable native provider backups, Workspace cohort admission proofs,
Original local Git sources for archived workspaces, Proactive agent chat
messages, Persistent case handoffs, Task query performance, Tool profile
reads, Execution identity row locks, Remote workspace row locks, Managed
adapter settings, Cloud configuration migration receipt, Cloud runtime
identity singleton, Unclaimed Cloud warm standby, Native finalization
history and idle sleep, Feedback history and idle sleep, Resource
membership tables, Decision training snapshot retention, Decision queues
and triage provenance, Native runner persistence, Chat communication
snapshots, Telegram private draft identities, Attachment upload
provenance, Question-response delivery receipts, Secret storage,
Persistent agent conversations, Resource lifecycle events, Legacy
controller ownership, Agent file persistence and legacy revisions, Large
API response snapshots, Runner API response reservations, Internal agent
commentary, Agent identity keys and backups, Slack app registration,
Slack manager workspace grants, Installed skill packages, Project
repository skill synchronization, Task workspace records, Execution
recovery query performance, Agent runtime synchronization journal,
Retired Pipelines storage, Prompt injection classification, Deferred
conversation requests, GitHub inherited review-wait reads, Sandbox
creation ownership, Object-backed run evidence, Cloud runtime
configuration singleton, Independent accounting and budget evaluation,
Standard workspace and Classic opt-in, Project environment rolling
compatibility.
- Also deleted from kept sections: the subscription-table note and the
issue-reference backfill note in the embedded mode section, the HTTP
access log note and the per-endpoint retry notes in the connection loss
section, and the notes on migrations `0355`, `0313`, and `0314` in the
authoring checklist.
- `AGENTS.md` now states what `doc/DATABASE.md` is for.

## Verification

- A script built the new files by line range from the old file. It
asserts the old line count and the first line of each range.
- A link check on every changed Markdown file, and on the four files
that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing
anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- The deleted notes were not all read in full before deletion. The
decision was by section: a section about one feature's tables, with no
inbound link, was deleted. The text stays in git history.
- Two deleted sections describe procedures for an operator: the
duplicate-account check before the Better Auth 1.7.16 migration, and the
secret storage tables. Restore them into a focused doc if operators
still need them.
- No runtime behavior changes.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-14 20:01:28 -06:00
1 parent 958eda6f16
commit d6acbf50e6
5 files changed
+346 -2296

No files matched your search

+1 -1
View File
@@ -210,5 +210,5 @@ A change is done when all are true:
1. Behavior matches what the task asked for and preserves the invariants in section 5
2. Relevant local verification and required CI typecheck, test, and build checks pass for the final revision, with any acceptance or platform-specific gaps resolved (section 7)
3. Contracts are synced across db/shared/server/ui
4. Docs updated when behavior or commands change. `doc/DEVELOPING.md` holds setup, commands, and everyday workflow only. Put a note about how a subsystem behaves in that subsystem's doc (see "More documentation" in `doc/DEVELOPING.md`), or leave it in the PR description when the code and tests already say it.
4. Docs updated when behavior or commands change. `doc/DEVELOPING.md` holds setup, commands, and everyday workflow only, and `doc/DATABASE.md` holds database setup and the rules for every schema change. Put a note about how a subsystem behaves in that subsystem's doc (see "More documentation" in `doc/DEVELOPING.md`), or leave it in the PR description when the code and tests already say it.
5. PR description follows the [PR template](.github/PULL_REQUEST_TEMPLATE.md) with all sections filled in (including Model Used)
+59 -2293
View File
File diff suppressed because it is too large. Load diff
+181
View File
@@ -0,0 +1,181 @@
# Database client internals
Diagnostics and driver details for `packages/db`. Setup and migration rules are in [DATABASE.md](DATABASE.md).
## Bounded response diagnostics
`PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW=<start>/<end>` enables one explicit UTC
window. Both timestamps use ISO format ending in `Z`. The window lasts at most
60 seconds and starts within 30 minutes. Unset, invalid, and expired windows
leave database construction unchanged. An expired window stays inactive after a
restart; remove the setting after the investigation.
The diagnostic measures awaited Postgres.js `unsafe` responses before Drizzle
column decoding. It issues no queries. Configuration is read when each database
client is created; setting the environment on an already-running process does
not install the capture on existing clients. Prepare the deployment artifact
before choosing the UTC window, then start the window after rollout and warm-up.
Server entry points use `withDatabaseOperation(owner, action)` to name the work
that executes queries. Owners are fixed semantic labels, never request paths,
company or run IDs, user input, or SQL. The execution owner is captured for each
query and retained through completion, including transactions and dedicated
pools. A prepared query reused by different operations keeps the owner of each
execution. The builder caller is separate from that execution owner.
Owner context is active while a valid future window is armed, up to 30 minutes
before its start, so work that crosses the start boundary keeps its owner. Query
accounting starts only at the window start. With no armed window, or after
expiry, the owner helper calls the action directly without a context scope.
The capture aggregates owner, static schema table, operation, calls, returned
rows, query errors, and total/maximum completion wall time. `totalWallMs` and
`maxWallMs` include pool, database, network, and event-loop waits; they are not
CPU measurements, and concurrent query durations overlap. Timing excludes the
capture's response sizing. Rank calls and returned rows both per second and per
completed service operation; a busy owner is not necessarily wasteful.
Response sizing samples successful results 1, 17, 33, and so on per aggregate
group (`sampleEvery: 16`). It examines at most
256 cells, 65,536 string code units, and 65,536 value bytes per sampled result;
an individual string over 16,384 code units is not measured. Parsed objects and
other complex values are not serialized. Oversized, complex, or unexamined
values leave an incomplete sample. `sampledValueBytes` is a lower bound over
sampled primitive values, not a total response-size estimate or network bytes.
Do not extrapolate it as the total bytes returned by the database. Inspect
`sampledResults`, `sampledRows`, `measuredValues`, `unmeasuredValues`, and
`incompleteSamples` alongside it. `sizingWallMs` records the diagnostic's sizing
work separately. Use a separate capture-disabled CPU profile to measure normal
processing cost.
The diagnostic emits at most 2,048 aggregate groups and a summary. Measurement
stops at the deadline. Environment-enabled
captures emit at most 64 records per second and can take another 33 seconds to
finish writing the bounded output. Wait for the summary before collecting it.
It never logs SQL, parameters, result
values, connection strings, or query errors. In-flight queries continue normally.
Semantic owners work in a bundled server without source maps. Specific owners
avoid stack capture. The generic `http.request` owner keeps its exact name but,
like an unowned query, also captures a bounded caller location during the window.
For wrapped query builders, a map-free hosted bundle retains one static generated
construction-site anchor (`dist-runtime/server.mjs`, `index.mjs`, or
`chunk-<build hash>.mjs`, with line and column). Correlate it with the exact serving
revision and bundle; it is not a TypeScript location or an execution-time CPU
owner. Capture wrapper frames are excluded before reading the stack, and deeper
frames in the same bundle are omitted.
Unscoped driver calls, including relational queries, do not accept generated
anchors: a bundled ORM helper could otherwise be mistaken for an application
caller. They can still retain existing public source/dist locations; linked
source maps and Node's `--enable-source-maps` may help resolve those locations.
Restore prior Node options after capture. Missing callers remain unknown and
still contribute to totals. Disabled, future and expired windows do not capture
stacks; wrappers installed for an armed window remain after expiry. Active census
stack capture, grouping and bounded sizing add measurement overhead. More caller
groups still share the existing group, page and output-rate caps. This diagnostic
is not a production performance improvement.
Look for `paperclip.db_response_capture.entry.v2` and
`paperclip.db_response_capture.summary.v2` in the process logs. Version 2 uses
sampled, bounded primitive-value sizing; do not interpret it as the version 1
all-result byte estimate. A log provider
can store `event` as a structured field instead of message text. Filter by that
field or read the bounded window from the exact deployment. Check the summary
and group totals, including the unattributed overflow group.
Fetch a complete capture as follows:
1. Read the summary events from the exact deployment and window. Keep each
distinct `captureId` separately.
2. Each entry has a unique `groupIndex` within that capture and a `groupPage`.
The summary gives `groupPageSize` (128) and `groupPages`. Fetch every page
from zero through `groupPages - 1`, filtering on the entry event, `captureId`,
and `groupPage`. Request at least `groupPageSize` matching records per page.
A structured query can use
`@event:paperclip.db_response_capture.entry.v2 AND @captureId:<id> AND @groupPage:0`.
3. If the provider's record limit is smaller than a page, fetch each
`groupIndex` separately with the same capture ID. Each index matches one
entry. Do not split by timestamp: entries can share a timestamp.
4. Deduplicate by `(captureId, groupIndex)`. Require every index from zero
through `groups - 1`, with no gaps. Entry count must equal `groups`, summed
`calls` must equal `completedQueries`, and summed `rows`, `sampledResults`,
and `sampledValueBytes` must equal their summary totals. A missing page is an
incomplete measurement. Keep overflow and unattributed groups visible.
Tagged raw-driver calls, streams, unawaited raw-driver
dispatch, and protocol overhead are not measured. Relational queries use a driver-level caller fallback.
Queries still in flight at the deadline are reported separately. The census does
not include every PostgreSQL statement or internal transaction command, so its
calls are not interchangeable with database-wide commit counters. It does not
measure SQL compilation, driver parsing CPU, or Drizzle row-mapping CPU. Use the
ranked owners to choose a narrow follow-up profile or synchronous-stage
measurement; do not attribute CPU by subtracting process CPU usage across an
asynchronous query. Remove the capture setting after collecting its final
summary; the absolute deadline already prevents further measurement.
## postgres.js patch
Source builds carry `patches/postgres@3.4.9.patch` for this behavior. It rejects
queued and later queries from a disconnected transaction or reserved connection,
and prevents a released, closed connection from returning to the open pool.
The patch covers both ESM and CommonJS. The regression suite terminates real
PostgreSQL backends and checks rejection, pool recovery, and transaction isolation.
Remove the patch when an upstream release passes these tests. Installs of the
unmodified `postgres` package outside this workspace do not include the patch.
The patch also preserves transaction connection reservation when pipelining is
disabled with `PGMAX_PIPELINE=0`. Without this fix, the driver's pipeline capacity
check skips the transaction reservation callback and `BEGIN` fails with
`UNSAFE_TRANSACTION`. Disabling pipelining makes a queued query wait for the next
available pool connection instead of sending it behind unrelated work on a busy
connection. This can help interactive requests that share a prepared-query pool
with slow background work. It does not make the SQL itself faster. The default
pipeline limit remains unchanged. Test transaction behavior before using this
setting with a driver build that does not include the patch.
The patch also releases a connection's receive buffer when the connection has
read every message in it. Without this fix, an idle connection keeps the buffer
of its last read until its next read: the socket chunk, or the whole buffer of
its largest recent message. At 340 open connections that each read a 1 MB row,
this held about 320 MB of `external` memory. The memory is outside the
JavaScript heap. `packages/db/src/postgres-receive-buffer.test.ts` checks that
the buffer is released. It also checks that results are reassembled correctly
when socket chunks end exactly at message boundaries, when they split messages
in the middle, and when the operating system chooses the chunks. Measure it with
`pnpm --filter @paperclipai/db exec node --expose-gc --import tsx scripts/benchmark-postgres-receive-buffer.ts`
(`CONNECTIONS` and `ROW_BYTES` are optional).
The patch also keeps the driver's connection queues proportional to the
connections they hold. Without this fix, the queue of idle connections keeps an
empty slot for every query it has handed out until it is completely empty. A
busy pool always has some idle connections, so that queue never empties. Each
new query then scans one slot for every query the process has already run. In a
simulation at 1,000 statements per second, dispatching a query used about 15%
of the main thread after 17 minutes, and the cost kept growing. The patched queue removes
the empty slots once there are more than 1,024 of them and more empty slots
than queued connections. It also searches only the queued connections.
`packages/db/src/postgres-queue.test.ts` checks the queue order and the work per
query, and `packages/db/src/postgres-queue-pool.test.ts` checks a real pool.
Measure it with
`pnpm --filter @paperclipai/db exec tsx scripts/benchmark-postgres-queue.ts`
(`SIMULATED_QUERIES` and `QUERIES` are optional).
## Relational queries inside transactions
Drizzle builds one relational query builder for every schema table when it
creates a database object, and it creates one for each transaction and each
savepoint. With the Paperclip schema that is about 100 KB per transaction.
`createDb` therefore opens transactions through a second Drizzle client on the
same postgres.js client, created without the schema, and builds `tx.query` the
first time a transaction reads it (`packages/db/src/lazy-transaction-queries.ts`).
`db.query`, `tx.query`, and `tx._` keep their contents. `BEGIN`, `SAVEPOINT`,
commit, rollback, and connection reservation remain Drizzle's and the driver's.
Measure the cost of a transaction and a savepoint with the command below. It
needs Node.js 25 or later, because Node.js 24 has no V8 allocation counter, and
it stops with that message on an older runtime:
```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts
```
+57 -1
View File
@@ -342,7 +342,7 @@ for storage, import, retention, and lifecycle regression checks. These checks do
not qualify a deployed Runner/provider or a zero-outage webserver deploy. Keep
the existing persistent volume until the other instance data paths and deploy
handoff are separately qualified. See the
[snapshot storage contract](DATABASE.md#experimental-remote-workspace-snapshots).
[snapshot storage contract](#experimental-remote-workspace-snapshots).
## S3-backed run evidence
@@ -408,3 +408,59 @@ existing protection: if the native controller check finds state the database doe
not prove, startup refuses and the error names the run, the task and the file,
because that file may be the only copy of the run's progress. Adopt it with the
native controller migration first.
## Experimental remote workspace snapshots
Migration [0421_bright_skaar.sql](../packages/db/src/migrations/0421_bright_skaar.sql)
adds the company-scoped `execution_workspace_snapshots` ledger. It records the
workspace, source operation/run, parent, generation, writer epoch, encrypted
object receipt, hashes, manifest digest, and retention state. Integrity and key
fields remain nullable while an upload is pending. Upload attempts record their
candidate object keys before I/O so a lost reply can be reconciled and abandoned
objects can be reclaimed. Object existence alone is not a committed save.
`execution_workspaces.metadata.remoteWorkspace`, schema
`paperclip.remote-workspace.v1`, holds the source, epoch, committed snapshot ID,
remote allocation/path, writer run, and dirty flag. Paths in this record belong
to the remote allocation. Save publication checks the current writer/epoch and
atomically commits the ledger row, authority pointer, and operation receipt.
`execution_workspaces.metadata.remoteWorkspaceSnapshotRepositories` records the
committed snapshot ID and the repository descriptors covered by that archive.
Reservation retains this coverage when it adds requested repositories to the
current source. Successful save publication advances coverage and the snapshot
pointer together. Older saved workspaces capture their previous source inventory
at the next reservation. Newly attached repositories use normal acquisition;
they cannot claim coverage from an older archive. No schema migration is needed.
`workspace_operations` journals preparation, save, restore, and local import;
large file bodies stay outside the database. Ledger identities deliberately
survive source-record deletion so object cleanup can finish.
Snapshots stream through authenticated encryption into S3. Portable archives
preserve supported files, modes, empty directories, confined symlinks, and Git
checkout/index/object state, including unpushed history. Linked worktrees and
object alternates are normalized into standalone Git storage. The versioned
exclusion policy omits runtime transport material and proven rebuildable
dependencies; unsupported entries fail the save.
The current snapshot and active run/restore/recovery/unfinished-operation
references pin saved data. Unreferenced superseded versions receive a seven-day
grace. Retention uses committed producer notifications and actual deadlines;
pinned history and an empty queue do not cause recurring scans. Deletion fences
new adoption, rechecks references, and removes every recorded upload candidate
and winning object within the company/workspace prefix.
When retention finds a pinned version, it records the first reference that holds
it in `metadata.retentionPin`: the current pointer, an importing parent, an
operation, an unfinished child, a run, or a recovery action. The candidate query
leaves that version out while the recorded reference still holds, so execution
notifications do not re-examine unchanged history. When the recorded reference
releases, the next pass checks every reference again. It records the next
reference that holds, or schedules or retires the version. The record is durable,
so a restarted server keeps the exclusion. A version without a record, such as
one saved before this change, is examined once. Wakes coalesce for up to one
second.
Database backups contain snapshot receipts, not S3 payloads or encryption keys.
Disaster recovery requires the database, object store, and original master key.
Unsaved live sandbox edits remain outside that saved-version guarantee. See
[setup and explicit local import](workspace-storage.md#experimental-native-remote-workspace-storage).
+48 -1
View File
@@ -63,4 +63,51 @@ internal policy commands and the deletion guards that project, agent, and
company deletion call. Settlement releases active charges but keeps each
admission's complete project set for later retained-session resolution.
Execution claims still belong to `services/heartbeat/queue.ts` and native
review dispatch. See [execution admission accounting](../../../doc/DATABASE.md#execution-admission-accounting).
review dispatch. See [execution admission accounting](#execution-admission-accounting).
## Execution admission accounting
The execution-admission module (`server/src/modules/execution-admission/`) owns
three tables. Nothing writes admissions or caps in production yet, and no claim
path enforces a project maximum.
- `project_task_execution_policies` stores a project's maximum concurrent tasks
apart from `projects`, so project readers never return an unenforced cap. No
row means never configured. A row with a null maximum means an explicit "no
limit", which a later legacy migration must not overwrite. Maximums are
positive. The row is deleted with its project.
- `execution_admissions` stores one admission attempt of a run. A run can be
released before dispatch and admitted again, so `(run_id, attempt)` is unique
and a partial unique index allows one unsettled attempt per run. The row keeps
the context revision, the nullable primary project as provenance (no foreign
key) and the settlement identity, which is set together with `settled_at`.
`admitted_project_ids` is the complete admitted project set, sorted, kept
after settlement and immutable (a trigger rejects any change). It has no
project foreign key, so settled history never blocks project deletion and
keeps the deleted project's ID. `{}` is a known empty set. Null means
unknown and is allowed only on settled rows: migration `0428` filled it from
the charges of unsettled rows, but settled `0426` rows had already lost
their set. A retained provider session resumes with the previous set ∪ the
current references; an unknown previous set requires a fresh context, never
an empty or primary-only guess.
- `execution_admission_project_charges` stores one row per distinct admitted
project, including unlimited projects. An admission without projects has no
rows. Composite `(company_id, …)` foreign keys keep admissions and projects in
one company. Rows are the only active capacity and exist only while the
admission is unsettled: settlement deletes the whole set in one transaction,
and `(company_id, project_id)` is indexed for active counts, so lookups
scale with active work, not history.
Archiving a project changes no charge. Deleting a project with a charge returns
`409 project_execution_admission_pending`; the foreign key is the backstop.
Deleting an agent refuses unsettled admissions on its runs and removes settled
ones, with their sets, together with the runs. Company deletion removes the company's admissions before its runs and
projects.
The proposed global lock order for the admission transaction is: company row
(`FOR NO KEY UPDATE`), native finalization recovery owner, the whole project set
in ascending ID order (`FOR NO KEY UPDATE`, one statement), issues, wake
requests, the run, the external-admission advisory lock, the admission row and
its charges, then budget reservations. Policy updates take the same project
lock. Settlement takes no project lock. Only the project-set step is implemented
so far; claim paths keep their current locks until they are integrated.