mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DATABASE.md` before a schema or migration change > - That file grew to 2,611 lines. About 1,950 of them were notes on the tables of one feature, added one pull request at a time > - The setup steps and the migration rules started at line 792, below those notes > - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and the rules that apply to every schema change > - The benefit is a guide that a person or an agent can read in full before a schema change ## Linked Issues or Issue Description Refs #21670 (the same cleanup for `doc/DEVELOPING.md`). **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DATABASE.md` **What's wrong?** The file was 2,611 lines. It opened with a Better Auth upgrade note and 492 lines on task query performance. The schema change workflow started at line 1367. **Suggested fix** Keep database setup and the general rules in `doc/DATABASE.md`. Remove the notes on individual tables. The schema and its tests describe those. ## What Changed - `doc/DATABASE.md` is now 377 lines. It keeps the three database modes, client tuning, the schema change workflow, migration compatibility, the migration authoring checklist, migration snapshots, the rule for retries after a lost connection, transaction-aware work signals, plugin namespaces, and backups. The wording of the kept text is unchanged. - New `doc/database-client-internals.md` holds the response-size diagnostic, the notes on the postgres.js patch, and the transaction memory note. The text is moved without wording changes. - "Experimental remote workspace snapshots" moved to `doc/workspace-storage.md`, which links to it. - "Execution admission accounting" moved to `server/src/modules/README.md`, which links to it. - Deleted about 1,950 lines of notes on one feature each. No other file linked to them. The deleted sections are: Better Auth 1.7.16 upgrade, Durable native provider backups, Workspace cohort admission proofs, Original local Git sources for archived workspaces, Proactive agent chat messages, Persistent case handoffs, Task query performance, Tool profile reads, Execution identity row locks, Remote workspace row locks, Managed adapter settings, Cloud configuration migration receipt, Cloud runtime identity singleton, Unclaimed Cloud warm standby, Native finalization history and idle sleep, Feedback history and idle sleep, Resource membership tables, Decision training snapshot retention, Decision queues and triage provenance, Native runner persistence, Chat communication snapshots, Telegram private draft identities, Attachment upload provenance, Question-response delivery receipts, Secret storage, Persistent agent conversations, Resource lifecycle events, Legacy controller ownership, Agent file persistence and legacy revisions, Large API response snapshots, Runner API response reservations, Internal agent commentary, Agent identity keys and backups, Slack app registration, Slack manager workspace grants, Installed skill packages, Project repository skill synchronization, Task workspace records, Execution recovery query performance, Agent runtime synchronization journal, Retired Pipelines storage, Prompt injection classification, Deferred conversation requests, GitHub inherited review-wait reads, Sandbox creation ownership, Object-backed run evidence, Cloud runtime configuration singleton, Independent accounting and budget evaluation, Standard workspace and Classic opt-in, Project environment rolling compatibility. - Also deleted from kept sections: the subscription-table note and the issue-reference backfill note in the embedded mode section, the HTTP access log note and the per-endpoint retry notes in the connection loss section, and the notes on migrations `0355`, `0313`, and `0314` in the authoring checklist. - `AGENTS.md` now states what `doc/DATABASE.md` is for. ## Verification - A script built the new files by line range from the old file. It asserts the old line count and the first line of each range. - A link check on every changed Markdown file, and on the four files that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - The deleted notes were not all read in full before deletion. The decision was by section: a section about one feature's tables, with no inbound link, was deleted. The text stays in git history. - Two deleted sections describe procedures for an operator: the duplicate-account check before the Better Auth 1.7.16 migration, and the secret storage tables. Restore them into a focused doc if operators still need them. - No runtime behavior changes. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
114 lines
6.5 KiB
Markdown
114 lines
6.5 KiB
Markdown
# Feature modules
|
||
|
||
A feature module under `server/src/modules/<name>/` uses three layers. Each
|
||
layer has one rule: which layer below it, it may import from.
|
||
|
||
```
|
||
adapters → application → domain
|
||
```
|
||
|
||
- **`domain/`** holds pure business rules. A domain file takes plain data in
|
||
and returns plain data out. A domain file must not import `drizzle-orm`,
|
||
`@paperclipai/db`, a service under `server/src/services/`, a route under
|
||
`server/src/routes/`, or a Node.js I/O module (`node:child_process`,
|
||
`node:fs`, `node:net`). A domain function must not read the system clock;
|
||
the caller passes `now` as an explicit `Date` value.
|
||
- **`application/`** holds use cases and the ports they need. A use case
|
||
takes its ports as constructor arguments and calls domain functions for
|
||
policy decisions. An application file must not import `drizzle-orm`, a SQL
|
||
client, a concrete adapter, or the server's HTTP error helpers. The outer
|
||
service or route translates application errors into transport responses.
|
||
- **`adapters/`** holds the concrete implementations of the ports:
|
||
Postgres queries, transactions, and process control. An adapter file may
|
||
import `drizzle-orm`, `@paperclipai/db`, and Node.js I/O modules.
|
||
|
||
A module exposes its commands through `index.ts`, which composes the adapters
|
||
and the use cases behind a factory function. Code outside the module imports
|
||
that entry point, never a file inside `domain/`, `application/`, or
|
||
`adapters/` directly.
|
||
|
||
`pnpm check:module-boundaries` enforces these rules for production source
|
||
files. It also checks access to the company deletion entry point below.
|
||
|
||
## Company deletion
|
||
|
||
The existing company deletion service owns the database transaction and deletion
|
||
order. A module can expose a separate `company-deletion.ts` entry point.
|
||
Only `services/company-deletion.ts` can import this entry point.
|
||
The entry point exports an implementation of `CompanyDeletionParticipant` from
|
||
`lib/company-deletion.ts`. Do not export it through the normal module index.
|
||
|
||
The service must complete external cleanup before it starts the transaction.
|
||
It must lock the company before it deletes dependent records or calls a module.
|
||
Creation commands must use the same company lock.
|
||
The service removes dependent records before it calls the module.
|
||
The module checks its deletion conditions and deletes the records it owns.
|
||
Use the supplied transaction. Do not start another transaction or call external
|
||
systems. Throw an error if deletion cannot proceed.
|
||
An error must roll back all database changes in the deletion transaction.
|
||
|
||
Call each module explicitly in dependency order. No module registry is required.
|
||
|
||
The agent lifecycle module also has a restricted `configuration.ts` entry point.
|
||
It saves configuration and invalidates verification in the caller's transaction.
|
||
The [lifecycle guide](../../../doc/AGENT-LIFECYCLE.md) lists its permitted callers.
|
||
Ordinary configuration updates use the root lifecycle command.
|
||
|
||
The [agent lifecycle module](../../../doc/AGENT-LIFECYCLE.md) owns agent creation,
|
||
pause, resume, and termination. Use its commands for these changes.
|
||
|
||
The execution admission module owns project task-execution policy and the
|
||
durable admission records that charge project capacity. Today it exposes only
|
||
internal policy commands and the deletion guards that project, agent, and
|
||
company deletion call. Settlement releases active charges but keeps each
|
||
admission's complete project set for later retained-session resolution.
|
||
Execution claims still belong to `services/heartbeat/queue.ts` and native
|
||
review dispatch. See [execution admission accounting](#execution-admission-accounting).
|
||
|
||
## Execution admission accounting
|
||
|
||
The execution-admission module (`server/src/modules/execution-admission/`) owns
|
||
three tables. Nothing writes admissions or caps in production yet, and no claim
|
||
path enforces a project maximum.
|
||
|
||
- `project_task_execution_policies` stores a project's maximum concurrent tasks
|
||
apart from `projects`, so project readers never return an unenforced cap. No
|
||
row means never configured. A row with a null maximum means an explicit "no
|
||
limit", which a later legacy migration must not overwrite. Maximums are
|
||
positive. The row is deleted with its project.
|
||
- `execution_admissions` stores one admission attempt of a run. A run can be
|
||
released before dispatch and admitted again, so `(run_id, attempt)` is unique
|
||
and a partial unique index allows one unsettled attempt per run. The row keeps
|
||
the context revision, the nullable primary project as provenance (no foreign
|
||
key) and the settlement identity, which is set together with `settled_at`.
|
||
`admitted_project_ids` is the complete admitted project set, sorted, kept
|
||
after settlement and immutable (a trigger rejects any change). It has no
|
||
project foreign key, so settled history never blocks project deletion and
|
||
keeps the deleted project's ID. `{}` is a known empty set. Null means
|
||
unknown and is allowed only on settled rows: migration `0428` filled it from
|
||
the charges of unsettled rows, but settled `0426` rows had already lost
|
||
their set. A retained provider session resumes with the previous set ∪ the
|
||
current references; an unknown previous set requires a fresh context, never
|
||
an empty or primary-only guess.
|
||
- `execution_admission_project_charges` stores one row per distinct admitted
|
||
project, including unlimited projects. An admission without projects has no
|
||
rows. Composite `(company_id, …)` foreign keys keep admissions and projects in
|
||
one company. Rows are the only active capacity and exist only while the
|
||
admission is unsettled: settlement deletes the whole set in one transaction,
|
||
and `(company_id, project_id)` is indexed for active counts, so lookups
|
||
scale with active work, not history.
|
||
|
||
Archiving a project changes no charge. Deleting a project with a charge returns
|
||
`409 project_execution_admission_pending`; the foreign key is the backstop.
|
||
Deleting an agent refuses unsettled admissions on its runs and removes settled
|
||
ones, with their sets, together with the runs. Company deletion removes the company's admissions before its runs and
|
||
projects.
|
||
|
||
The proposed global lock order for the admission transaction is: company row
|
||
(`FOR NO KEY UPDATE`), native finalization recovery owner, the whole project set
|
||
in ascending ID order (`FOR NO KEY UPDATE`, one statement), issues, wake
|
||
requests, the run, the external-admission advisory lock, the admission row and
|
||
its charges, then budget reservations. Policy updates take the same project
|
||
lock. Settlement takes no project lock. Only the project-set step is implemented
|
||
so far; claim paths keep their current locks until they are integrated.
|