From ac920fad12950b921487d52b42d49f481bb90bbc Mon Sep 17 00:00:00 2001 From: Dotta Date: Sat, 9 May 2026 15:16:19 -0500 Subject: [PATCH] Update local-space docs for canonical spaces/default layout PAP-9151. Reflects the resolver and migration command shipped in PAP-9149: docs now describe the instance root as cross-space metadata, the active space as the home for db/data/logs/secrets/workspaces, and the paperclipai spaces migrate-default command for legacy root-shaped installs. Co-Authored-By: Paperclip --- doc/CLI.md | 78 ++++++++++++++++++++++++++++++++++---- doc/DATABASE.md | 10 ++--- doc/DEVELOPING.md | 55 ++++++++++++++++++++------- doc/SPEC-implementation.md | 6 +-- 4 files changed, 121 insertions(+), 28 deletions(-) diff --git a/doc/CLI.md b/doc/CLI.md index 9f2aba86450..e854a522d9e 100644 --- a/doc/CLI.md +++ b/doc/CLI.md @@ -178,20 +178,84 @@ pnpm paperclipai heartbeat run --agent-id [--api-base http://localhos ## Local Storage Defaults -Default local instance root is `~/.paperclip/instances/default`: +Local Paperclip data is split between an instance root and one or more spaces inside it. The default install creates a single space called `default` under `spaces/default`. -- config: `~/.paperclip/instances/default/config.json` -- embedded db: `~/.paperclip/instances/default/db` -- logs: `~/.paperclip/instances/default/logs` -- storage: `~/.paperclip/instances/default/data/storage` -- secrets key: `~/.paperclip/instances/default/secrets/master.key` +```text +~/.paperclip/ # PAPERCLIP_HOME +└── instances/ + └── default/ # instance root (PAPERCLIP_INSTANCE_ID) + ├── config.json # space registry (activeSpaceId, known spaces) + └── spaces/ + └── default/ # active space root (PAPERCLIP_SPACE_ID) + ├── config.json # runtime config for this space + ├── .env # space-scoped env file + ├── db/ # embedded PostgreSQL data + ├── data/ + │ ├── storage/ # local_disk uploads + │ └── backups/ # automatic DB backups + ├── logs/ + ├── secrets/ + │ └── master.key # local_encrypted master key + ├── workspaces/ # default agent workspaces + ├── projects/ # project execution workspaces + ├── companies/ # per-company adapter homes (e.g. codex-home) + └── codex-home/ # per-instance codex home (when not company-scoped) +``` -Override base home or instance with env vars: +Default paths for the canonical install: + +- instance registry: `~/.paperclip/instances/default/config.json` +- space config: `~/.paperclip/instances/default/spaces/default/config.json` +- embedded db: `~/.paperclip/instances/default/spaces/default/db` +- logs: `~/.paperclip/instances/default/spaces/default/logs` +- storage: `~/.paperclip/instances/default/spaces/default/data/storage` +- secrets key: `~/.paperclip/instances/default/spaces/default/secrets/master.key` + +The instance root holds only cross-space metadata. All runtime state — database, storage, secrets, logs, agent workspaces — lives inside the active space. + +Override base home, instance, or active space with env vars: ```sh PAPERCLIP_HOME=/custom/home PAPERCLIP_INSTANCE_ID=dev pnpm paperclipai run +PAPERCLIP_SPACE_ID=staging pnpm paperclipai run # use an alternate space inside the same instance ``` +`PAPERCLIP_SPACE_ID` overrides whatever `activeSpaceId` is recorded in the instance registry. When unset, Paperclip resolves the active space from `instances//config.json` and falls back to `default`. + +## Migrating a Legacy Default-Space Install + +Earlier Paperclip versions stored default-space data directly at the instance root (for example `~/.paperclip/instances/default/db`). Those installs continue to start through a compatibility resolver, but new code paths assume `spaces/default`. Migrate explicitly with: + +```sh +# Stop Paperclip first; the migration refuses to run while a local server is up. +pnpm paperclipai spaces migrate-default + +# Preview the move without touching files. +pnpm paperclipai spaces migrate-default --dry-run + +# Operate on a non-default instance. +pnpm paperclipai spaces migrate-default --instance dev +``` + +The command: + +- detects a legacy root-shaped install by inspecting `instances//config.json` +- preflight-checks `http://:/api/health` and refuses to run when the server responds +- refuses to merge if the destination already contains conflicting paths under `spaces/default/` +- moves only known space-owned paths: `config.json`, `.env`, `db`, `data`, `logs`, `secrets`, `workspaces`, `projects`, `companies`, and the legacy top-level `codex-home` +- rewrites absolute paths inside the migrated `config.json` from the instance root to the new space root (`embeddedPostgresDataDir`, backup `dir`, `logging.logDir`, `storage.localDisk.baseDir`, `secrets.localEncrypted.keyFilePath`) +- writes the registry/marker back at `instances//config.json` recording `activeSpaceId`, the migration timestamp, and the moved paths + +Available flags: + +| Option | Description | +|---|---| +| `-i, --instance ` | Local instance id (default: `default`) | +| `--dry-run` | Show the migration plan without moving files | +| `--skip-server-check` | Skip the `/api/health` preflight (only for offline installs) | + +Restart Paperclip after a successful migration. Subsequent commands resolve the active space root from the new registry automatically. + ## Storage Configuration Configure storage provider and settings: diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 23abd32d415..cad776c3dc3 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -12,12 +12,12 @@ pnpm dev That's it. On first start the server: -1. Creates a `~/.paperclip/instances/default/db/` directory for storage +1. Creates a `~/.paperclip/instances/default/spaces/default/db/` directory for storage 2. Ensures the `paperclip` database exists 3. Runs migrations automatically for empty databases 4. Starts serving requests -Data persists across restarts in `~/.paperclip/instances/default/db/`. To reset local dev data, delete that directory. +Data persists across restarts in `~/.paperclip/instances/default/spaces/default/db/`. To reset local dev data, delete that directory. Older installs that still keep data at `~/.paperclip/instances/default/db/` continue to start through the legacy resolver — see [Migrating a Legacy Default-Space Install](CLI.md#migrating-a-legacy-default-space-install). If you need to apply pending migrations manually, run: @@ -137,7 +137,7 @@ The database mode is controlled by `DATABASE_URL`: | `DATABASE_URL` | Mode | |---|---| -| Not set | Embedded PostgreSQL (`~/.paperclip/instances/default/db/`) | +| Not set | Embedded PostgreSQL (`~/.paperclip/instances/default/spaces/default/db/`) | | `postgres://...localhost...` | Local Docker PostgreSQL | | `postgres://...supabase.com...` | Hosted Supabase | @@ -169,8 +169,8 @@ Paperclip stores secret metadata and versions in: For local/default installs, the active provider is `local_encrypted`: - Secret material is encrypted at rest with a local master key. -- Default key file: `~/.paperclip/instances/default/secrets/master.key` (auto-created if missing). -- CLI config location: `~/.paperclip/instances/default/config.json` under `secrets.localEncrypted.keyFilePath`. +- Default key file: `~/.paperclip/instances/default/spaces/default/secrets/master.key` (auto-created if missing). +- CLI config location: `~/.paperclip/instances/default/spaces/default/config.json` under `secrets.localEncrypted.keyFilePath`. (The instance-level `~/.paperclip/instances/default/config.json` is the cross-space registry that records which space is active.) Optional overrides: diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index d95bb04dd78..8bd759290af 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -157,17 +157,44 @@ See `doc/DOCKER.md` for API key wiring (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY`) For a separate review-oriented container that keeps `codex`/`claude` login state in Docker volumes and checks out PRs into an isolated scratch workspace, see `doc/UNTRUSTED-PR-REVIEW.md`. +## Local Instance Layout + +Every local install nests instance metadata, the active space, and runtime state under `~/.paperclip/`: + +```text +~/.paperclip/instances/default/ # instance root (cross-space metadata only) + config.json # space registry (activeSpaceId, known spaces) + spaces/ + default/ # active space root + config.json # space runtime config + .env # space-scoped env file + db/ # embedded PostgreSQL data + data/ + storage/ # local_disk uploads + backups/ # automatic DB backups + logs/ + secrets/master.key # local_encrypted master key + workspaces// # default agent workspaces + projects/ # project execution workspaces + companies//codex-home/ # per-company codex_local home +``` + +The instance root no longer doubles as the default space. New installs created by `paperclipai onboard`, `paperclipai run`, and `paperclipai configure` always write the registry at `instances//config.json` and put runtime data under `spaces/default/`. To migrate an older root-shaped install in place, see [Migrating a Legacy Default-Space Install](CLI.md#migrating-a-legacy-default-space-install). + +`PAPERCLIP_HOME`, `PAPERCLIP_INSTANCE_ID`, and `PAPERCLIP_SPACE_ID` override the home root, instance id, and active space respectively. `paperclipai onboard` echoes the resolved values in its banner (`Local home: | instance: | space: | config: `) so you can confirm where state will land before continuing. + ## Database in Dev (Auto-Handled) For local development, leave `DATABASE_URL` unset. The server will automatically use embedded PostgreSQL and persist data at: -- `~/.paperclip/instances/default/db` +- `~/.paperclip/instances/default/spaces/default/db` -Override home and instance: +Override home, instance, or space: ```sh PAPERCLIP_HOME=/custom/path PAPERCLIP_INSTANCE_ID=dev pnpm paperclipai run +PAPERCLIP_SPACE_ID=staging pnpm paperclipai run ``` No Docker or external database is required for this mode. @@ -176,7 +203,7 @@ No Docker or external database is required for this mode. For local development, the default storage provider is `local_disk`, which persists uploaded images/attachments at: -- `~/.paperclip/instances/default/data/storage` +- `~/.paperclip/instances/default/spaces/default/data/storage` Configure storage provider/settings: @@ -186,15 +213,15 @@ pnpm paperclipai configure --section storage ## Default Agent Workspaces -When a local agent run has no resolved project/session workspace, Paperclip falls back to an agent home workspace under the instance root: +When a local agent run has no resolved project/session workspace, Paperclip falls back to an agent home workspace under the active space: -- `~/.paperclip/instances/default/workspaces/` +- `~/.paperclip/instances/default/spaces/default/workspaces/` -This path honors `PAPERCLIP_HOME` and `PAPERCLIP_INSTANCE_ID` in non-default setups. +This path honors `PAPERCLIP_HOME`, `PAPERCLIP_INSTANCE_ID`, and `PAPERCLIP_SPACE_ID` in non-default setups. -For `codex_local`, Paperclip also manages a per-company Codex home under the instance root and seeds it from the shared Codex login/config home (`$CODEX_HOME` or `~/.codex`): +For `codex_local`, Paperclip also manages a per-company Codex home under the active space and seeds it from the shared Codex login/config home (`$CODEX_HOME` or `~/.codex`): -- `~/.paperclip/instances/default/companies//codex-home` +- `~/.paperclip/instances/default/spaces/default/companies//codex-home` If the `codex` CLI is not installed or not on `PATH`, `codex_local` agent runs fail at execution time with a clear adapter error. Quota polling uses a short-lived `codex app-server` subprocess: when `codex` cannot be spawned, that provider reports `ok: false` in aggregated quota results and the API server keeps running (it must not exit on a missing binary). @@ -280,13 +307,13 @@ paperclipai worktree init --from-data-dir ~/.paperclip paperclipai worktree init --force ``` -Repair an already-created repo-managed worktree and reseed its isolated instance from the main default install: +Repair an already-created repo-managed worktree and reseed its isolated instance from the main default install. Point `--from-config` at the active space config (`spaces/default/config.json` for post-migration installs; the legacy `~/.paperclip/instances/default/config.json` still works on root-shaped installs that haven't run `paperclipai spaces migrate-default` yet): ```sh cd /path/to/paperclip/.paperclip/worktrees/PAP-884-ai-commits-component pnpm paperclipai worktree init --force --seed-mode minimal \ --name PAP-884-ai-commits-component \ - --from-config ~/.paperclip/instances/default/config.json + --from-config ~/.paperclip/instances/default/spaces/default/config.json ``` That rewrites the worktree-local `.paperclip/config.json` + `.paperclip/.env`, recreates the isolated instance under `~/.paperclip-worktrees/instances//`, and preserves the git worktree contents themselves. @@ -411,10 +438,12 @@ Expected: To wipe local dev data and start fresh: ```sh -rm -rf ~/.paperclip/instances/default/db +rm -rf ~/.paperclip/instances/default/spaces/default/db pnpm dev ``` +If you migrated from a legacy root-shaped install and still have a stale `~/.paperclip/instances/default/db`, remove that too — but only after confirming `paperclipai spaces migrate-default` ran and the active database lives under `spaces/default/`. + ## Optional: Use External Postgres If you set `DATABASE_URL`, the server will use that instead of embedded PostgreSQL. @@ -428,7 +457,7 @@ schemas. Defaults: - enabled - every 60 minutes - retain 30 days -- backup dir: `~/.paperclip/instances/default/data/backups` +- backup dir: `~/.paperclip/instances/default/spaces/default/data/backups` Configure these in: @@ -459,7 +488,7 @@ those providers are enabled. Agent env vars now support secret references. By default, secret values are stored with local encryption and only secret refs are persisted in agent config. -- Default local key path: `~/.paperclip/instances/default/secrets/master.key` +- Default local key path: `~/.paperclip/instances/default/spaces/default/secrets/master.key` - Override key material directly: `PAPERCLIP_SECRETS_MASTER_KEY` - Override key file path: `PAPERCLIP_SECRETS_MASTER_KEY_FILE` diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 22075e0453a..8ae2df78d66 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -93,11 +93,11 @@ V1 implementation extends this baseline into a company-centric, governance-aware ## 6.2 Data Stores - Primary: PostgreSQL -- Local default: embedded PostgreSQL at `~/.paperclip/instances/default/db` +- Local default: embedded PostgreSQL at `~/.paperclip/instances/default/spaces/default/db` - Optional local prod-like: Docker Postgres - Optional hosted: Supabase/Postgres-compatible - File/object storage: - - local default: `~/.paperclip/instances/default/data/storage` (`local_disk`) + - local default: `~/.paperclip/instances/default/spaces/default/data/storage` (`local_disk`) - cloud: S3-compatible object storage (`s3`) ## 6.3 Background Processing @@ -785,7 +785,7 @@ Required UX behaviors: - Node 20+ - `DATABASE_URL` optional -- if unset, auto-use embedded PostgreSQL under `~/.paperclip/instances/default/db` +- if unset, auto-use embedded PostgreSQL under `~/.paperclip/instances/default/spaces/default/db` ## 15.2 Migrations