DottaandPaperclip
76fde891cc
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accounting, credential recovery, and sandbox cleanup can retain legacy files on an instance volume. > - Replacing that volume while files remain can lose unresolved work. > - There is no verified production backlog, so an empty installation needs no migration. > - The recovery owners must decide whether each record is valid and settled. > - This pull request adds a bounded file inventory and an optional cutover gate. > - The benefit is a safe check that preserves obligations without reading secret material. ## Linked Issues or Issue Description **Problem or motivation** Operators need to check the actual affected instance directories before replacing a web server. A skipped corrupt file or failed read must not appear clear. **Proposed solution** Add a Node script that inspects only the three legacy recovery directories for an explicitly chosen instance. Default mode reports counts. The optional check mode blocks when files or uncertainty remain. **Alternatives considered** A bulk migration system has no established backlog to migrate. Reimplementing the owner parsers could expose credentials and still would not prove settlement. This check never reads file bytes. **Roadmap alignment** This is a small operational prerequisite for safe cloud server replacement. It does not add a recovery backend or change resource ownership. Related recovery-owner changes: #18354 (accounting), #18352 (credentials), and #18357 (sandbox ownership). This inventory remains independently usable with existing files. ## What Changed - Add a pure classification policy and a bounded filesystem boundary. - Add `node scripts/recovery-cutover.mjs --instance-root /absolute/instance` with optional `--check`, directory override, and entry budget. - Report observed lower-bound counts and safe status summaries. Do not print paths, filenames, identifiers, raw errors, or file contents. - Add one dependency-free CLI acceptance suite and a Vitest bridge so the same cases run in normal server CI. - Document existing owner entry points, override selection, stopped producers, and retained-volume requirements. ## Verification - `node --test scripts/recovery-cutover.test.mjs`: 9 cases pass. The suite starts the actual CLI with synthetic legacy directories. - Pending receipts, nested credential files, orphan records, malformed bytes, temporary files, unreadable files, unsafe paths, symlinks, read failures, bounds, and configured overrides all block safely. - Empty or absent recovery directories clear. Source file bytes remain unchanged. Report-only mode and check-mode exit codes are tested. - `pnpm exec vitest run server/src/__tests__/recovery-cutover.test.ts --pool=forks --maxWorkers=1`: the normal CI bridge passes and runs the same 9 cases. - Cached frozen dependency installation, script syntax checks, and `git diff --check` pass. The lockfile is unchanged. - Independent source review found the read-only boundary sound. - [Complete current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41097614666) passed at `bf5cc7f067041c10cd0294ff26756678da943976`: 53 successful checks and two intended skips, including typecheck, full tests, build, and E2E. Greptile is 5/5 with no review threads. - One failed-job retry was needed for an existing heartbeat timing failure in server shard 4. The relevant heartbeat source and tests match the base branch. Local diagnostic runs had 61 pass/2 fail and 7 pass/1 fail, with changing timing failures; neither run is claimed as green. The bounded retry and all failed evidence are [recorded here](https://github.com/paperclipai/paperclip/pull/18347#issuecomment-6529629804). No heartbeat code or test was changed. ## Risks - The scan is a point-in-time inventory. Stop producers for the final gate. It is not an atomic drain certificate or protection against hostile path mutation. - All files remain unverified. The check cannot certify validity or Postgres settlement. Existing owners retain that responsibility. - The scan does not inspect other instances, volumes, Runner delivery, or database obligations. It does not establish that production cutover is ready. - No migration, deletion, decryption, or replay command is added. Symlink paths are rejected, including system aliases; use canonical absolute paths. ## Model Used - OpenAI `gpt-6.1-sol`, with reasoning, tool use, and code execution. Context window size was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6.2 KiB
Check existing recovery files before web-server replacement
This check covers only observed legacy files on one explicitly selected instance. There is no verified production backlog. An empty installation needs no migration. This check does not inspect Postgres obligations, Runner delivery, other volumes, other instances, or the rest of web-server replacement readiness.
Use the retained volume and the instance root from the old server configuration.
The instance root is the directory that contains accounting-receipts/ and
quota-credential-recovery/. It is not the parent directory of all instances.
Use canonical absolute paths. The check rejects symlink components, including
system aliases such as /var on macOS; use the actual /private/var path there.
The chosen instance directory must exist. Missing recovery directories are normal.
Run this command from a checkout that includes the check. Node is the only runtime dependency. It needs no database connection, master key, or package installation.
node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance
The default command reports JSON and exits successfully even when work remains.
For an automation gate, add --check:
node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance --check
| Exit code | Meaning |
|---|---|
0 |
Report produced; with --check, all three scans are clear |
1 |
With --check, unresolved files or an unknown/incomplete scan block cutover |
2 |
Invalid arguments or the inventory could not run |
The sandbox directory defaults to data/sandbox-orphan-cleanup/ below the chosen
instance. If the old server uses SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR, supply that same
environment value when running the check. An explicit --orphan-directory takes
precedence. A relative or empty override is rejected; supply its canonical absolute
path for this check. An override is an additional explicitly selected root and can
be outside the instance directory. The check does not search for other overrides.
node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance \
--orphan-directory /absolute/retained/orphans --check
The report gives one summary for each of the three recovery kinds. It never
prints filenames, input paths, file contents, identifiers, token values, decrypted
material, or raw filesystem errors. unverifiedFiles includes every regular file,
including JSON, temporary, probe, truncated, corrupt, and unreadable files. The
check does not read file bytes and cannot claim that a file is valid or settled.
That decision belongs to the existing recovery owner and its format definitions.
Only the credential directory's immediate UUID company directories are traversed;
empty company directories are normal after recovery. Accounting and orphan files
use flat directories. Unexpected directories, deeper nesting, symlinks, and
special entries count as unknownEntries and block. The check does not follow
symlinks or inspect unexpected subtrees. Directory/stat/close failures increment
readFailures or leave the scan incomplete. An invalid or missing instance root
also blocks; it is not an empty installation.
The default shared budget is 10,000 observed entries across all three scans,
including company directories. --max-entries accepts 1 through 100,000. At the
limit the scanner reads at most one extra entry per open directory to detect
truncation. An unfinished scan reports complete: false and status unknown.
Counts are observed lower bounds, not total records or total pending operations.
A complete scan with any files reports blocked; any uncertainty reports
unknown. Only a complete empty scan reports clear.
For the final cutover gate, stop the old producers first and retain the old volume. Run the check after the relevant recovery owners finish their work and producers are stopped again. A live scan is a point-in-time inventory, not an atomic drain certificate. Concurrent renames or writes invalidate the snapshot; the tool does not lock writers or protect against a hostile process that changes paths during inspection. Do not use it to certify a live writable or untrusted volume.
If files exist, keep the old volume and let the existing owners settle actual entries with their existing permissions and database checks:
- Accounting:
run-cost-accounting.tsuses theusage-receipts.tsreplay and per-run recovery paths. The receipt owner validates the accounting envelope and handles database conflicts and settlement. - Credentials:
ai-connections.tsandai-connection-runtime.tsusequota-credential-recovery.tsduring account credential access and refresh. The owner validates encrypted material with the original instance master key and preserves grant, company and credential version fences. - Orphan cleanup:
environment-runtime.tsexposesflushDeferredOrphanCleanupsto the existing cleanup lifecycle.sandbox-orphan-cleanup-spool.tsowns the record format and removes a file only after the recovery path saves ownership.
These are service entry points, not standalone operator commands. This change adds no import, deletion, replay, decryption, or bulk migration command. Do not trigger new paid runs or token exchanges merely to empty the report. If an owner cannot settle a record, preserve it and record an explicit reconciliation outcome through the existing incident/operations process. Corrupt files need owner review; they do not disappear from the gate because a recovery reader skipped them. Never delete unresolved files to obtain a clear result. An approved exception requires separate evidence; this tool continues to block while the files remain. Retain the volume and required secrets through rollback compatibility review.
Verify the check without any real instance data:
node --test scripts/recovery-cutover.test.mjs
The tests use disposable synthetic directories and spawn the actual CLI. They cover pending legacy layouts, malformed bytes, temporary files, empty directories, symlinks, overrides, scan limits, safe diagnostics, failure injection, exit codes, and preservation of source bytes. They establish file inventory behavior only. They do not establish that any production volume is clear or that domain recovery has completed.