Add bounded recovery file inventory before server cutover (#18347)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Accounting, credential recovery, and sandbox cleanup can retain
legacy files on an instance volume.
> - Replacing that volume while files remain can lose unresolved work.
> - There is no verified production backlog, so an empty installation
needs no migration.
> - The recovery owners must decide whether each record is valid and
settled.
> - This pull request adds a bounded file inventory and an optional
cutover gate.
> - The benefit is a safe check that preserves obligations without
reading secret material.

## Linked Issues or Issue Description

**Problem or motivation**

Operators need to check the actual affected instance directories before
replacing a web server. A skipped corrupt file or failed read must not
appear clear.

**Proposed solution**

Add a Node script that inspects only the three legacy recovery
directories for an explicitly chosen instance. Default mode reports
counts. The optional check mode blocks when files or uncertainty remain.

**Alternatives considered**

A bulk migration system has no established backlog to migrate.
Reimplementing the owner parsers could expose credentials and still
would not prove settlement. This check never reads file bytes.

**Roadmap alignment**

This is a small operational prerequisite for safe cloud server
replacement. It does not add a recovery backend or change resource
ownership.

Related recovery-owner changes: #18354 (accounting), #18352
(credentials), and #18357 (sandbox ownership). This inventory remains
independently usable with existing files.

## What Changed

- Add a pure classification policy and a bounded filesystem boundary.
- Add `node scripts/recovery-cutover.mjs --instance-root
/absolute/instance` with optional `--check`, directory override, and
entry budget.
- Report observed lower-bound counts and safe status summaries. Do not
print paths, filenames, identifiers, raw errors, or file contents.
- Add one dependency-free CLI acceptance suite and a Vitest bridge so
the same cases run in normal server CI.
- Document existing owner entry points, override selection, stopped
producers, and retained-volume requirements.

## Verification

- `node --test scripts/recovery-cutover.test.mjs`: 9 cases pass. The
suite starts the actual CLI with synthetic legacy directories.
- Pending receipts, nested credential files, orphan records, malformed
bytes, temporary files, unreadable files, unsafe paths, symlinks, read
failures, bounds, and configured overrides all block safely.
- Empty or absent recovery directories clear. Source file bytes remain
unchanged. Report-only mode and check-mode exit codes are tested.
- `pnpm exec vitest run server/src/__tests__/recovery-cutover.test.ts
--pool=forks --maxWorkers=1`: the normal CI bridge passes and runs the
same 9 cases.
- Cached frozen dependency installation, script syntax checks, and `git
diff --check` pass. The lockfile is unchanged.
- Independent source review found the read-only boundary sound.
- [Complete current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41097614666)
passed at `bf5cc7f067041c10cd0294ff26756678da943976`: 53 successful
checks and two intended skips, including typecheck, full tests, build,
and E2E. Greptile is 5/5 with no review threads.
- One failed-job retry was needed for an existing heartbeat timing
failure in server shard 4. The relevant heartbeat source and tests match
the base branch. Local diagnostic runs had 61 pass/2 fail and 7 pass/1
fail, with changing timing failures; neither run is claimed as green.
The bounded retry and all failed evidence are [recorded
here](https://github.com/paperclipai/paperclip/pull/18347#issuecomment-6529629804).
No heartbeat code or test was changed.

## Risks

- The scan is a point-in-time inventory. Stop producers for the final
gate. It is not an atomic drain certificate or protection against
hostile path mutation.
- All files remain unverified. The check cannot certify validity or
Postgres settlement. Existing owners retain that responsibility.
- The scan does not inspect other instances, volumes, Runner delivery,
or database obligations. It does not establish that production cutover
is ready.
- No migration, deletion, decryption, or replay command is added.
Symlink paths are rejected, including system aliases; use canonical
absolute paths.

## Model Used

- OpenAI `gpt-6.1-sol`, with reasoning, tool use, and code execution.
Context window size was not supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

scripts/lib/recovery-cutover-inventory.mjs

@@ -0,0 +1,99 @@
+import { promises as fs } from "node:fs";
+import path from "node:path";
+import { classifyRecoveryEntry, cutoverStatus, recoveryKinds } from "./recovery-cutover-policy.mjs";
+
+const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+
+// Inspect every ancestor: lstat on only the final component would follow an
+// intermediate symlink. No directory/file names or OS errors enter the report.
+async function directoryState(directory, io) {
+  const parsed = path.parse(directory);
+  let current = parsed.root;
+  for (const part of directory.slice(parsed.root.length).split(path.sep).filter(Boolean)) {
+    current = path.join(current, part);
+    try {
+      const stat = await io.lstat(current);
+      if (!stat.isDirectory() || stat.isSymbolicLink()) return "unsafe";
+    } catch (error) {
+      return error?.code === "ENOENT" ? "absent" : "unreadable";
+    }
+  }
+  return "directory";
+}
+
+/** Read-only, bounded entry inventory. Run with producers stopped on the retained
+ * volume. This is not an atomic snapshot and does not establish DB settlement.
+ * io exists for deterministic fault tests; production uses the actual filesystem. */
+export async function inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries = 10_000 }, io = fs) {
+  if (!path.isAbsolute(instanceRoot) || (orphanDirectory !== undefined && !path.isAbsolute(orphanDirectory))) {
+    throw new Error("Recovery inventory requires absolute directories");
+  }
+  if (!Number.isSafeInteger(maxEntries) || maxEntries < 1 || maxEntries > 100_000) {
+    throw new Error("Entry limit must be between 1 and 100000");
+  }
+  const root = path.resolve(instanceRoot);
+  const rootState = await directoryState(root, io);
+  const directories = [path.join(root, "accounting-receipts"), path.join(root, "quota-credential-recovery"),
+    orphanDirectory === undefined ? path.join(root, "data", "sandbox-orphan-cleanup") : path.resolve(orphanDirectory)];
+  let remaining = maxEntries;
+  const summaries = [];
+  for (const [index, directory] of directories.entries()) {
+    const summary = { kind: recoveryKinds[index], presence: "unknown", observedEntries: 0,
+      unverifiedFiles: 0, unknownEntries: 0, readFailures: 0, complete: true, status: "unknown" };
+    if (rootState !== "directory") {
+      summary.complete = false;
+    } else {
+      const state = await directoryState(directory, io);
+      if (state === "absent") summary.presence = "absent";
+      else if (state !== "directory") {
+        summary.unknownEntries++;
+        summary.complete = false;
+        if (state === "unreadable") summary.readFailures++;
+      } else {
+        summary.presence = "present";
+        async function scan(target, depth) {
+          let handle;
+          try {
+            handle = await io.opendir(target);
+            while (true) {
+              // Read one sentinel entry to distinguish exactly-at-limit from
+              // truncated scans. The sentinel is not included in observed counts.
+              const entry = await handle.read();
+              if (entry === null) break;
+              if (remaining === 0) { summary.complete = false; break; }
+              remaining--;
+              summary.observedEntries++;
+              const entryKind = entry.isFile() ? "file"
+                : entry.isDirectory() && uuid.test(entry.name) ? "company-directory" : "other";
+              const classification = classifyRecoveryEntry(summary.kind, depth, entryKind);
+              if (classification === "descend") {
+                const child = path.join(target, entry.name);
+                // Recheck the entry before traversing; never intentionally follow
+                // links. Operators must quiesce writers for the snapshot to hold.
+                if (await directoryState(child, io) !== "directory") {
+                  summary.unknownEntries++;
+                  summary.complete = false;
+                } else await scan(child, depth + 1);
+              } else summary[classification]++;
+              if (!summary.complete && remaining === 0) break;
+            }
+          } catch {
+            summary.readFailures++;
+            summary.complete = false;
+          } finally {
+            if (handle) {
+              try { await handle.close(); }
+              catch { summary.readFailures++; summary.complete = false; }
+            }
+          }
+        }
+        await scan(directory, 0);
+      }
+    }
+    summary.status = cutoverStatus(summary);
+    summaries.push(summary);
+  }
+  return { schema: "paperclip/recovery-cutover-inventory/v1", status:
+    summaries.some(s => s.status === "unknown") ? "unknown" : summaries.some(s => s.status === "blocked") ? "blocked" : "clear",
+    instanceStatus: rootState, contentInspection: "none", counts: "observed-lower-bounds", maxEntries, directories: summaries };
+}

scripts/lib/recovery-cutover-policy.mjs

@@ -0,0 +1,14 @@
+/** File contents belong to each recovery owner. Inventory never parses them:
+ * even an apparently valid file cannot establish that its obligation settled. */
+export const recoveryKinds = ["accounting-receipts", "quota-credential-recovery", "sandbox-orphan-cleanup"];
+
+export function classifyRecoveryEntry(kind, depth, entry) {
+  if (entry === "file") return "unverifiedFiles";
+  if (entry === "company-directory" && kind === "quota-credential-recovery" && depth === 0) return "descend";
+  return "unknownEntries";
+}
+
+export function cutoverStatus(summary) {
+  if (!summary.complete || summary.unknownEntries > 0 || summary.readFailures > 0) return "unknown";
+  return summary.unverifiedFiles > 0 ? "blocked" : "clear";
+}

scripts/recovery-cutover.mjs

@@ -0,0 +1,33 @@
+#!/usr/bin/env node
+import { inventoryRecoveryFiles } from "./lib/recovery-cutover-inventory.mjs";
+
+const usage = "Usage: node scripts/recovery-cutover.mjs --instance-root /absolute/instance [--orphan-directory /absolute/override] [--max-entries 10000] [--check]";
+const args = process.argv.slice(2);
+try {
+  let instanceRoot;
+  let orphanDirectory = process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR;
+  let maxEntries = 10_000;
+  let check = false;
+  const seen = new Set();
+  for (let i = 0; i < args.length; i++) {
+    const argument = args[i];
+    if (seen.has(argument)) throw new Error("duplicate");
+    seen.add(argument);
+    if (argument === "--check") check = true;
+    else if (["--instance-root", "--orphan-directory", "--max-entries"].includes(argument)) {
+      const value = args[++i];
+      if (value === undefined || value.startsWith("--")) throw new Error("missing");
+      if (argument === "--instance-root") instanceRoot = value;
+      else if (argument === "--orphan-directory") orphanDirectory = value;
+      else maxEntries = Number(value);
+    } else throw new Error("unknown");
+  }
+  if (!instanceRoot) throw new Error("missing");
+  const report = await inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries });
+  process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
+  if (check && report.status !== "clear") process.exitCode = 1;
+} catch {
+  // Never echo args, filesystem errors or paths: they may contain secrets.
+  process.stderr.write(`Recovery inventory could not run. ${usage}\n`);
+  process.exitCode = 2;
+}

scripts/recovery-cutover.test.mjs

@@ -0,0 +1,163 @@
+import { afterEach, test } from "node:test";
+import assert from "node:assert/strict";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { spawnSync } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import { inventoryRecoveryFiles } from "./lib/recovery-cutover-inventory.mjs";
+import { classifyRecoveryEntry, cutoverStatus } from "./lib/recovery-cutover-policy.mjs";
+
+const cli = fileURLToPath(new URL("./recovery-cutover.mjs", import.meta.url));
+const roots = [];
+const company = "00000000-0000-4000-8000-000000000001";
+const grant = "00000000-0000-4000-8000-000000000002";
+async function fixture() {
+  // Canonicalize macOS /var -> /private/var, since the check rejects links.
+  const root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "cutover-")));
+  roots.push(root);
+  return root;
+}
+async function write(root, relative, content = "secret-token-must-never-appear") {
+  const file = path.join(root, relative);
+  await fs.mkdir(path.dirname(file), { recursive: true });
+  await fs.writeFile(file, content);
+  return file;
+}
+function run(root, args = [], env = {}) {
+  return spawnSync(process.execPath, [cli, "--instance-root", root, ...args], {
+    encoding: "utf8", env: { ...process.env, SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: "", ...env },
+  });
+}
+// An unset override differs from an invalid empty override.
+function cleanRun(root, args = []) {
+  const env = { ...process.env };
+  delete env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR;
+  return spawnSync(process.execPath, [cli, "--instance-root", root, ...args], { encoding: "utf8", env });
+}
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); });
+
+test("pure policy never clears files or unknown entries", () => {
+  assert.equal(classifyRecoveryEntry("accounting-receipts", 0, "file"), "unverifiedFiles");
+  assert.equal(classifyRecoveryEntry("quota-credential-recovery", 0, "company-directory"), "descend");
+  assert.equal(classifyRecoveryEntry("quota-credential-recovery", 1, "company-directory"), "unknownEntries");
+  assert.equal(cutoverStatus({ complete: false, unknownEntries: 0, readFailures: 0, unverifiedFiles: 0 }), "unknown");
+});
+
+test("CLI: absent and empty legacy layouts clear without migration", async () => {
+  const root = await fixture();
+  let result = cleanRun(root, ["--check"]);
+  assert.equal(result.status, 0);
+  assert.equal(JSON.parse(result.stdout).status, "clear");
+  for (const directory of ["accounting-receipts", `quota-credential-recovery/${company}`, "data/sandbox-orphan-cleanup"]) {
+    await fs.mkdir(path.join(root, directory), { recursive: true });
+  }
+  result = cleanRun(root, ["--check"]);
+  assert.equal(result.status, 0);
+  assert.equal(JSON.parse(result.stdout).status, "clear");
+});
+
+test("CLI: real pending legacy files block, preserve bytes and reveal no secrets or names", async () => {
+  const root = await fixture();
+  const receipt = JSON.stringify({ schema: "paperclip/accounting-receipt/v1", id: grant, companyId: company,
+    runId: company, sourceId: company, sequence: 1, receivedAt: "2026-10-30T00:00:00.000Z", adapterType: "codex_local",
+    receipt: { complete: true, usage: { inputTokens: 1, outputTokens: 1 } } });
+  const files = await Promise.all([
+    write(root, `accounting-receipts/${grant}.json`, receipt),
+    write(root, `quota-credential-recovery/${company}/${grant}.json`, '{"ciphertext":"secret-token-must-never-appear"}'),
+    write(root, "data/sandbox-orphan-cleanup/secret-filename.json", JSON.stringify({ companyId: company,
+      environmentId: company, executionWorkspaceId: null, issueId: null, heartbeatRunId: null, provider: "daytona",
+      providerLeaseId: "private-resource", metadata: {} })),
+  ]);
+  const before = await Promise.all(files.map(file => fs.readFile(file)));
+  const result = cleanRun(root, ["--check"]);
+  assert.equal(result.status, 1);
+  const report = JSON.parse(result.stdout);
+  assert.equal(report.status, "blocked");
+  assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [1, 1, 1]);
+  assert.doesNotMatch(result.stdout + result.stderr, /secret-token|secret-filename|private-resource|ciphertext|00000000-/);
+  assert.deepEqual(await Promise.all(files.map(file => fs.readFile(file))), before);
+  assert.equal(cleanRun(root).status, 0, "report-only mode does not fail on obligations");
+});
+
+test("truncated, temporary, and unreadable files are unverified, never clear", async () => {
+  const root = await fixture();
+  await write(root, "accounting-receipts/truncated.json", '{"schema":');
+  const file = await write(root, `quota-credential-recovery/${company}/${grant}.json.probe`);
+  await fs.chmod(file, 0);
+  const report = await inventoryRecoveryFiles({ instanceRoot: root });
+  assert.equal(report.status, "blocked");
+  assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [1, 1, 0]);
+});
+
+test("unknown nesting and symlinks block without traversing outside the chosen roots", async () => {
+  const root = await fixture();
+  const outside = await fixture();
+  await write(outside, "do-not-read.json");
+  await fs.mkdir(path.join(root, "accounting-receipts"));
+  await fs.symlink(outside, path.join(root, "accounting-receipts", "link"));
+  await fs.mkdir(path.join(root, "quota-credential-recovery", company, "unexpected"), { recursive: true });
+  const report = await inventoryRecoveryFiles({ instanceRoot: root });
+  assert.equal(report.status, "unknown");
+  assert.deepEqual(report.directories.map(s => s.unknownEntries), [1, 1, 0]);
+  assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [0, 0, 0]);
+  await fs.symlink(outside, path.join(root, "data"));
+  assert.equal((await inventoryRecoveryFiles({ instanceRoot: root })).directories[2].status, "unknown");
+  await fs.symlink(outside, path.join(root, "instance-link"));
+  assert.equal((await inventoryRecoveryFiles({ instanceRoot: path.join(root, "instance-link") })).status, "unknown");
+});
+
+test("entry budget gives incomplete lower-bound counts and never clears omitted work", async () => {
+  const root = await fixture();
+  for (let i = 0; i < 4; i++) await write(root, `accounting-receipts/${i}.json`);
+  const report = await inventoryRecoveryFiles({ instanceRoot: root, maxEntries: 2 });
+  assert.equal(report.status, "unknown");
+  assert.equal(report.directories[0].observedEntries, 2);
+  assert.equal(report.directories[0].complete, false);
+  assert.equal(report.directories[0].unverifiedFiles, 2);
+  const exact = await inventoryRecoveryFiles({ instanceRoot: root, maxEntries: 4 });
+  assert.equal(exact.directories[0].complete, true);
+});
+
+test("CLI respects configured orphan override and explicit override precedence", async () => {
+  const root = await fixture();
+  const override = await fixture();
+  await write(override, "pending.json");
+  let result = run(root, ["--check"], { SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: override });
+  assert.equal(result.status, 1);
+  assert.equal(JSON.parse(result.stdout).directories[2].unverifiedFiles, 1);
+  const emptyOverride = await fixture();
+  result = run(root, ["--check", "--orphan-directory", emptyOverride], { SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: override });
+  assert.equal(result.status, 0);
+  assert.equal(JSON.parse(result.stdout).status, "clear");
+});
+
+test("fault boundary reports stat, directory read and close failures without error data", async () => {
+  const root = await fixture();
+  await fs.mkdir(path.join(root, "accounting-receipts"));
+  for (const operation of ["lstat", "opendir", "read", "close"]) {
+    const sensitiveError = Object.assign(new Error("secret-token-must-never-appear"), { code: "EACCES" });
+    const io = { ...fs };
+    if (operation === "lstat" || operation === "opendir") io[operation] = async () => { throw sensitiveError; };
+    else io.opendir = async () => ({
+      read: async () => { if (operation === "read") throw sensitiveError; return null; },
+      close: async () => { if (operation === "close") throw sensitiveError; },
+    });
+    const report = await inventoryRecoveryFiles({ instanceRoot: root }, io);
+    assert.equal(report.status, "unknown");
+    if (operation === "lstat") assert.equal(report.instanceStatus, "unreadable");
+    else assert.ok(report.directories[0].readFailures > 0);
+    assert.doesNotMatch(JSON.stringify(report), /secret-token|EACCES/);
+  }
+});
+
+test("CLI fails closed for missing instance, unsafe roots and bad arguments", async () => {
+  const root = await fixture();
+  assert.equal(cleanRun(path.join(root, "absent"), ["--check"]).status, 1);
+  for (const args of [["--max-entries", "0"], ["--max-entries", "100001"], ["--max-entries", "NaN"],
+    ["--orphan-directory", "relative-secret"], ["--bad-secret"], ["--check", "--check"]]) {
+    const result = cleanRun(root, args);
+    assert.equal(result.status, 2);
+    assert.doesNotMatch(result.stdout + result.stderr, /relative-secret|bad-secret/);
+  }
+});