diff --git a/doc/recovery-file-cutover.md b/doc/recovery-file-cutover.md new file mode 100644 index 00000000000..225a81da1bd --- /dev/null +++ b/doc/recovery-file-cutover.md @@ -0,0 +1,112 @@ +# Check existing recovery files before web-server replacement + +This check covers only observed legacy files on one explicitly selected instance. +There is no verified production backlog. An empty installation needs no migration. +This check does not inspect Postgres obligations, Runner delivery, other volumes, +other instances, or the rest of web-server replacement readiness. + +Use the retained volume and the instance root from the old server configuration. +The instance root is the directory that contains `accounting-receipts/` and +`quota-credential-recovery/`. It is not the parent directory of all instances. +Use canonical absolute paths. The check rejects symlink components, including +system aliases such as `/var` on macOS; use the actual `/private/var` path there. +The chosen instance directory must exist. Missing recovery directories are normal. + +Run this command from a checkout that includes the check. Node is the only runtime +dependency. It needs no database connection, master key, or package installation. + +```sh +node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance +``` + +The default command reports JSON and exits successfully even when work remains. +For an automation gate, add `--check`: + +```sh +node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance --check +``` + +| Exit code | Meaning | +| --- | --- | +| `0` | Report produced; with `--check`, all three scans are clear | +| `1` | With `--check`, unresolved files or an unknown/incomplete scan block cutover | +| `2` | Invalid arguments or the inventory could not run | + +The sandbox directory defaults to `data/sandbox-orphan-cleanup/` below the chosen +instance. If the old server uses `SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR`, supply that same +environment value when running the check. An explicit `--orphan-directory` takes +precedence. A relative or empty override is rejected; supply its canonical absolute +path for this check. An override is an additional explicitly selected root and can +be outside the instance directory. The check does not search for other overrides. + +```sh +node scripts/recovery-cutover.mjs --instance-root /absolute/retained/instance \ + --orphan-directory /absolute/retained/orphans --check +``` + +The report gives one summary for each of the three recovery kinds. It never +prints filenames, input paths, file contents, identifiers, token values, decrypted +material, or raw filesystem errors. `unverifiedFiles` includes every regular file, +including JSON, temporary, probe, truncated, corrupt, and unreadable files. The +check does not read file bytes and cannot claim that a file is valid or settled. +That decision belongs to the existing recovery owner and its format definitions. + +Only the credential directory's immediate UUID company directories are traversed; +empty company directories are normal after recovery. Accounting and orphan files +use flat directories. Unexpected directories, deeper nesting, symlinks, and +special entries count as `unknownEntries` and block. The check does not follow +symlinks or inspect unexpected subtrees. Directory/stat/close failures increment +`readFailures` or leave the scan incomplete. An invalid or missing instance root +also blocks; it is not an empty installation. + +The default shared budget is 10,000 observed entries across all three scans, +including company directories. `--max-entries` accepts 1 through 100,000. At the +limit the scanner reads at most one extra entry per open directory to detect +truncation. An unfinished scan reports `complete: false` and status `unknown`. +Counts are observed lower bounds, not total records or total pending operations. +A complete scan with any files reports `blocked`; any uncertainty reports +`unknown`. Only a complete empty scan reports `clear`. + +For the final cutover gate, stop the old producers first and retain the old volume. +Run the check after the relevant recovery owners finish their work and producers +are stopped again. A live scan is a point-in-time inventory, not an atomic drain +certificate. Concurrent renames or writes invalidate the snapshot; the tool does +not lock writers or protect against a hostile process that changes paths during +inspection. Do not use it to certify a live writable or untrusted volume. + +If files exist, keep the old volume and let the existing owners settle actual +entries with their existing permissions and database checks: + +- Accounting: `run-cost-accounting.ts` uses the `usage-receipts.ts` replay and + per-run recovery paths. The receipt owner validates the accounting envelope and + handles database conflicts and settlement. +- Credentials: `ai-connections.ts` and `ai-connection-runtime.ts` use + `quota-credential-recovery.ts` during account credential access and refresh. + The owner validates encrypted material with the original instance master key + and preserves grant, company and credential version fences. +- Orphan cleanup: `environment-runtime.ts` exposes `flushDeferredOrphanCleanups` + to the existing cleanup lifecycle. `sandbox-orphan-cleanup-spool.ts` owns the + record format and removes a file only after the recovery path saves ownership. + +These are service entry points, not standalone operator commands. This change +adds no import, deletion, replay, decryption, or bulk migration command. Do not +trigger new paid runs or token exchanges merely to empty the report. If an owner +cannot settle a record, preserve it and record an explicit reconciliation outcome +through the existing incident/operations process. Corrupt files need owner review; +they do not disappear from the gate because a recovery reader skipped them. +Never delete unresolved files to obtain a clear result. An approved exception +requires separate evidence; this tool continues to block while the files remain. +Retain the volume and required secrets through rollback compatibility review. + +Verify the check without any real instance data: + +```sh +node --test scripts/recovery-cutover.test.mjs +``` + +The tests use disposable synthetic directories and spawn the actual CLI. They +cover pending legacy layouts, malformed bytes, temporary files, empty directories, +symlinks, overrides, scan limits, safe diagnostics, failure injection, exit codes, +and preservation of source bytes. They establish file inventory behavior only. +They do not establish that any production volume is clear or that domain recovery +has completed. diff --git a/scripts/lib/recovery-cutover-inventory.mjs b/scripts/lib/recovery-cutover-inventory.mjs new file mode 100644 index 00000000000..6c9d74d2b17 --- /dev/null +++ b/scripts/lib/recovery-cutover-inventory.mjs @@ -0,0 +1,99 @@ +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { classifyRecoveryEntry, cutoverStatus, recoveryKinds } from "./recovery-cutover-policy.mjs"; + +const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +// Inspect every ancestor: lstat on only the final component would follow an +// intermediate symlink. No directory/file names or OS errors enter the report. +async function directoryState(directory, io) { + const parsed = path.parse(directory); + let current = parsed.root; + for (const part of directory.slice(parsed.root.length).split(path.sep).filter(Boolean)) { + current = path.join(current, part); + try { + const stat = await io.lstat(current); + if (!stat.isDirectory() || stat.isSymbolicLink()) return "unsafe"; + } catch (error) { + return error?.code === "ENOENT" ? "absent" : "unreadable"; + } + } + return "directory"; +} + +/** Read-only, bounded entry inventory. Run with producers stopped on the retained + * volume. This is not an atomic snapshot and does not establish DB settlement. + * io exists for deterministic fault tests; production uses the actual filesystem. */ +export async function inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries = 10_000 }, io = fs) { + if (!path.isAbsolute(instanceRoot) || (orphanDirectory !== undefined && !path.isAbsolute(orphanDirectory))) { + throw new Error("Recovery inventory requires absolute directories"); + } + if (!Number.isSafeInteger(maxEntries) || maxEntries < 1 || maxEntries > 100_000) { + throw new Error("Entry limit must be between 1 and 100000"); + } + const root = path.resolve(instanceRoot); + const rootState = await directoryState(root, io); + const directories = [path.join(root, "accounting-receipts"), path.join(root, "quota-credential-recovery"), + orphanDirectory === undefined ? path.join(root, "data", "sandbox-orphan-cleanup") : path.resolve(orphanDirectory)]; + let remaining = maxEntries; + const summaries = []; + for (const [index, directory] of directories.entries()) { + const summary = { kind: recoveryKinds[index], presence: "unknown", observedEntries: 0, + unverifiedFiles: 0, unknownEntries: 0, readFailures: 0, complete: true, status: "unknown" }; + if (rootState !== "directory") { + summary.complete = false; + } else { + const state = await directoryState(directory, io); + if (state === "absent") summary.presence = "absent"; + else if (state !== "directory") { + summary.unknownEntries++; + summary.complete = false; + if (state === "unreadable") summary.readFailures++; + } else { + summary.presence = "present"; + async function scan(target, depth) { + let handle; + try { + handle = await io.opendir(target); + while (true) { + // Read one sentinel entry to distinguish exactly-at-limit from + // truncated scans. The sentinel is not included in observed counts. + const entry = await handle.read(); + if (entry === null) break; + if (remaining === 0) { summary.complete = false; break; } + remaining--; + summary.observedEntries++; + const entryKind = entry.isFile() ? "file" + : entry.isDirectory() && uuid.test(entry.name) ? "company-directory" : "other"; + const classification = classifyRecoveryEntry(summary.kind, depth, entryKind); + if (classification === "descend") { + const child = path.join(target, entry.name); + // Recheck the entry before traversing; never intentionally follow + // links. Operators must quiesce writers for the snapshot to hold. + if (await directoryState(child, io) !== "directory") { + summary.unknownEntries++; + summary.complete = false; + } else await scan(child, depth + 1); + } else summary[classification]++; + if (!summary.complete && remaining === 0) break; + } + } catch { + summary.readFailures++; + summary.complete = false; + } finally { + if (handle) { + try { await handle.close(); } + catch { summary.readFailures++; summary.complete = false; } + } + } + } + await scan(directory, 0); + } + } + summary.status = cutoverStatus(summary); + summaries.push(summary); + } + return { schema: "paperclip/recovery-cutover-inventory/v1", status: + summaries.some(s => s.status === "unknown") ? "unknown" : summaries.some(s => s.status === "blocked") ? "blocked" : "clear", + instanceStatus: rootState, contentInspection: "none", counts: "observed-lower-bounds", maxEntries, directories: summaries }; +} diff --git a/scripts/lib/recovery-cutover-policy.mjs b/scripts/lib/recovery-cutover-policy.mjs new file mode 100644 index 00000000000..19f5a7568fe --- /dev/null +++ b/scripts/lib/recovery-cutover-policy.mjs @@ -0,0 +1,14 @@ +/** File contents belong to each recovery owner. Inventory never parses them: + * even an apparently valid file cannot establish that its obligation settled. */ +export const recoveryKinds = ["accounting-receipts", "quota-credential-recovery", "sandbox-orphan-cleanup"]; + +export function classifyRecoveryEntry(kind, depth, entry) { + if (entry === "file") return "unverifiedFiles"; + if (entry === "company-directory" && kind === "quota-credential-recovery" && depth === 0) return "descend"; + return "unknownEntries"; +} + +export function cutoverStatus(summary) { + if (!summary.complete || summary.unknownEntries > 0 || summary.readFailures > 0) return "unknown"; + return summary.unverifiedFiles > 0 ? "blocked" : "clear"; +} diff --git a/scripts/recovery-cutover.mjs b/scripts/recovery-cutover.mjs new file mode 100644 index 00000000000..138a5f17947 --- /dev/null +++ b/scripts/recovery-cutover.mjs @@ -0,0 +1,33 @@ +#!/usr/bin/env node +import { inventoryRecoveryFiles } from "./lib/recovery-cutover-inventory.mjs"; + +const usage = "Usage: node scripts/recovery-cutover.mjs --instance-root /absolute/instance [--orphan-directory /absolute/override] [--max-entries 10000] [--check]"; +const args = process.argv.slice(2); +try { + let instanceRoot; + let orphanDirectory = process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR; + let maxEntries = 10_000; + let check = false; + const seen = new Set(); + for (let i = 0; i < args.length; i++) { + const argument = args[i]; + if (seen.has(argument)) throw new Error("duplicate"); + seen.add(argument); + if (argument === "--check") check = true; + else if (["--instance-root", "--orphan-directory", "--max-entries"].includes(argument)) { + const value = args[++i]; + if (value === undefined || value.startsWith("--")) throw new Error("missing"); + if (argument === "--instance-root") instanceRoot = value; + else if (argument === "--orphan-directory") orphanDirectory = value; + else maxEntries = Number(value); + } else throw new Error("unknown"); + } + if (!instanceRoot) throw new Error("missing"); + const report = await inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries }); + process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); + if (check && report.status !== "clear") process.exitCode = 1; +} catch { + // Never echo args, filesystem errors or paths: they may contain secrets. + process.stderr.write(`Recovery inventory could not run. ${usage}\n`); + process.exitCode = 2; +} diff --git a/scripts/recovery-cutover.test.mjs b/scripts/recovery-cutover.test.mjs new file mode 100644 index 00000000000..56bd6a0998d --- /dev/null +++ b/scripts/recovery-cutover.test.mjs @@ -0,0 +1,163 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { inventoryRecoveryFiles } from "./lib/recovery-cutover-inventory.mjs"; +import { classifyRecoveryEntry, cutoverStatus } from "./lib/recovery-cutover-policy.mjs"; + +const cli = fileURLToPath(new URL("./recovery-cutover.mjs", import.meta.url)); +const roots = []; +const company = "00000000-0000-4000-8000-000000000001"; +const grant = "00000000-0000-4000-8000-000000000002"; +async function fixture() { + // Canonicalize macOS /var -> /private/var, since the check rejects links. + const root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "cutover-"))); + roots.push(root); + return root; +} +async function write(root, relative, content = "secret-token-must-never-appear") { + const file = path.join(root, relative); + await fs.mkdir(path.dirname(file), { recursive: true }); + await fs.writeFile(file, content); + return file; +} +function run(root, args = [], env = {}) { + return spawnSync(process.execPath, [cli, "--instance-root", root, ...args], { + encoding: "utf8", env: { ...process.env, SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: "", ...env }, + }); +} +// An unset override differs from an invalid empty override. +function cleanRun(root, args = []) { + const env = { ...process.env }; + delete env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR; + return spawnSync(process.execPath, [cli, "--instance-root", root, ...args], { encoding: "utf8", env }); +} +afterEach(async () => { await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); }); + +test("pure policy never clears files or unknown entries", () => { + assert.equal(classifyRecoveryEntry("accounting-receipts", 0, "file"), "unverifiedFiles"); + assert.equal(classifyRecoveryEntry("quota-credential-recovery", 0, "company-directory"), "descend"); + assert.equal(classifyRecoveryEntry("quota-credential-recovery", 1, "company-directory"), "unknownEntries"); + assert.equal(cutoverStatus({ complete: false, unknownEntries: 0, readFailures: 0, unverifiedFiles: 0 }), "unknown"); +}); + +test("CLI: absent and empty legacy layouts clear without migration", async () => { + const root = await fixture(); + let result = cleanRun(root, ["--check"]); + assert.equal(result.status, 0); + assert.equal(JSON.parse(result.stdout).status, "clear"); + for (const directory of ["accounting-receipts", `quota-credential-recovery/${company}`, "data/sandbox-orphan-cleanup"]) { + await fs.mkdir(path.join(root, directory), { recursive: true }); + } + result = cleanRun(root, ["--check"]); + assert.equal(result.status, 0); + assert.equal(JSON.parse(result.stdout).status, "clear"); +}); + +test("CLI: real pending legacy files block, preserve bytes and reveal no secrets or names", async () => { + const root = await fixture(); + const receipt = JSON.stringify({ schema: "paperclip/accounting-receipt/v1", id: grant, companyId: company, + runId: company, sourceId: company, sequence: 1, receivedAt: "2026-10-30T00:00:00.000Z", adapterType: "codex_local", + receipt: { complete: true, usage: { inputTokens: 1, outputTokens: 1 } } }); + const files = await Promise.all([ + write(root, `accounting-receipts/${grant}.json`, receipt), + write(root, `quota-credential-recovery/${company}/${grant}.json`, '{"ciphertext":"secret-token-must-never-appear"}'), + write(root, "data/sandbox-orphan-cleanup/secret-filename.json", JSON.stringify({ companyId: company, + environmentId: company, executionWorkspaceId: null, issueId: null, heartbeatRunId: null, provider: "daytona", + providerLeaseId: "private-resource", metadata: {} })), + ]); + const before = await Promise.all(files.map(file => fs.readFile(file))); + const result = cleanRun(root, ["--check"]); + assert.equal(result.status, 1); + const report = JSON.parse(result.stdout); + assert.equal(report.status, "blocked"); + assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [1, 1, 1]); + assert.doesNotMatch(result.stdout + result.stderr, /secret-token|secret-filename|private-resource|ciphertext|00000000-/); + assert.deepEqual(await Promise.all(files.map(file => fs.readFile(file))), before); + assert.equal(cleanRun(root).status, 0, "report-only mode does not fail on obligations"); +}); + +test("truncated, temporary, and unreadable files are unverified, never clear", async () => { + const root = await fixture(); + await write(root, "accounting-receipts/truncated.json", '{"schema":'); + const file = await write(root, `quota-credential-recovery/${company}/${grant}.json.probe`); + await fs.chmod(file, 0); + const report = await inventoryRecoveryFiles({ instanceRoot: root }); + assert.equal(report.status, "blocked"); + assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [1, 1, 0]); +}); + +test("unknown nesting and symlinks block without traversing outside the chosen roots", async () => { + const root = await fixture(); + const outside = await fixture(); + await write(outside, "do-not-read.json"); + await fs.mkdir(path.join(root, "accounting-receipts")); + await fs.symlink(outside, path.join(root, "accounting-receipts", "link")); + await fs.mkdir(path.join(root, "quota-credential-recovery", company, "unexpected"), { recursive: true }); + const report = await inventoryRecoveryFiles({ instanceRoot: root }); + assert.equal(report.status, "unknown"); + assert.deepEqual(report.directories.map(s => s.unknownEntries), [1, 1, 0]); + assert.deepEqual(report.directories.map(s => s.unverifiedFiles), [0, 0, 0]); + await fs.symlink(outside, path.join(root, "data")); + assert.equal((await inventoryRecoveryFiles({ instanceRoot: root })).directories[2].status, "unknown"); + await fs.symlink(outside, path.join(root, "instance-link")); + assert.equal((await inventoryRecoveryFiles({ instanceRoot: path.join(root, "instance-link") })).status, "unknown"); +}); + +test("entry budget gives incomplete lower-bound counts and never clears omitted work", async () => { + const root = await fixture(); + for (let i = 0; i < 4; i++) await write(root, `accounting-receipts/${i}.json`); + const report = await inventoryRecoveryFiles({ instanceRoot: root, maxEntries: 2 }); + assert.equal(report.status, "unknown"); + assert.equal(report.directories[0].observedEntries, 2); + assert.equal(report.directories[0].complete, false); + assert.equal(report.directories[0].unverifiedFiles, 2); + const exact = await inventoryRecoveryFiles({ instanceRoot: root, maxEntries: 4 }); + assert.equal(exact.directories[0].complete, true); +}); + +test("CLI respects configured orphan override and explicit override precedence", async () => { + const root = await fixture(); + const override = await fixture(); + await write(override, "pending.json"); + let result = run(root, ["--check"], { SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: override }); + assert.equal(result.status, 1); + assert.equal(JSON.parse(result.stdout).directories[2].unverifiedFiles, 1); + const emptyOverride = await fixture(); + result = run(root, ["--check", "--orphan-directory", emptyOverride], { SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR: override }); + assert.equal(result.status, 0); + assert.equal(JSON.parse(result.stdout).status, "clear"); +}); + +test("fault boundary reports stat, directory read and close failures without error data", async () => { + const root = await fixture(); + await fs.mkdir(path.join(root, "accounting-receipts")); + for (const operation of ["lstat", "opendir", "read", "close"]) { + const sensitiveError = Object.assign(new Error("secret-token-must-never-appear"), { code: "EACCES" }); + const io = { ...fs }; + if (operation === "lstat" || operation === "opendir") io[operation] = async () => { throw sensitiveError; }; + else io.opendir = async () => ({ + read: async () => { if (operation === "read") throw sensitiveError; return null; }, + close: async () => { if (operation === "close") throw sensitiveError; }, + }); + const report = await inventoryRecoveryFiles({ instanceRoot: root }, io); + assert.equal(report.status, "unknown"); + if (operation === "lstat") assert.equal(report.instanceStatus, "unreadable"); + else assert.ok(report.directories[0].readFailures > 0); + assert.doesNotMatch(JSON.stringify(report), /secret-token|EACCES/); + } +}); + +test("CLI fails closed for missing instance, unsafe roots and bad arguments", async () => { + const root = await fixture(); + assert.equal(cleanRun(path.join(root, "absent"), ["--check"]).status, 1); + for (const args of [["--max-entries", "0"], ["--max-entries", "100001"], ["--max-entries", "NaN"], + ["--orphan-directory", "relative-secret"], ["--bad-secret"], ["--check", "--check"]]) { + const result = cleanRun(root, args); + assert.equal(result.status, 2); + assert.doesNotMatch(result.stdout + result.stderr, /relative-secret|bad-secret/); + } +}); diff --git a/server/src/__tests__/recovery-cutover.test.ts b/server/src/__tests__/recovery-cutover.test.ts new file mode 100644 index 00000000000..ee9aca31a51 --- /dev/null +++ b/server/src/__tests__/recovery-cutover.test.ts @@ -0,0 +1,15 @@ +import { execFile } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { it } from "vitest"; + +const run = promisify(execFile); + +// Keep the dependency-free, actual-CLI acceptance suite usable on a retained +// volume while also exercising the same cases in the normal CI test lane. +it("passes the read-only recovery cutover CLI acceptance suite", async () => { + await run(process.execPath, ["--test", fileURLToPath(new URL("../../../scripts/recovery-cutover.test.mjs", import.meta.url))], { + timeout: 45_000, + maxBuffer: 1024 * 1024, + }); +}, 60_000);