mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 06:42:06 +01:00
Add bounded recovery file inventory before server cutover (#18347)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accounting, credential recovery, and sandbox cleanup can retain legacy files on an instance volume. > - Replacing that volume while files remain can lose unresolved work. > - There is no verified production backlog, so an empty installation needs no migration. > - The recovery owners must decide whether each record is valid and settled. > - This pull request adds a bounded file inventory and an optional cutover gate. > - The benefit is a safe check that preserves obligations without reading secret material. ## Linked Issues or Issue Description **Problem or motivation** Operators need to check the actual affected instance directories before replacing a web server. A skipped corrupt file or failed read must not appear clear. **Proposed solution** Add a Node script that inspects only the three legacy recovery directories for an explicitly chosen instance. Default mode reports counts. The optional check mode blocks when files or uncertainty remain. **Alternatives considered** A bulk migration system has no established backlog to migrate. Reimplementing the owner parsers could expose credentials and still would not prove settlement. This check never reads file bytes. **Roadmap alignment** This is a small operational prerequisite for safe cloud server replacement. It does not add a recovery backend or change resource ownership. Related recovery-owner changes: #18354 (accounting), #18352 (credentials), and #18357 (sandbox ownership). This inventory remains independently usable with existing files. ## What Changed - Add a pure classification policy and a bounded filesystem boundary. - Add `node scripts/recovery-cutover.mjs --instance-root /absolute/instance` with optional `--check`, directory override, and entry budget. - Report observed lower-bound counts and safe status summaries. Do not print paths, filenames, identifiers, raw errors, or file contents. - Add one dependency-free CLI acceptance suite and a Vitest bridge so the same cases run in normal server CI. - Document existing owner entry points, override selection, stopped producers, and retained-volume requirements. ## Verification - `node --test scripts/recovery-cutover.test.mjs`: 9 cases pass. The suite starts the actual CLI with synthetic legacy directories. - Pending receipts, nested credential files, orphan records, malformed bytes, temporary files, unreadable files, unsafe paths, symlinks, read failures, bounds, and configured overrides all block safely. - Empty or absent recovery directories clear. Source file bytes remain unchanged. Report-only mode and check-mode exit codes are tested. - `pnpm exec vitest run server/src/__tests__/recovery-cutover.test.ts --pool=forks --maxWorkers=1`: the normal CI bridge passes and runs the same 9 cases. - Cached frozen dependency installation, script syntax checks, and `git diff --check` pass. The lockfile is unchanged. - Independent source review found the read-only boundary sound. - [Complete current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41097614666) passed at `bf5cc7f067041c10cd0294ff26756678da943976`: 53 successful checks and two intended skips, including typecheck, full tests, build, and E2E. Greptile is 5/5 with no review threads. - One failed-job retry was needed for an existing heartbeat timing failure in server shard 4. The relevant heartbeat source and tests match the base branch. Local diagnostic runs had 61 pass/2 fail and 7 pass/1 fail, with changing timing failures; neither run is claimed as green. The bounded retry and all failed evidence are [recorded here](https://github.com/paperclipai/paperclip/pull/18347#issuecomment-6529629804). No heartbeat code or test was changed. ## Risks - The scan is a point-in-time inventory. Stop producers for the final gate. It is not an atomic drain certificate or protection against hostile path mutation. - All files remain unverified. The check cannot certify validity or Postgres settlement. Existing owners retain that responsibility. - The scan does not inspect other instances, volumes, Runner delivery, or database obligations. It does not establish that production cutover is ready. - No migration, deletion, decryption, or replay command is added. Symlink paths are rejected, including system aliases; use canonical absolute paths. ## Model Used - OpenAI `gpt-6.1-sol`, with reasoning, tool use, and code execution. Context window size was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
f53db0c70f
commit
76fde891cc
6 files changed
+436
No files matched your search
@@ -0,0 +1,99 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { classifyRecoveryEntry, cutoverStatus, recoveryKinds } from "./recovery-cutover-policy.mjs";
|
||||
|
||||
const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
|
||||
// Inspect every ancestor: lstat on only the final component would follow an
|
||||
// intermediate symlink. No directory/file names or OS errors enter the report.
|
||||
async function directoryState(directory, io) {
|
||||
const parsed = path.parse(directory);
|
||||
let current = parsed.root;
|
||||
for (const part of directory.slice(parsed.root.length).split(path.sep).filter(Boolean)) {
|
||||
current = path.join(current, part);
|
||||
try {
|
||||
const stat = await io.lstat(current);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) return "unsafe";
|
||||
} catch (error) {
|
||||
return error?.code === "ENOENT" ? "absent" : "unreadable";
|
||||
}
|
||||
}
|
||||
return "directory";
|
||||
}
|
||||
|
||||
/** Read-only, bounded entry inventory. Run with producers stopped on the retained
|
||||
* volume. This is not an atomic snapshot and does not establish DB settlement.
|
||||
* io exists for deterministic fault tests; production uses the actual filesystem. */
|
||||
export async function inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries = 10_000 }, io = fs) {
|
||||
if (!path.isAbsolute(instanceRoot) || (orphanDirectory !== undefined && !path.isAbsolute(orphanDirectory))) {
|
||||
throw new Error("Recovery inventory requires absolute directories");
|
||||
}
|
||||
if (!Number.isSafeInteger(maxEntries) || maxEntries < 1 || maxEntries > 100_000) {
|
||||
throw new Error("Entry limit must be between 1 and 100000");
|
||||
}
|
||||
const root = path.resolve(instanceRoot);
|
||||
const rootState = await directoryState(root, io);
|
||||
const directories = [path.join(root, "accounting-receipts"), path.join(root, "quota-credential-recovery"),
|
||||
orphanDirectory === undefined ? path.join(root, "data", "sandbox-orphan-cleanup") : path.resolve(orphanDirectory)];
|
||||
let remaining = maxEntries;
|
||||
const summaries = [];
|
||||
for (const [index, directory] of directories.entries()) {
|
||||
const summary = { kind: recoveryKinds[index], presence: "unknown", observedEntries: 0,
|
||||
unverifiedFiles: 0, unknownEntries: 0, readFailures: 0, complete: true, status: "unknown" };
|
||||
if (rootState !== "directory") {
|
||||
summary.complete = false;
|
||||
} else {
|
||||
const state = await directoryState(directory, io);
|
||||
if (state === "absent") summary.presence = "absent";
|
||||
else if (state !== "directory") {
|
||||
summary.unknownEntries++;
|
||||
summary.complete = false;
|
||||
if (state === "unreadable") summary.readFailures++;
|
||||
} else {
|
||||
summary.presence = "present";
|
||||
async function scan(target, depth) {
|
||||
let handle;
|
||||
try {
|
||||
handle = await io.opendir(target);
|
||||
while (true) {
|
||||
// Read one sentinel entry to distinguish exactly-at-limit from
|
||||
// truncated scans. The sentinel is not included in observed counts.
|
||||
const entry = await handle.read();
|
||||
if (entry === null) break;
|
||||
if (remaining === 0) { summary.complete = false; break; }
|
||||
remaining--;
|
||||
summary.observedEntries++;
|
||||
const entryKind = entry.isFile() ? "file"
|
||||
: entry.isDirectory() && uuid.test(entry.name) ? "company-directory" : "other";
|
||||
const classification = classifyRecoveryEntry(summary.kind, depth, entryKind);
|
||||
if (classification === "descend") {
|
||||
const child = path.join(target, entry.name);
|
||||
// Recheck the entry before traversing; never intentionally follow
|
||||
// links. Operators must quiesce writers for the snapshot to hold.
|
||||
if (await directoryState(child, io) !== "directory") {
|
||||
summary.unknownEntries++;
|
||||
summary.complete = false;
|
||||
} else await scan(child, depth + 1);
|
||||
} else summary[classification]++;
|
||||
if (!summary.complete && remaining === 0) break;
|
||||
}
|
||||
} catch {
|
||||
summary.readFailures++;
|
||||
summary.complete = false;
|
||||
} finally {
|
||||
if (handle) {
|
||||
try { await handle.close(); }
|
||||
catch { summary.readFailures++; summary.complete = false; }
|
||||
}
|
||||
}
|
||||
}
|
||||
await scan(directory, 0);
|
||||
}
|
||||
}
|
||||
summary.status = cutoverStatus(summary);
|
||||
summaries.push(summary);
|
||||
}
|
||||
return { schema: "paperclip/recovery-cutover-inventory/v1", status:
|
||||
summaries.some(s => s.status === "unknown") ? "unknown" : summaries.some(s => s.status === "blocked") ? "blocked" : "clear",
|
||||
instanceStatus: rootState, contentInspection: "none", counts: "observed-lower-bounds", maxEntries, directories: summaries };
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
/** File contents belong to each recovery owner. Inventory never parses them:
|
||||
* even an apparently valid file cannot establish that its obligation settled. */
|
||||
export const recoveryKinds = ["accounting-receipts", "quota-credential-recovery", "sandbox-orphan-cleanup"];
|
||||
|
||||
export function classifyRecoveryEntry(kind, depth, entry) {
|
||||
if (entry === "file") return "unverifiedFiles";
|
||||
if (entry === "company-directory" && kind === "quota-credential-recovery" && depth === 0) return "descend";
|
||||
return "unknownEntries";
|
||||
}
|
||||
|
||||
export function cutoverStatus(summary) {
|
||||
if (!summary.complete || summary.unknownEntries > 0 || summary.readFailures > 0) return "unknown";
|
||||
return summary.unverifiedFiles > 0 ? "blocked" : "clear";
|
||||
}
|
||||
Reference in new issue
Block a user