mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 23:43:16 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accounting, credential recovery, and sandbox cleanup can retain legacy files on an instance volume. > - Replacing that volume while files remain can lose unresolved work. > - There is no verified production backlog, so an empty installation needs no migration. > - The recovery owners must decide whether each record is valid and settled. > - This pull request adds a bounded file inventory and an optional cutover gate. > - The benefit is a safe check that preserves obligations without reading secret material. ## Linked Issues or Issue Description **Problem or motivation** Operators need to check the actual affected instance directories before replacing a web server. A skipped corrupt file or failed read must not appear clear. **Proposed solution** Add a Node script that inspects only the three legacy recovery directories for an explicitly chosen instance. Default mode reports counts. The optional check mode blocks when files or uncertainty remain. **Alternatives considered** A bulk migration system has no established backlog to migrate. Reimplementing the owner parsers could expose credentials and still would not prove settlement. This check never reads file bytes. **Roadmap alignment** This is a small operational prerequisite for safe cloud server replacement. It does not add a recovery backend or change resource ownership. Related recovery-owner changes: #18354 (accounting), #18352 (credentials), and #18357 (sandbox ownership). This inventory remains independently usable with existing files. ## What Changed - Add a pure classification policy and a bounded filesystem boundary. - Add `node scripts/recovery-cutover.mjs --instance-root /absolute/instance` with optional `--check`, directory override, and entry budget. - Report observed lower-bound counts and safe status summaries. Do not print paths, filenames, identifiers, raw errors, or file contents. - Add one dependency-free CLI acceptance suite and a Vitest bridge so the same cases run in normal server CI. - Document existing owner entry points, override selection, stopped producers, and retained-volume requirements. ## Verification - `node --test scripts/recovery-cutover.test.mjs`: 9 cases pass. The suite starts the actual CLI with synthetic legacy directories. - Pending receipts, nested credential files, orphan records, malformed bytes, temporary files, unreadable files, unsafe paths, symlinks, read failures, bounds, and configured overrides all block safely. - Empty or absent recovery directories clear. Source file bytes remain unchanged. Report-only mode and check-mode exit codes are tested. - `pnpm exec vitest run server/src/__tests__/recovery-cutover.test.ts --pool=forks --maxWorkers=1`: the normal CI bridge passes and runs the same 9 cases. - Cached frozen dependency installation, script syntax checks, and `git diff --check` pass. The lockfile is unchanged. - Independent source review found the read-only boundary sound. - [Complete current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41097614666) passed at `bf5cc7f067041c10cd0294ff26756678da943976`: 53 successful checks and two intended skips, including typecheck, full tests, build, and E2E. Greptile is 5/5 with no review threads. - One failed-job retry was needed for an existing heartbeat timing failure in server shard 4. The relevant heartbeat source and tests match the base branch. Local diagnostic runs had 61 pass/2 fail and 7 pass/1 fail, with changing timing failures; neither run is claimed as green. The bounded retry and all failed evidence are [recorded here](https://github.com/paperclipai/paperclip/pull/18347#issuecomment-6529629804). No heartbeat code or test was changed. ## Risks - The scan is a point-in-time inventory. Stop producers for the final gate. It is not an atomic drain certificate or protection against hostile path mutation. - All files remain unverified. The check cannot certify validity or Postgres settlement. Existing owners retain that responsibility. - The scan does not inspect other instances, volumes, Runner delivery, or database obligations. It does not establish that production cutover is ready. - No migration, deletion, decryption, or replay command is added. Symlink paths are rejected, including system aliases; use canonical absolute paths. ## Model Used - OpenAI `gpt-6.1-sol`, with reasoning, tool use, and code execution. Context window size was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
100 lines
4.7 KiB
JavaScript
100 lines
4.7 KiB
JavaScript
import { promises as fs } from "node:fs";
|
|
import path from "node:path";
|
|
import { classifyRecoveryEntry, cutoverStatus, recoveryKinds } from "./recovery-cutover-policy.mjs";
|
|
|
|
const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
|
|
|
// Inspect every ancestor: lstat on only the final component would follow an
|
|
// intermediate symlink. No directory/file names or OS errors enter the report.
|
|
async function directoryState(directory, io) {
|
|
const parsed = path.parse(directory);
|
|
let current = parsed.root;
|
|
for (const part of directory.slice(parsed.root.length).split(path.sep).filter(Boolean)) {
|
|
current = path.join(current, part);
|
|
try {
|
|
const stat = await io.lstat(current);
|
|
if (!stat.isDirectory() || stat.isSymbolicLink()) return "unsafe";
|
|
} catch (error) {
|
|
return error?.code === "ENOENT" ? "absent" : "unreadable";
|
|
}
|
|
}
|
|
return "directory";
|
|
}
|
|
|
|
/** Read-only, bounded entry inventory. Run with producers stopped on the retained
|
|
* volume. This is not an atomic snapshot and does not establish DB settlement.
|
|
* io exists for deterministic fault tests; production uses the actual filesystem. */
|
|
export async function inventoryRecoveryFiles({ instanceRoot, orphanDirectory, maxEntries = 10_000 }, io = fs) {
|
|
if (!path.isAbsolute(instanceRoot) || (orphanDirectory !== undefined && !path.isAbsolute(orphanDirectory))) {
|
|
throw new Error("Recovery inventory requires absolute directories");
|
|
}
|
|
if (!Number.isSafeInteger(maxEntries) || maxEntries < 1 || maxEntries > 100_000) {
|
|
throw new Error("Entry limit must be between 1 and 100000");
|
|
}
|
|
const root = path.resolve(instanceRoot);
|
|
const rootState = await directoryState(root, io);
|
|
const directories = [path.join(root, "accounting-receipts"), path.join(root, "quota-credential-recovery"),
|
|
orphanDirectory === undefined ? path.join(root, "data", "sandbox-orphan-cleanup") : path.resolve(orphanDirectory)];
|
|
let remaining = maxEntries;
|
|
const summaries = [];
|
|
for (const [index, directory] of directories.entries()) {
|
|
const summary = { kind: recoveryKinds[index], presence: "unknown", observedEntries: 0,
|
|
unverifiedFiles: 0, unknownEntries: 0, readFailures: 0, complete: true, status: "unknown" };
|
|
if (rootState !== "directory") {
|
|
summary.complete = false;
|
|
} else {
|
|
const state = await directoryState(directory, io);
|
|
if (state === "absent") summary.presence = "absent";
|
|
else if (state !== "directory") {
|
|
summary.unknownEntries++;
|
|
summary.complete = false;
|
|
if (state === "unreadable") summary.readFailures++;
|
|
} else {
|
|
summary.presence = "present";
|
|
async function scan(target, depth) {
|
|
let handle;
|
|
try {
|
|
handle = await io.opendir(target);
|
|
while (true) {
|
|
// Read one sentinel entry to distinguish exactly-at-limit from
|
|
// truncated scans. The sentinel is not included in observed counts.
|
|
const entry = await handle.read();
|
|
if (entry === null) break;
|
|
if (remaining === 0) { summary.complete = false; break; }
|
|
remaining--;
|
|
summary.observedEntries++;
|
|
const entryKind = entry.isFile() ? "file"
|
|
: entry.isDirectory() && uuid.test(entry.name) ? "company-directory" : "other";
|
|
const classification = classifyRecoveryEntry(summary.kind, depth, entryKind);
|
|
if (classification === "descend") {
|
|
const child = path.join(target, entry.name);
|
|
// Recheck the entry before traversing; never intentionally follow
|
|
// links. Operators must quiesce writers for the snapshot to hold.
|
|
if (await directoryState(child, io) !== "directory") {
|
|
summary.unknownEntries++;
|
|
summary.complete = false;
|
|
} else await scan(child, depth + 1);
|
|
} else summary[classification]++;
|
|
if (!summary.complete && remaining === 0) break;
|
|
}
|
|
} catch {
|
|
summary.readFailures++;
|
|
summary.complete = false;
|
|
} finally {
|
|
if (handle) {
|
|
try { await handle.close(); }
|
|
catch { summary.readFailures++; summary.complete = false; }
|
|
}
|
|
}
|
|
}
|
|
await scan(directory, 0);
|
|
}
|
|
}
|
|
summary.status = cutoverStatus(summary);
|
|
summaries.push(summary);
|
|
}
|
|
return { schema: "paperclip/recovery-cutover-inventory/v1", status:
|
|
summaries.some(s => s.status === "unknown") ? "unknown" : summaries.some(s => s.status === "blocked") ? "blocked" : "clear",
|
|
instanceStatus: rootState, contentInspection: "none", counts: "observed-lower-bounds", maxEntries, directories: summaries };
|
|
}
|