mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
## Thinking Path > - Paperclip preserves the files and recovery state produced by its agents. > - Canonical online migration can preserve selected local workspaces while unrelated work continues. > - Operators sometimes need a newer migration owner before the serving web process can take that release. > - The existing packaged maintenance entry requires stopped writers. > - This pull request adds a finite online entry from one coherent reviewed release tree. > - The entry delegates capture and publication to the existing owners and keeps tool and serving identities separate. ## Linked Issues or Issue Description Refs #19854, #19855 and #19872. The duplicate search found related owner fixes and no separate versioned online entry. **What happened?** The finite online workspace importer was available as a library in the serving image. The packaged all-phase migration entry required maintenance mode. **Expected behavior** Run the reviewed online owner from a separate complete release tree without replacing the web process, stopping unrelated work or changing its release policy. **Deployment mode** A managed instance with its original local volume, database, existing encryption key and shared S3 storage. ## What Changed - Add strict inspect, single, cohort and exact single-journal resume requests with explicit company, workspace, deadline and concurrency bounds. - Verify the private tool root and its own build stamp separately from the serving source commit and deployment. - Reuse the nonmutating full migration-manifest inspector and require the additive controller host-identity column before publication. - Hydrate existing configuration and key bindings, then use the original ESM database pool and canonical storage, source locks and migration owners. - Keep multi-member inspection with the optional canonical cohort preflight in a read-only transaction with denied storage. Releases without that owner refuse cohort operations; single import and resume remain available. - Register the optional runtime bundle entry and document the supported full-tree source mode, resource admission and reconciliation. - Add hash-bound old-writer/reader compatibility and encrypted restore tests. ## Verification Final local revision: `c9345b72f27aee1158047dccd0b1060f9a0be6df`, on master `fbe689938c3bf8c5849d2cd56b9da90029db9389`, macOS arm64 with Node 24.22.2. - Entry and runtime packaging: **34/34 PASS**. - Real PostgreSQL/filesystem compatibility: **6/6 PASS**, disposable database wrapper exit 0 and cleanup complete. The frozen old source lock owner contends with the new entry; unrelated work proceeds. The old authority and snapshot readers decrypt and restore newly published bytes. - Pending and uploaded journal cases use the unchanged old persisted format and preserve operation/snapshot IDs through resume. - Full layered server source and test typecheck: **PASS**. - Module boundaries and `git diff --check`: **PASS**. - Actual source-mode module import under Node 24 succeeds without starting the application. - A separate temporary full-old-producer fixture did not get past module setup: 90 seconds under Vitest and a 15-second direct Node control. Neither is counted as an executed old-producer test or a product failure. The smaller frozen old-writer/reader proof and old-format journal cases are the local evidence. - Hosted resume of the actual saved operation and Linux artifact qualification remain operational acceptance gates. This PR does not claim those ran. - Review regression: a real successful import returned a null operation ID (genuine RED). The repaired entry binds the exact returned snapshot to its succeeded migration journal after commit (GREEN). Committed or superseded snapshots are accepted; a missing readback requires reconciliation rather than a blind retry. - Final affected check: **40/40 PASS** across the entry, packaging and actual PostgreSQL compatibility suites; wrapper exit 0 and cleanup complete. Full layered typecheck and boundaries pass after the user-requested rebase and current Runner rebuild. - A second real postcommit database-read fault also has genuine RED/GREEN proof. The succeeded journal, saved authority and original bytes remain intact, and every readback error now preserves the reconcile-required warning. - Final-head [CI run 42404116001](https://github.com/paperclipai/paperclip/actions/runs/42404116001): **PASS**, with all 59 check contexts terminal (55 success and 4 intentional skips). Fresh Greptile review is **5/5** on `c9345b72f27aee1158047dccd0b1060f9a0be6df`, with no unresolved threads; check and thread pagination are complete. ## Risks - The tool must come from one authenticated complete release tree with its own dependencies. A copied bundle chunk can retain old absolute paths and is not a portable release. - Concurrency is not resource admission. The operator must check memory, CPU and scratch headroom and start with one import or two cohort members. - Schema must be prepared by the existing schema owner. This entry never applies DDL. - Cancellation awaits the canonical owner. An SSH interruption does not prove the process stopped; durable journals must be reconciled before retry. - The entry preserves remote obligations and source files. It does not prove enrollment readiness or zero-downtime operation. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact serving snapshot and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
121 lines
5.2 KiB
JavaScript
121 lines
5.2 KiB
JavaScript
|
|
import { readFile, rm } from "node:fs/promises";
|
|
import { dirname, join } from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import { build } from "esbuild";
|
|
|
|
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
|
const serverDir = join(scriptDir, "..");
|
|
|
|
const REQUIRE_SHIM = "import { createRequire as __paperclipCreateRequire } from 'node:module'; const require = __paperclipCreateRequire(import.meta.url);";
|
|
|
|
export function preserveModuleLocation(source, file) {
|
|
return source
|
|
.replace(/\bimport\.meta\.url\b/g, JSON.stringify(pathToFileURL(file).href))
|
|
.replace(/\bimport\.meta\.dirname\b/g, JSON.stringify(dirname(file)))
|
|
.replace(/\bimport\.meta\.filename\b/g, JSON.stringify(file));
|
|
}
|
|
|
|
function runtimeEntry(entry) {
|
|
return {
|
|
name: "runtime-entry",
|
|
setup(builder) {
|
|
builder.onResolve({ filter: /^paperclip-runtime-entry$/ }, () => ({ path: "paperclip-runtime-entry", namespace: "runtime-entry" }));
|
|
builder.onLoad({ filter: /.*/, namespace: "runtime-entry" }, () => ({
|
|
contents: `process.argv[1] = ${JSON.stringify(entry)};\nawait import(${JSON.stringify(entry)});\n`,
|
|
loader: "js",
|
|
resolveDir: dirname(entry),
|
|
}));
|
|
},
|
|
};
|
|
}
|
|
|
|
const keepModuleLocations = {
|
|
name: "keep-module-locations",
|
|
setup(builder) {
|
|
builder.onLoad({ filter: /\.(?:[cm]?js|[cm]?ts)$/ }, async (args) => {
|
|
if (args.path.includes("/node_modules/")) return undefined;
|
|
const source = await readFile(args.path, "utf8");
|
|
if (!source.includes("import.meta")) return undefined;
|
|
return {
|
|
contents: preserveModuleLocation(source, args.path),
|
|
loader: /\.[cm]?ts$/.test(args.path) ? "ts" : "js",
|
|
resolveDir: dirname(args.path),
|
|
};
|
|
});
|
|
},
|
|
};
|
|
|
|
function externalizeThirdParty(isBundled, runtimeDir) {
|
|
return {
|
|
name: "externalize-third-party",
|
|
setup(builder) {
|
|
builder.onResolve({ filter: /^[^./]/ }, async (args) => {
|
|
if (args.pluginData?.externalizeThirdParty || isBundled(args.path)) return undefined;
|
|
const resolved = await builder.resolve(args.path, {
|
|
kind: args.kind,
|
|
resolveDir: args.resolveDir,
|
|
importer: args.importer,
|
|
pluginData: { externalizeThirdParty: true },
|
|
});
|
|
if (resolved.errors.length > 0 || resolved.external) return { path: args.path, external: true };
|
|
const fromServer = await builder.resolve(args.path, {
|
|
kind: args.kind,
|
|
resolveDir: runtimeDir,
|
|
pluginData: { externalizeThirdParty: true },
|
|
});
|
|
if (fromServer.errors.length === 0 && fromServer.path === resolved.path) return { path: args.path, external: true };
|
|
return { path: resolved.path, external: true };
|
|
});
|
|
},
|
|
};
|
|
}
|
|
|
|
/** @param {{ entry: string, outdir: string, maintenanceEntry?: string, ephemeralMigrationEntry?: string,
|
|
* nativeHarnessPrestageEntry?: string, onlineWorkspaceMigrationEntry?: string, isBundled?: (specifier: string) => boolean }} options */
|
|
export async function bundleRuntime({ entry, outdir, maintenanceEntry, ephemeralMigrationEntry, nativeHarnessPrestageEntry, onlineWorkspaceMigrationEntry, isBundled = (specifier) => specifier.startsWith("@paperclipai/") }) {
|
|
await rm(outdir, { recursive: true, force: true });
|
|
const result = await build({
|
|
entryPoints: { index: "paperclip-runtime-entry", server: entry, ...(maintenanceEntry ? { "agent-files": maintenanceEntry } : {}),
|
|
...(ephemeralMigrationEntry ? { "ephemeral-migration": ephemeralMigrationEntry } : {}),
|
|
...(nativeHarnessPrestageEntry ? { "native-harness-prestage": nativeHarnessPrestageEntry } : {}),
|
|
...(onlineWorkspaceMigrationEntry ? { "online-workspace-migration": onlineWorkspaceMigrationEntry } : {}) },
|
|
outdir,
|
|
outExtension: { ".js": ".mjs" },
|
|
chunkNames: "chunk-[hash]",
|
|
bundle: true,
|
|
splitting: true,
|
|
format: "esm",
|
|
platform: "node",
|
|
target: "node24",
|
|
mainFields: ["main"],
|
|
conditions: ["node"],
|
|
banner: { js: REQUIRE_SHIM },
|
|
sourcemap: "linked",
|
|
metafile: true,
|
|
logLevel: "error",
|
|
plugins: [runtimeEntry(entry), keepModuleLocations, externalizeThirdParty(isBundled, dirname(entry))],
|
|
});
|
|
const outputs = Object.keys(result.metafile.outputs).filter((file) => file.endsWith(".mjs"));
|
|
return {
|
|
outfile: join(outdir, "index.mjs"),
|
|
serverModule: join(outdir, "server.mjs"),
|
|
inputs: Object.keys(result.metafile.inputs).length,
|
|
chunks: outputs.length,
|
|
};
|
|
}
|
|
|
|
async function main() {
|
|
const started = performance.now();
|
|
const summary = await bundleRuntime({ entry: join(serverDir, "dist", "index.js"), outdir: join(serverDir, "dist-runtime"),
|
|
maintenanceEntry: join(serverDir, "dist", "maintenance", "agent-files.js"),
|
|
ephemeralMigrationEntry: join(serverDir, "dist", "maintenance", "ephemeral-migration.js"),
|
|
nativeHarnessPrestageEntry: join(serverDir, "dist", "maintenance", "native-harness-prestage.js"),
|
|
onlineWorkspaceMigrationEntry: join(serverDir, "dist", "maintenance", "online-workspace-migration.js") });
|
|
console.log(`[bundle-runtime] ${summary.inputs} inputs -> ${summary.chunks} chunks in ${Math.round(performance.now() - started)} ms: ${summary.outfile}`);
|
|
}
|
|
|
|
if (process.argv[1] && pathToFileURL(process.argv[1]).href === import.meta.url) {
|
|
await main();
|
|
}
|