mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 16:58:52 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud web servers can now keep durable state in PostgreSQL and S3.
> - Existing servers still have files and a decision signing key on
their volume.
> - The volume must stay attached until every data owner verifies its
migration.
> - This pull request adds a dedicated maintenance process for that
transition.
> - Cloud can preserve the original key and remove the volume only after
a matching receipt.
## Linked Issues or Issue Description
Refs #18518 and #18526. Existing-volume enrollment is a required step in
the zero-downtime web deployment work. This is the Core maintenance
side; the Cloud transition is a separate private change.
## What Changed
- Package a standalone maintenance entry that starts no normal app
routes or execution workers.
- Coordinate the existing agent-file, run-evidence and skill migration
owners. Follow every page and verify shared bytes before final
eligibility.
- Require stopped writers. Refuse active execution, missing local bytes,
local workspaces and external instructions.
- Read the existing decision key without generating a replacement.
Preserve normal key creation for ordinary server startup.
- Require an attempt token for status and export. Export the original
key only after successful migration. Keep status and errors free of
secret material.
- Document the Cloud ownership boundary, deadline, retry and retention
requirements.
## Verification
- Full workspace typecheck and build passed before the review fixes. The
final server TypeScript compilation passed after correcting Drizzle's
optional predicate type.
- Production runtime bundling passed: 2,702 inputs, 122 chunks,
including the maintenance executable.
- The focused suite passed 21 cases using PostgreSQL and real
S3-compatible storage. It covers multi-page migration of 26 agents,
exact reads after deleting the predecessor home, corrupt-object
rejection, active/missing-data refusal, and the bundled plain-Node
process with authenticated export and clean SIGTERM exit.
- Regression cases refuse unsaved terminal working copies, require
explicit stopped/cleaned receipts for unavailable copies, clean 101
expired traces through the owning service, and enforce the deadline
after migration succeeds.
- Initial integration attempts exposed fixture cleanup ordering
problems; corrected runs pass. CI also caught a Drizzle predicate type
error, now fixed and compiled locally.
- All repository CI gates passed on a4db955403, including typecheck,
build, browser suites and serialized tests. Greptile reviewed that head
at 5/5; all four review threads are resolved. Hosted enrollment and
active-agent continuity remain separate acceptance work.
## Risks
- This is an explicit maintenance entry. The caller must confirm all
prior deployments stopped and keep normal tenant routing closed through
detachment.
- A timeout blocks export but an owner may finish its current bounded
operation before observing cancellation. Cloud must confirm process exit
before detaching. SIGTERM has a ten-second hard exit deadline.
- The authenticated export contains the original signing key. The Cloud
caller must put it only in the stack vault and must not log the body.
- Local-workspace and external-instruction migrations remain their
existing owners' responsibility. Enrollment fails closed until those
data are eligible.
- No database schema changes. Ordinary server startup is unchanged
unless this maintenance entry is explicitly launched.
## Model Used
OpenAI Codex, GPT-6. Tool use, code execution and repository review. The
runtime does not expose a more specific model ID or context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
169 lines
10 KiB
JavaScript
169 lines
10 KiB
JavaScript
import { execFile } from "node:child_process";
|
|
import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import { promisify } from "node:util";
|
|
|
|
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
|
|
|
import { bundleRuntime, preserveModuleLocation } from "./bundle-runtime.mjs";
|
|
|
|
const execFileAsync = promisify(execFile);
|
|
|
|
function write(file, content) {
|
|
mkdirSync(path.dirname(file), { recursive: true });
|
|
writeFileSync(file, content);
|
|
}
|
|
|
|
describe("bundleRuntime", () => {
|
|
let root;
|
|
|
|
beforeEach(() => {
|
|
root = realpathSync(mkdtempSync(path.join(tmpdir(), "paperclip-bundle-runtime-")));
|
|
const pkg = path.join(root, "packages", "fixture");
|
|
write(path.join(pkg, "package.json"), JSON.stringify({ name: "@paperclipai/fixture", type: "module", exports: { ".": "./src/index.ts" } }));
|
|
write(path.join(pkg, "src", "asset.txt"), "asset beside its source\n");
|
|
write(path.join(pkg, "src", "helper.ts"), "export const double = (value: number): number => value * 2;\n");
|
|
write(path.join(pkg, "src", "index.ts"), [
|
|
"import { readFileSync } from \"node:fs\";",
|
|
"import { double } from \"./helper.js\";",
|
|
"import { onlyForPackage } from \"package-private-dep\";",
|
|
"import { nodeBuild } from \"dual-build-dep\";",
|
|
"export const asset = readFileSync(new URL(\"./asset.txt\", import.meta.url), \"utf8\").trim();",
|
|
"export const location = import.meta.dirname;",
|
|
"export const values = { doubled: double(21), onlyForPackage, nodeBuild };",
|
|
"",
|
|
].join("\n"));
|
|
write(path.join(pkg, "node_modules", "package-private-dep", "package.json"), JSON.stringify({ name: "package-private-dep", type: "module", main: "index.js" }));
|
|
write(path.join(pkg, "node_modules", "package-private-dep", "index.js"), "export const onlyForPackage = \"resolved from the package\";\n");
|
|
write(path.join(pkg, "node_modules", "dual-build-dep", "package.json"), JSON.stringify({
|
|
name: "dual-build-dep", exports: { ".": { module: "./bundler/index.js", default: "./node.cjs" } },
|
|
}));
|
|
write(path.join(pkg, "node_modules", "dual-build-dep", "node.cjs"), "exports.nodeBuild = \"node build\";\n");
|
|
write(path.join(pkg, "node_modules", "dual-build-dep", "bundler", "index.js"), "export * from \"./extensionless\";\n");
|
|
|
|
const server = path.join(root, "server");
|
|
mkdirSync(path.join(server, "node_modules", "@paperclipai"), { recursive: true });
|
|
symlinkSync(pkg, path.join(server, "node_modules", "@paperclipai", "fixture"), "dir");
|
|
write(path.join(server, "package.json"), JSON.stringify({ name: "server", type: "module" }));
|
|
for (const [version, flavor] of [["1", "installed at build"], ["2", "replaced after build"]]) {
|
|
const store = path.join(root, "node_modules", ".pnpm", `server-dep@${version}`);
|
|
write(path.join(store, "package.json"), JSON.stringify({ name: "server-dep", type: "module", main: "index.js" }));
|
|
write(path.join(store, "index.js"), `export const flavor = ${JSON.stringify(flavor)};\n`);
|
|
}
|
|
symlinkSync(path.join(root, "node_modules", ".pnpm", "server-dep@1"), path.join(server, "node_modules", "server-dep"), "dir");
|
|
write(path.join(server, "dist", "late.js"), "import { values } from \"@paperclipai/fixture\";\nexport const late = () => values.doubled + 1;\n");
|
|
write(path.join(server, "dist", "index.js"), [
|
|
"import { asset, location, values } from \"@paperclipai/fixture\";",
|
|
"import { flavor } from \"server-dep\";",
|
|
"import { pathToFileURL } from \"node:url\";",
|
|
"const { late } = await import(\"./late.js\");",
|
|
"// The server only starts when it is the main module.",
|
|
"if (pathToFileURL(process.argv[1]).href === import.meta.url) {",
|
|
" console.log(JSON.stringify({ asset, location, values, flavor, late: late(), entryDir: import.meta.dirname }));",
|
|
"}",
|
|
"",
|
|
].join("\n"));
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmSync(root, { recursive: true, force: true });
|
|
});
|
|
|
|
it("packages a maintenance entry without starting the application or needing a TypeScript loader", async () => {
|
|
const maintenanceEntry = path.join(root, "server", "dist", "maintenance.js");
|
|
write(maintenanceEntry, 'import { asset } from "@paperclipai/fixture"; console.log(JSON.stringify({ asset, phase: process.argv.at(-1) }));');
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir, maintenanceEntry });
|
|
const { stdout } = await execFileAsync(process.execPath, [path.join(outdir, "agent-files.mjs"), "--phase=eligibility"], { timeout: 30_000 });
|
|
expect(JSON.parse(stdout)).toEqual({ asset: "asset beside its source", phase: "--phase=eligibility" });
|
|
});
|
|
|
|
it("packages standalone enrollment without starting the app", async () => {
|
|
const ephemeralMigrationEntry = path.join(root, "server", "dist", "migration.js");
|
|
write(ephemeralMigrationEntry, 'import { values } from "@paperclipai/fixture"; console.log(JSON.stringify({ migrated: values.doubled }));');
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir, ephemeralMigrationEntry });
|
|
const { stdout } = await execFileAsync(process.execPath, [path.join(outdir, "ephemeral-migration.mjs")], { timeout: 30_000 });
|
|
expect(JSON.parse(stdout)).toEqual({ migrated: 42 });
|
|
});
|
|
|
|
it("bundles workspace TypeScript and runs with plain node", async () => {
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir });
|
|
expect(summary.outfile).toBe(path.join(outdir, "index.mjs"));
|
|
|
|
const { stdout } = await execFileAsync(process.execPath, [summary.outfile], { cwd: path.join(root, "server"), timeout: 30_000 });
|
|
const output = JSON.parse(stdout);
|
|
expect(output.values).toEqual({ doubled: 42, onlyForPackage: "resolved from the package", nodeBuild: "node build" });
|
|
expect(output.late).toBe(43);
|
|
expect(output.asset).toBe("asset beside its source");
|
|
expect(output.location).toBe(path.join(root, "packages", "fixture", "src"));
|
|
expect(output.entryDir).toBe(path.join(root, "server", "dist"));
|
|
});
|
|
|
|
it("keeps database caller locations public in a source-mapped runtime bundle", async () => {
|
|
const captureModule = fileURLToPath(new URL("../../packages/db/src/response-capture.ts", import.meta.url));
|
|
write(path.join(root, "server", "dist", "services", "response-caller.js"), [
|
|
`import { publicDatabaseCallers } from ${JSON.stringify(captureModule)};`,
|
|
'export function callers() { return publicDatabaseCallers(new Error("private-capture-message").stack); }',
|
|
].join("\n"));
|
|
write(path.join(root, "server", "dist", "index.js"), [
|
|
'import { callers } from "./services/response-caller.js";',
|
|
'console.log(JSON.stringify({ callers: callers() }));',
|
|
].join("\n"));
|
|
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir: path.join(root, "server", "dist-runtime") });
|
|
const env = { ...process.env };
|
|
delete env.NODE_OPTIONS;
|
|
delete env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW;
|
|
const options = { cwd: path.join(root, "server"), env, timeout: 30_000 };
|
|
const plain = JSON.parse((await execFileAsync(process.execPath, [summary.outfile], options)).stdout);
|
|
const mapped = JSON.parse((await execFileAsync(process.execPath, ["--enable-source-maps", summary.outfile], options)).stdout);
|
|
expect(plain.callers).toEqual([]);
|
|
expect(mapped.callers).toEqual([expect.stringMatching(/^services\/response-caller\.js:\d+:\d+$/)]);
|
|
for (const result of [plain, mapped]) {
|
|
const output = JSON.stringify(result);
|
|
expect(output).not.toContain(root);
|
|
expect(output).not.toContain(captureModule);
|
|
expect(output).not.toContain("private-capture-message");
|
|
}
|
|
});
|
|
|
|
it("emits a server module that loads without starting the server", async () => {
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir });
|
|
expect(summary.serverModule).toBe(path.join(outdir, "server.mjs"));
|
|
const probe = path.join(root, "server", "probe.mjs");
|
|
write(probe, `await import(${JSON.stringify(pathToFileURL(summary.serverModule).href)});\nconsole.log("loaded");\n`);
|
|
const loaded = await execFileAsync(process.execPath, [probe], { cwd: path.join(root, "server"), timeout: 30_000 });
|
|
expect(loaded.stdout.trim()).toBe("loaded");
|
|
const started = await execFileAsync(process.execPath, [summary.outfile], { cwd: path.join(root, "server"), timeout: 30_000 });
|
|
expect(JSON.parse(started.stdout).late).toBe(43);
|
|
});
|
|
|
|
it("leaves the server's own dependencies to runtime resolution", async () => {
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir });
|
|
rmSync(path.join(root, "server", "node_modules", "server-dep"));
|
|
symlinkSync(path.join(root, "node_modules", ".pnpm", "server-dep@2"), path.join(root, "server", "node_modules", "server-dep"), "dir");
|
|
const { stdout } = await execFileAsync(process.execPath, [summary.outfile], { cwd: path.join(root, "server"), timeout: 30_000 });
|
|
expect(JSON.parse(stdout).flavor).toBe("replaced after build");
|
|
});
|
|
|
|
it("keeps third-party packages external", async () => {
|
|
const outdir = path.join(root, "server", "dist-runtime");
|
|
await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir });
|
|
const { stdout } = await execFileAsync("grep", ["-rl", "resolved from the package", outdir], { encoding: "utf8" }).catch(() => ({ stdout: "" }));
|
|
expect(stdout.trim()).toBe("");
|
|
});
|
|
});
|
|
|
|
describe("preserveModuleLocation", () => {
|
|
it("replaces import.meta location properties with the original path", () => {
|
|
const file = "/srv/app/packages/x/src/mod.ts";
|
|
const out = preserveModuleLocation("a(import.meta.url); b(import.meta.dirname); c(import.meta.filename); d(import.meta.resolve);", file);
|
|
expect(out).toBe(`a("file:///srv/app/packages/x/src/mod.ts"); b("/srv/app/packages/x/src"); c("/srv/app/packages/x/src/mod.ts"); d(import.meta.resolve);`);
|
|
});
|
|
});
|