mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 11:23:40 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at ce63517d0c
passed: 54 successful checks, two skipped and one neutral, with no
failed or pending statuses. Typecheck, tests and build pass. CI:
https://github.com/paperclipai/paperclip/actions/runs/41970370960. The
earlier superseded run lost a spot runner after a shutdown signal; that
infrastructure cancellation is retained and is not counted as a passing
run.
- Greptile and Ponytail both rate this exact head 5/5; all actionable
review threads are resolved. Comment-only cleanup removes newly added
prose comments under CONTRIBUTING.md. TypeScript output with
removeComments:true is byte-identical for all seven changed TypeScript
files; 20 existing command, runtime and bundle tests pass after cleanup.
- Serving enrollment, full normal-image migration, empty-home
restoration and active-agent release handoffs remain separate acceptance
gates.
## Risks
- Preparation still consumes CPU, disk and storage requests. It uses one
transfer and at most two database connections. Check memory headroom for
a separate operator process.
- A task can gain work during upload. The final fence rejects that
publication and retains the previous current object.
- Deferred sources remain migration obligations. Preparation never
permits detaching the volume.
- Ordinary backup callers do not opt into the new preparation constraint
and keep their existing behavior.
## Model Used
OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning,
code editing, shell tools and test execution. The runtime does not
expose its context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
119 lines
4.9 KiB
JavaScript
119 lines
4.9 KiB
JavaScript
|
|
import { readFile, rm } from "node:fs/promises";
|
|
import { dirname, join } from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import { build } from "esbuild";
|
|
|
|
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
|
const serverDir = join(scriptDir, "..");
|
|
|
|
const REQUIRE_SHIM = "import { createRequire as __paperclipCreateRequire } from 'node:module'; const require = __paperclipCreateRequire(import.meta.url);";
|
|
|
|
export function preserveModuleLocation(source, file) {
|
|
return source
|
|
.replace(/\bimport\.meta\.url\b/g, JSON.stringify(pathToFileURL(file).href))
|
|
.replace(/\bimport\.meta\.dirname\b/g, JSON.stringify(dirname(file)))
|
|
.replace(/\bimport\.meta\.filename\b/g, JSON.stringify(file));
|
|
}
|
|
|
|
function runtimeEntry(entry) {
|
|
return {
|
|
name: "runtime-entry",
|
|
setup(builder) {
|
|
builder.onResolve({ filter: /^paperclip-runtime-entry$/ }, () => ({ path: "paperclip-runtime-entry", namespace: "runtime-entry" }));
|
|
builder.onLoad({ filter: /.*/, namespace: "runtime-entry" }, () => ({
|
|
contents: `process.argv[1] = ${JSON.stringify(entry)};\nawait import(${JSON.stringify(entry)});\n`,
|
|
loader: "js",
|
|
resolveDir: dirname(entry),
|
|
}));
|
|
},
|
|
};
|
|
}
|
|
|
|
const keepModuleLocations = {
|
|
name: "keep-module-locations",
|
|
setup(builder) {
|
|
builder.onLoad({ filter: /\.(?:[cm]?js|[cm]?ts)$/ }, async (args) => {
|
|
if (args.path.includes("/node_modules/")) return undefined;
|
|
const source = await readFile(args.path, "utf8");
|
|
if (!source.includes("import.meta")) return undefined;
|
|
return {
|
|
contents: preserveModuleLocation(source, args.path),
|
|
loader: /\.[cm]?ts$/.test(args.path) ? "ts" : "js",
|
|
resolveDir: dirname(args.path),
|
|
};
|
|
});
|
|
},
|
|
};
|
|
|
|
function externalizeThirdParty(isBundled, runtimeDir) {
|
|
return {
|
|
name: "externalize-third-party",
|
|
setup(builder) {
|
|
builder.onResolve({ filter: /^[^./]/ }, async (args) => {
|
|
if (args.pluginData?.externalizeThirdParty || isBundled(args.path)) return undefined;
|
|
const resolved = await builder.resolve(args.path, {
|
|
kind: args.kind,
|
|
resolveDir: args.resolveDir,
|
|
importer: args.importer,
|
|
pluginData: { externalizeThirdParty: true },
|
|
});
|
|
if (resolved.errors.length > 0 || resolved.external) return { path: args.path, external: true };
|
|
const fromServer = await builder.resolve(args.path, {
|
|
kind: args.kind,
|
|
resolveDir: runtimeDir,
|
|
pluginData: { externalizeThirdParty: true },
|
|
});
|
|
if (fromServer.errors.length === 0 && fromServer.path === resolved.path) return { path: args.path, external: true };
|
|
return { path: resolved.path, external: true };
|
|
});
|
|
},
|
|
};
|
|
}
|
|
|
|
/** @param {{ entry: string, outdir: string, maintenanceEntry?: string, ephemeralMigrationEntry?: string,
|
|
* nativeHarnessPrestageEntry?: string, isBundled?: (specifier: string) => boolean }} options */
|
|
export async function bundleRuntime({ entry, outdir, maintenanceEntry, ephemeralMigrationEntry, nativeHarnessPrestageEntry, isBundled = (specifier) => specifier.startsWith("@paperclipai/") }) {
|
|
await rm(outdir, { recursive: true, force: true });
|
|
const result = await build({
|
|
entryPoints: { index: "paperclip-runtime-entry", server: entry, ...(maintenanceEntry ? { "agent-files": maintenanceEntry } : {}),
|
|
...(ephemeralMigrationEntry ? { "ephemeral-migration": ephemeralMigrationEntry } : {}),
|
|
...(nativeHarnessPrestageEntry ? { "native-harness-prestage": nativeHarnessPrestageEntry } : {}) },
|
|
outdir,
|
|
outExtension: { ".js": ".mjs" },
|
|
chunkNames: "chunk-[hash]",
|
|
bundle: true,
|
|
splitting: true,
|
|
format: "esm",
|
|
platform: "node",
|
|
target: "node24",
|
|
mainFields: ["main"],
|
|
conditions: ["node"],
|
|
banner: { js: REQUIRE_SHIM },
|
|
sourcemap: "linked",
|
|
metafile: true,
|
|
logLevel: "error",
|
|
plugins: [runtimeEntry(entry), keepModuleLocations, externalizeThirdParty(isBundled, dirname(entry))],
|
|
});
|
|
const outputs = Object.keys(result.metafile.outputs).filter((file) => file.endsWith(".mjs"));
|
|
return {
|
|
outfile: join(outdir, "index.mjs"),
|
|
serverModule: join(outdir, "server.mjs"),
|
|
inputs: Object.keys(result.metafile.inputs).length,
|
|
chunks: outputs.length,
|
|
};
|
|
}
|
|
|
|
async function main() {
|
|
const started = performance.now();
|
|
const summary = await bundleRuntime({ entry: join(serverDir, "dist", "index.js"), outdir: join(serverDir, "dist-runtime"),
|
|
maintenanceEntry: join(serverDir, "dist", "maintenance", "agent-files.js"),
|
|
ephemeralMigrationEntry: join(serverDir, "dist", "maintenance", "ephemeral-migration.js"),
|
|
nativeHarnessPrestageEntry: join(serverDir, "dist", "maintenance", "native-harness-prestage.js") });
|
|
console.log(`[bundle-runtime] ${summary.inputs} inputs -> ${summary.chunks} chunks in ${Math.round(performance.now() - started)} ms: ${summary.outfile}`);
|
|
}
|
|
|
|
if (process.argv[1] && pathToFileURL(process.argv[1]).href === import.meta.url) {
|
|
await main();
|
|
}
|