mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary runs at a time, so its duration sets how often merged work ships > - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`, and 4 min 40 s of that is an uncached `cargo build --release` of `paperclip-runnerd` > - The release then replaces that binary with the exact-source daemons that `release_runner_binaries` already built and verified for all four targets > - This pull request stages the verified daemon for the publisher's own target during a release, instead of compiling it > - The benefit is about 4.5 min less per canary, with the same verified bytes published ## Linked Issues or Issue Description **What happened?** In canary run 43487884481 (2026-11-13), the `Publish canary` step took 28.6 min. Its `pnpm build` ran 19:42-19:50. The `packages/paperclip-runner build` took 5.3 min, and its `cargo build` reported `Finished release profile ... in 4m 40s`. Every other package waited for it. `release.sh` then runs `assemble-daemons`, which overwrites `dist/bin` with the four exact-source release daemons. **Expected behavior** A release does not compile a binary that it discards. The Runner build stamp stays valid, so the server build still reuses the Runner build. Refs #21554 (the other canary speed-up) ## What Changed - `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new): when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the release daemons with the existing `releaseDaemonManifest` (source revision equal to `HEAD`, digest, architecture, build metadata) and stages the host target's daemon at `dist/bin` with the existing `stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as before. - The Runner `build` chain calls this script in place of `pnpm run build:binary`. `build:binary` itself and the image build are unchanged. - `stage-release-runner-binaries.mjs` exports `metadataContract`. - Tests: `stage-runner-binary.test.mjs` (staging per target, wrong source, bad digest, missing target, non-release fallback); `runner-image-identity.test.ts` step list. - `doc/RELEASING.md` describes the behavior. ## Verification - `node --test packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9 pass. - `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42 of 42 pass. - With the real daemon artifacts of canary run 43487884481 (receipts pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=... pnpm run build` in `packages/paperclip-runner` finished in 15 s with no `cargo` line. The build stamp was recorded, and `server/scripts/prepare-runner-vendor.mjs` then reported `action:reuse,reason:inputs_unchanged`. The staged daemon runs. - After merge: the next canary's `packages/paperclip-runner build` log shows "Staged the verified exact-source linux-x64 release daemon" and no `Compiling` lines. ## Risks - Low. A release still fails closed: a daemon with the wrong source, digest, architecture or metadata stops the build, and `assemble-daemons` checks all four again afterwards. - Local, Docker and CI builds without the variable are unchanged. - On macOS, staging re-signs the binary, as it does for a compiled one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8cd2d6a73d
commit
e7f31705da
6 files changed
+120
-3
No files matched your search
@@ -137,6 +137,13 @@ discarded. The native daemon build cache holds dependency crates only. Its key
|
||||
covers the Cargo manifest, Cargo configuration, host and target, but not the
|
||||
Runner's own source, so a Runner change does not discard the dependencies.
|
||||
|
||||
During a release (`PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` set), the Runner
|
||||
package build does not compile `paperclip-runnerd`. It verifies the
|
||||
exact-source release daemon for the publisher's own target (source, digest,
|
||||
architecture and build metadata) and stages it at `dist/bin`, so the build
|
||||
stamp and the server's vendored Runner stay current. Every other build still
|
||||
runs `build:binary`.
|
||||
|
||||
After publication, the onboarding smoke checks out the same source SHA and
|
||||
installs its test dependencies from the publisher's resolved lockfile with
|
||||
`--frozen-lockfile`. The same-run artifact is captured before tracked files are
|
||||
|
||||
@@ -67,7 +67,7 @@
|
||||
},
|
||||
"sideEffects": false,
|
||||
"scripts": {
|
||||
"build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish",
|
||||
"build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && node scripts/build-runner-binary.mjs && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish",
|
||||
"build:typescript": "pnpm run check:acpx-profiles && pnpm run ensure:eval-build-deps && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json && node scripts/build-verified-provider-entrypoints.mjs && node scripts/install-copilot-assets.mjs",
|
||||
"build:rust": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins",
|
||||
"build:binary": "cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/stage-runner-binary.mjs",
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { stageRunnerBinary } from "./stage-runner-binary.mjs";
|
||||
|
||||
export const RELEASE_ARTIFACTS_VARIABLE = "PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR";
|
||||
|
||||
export function hostRunnerTarget(platform = process.platform, arch = process.arch) {
|
||||
return `${platform}-${arch}`;
|
||||
}
|
||||
|
||||
export async function stageReleaseHostBinary({ artifactDirectory, sourceRevision, packageRoot, verifyMetadata, target = hostRunnerTarget(), stage = stageRunnerBinary }) {
|
||||
const { releaseDaemonManifest } = await import("./stage-release-runner-binaries.mjs");
|
||||
const manifest = releaseDaemonManifest(artifactDirectory, sourceRevision, verifyMetadata);
|
||||
const artifact = manifest.platforms[target];
|
||||
if (!artifact) throw new Error(`No exact-source release daemon for ${target}`);
|
||||
const executable = target.startsWith("win32-") ? "paperclip-runnerd.exe" : "paperclip-runnerd";
|
||||
await stage(artifact.path, path.join(packageRoot, "dist", "bin", executable));
|
||||
return { target, sha256: artifact.sha256 };
|
||||
}
|
||||
|
||||
function pnpmInvocation(env) {
|
||||
const npmExecPath = env.npm_execpath?.trim();
|
||||
if (npmExecPath && npmExecPath.toLowerCase().includes("pnpm")) {
|
||||
return /\.(c|m)?js$/i.test(npmExecPath) ? { command: process.execPath, args: [npmExecPath] } : { command: npmExecPath, args: [] };
|
||||
}
|
||||
return { command: "pnpm", args: [] };
|
||||
}
|
||||
|
||||
export async function main({ env = process.env, packageRoot, log = console.log } = {}) {
|
||||
const root = packageRoot ?? path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const artifactDirectory = env[RELEASE_ARTIFACTS_VARIABLE]?.trim();
|
||||
if (artifactDirectory) {
|
||||
const { metadataContract } = await import("./stage-release-runner-binaries.mjs");
|
||||
const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim();
|
||||
const staged = await stageReleaseHostBinary({ artifactDirectory, sourceRevision, packageRoot: root, verifyMetadata: await metadataContract() });
|
||||
log(`Staged the verified exact-source ${staged.target} release daemon (${staged.sha256}) instead of compiling it.`);
|
||||
return { reused: true, ...staged };
|
||||
}
|
||||
const { command, args } = pnpmInvocation(env);
|
||||
const result = spawnSync(command, [...args, "run", "build:binary"], { cwd: root, env, stdio: "inherit", shell: process.platform === "win32" && command === "pnpm" });
|
||||
if (result.error) throw result.error;
|
||||
if (result.status !== 0) throw new Error(`build:binary exited with ${result.status ?? result.signal}`);
|
||||
return { reused: false };
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch((error) => {
|
||||
console.error(error.message);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -31,7 +31,7 @@ function regularFile(path, maximumSize) {
|
||||
return readFileSync(path);
|
||||
}
|
||||
|
||||
async function metadataContract() {
|
||||
export async function metadataContract() {
|
||||
const { PAPERCLIP_RUNNER_BUILD_METADATA } = await import("../dist/evals/build-metadata.js");
|
||||
const { parsePaperclipRunnerdBuildMetadata } = await import("../dist/evals/runnerd-artifact.js");
|
||||
return metadata => verifyReleaseRunnerMetadata(parsePaperclipRunnerdBuildMetadata(metadata), PAPERCLIP_RUNNER_BUILD_METADATA);
|
||||
|
||||
@@ -7,6 +7,7 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { test } from "node:test";
|
||||
import { stageBuiltRunnerBinary, stageRunnerBinary } from "./stage-runner-binary.mjs";
|
||||
import { main as buildRunnerBinary, RELEASE_ARTIFACTS_VARIABLE, stageReleaseHostBinary } from "./build-runner-binary.mjs";
|
||||
import { RELEASE_RUNNER_TARGETS, releaseRunnerTargets, releaseDaemonManifest, verifyReleaseRunnerMetadata } from "./stage-release-runner-binaries.mjs";
|
||||
import { paperclipRunnerBinaryNeedsBuild, resolveNativeRunnerDevBinary } from "../../../scripts/dev-runner-native-binary.mjs";
|
||||
import { resolvePackagedRunnerBinary } from "../src/live/runner-binary.ts";
|
||||
@@ -231,3 +232,60 @@ test("failed preparation preserves the published bytes and mode", async () => {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
async function writeReleaseDaemons(root, source) {
|
||||
for (const target of RELEASE_RUNNER_TARGETS) {
|
||||
await mkdir(path.join(root, target), { recursive: true });
|
||||
const bytes = Buffer.alloc(64);
|
||||
if (target.startsWith("linux-")) {
|
||||
bytes.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); bytes.writeUInt16LE(target === "linux-x64" ? 62 : 183, 18);
|
||||
} else { bytes.writeUInt32LE(0xfeedfacf); bytes.writeUInt32LE(target === "darwin-x64" ? 0x01000007 : 0x0100000c, 4); }
|
||||
await writeFile(path.join(root, target, "paperclip-runnerd"), bytes);
|
||||
await writeFile(path.join(root, target, "receipt.json"), JSON.stringify({ schema: "paperclip.runner.release-artifact.v1", sourceRevision: source,
|
||||
target, sha256: "sha256:" + createHash("sha256").update(bytes).digest("hex"), buildMetadata: metadata }));
|
||||
}
|
||||
}
|
||||
|
||||
test("a release build stages the verified exact-source host daemon instead of compiling it", async () => {
|
||||
const root = await realpath(await mkdtemp(path.join(tmpdir(), "runner-release-host-"))), source = "a".repeat(40);
|
||||
const artifacts = path.join(root, "artifacts"), packageRoot = path.join(root, "package");
|
||||
const verifyMetadata = value => verifyReleaseRunnerMetadata(value, contract);
|
||||
const staged = [];
|
||||
const stage = async (from, to) => { staged.push([from, to]); await mkdir(path.dirname(to), { recursive: true }); await copyFile(from, to); };
|
||||
try {
|
||||
await writeReleaseDaemons(artifacts, source);
|
||||
for (const target of RELEASE_RUNNER_TARGETS) {
|
||||
const result = await stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target, stage });
|
||||
assert.equal(result.target, target);
|
||||
const bytes = await readFile(path.join(packageRoot, "dist", "bin", "paperclip-runnerd"));
|
||||
assert.deepEqual(bytes, await readFile(path.join(artifacts, target, "paperclip-runnerd")));
|
||||
assert.deepEqual(staged.at(-1), [path.join(artifacts, target, "paperclip-runnerd"), path.join(packageRoot, "dist", "bin", "paperclip-runnerd")]);
|
||||
}
|
||||
await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: "b".repeat(40), packageRoot, verifyMetadata, target: "linux-x64", stage }),
|
||||
/source, digest or architecture/);
|
||||
await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target: "win32-x64", stage }),
|
||||
/No exact-source release daemon for win32-x64/);
|
||||
await writeFile(path.join(artifacts, "linux-x64", "paperclip-runnerd"), Buffer.alloc(64, 1));
|
||||
await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target: "linux-x64", stage }),
|
||||
/digest or architecture/);
|
||||
assert.equal(staged.length, RELEASE_RUNNER_TARGETS.length, "rejected artifacts are never staged");
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("a build without release daemons still compiles the Runner binary", async () => {
|
||||
const root = await realpath(await mkdtemp(path.join(tmpdir(), "runner-local-build-")));
|
||||
try {
|
||||
const marker = path.join(root, "ran");
|
||||
await writeFile(path.join(root, "build.cjs"), `require("node:fs").writeFileSync(${JSON.stringify(marker)}, "yes");\n`);
|
||||
await writeFile(path.join(root, "package.json"), JSON.stringify({ name: "fixture", private: true, scripts: { "build:binary": "node build.cjs" } }));
|
||||
const env = { ...process.env };
|
||||
delete env[RELEASE_ARTIFACTS_VARIABLE];
|
||||
const result = await buildRunnerBinary({ env, packageRoot: root, log: () => {} });
|
||||
assert.deepEqual(result, { reused: false });
|
||||
assert.equal(await readFile(marker, "utf8"), "yes");
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -348,7 +348,7 @@ describe("Runner image stages", () => {
|
||||
"pnpm run check:protocol-coverage",
|
||||
"node scripts/generate-semantic-contracts.mjs --check",
|
||||
"node scripts/check-runner-workflow-traceability.mjs",
|
||||
"pnpm run build:binary",
|
||||
"node scripts/build-runner-binary.mjs",
|
||||
"node scripts/generate-replay-goldens.mjs --check",
|
||||
"node scripts/generate-semantic-action-catalog.mjs --check",
|
||||
"node scripts/record-build-inputs.mjs --finish",
|
||||
|
||||
Reference in new issue
Block a user