From e7f31705dad6dfdfd0f0f3f9e017ed2e0afc584b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 06:40:21 -0600 Subject: [PATCH] perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary runs at a time, so its duration sets how often merged work ships > - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`, and 4 min 40 s of that is an uncached `cargo build --release` of `paperclip-runnerd` > - The release then replaces that binary with the exact-source daemons that `release_runner_binaries` already built and verified for all four targets > - This pull request stages the verified daemon for the publisher's own target during a release, instead of compiling it > - The benefit is about 4.5 min less per canary, with the same verified bytes published ## Linked Issues or Issue Description **What happened?** In canary run 43487884481 (2026-11-13), the `Publish canary` step took 28.6 min. Its `pnpm build` ran 19:42-19:50. The `packages/paperclip-runner build` took 5.3 min, and its `cargo build` reported `Finished release profile ... in 4m 40s`. Every other package waited for it. `release.sh` then runs `assemble-daemons`, which overwrites `dist/bin` with the four exact-source release daemons. **Expected behavior** A release does not compile a binary that it discards. The Runner build stamp stays valid, so the server build still reuses the Runner build. Refs #21554 (the other canary speed-up) ## What Changed - `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new): when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the release daemons with the existing `releaseDaemonManifest` (source revision equal to `HEAD`, digest, architecture, build metadata) and stages the host target's daemon at `dist/bin` with the existing `stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as before. - The Runner `build` chain calls this script in place of `pnpm run build:binary`. `build:binary` itself and the image build are unchanged. - `stage-release-runner-binaries.mjs` exports `metadataContract`. - Tests: `stage-runner-binary.test.mjs` (staging per target, wrong source, bad digest, missing target, non-release fallback); `runner-image-identity.test.ts` step list. - `doc/RELEASING.md` describes the behavior. ## Verification - `node --test packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9 pass. - `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42 of 42 pass. - With the real daemon artifacts of canary run 43487884481 (receipts pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=... pnpm run build` in `packages/paperclip-runner` finished in 15 s with no `cargo` line. The build stamp was recorded, and `server/scripts/prepare-runner-vendor.mjs` then reported `action:reuse,reason:inputs_unchanged`. The staged daemon runs. - After merge: the next canary's `packages/paperclip-runner build` log shows "Staged the verified exact-source linux-x64 release daemon" and no `Compiling` lines. ## Risks - Low. A release still fails closed: a daemon with the wrong source, digest, architecture or metadata stops the build, and `assemble-daemons` checks all four again afterwards. - Local, Docker and CI builds without the variable are unchanged. - On macOS, staging re-signs the binary, as it does for a compiled one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/RELEASING.md | 7 +++ packages/paperclip-runner/package.json | 2 +- .../scripts/build-runner-binary.mjs | 52 +++++++++++++++++ .../scripts/stage-release-runner-binaries.mjs | 2 +- .../scripts/stage-runner-binary.test.mjs | 58 +++++++++++++++++++ .../__tests__/runner-image-identity.test.ts | 2 +- 6 files changed, 120 insertions(+), 3 deletions(-) create mode 100644 packages/paperclip-runner/scripts/build-runner-binary.mjs diff --git a/doc/RELEASING.md b/doc/RELEASING.md index 68230fcb677..d88794e7232 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -137,6 +137,13 @@ discarded. The native daemon build cache holds dependency crates only. Its key covers the Cargo manifest, Cargo configuration, host and target, but not the Runner's own source, so a Runner change does not discard the dependencies. +During a release (`PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` set), the Runner +package build does not compile `paperclip-runnerd`. It verifies the +exact-source release daemon for the publisher's own target (source, digest, +architecture and build metadata) and stages it at `dist/bin`, so the build +stamp and the server's vendored Runner stay current. Every other build still +runs `build:binary`. + After publication, the onboarding smoke checks out the same source SHA and installs its test dependencies from the publisher's resolved lockfile with `--frozen-lockfile`. The same-run artifact is captured before tracked files are diff --git a/packages/paperclip-runner/package.json b/packages/paperclip-runner/package.json index c89d6b11040..b57658f5b54 100644 --- a/packages/paperclip-runner/package.json +++ b/packages/paperclip-runner/package.json @@ -67,7 +67,7 @@ }, "sideEffects": false, "scripts": { - "build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish", + "build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && node scripts/build-runner-binary.mjs && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish", "build:typescript": "pnpm run check:acpx-profiles && pnpm run ensure:eval-build-deps && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json && node scripts/build-verified-provider-entrypoints.mjs && node scripts/install-copilot-assets.mjs", "build:rust": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins", "build:binary": "cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/stage-runner-binary.mjs", diff --git a/packages/paperclip-runner/scripts/build-runner-binary.mjs b/packages/paperclip-runner/scripts/build-runner-binary.mjs new file mode 100644 index 00000000000..dd9b25be093 --- /dev/null +++ b/packages/paperclip-runner/scripts/build-runner-binary.mjs @@ -0,0 +1,52 @@ +import { execFileSync, spawnSync } from "node:child_process"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { stageRunnerBinary } from "./stage-runner-binary.mjs"; + +export const RELEASE_ARTIFACTS_VARIABLE = "PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR"; + +export function hostRunnerTarget(platform = process.platform, arch = process.arch) { + return `${platform}-${arch}`; +} + +export async function stageReleaseHostBinary({ artifactDirectory, sourceRevision, packageRoot, verifyMetadata, target = hostRunnerTarget(), stage = stageRunnerBinary }) { + const { releaseDaemonManifest } = await import("./stage-release-runner-binaries.mjs"); + const manifest = releaseDaemonManifest(artifactDirectory, sourceRevision, verifyMetadata); + const artifact = manifest.platforms[target]; + if (!artifact) throw new Error(`No exact-source release daemon for ${target}`); + const executable = target.startsWith("win32-") ? "paperclip-runnerd.exe" : "paperclip-runnerd"; + await stage(artifact.path, path.join(packageRoot, "dist", "bin", executable)); + return { target, sha256: artifact.sha256 }; +} + +function pnpmInvocation(env) { + const npmExecPath = env.npm_execpath?.trim(); + if (npmExecPath && npmExecPath.toLowerCase().includes("pnpm")) { + return /\.(c|m)?js$/i.test(npmExecPath) ? { command: process.execPath, args: [npmExecPath] } : { command: npmExecPath, args: [] }; + } + return { command: "pnpm", args: [] }; +} + +export async function main({ env = process.env, packageRoot, log = console.log } = {}) { + const root = packageRoot ?? path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); + const artifactDirectory = env[RELEASE_ARTIFACTS_VARIABLE]?.trim(); + if (artifactDirectory) { + const { metadataContract } = await import("./stage-release-runner-binaries.mjs"); + const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim(); + const staged = await stageReleaseHostBinary({ artifactDirectory, sourceRevision, packageRoot: root, verifyMetadata: await metadataContract() }); + log(`Staged the verified exact-source ${staged.target} release daemon (${staged.sha256}) instead of compiling it.`); + return { reused: true, ...staged }; + } + const { command, args } = pnpmInvocation(env); + const result = spawnSync(command, [...args, "run", "build:binary"], { cwd: root, env, stdio: "inherit", shell: process.platform === "win32" && command === "pnpm" }); + if (result.error) throw result.error; + if (result.status !== 0) throw new Error(`build:binary exited with ${result.status ?? result.signal}`); + return { reused: false }; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch((error) => { + console.error(error.message); + process.exitCode = 1; + }); +} diff --git a/packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs b/packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs index ab6153b7ce7..0b5b1224572 100644 --- a/packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs +++ b/packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs @@ -31,7 +31,7 @@ function regularFile(path, maximumSize) { return readFileSync(path); } -async function metadataContract() { +export async function metadataContract() { const { PAPERCLIP_RUNNER_BUILD_METADATA } = await import("../dist/evals/build-metadata.js"); const { parsePaperclipRunnerdBuildMetadata } = await import("../dist/evals/runnerd-artifact.js"); return metadata => verifyReleaseRunnerMetadata(parsePaperclipRunnerdBuildMetadata(metadata), PAPERCLIP_RUNNER_BUILD_METADATA); diff --git a/packages/paperclip-runner/scripts/stage-runner-binary.test.mjs b/packages/paperclip-runner/scripts/stage-runner-binary.test.mjs index ddffc53e4c8..e262d64a213 100644 --- a/packages/paperclip-runner/scripts/stage-runner-binary.test.mjs +++ b/packages/paperclip-runner/scripts/stage-runner-binary.test.mjs @@ -7,6 +7,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { test } from "node:test"; import { stageBuiltRunnerBinary, stageRunnerBinary } from "./stage-runner-binary.mjs"; +import { main as buildRunnerBinary, RELEASE_ARTIFACTS_VARIABLE, stageReleaseHostBinary } from "./build-runner-binary.mjs"; import { RELEASE_RUNNER_TARGETS, releaseRunnerTargets, releaseDaemonManifest, verifyReleaseRunnerMetadata } from "./stage-release-runner-binaries.mjs"; import { paperclipRunnerBinaryNeedsBuild, resolveNativeRunnerDevBinary } from "../../../scripts/dev-runner-native-binary.mjs"; import { resolvePackagedRunnerBinary } from "../src/live/runner-binary.ts"; @@ -231,3 +232,60 @@ test("failed preparation preserves the published bytes and mode", async () => { await rm(root, { recursive: true, force: true }); } }); + +async function writeReleaseDaemons(root, source) { + for (const target of RELEASE_RUNNER_TARGETS) { + await mkdir(path.join(root, target), { recursive: true }); + const bytes = Buffer.alloc(64); + if (target.startsWith("linux-")) { + bytes.set([0x7f, 0x45, 0x4c, 0x46, 2, 1]); bytes.writeUInt16LE(target === "linux-x64" ? 62 : 183, 18); + } else { bytes.writeUInt32LE(0xfeedfacf); bytes.writeUInt32LE(target === "darwin-x64" ? 0x01000007 : 0x0100000c, 4); } + await writeFile(path.join(root, target, "paperclip-runnerd"), bytes); + await writeFile(path.join(root, target, "receipt.json"), JSON.stringify({ schema: "paperclip.runner.release-artifact.v1", sourceRevision: source, + target, sha256: "sha256:" + createHash("sha256").update(bytes).digest("hex"), buildMetadata: metadata })); + } +} + +test("a release build stages the verified exact-source host daemon instead of compiling it", async () => { + const root = await realpath(await mkdtemp(path.join(tmpdir(), "runner-release-host-"))), source = "a".repeat(40); + const artifacts = path.join(root, "artifacts"), packageRoot = path.join(root, "package"); + const verifyMetadata = value => verifyReleaseRunnerMetadata(value, contract); + const staged = []; + const stage = async (from, to) => { staged.push([from, to]); await mkdir(path.dirname(to), { recursive: true }); await copyFile(from, to); }; + try { + await writeReleaseDaemons(artifacts, source); + for (const target of RELEASE_RUNNER_TARGETS) { + const result = await stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target, stage }); + assert.equal(result.target, target); + const bytes = await readFile(path.join(packageRoot, "dist", "bin", "paperclip-runnerd")); + assert.deepEqual(bytes, await readFile(path.join(artifacts, target, "paperclip-runnerd"))); + assert.deepEqual(staged.at(-1), [path.join(artifacts, target, "paperclip-runnerd"), path.join(packageRoot, "dist", "bin", "paperclip-runnerd")]); + } + await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: "b".repeat(40), packageRoot, verifyMetadata, target: "linux-x64", stage }), + /source, digest or architecture/); + await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target: "win32-x64", stage }), + /No exact-source release daemon for win32-x64/); + await writeFile(path.join(artifacts, "linux-x64", "paperclip-runnerd"), Buffer.alloc(64, 1)); + await assert.rejects(stageReleaseHostBinary({ artifactDirectory: artifacts, sourceRevision: source, packageRoot, verifyMetadata, target: "linux-x64", stage }), + /digest or architecture/); + assert.equal(staged.length, RELEASE_RUNNER_TARGETS.length, "rejected artifacts are never staged"); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); + +test("a build without release daemons still compiles the Runner binary", async () => { + const root = await realpath(await mkdtemp(path.join(tmpdir(), "runner-local-build-"))); + try { + const marker = path.join(root, "ran"); + await writeFile(path.join(root, "build.cjs"), `require("node:fs").writeFileSync(${JSON.stringify(marker)}, "yes");\n`); + await writeFile(path.join(root, "package.json"), JSON.stringify({ name: "fixture", private: true, scripts: { "build:binary": "node build.cjs" } })); + const env = { ...process.env }; + delete env[RELEASE_ARTIFACTS_VARIABLE]; + const result = await buildRunnerBinary({ env, packageRoot: root, log: () => {} }); + assert.deepEqual(result, { reused: false }); + assert.equal(await readFile(marker, "utf8"), "yes"); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/server/src/__tests__/runner-image-identity.test.ts b/server/src/__tests__/runner-image-identity.test.ts index 43f497eac14..18e73885d62 100644 --- a/server/src/__tests__/runner-image-identity.test.ts +++ b/server/src/__tests__/runner-image-identity.test.ts @@ -348,7 +348,7 @@ describe("Runner image stages", () => { "pnpm run check:protocol-coverage", "node scripts/generate-semantic-contracts.mjs --check", "node scripts/check-runner-workflow-traceability.mjs", - "pnpm run build:binary", + "node scripts/build-runner-binary.mjs", "node scripts/generate-replay-goldens.mjs --check", "node scripts/generate-semantic-action-catalog.mjs --check", "node scripts/record-build-inputs.mjs --finish",