Files
PaperClipAI/server/src/__tests__/runner-image-identity.test.ts
T
DottaandClaude Opus 5.5 e7f31705da perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary runs at a time,
so its duration sets how often merged work ships
> - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`,
and 4 min 40 s of that is an uncached `cargo build --release` of
`paperclip-runnerd`
> - The release then replaces that binary with the exact-source daemons
that `release_runner_binaries` already built and verified for all four
targets
> - This pull request stages the verified daemon for the publisher's own
target during a release, instead of compiling it
> - The benefit is about 4.5 min less per canary, with the same verified
bytes published

## Linked Issues or Issue Description

**What happened?**
In canary run 43487884481 (2026-11-13), the `Publish canary` step took
28.6 min. Its `pnpm build` ran 19:42-19:50. The
`packages/paperclip-runner build` took 5.3 min, and its `cargo build`
reported `Finished release profile ... in 4m 40s`. Every other package
waited for it. `release.sh` then runs `assemble-daemons`, which
overwrites `dist/bin` with the four exact-source release daemons.

**Expected behavior**
A release does not compile a binary that it discards. The Runner build
stamp stays valid, so the server build still reuses the Runner build.

Refs #21554 (the other canary speed-up)

## What Changed

- `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new):
when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the
release daemons with the existing `releaseDaemonManifest` (source
revision equal to `HEAD`, digest, architecture, build metadata) and
stages the host target's daemon at `dist/bin` with the existing
`stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as
before.
- The Runner `build` chain calls this script in place of `pnpm run
build:binary`. `build:binary` itself and the image build are unchanged.
- `stage-release-runner-binaries.mjs` exports `metadataContract`.
- Tests: `stage-runner-binary.test.mjs` (staging per target, wrong
source, bad digest, missing target, non-release fallback);
`runner-image-identity.test.ts` step list.
- `doc/RELEASING.md` describes the behavior.

## Verification

- `node --test
packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9
pass.
- `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42
of 42 pass.
- With the real daemon artifacts of canary run 43487884481 (receipts
pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=...
pnpm run build` in `packages/paperclip-runner` finished in 15 s with no
`cargo` line. The build stamp was recorded, and
`server/scripts/prepare-runner-vendor.mjs` then reported
`action:reuse,reason:inputs_unchanged`. The staged daemon runs.
- After merge: the next canary's `packages/paperclip-runner build` log
shows "Staged the verified exact-source linux-x64 release daemon" and no
`Compiling` lines.

## Risks

- Low. A release still fails closed: a daemon with the wrong source,
digest, architecture or metadata stops the build, and `assemble-daemons`
checks all four again afterwards.
- Local, Docker and CI builds without the variable are unchanged.
- On macOS, staging re-signs the binary, as it does for a compiled one.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:40:21 -06:00

363 lines
20 KiB
TypeScript

import { chmodSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { afterEach, describe, expect, it } from "vitest";
import { computeBuildInputsDigest } from "../../../packages/paperclip-runner/scripts/build-inputs.mjs";
import {
RUNNER_IMAGE_CONTEXT_INPUTS,
RUNNER_IMAGE_INPUTS_SCHEMA,
computeContextTreeDigest,
computeRunnerImageIdentity,
} from "../../../scripts/runner-image-identity.mjs";
/**
* The Runner image is named after its identity. An identity that misses an
* input gives a changed Runner the name of an older one.
*/
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function write(root: string, relative: string, content: string) {
const file = path.join(root, ...relative.split("/"));
mkdirSync(path.dirname(file), { recursive: true });
writeFileSync(file, content);
return file;
}
/** A repository with the real Dockerfile and one small file for every kind of input. */
function repository(): string {
const root = mkdtempSync(path.join(tmpdir(), "runner-image-identity-"));
roots.push(root);
write(root, "Dockerfile", dockerfile);
write(root, ".dockerignore", "node_modules\n");
for (const input of RUNNER_IMAGE_CONTEXT_INPUTS) {
if (!input.startsWith("packages/paperclip-") && input !== "patches") write(root, input, `input ${input}\n`);
}
write(root, "packages/paperclip-runner/package.json", "{}\n");
write(root, "packages/paperclip-runner/src/index.ts", "export const runner = 1;\n");
write(root, "packages/paperclip-runner/runner/crates/runner-core/src/lib.rs", "pub fn runner() {}\n");
write(root, "packages/paperclip-eval-kernel/src/index.ts", "export const kernel = 1;\n");
write(root, "patches/acpx@0.22.0.patch", "patch\n");
write(root, "ui/src/App.tsx", "export const app = 1;\n");
write(root, "server/src/index.ts", "export const server = 1;\n");
write(root, "packages/shared/src/index.ts", "export const shared = 1;\n");
return root;
}
const digestOf = (root: string) => computeRunnerImageIdentity(root).digest as string;
describe("Runner image identity", () => {
it("reports the tag, the context tree, the Dockerfile and the Runner build inputs", () => {
const identity = computeRunnerImageIdentity(repository());
expect(identity.schema).toBe(RUNNER_IMAGE_INPUTS_SCHEMA);
expect(identity.digest).toMatch(/^[0-9a-f]{64}$/);
expect(identity.tag).toBe(`inputs-${identity.digest}`);
expect(identity.image).toBe("ghcr.io/paperclipai/paperclip-runner");
expect(identity.platforms).toEqual(["linux/amd64", "linux/arm64"]);
expect(identity.context.inputs).toEqual(RUNNER_IMAGE_CONTEXT_INPUTS);
expect(identity.context.digest).toMatch(/^[0-9a-f]{64}$/);
expect(identity.dockerfile.sha256).toMatch(/^[0-9a-f]{64}$/);
expect(identity.buildInputs.schema).toBe("paperclip.runner-build-inputs.v2");
});
it("is the same for the same tree", () => {
const root = repository();
expect(digestOf(root)).toBe(digestOf(root));
expect(digestOf(repository())).toBe(digestOf(root));
});
it.each([
["a Runner TypeScript source file", "packages/paperclip-runner/src/index.ts"],
["a Runner Rust source file", "packages/paperclip-runner/runner/crates/runner-core/src/lib.rs"],
["the eval kernel the Runner builds first", "packages/paperclip-eval-kernel/src/index.ts"],
["a dependency patch", "patches/acpx@0.22.0.patch"],
["the lockfile", "pnpm-lock.yaml"],
["the root manifest", "package.json"],
["the pnpm configuration", ".npmrc"],
["the pack archive script", "scripts/create-provider-pack-archive.mjs"],
["the layout script", "scripts/runner-image-layout.mjs"],
["the contract copy that build:typescript checks", "packages/shared/src/provider-failure.generated.ts"],
["the Dockerfile, in a stage the Runner is not built from", "Dockerfile"],
])("changes when %s changes", (_label, file) => {
const root = repository();
const before = digestOf(root);
write(root, file, file === "Dockerfile" ? dockerfile.replace("FROM base AS production\n", "FROM base AS production\nRUN true\n") : "changed\n");
expect(digestOf(root)).not.toBe(before);
});
it.each([
["a UI source file", "ui/src/App.tsx"],
["a server source file", "server/src/index.ts"],
["another package's source file", "packages/shared/src/index.ts"],
["a new file outside the Runner's inputs", "server/src/services/new-service.ts"],
])("does not change when %s changes", (_label, file) => {
const root = repository();
const before = digestOf(root);
write(root, file, "changed\n");
expect(digestOf(root)).toBe(before);
});
it("refuses an input that is missing from the build context", () => {
const root = repository();
rmSync(path.join(root, "scripts/create-provider-pack-archive.mjs"));
expect(() => computeRunnerImageIdentity(root)).toThrow(/scripts\/create-provider-pack-archive\.mjs, which is missing/);
});
describe("context tree", () => {
// The Runner's build-input digest skips dot-directories and any directory
// named dist, and it does not read file modes. The image build copies
// those paths and modes all the same, so the identity must see them.
it.each([
["a file in a dot-directory", "packages/paperclip-runner/runner/.cargo/config.toml"],
["a file in a directory named dist", "packages/paperclip-runner/src/vendor/dist/bundle.js"],
])("changes for %s, which the Runner build-input digest does not read", (_label, file) => {
const root = repository();
const before = computeRunnerImageIdentity(root);
write(root, file, "[build]\n");
const after = computeRunnerImageIdentity(root);
expect(after.buildInputs.digest).toBe(before.buildInputs.digest);
expect(after.digest).not.toBe(before.digest);
});
it("changes when only the executable bit of a file changes", () => {
const root = repository();
const file = write(root, "packages/paperclip-runner/scripts/provision.sh", "#!/bin/sh\n");
chmodSync(file, 0o644);
const before = computeRunnerImageIdentity(root);
chmodSync(file, 0o755);
expect(computeBuildInputsDigest(root).digest).toBe(before.buildInputs.digest);
expect(digestOf(root)).not.toBe(before.digest);
});
it("records a symbolic link by its target and does not follow it", () => {
const root = repository();
const link = path.join(root, "packages/paperclip-runner/current");
symlinkSync("src", link);
const before = digestOf(root);
rmSync(link);
symlinkSync("../../server/src", link);
const after = digestOf(root);
expect(after).not.toBe(before);
write(root, "server/src/index.ts", "changed\n");
expect(digestOf(root)).toBe(after);
});
it("changes when a file is added or removed, and not with the order of a directory listing", () => {
const root = repository();
const before = digestOf(root);
write(root, "packages/paperclip-eval-kernel/src/extra.ts", "export {};\n");
expect(digestOf(root)).not.toBe(before);
rmSync(path.join(root, "packages/paperclip-eval-kernel/src/extra.ts"));
expect(digestOf(root)).toBe(before);
const other = repository();
for (const name of ["b.ts", "Z.ts", "a.ts", "_c.ts"]) write(root, `packages/paperclip-runner/src/${name}`, "x\n");
for (const name of ["_c.ts", "a.ts", "Z.ts", "b.ts"]) write(other, `packages/paperclip-runner/src/${name}`, "x\n");
expect(computeContextTreeDigest(root)).toEqual(computeContextTreeDigest(other));
});
});
describe(".dockerignore", () => {
// Its rules decide which files the build receives, and nothing else. The
// identity is computed inside the build, from the files it received.
it("does not change the identity when its text changes and the received files do not", () => {
const root = repository();
const before = digestOf(root);
write(root, ".dockerignore", "node_modules\nui/dist\nserver/tmp\n");
expect(digestOf(root)).toBe(before);
});
it("changes the identity when a rule removes a file from the Runner's context", () => {
const root = repository();
write(root, "packages/paperclip-runner/styles.css", "body {}\n");
const before = digestOf(root);
// What a new `packages/paperclip-runner/styles.css` rule does to the build context.
rmSync(path.join(root, "packages/paperclip-runner/styles.css"));
expect(digestOf(root)).not.toBe(before);
});
it("keeps out of the build context every directory a build script or a Vite configuration of a package among the identity's inputs writes its output into, so a tree with a local build and a clean checkout of one commit give the build the same files", () => {
const rules = readFileSync(path.join(repoRoot, ".dockerignore"), "utf8").split(/\r?\n/).filter((rule) => rule && !rule.startsWith("#"));
const lastRuleFor = (target: string) => rules.filter((rule) => target.split("/").some((_, index, segments) =>
path.matchesGlob(segments.slice(0, index + 1).join("/"), rule.replace(/^!/, "").replace(/^\//, "")))).at(-1);
const outputs = RUNNER_IMAGE_CONTEXT_INPUTS.filter((input) => existsSync(path.join(repoRoot, input, "package.json"))).flatMap((input) => {
const manifest = JSON.parse(readFileSync(path.join(repoRoot, input, "package.json"), "utf8"));
const scripts = Object.values(manifest.scripts as Record<string, string>).join("\n");
const emitted = [...scripts.matchAll(/\btsc -p (\S+)(?!\S| --noEmit)/g)].map((match) =>
JSON.parse(readFileSync(path.join(repoRoot, input, match[1]), "utf8")).compilerOptions.outDir as string);
const compiled = [...scripts.matchAll(/--manifest-path (\S+)\/Cargo\.toml/g)].map((match) => `${match[1]}/target`);
const bundled = readdirSync(path.join(repoRoot, input)).filter((name) => /^vite\..*config\.[cm]?[jt]s$/.test(name)).flatMap((name) =>
[...readFileSync(path.join(repoRoot, input, name), "utf8").matchAll(/\boutDir:\s*(?:resolve\(packageRoot,\s*)?"([^"]+)"/g)].map((match) => match[1]));
return [...emitted, ...compiled, ...bundled].map((output) => path.posix.join(input, output));
});
expect(outputs).toEqual(expect.arrayContaining([
"packages/paperclip-eval-kernel/dist",
"packages/paperclip-runner/dist",
"packages/paperclip-runner/dist-browser",
"packages/paperclip-runner/dist-issue-thread",
"packages/paperclip-runner/dist-scenarios",
"packages/paperclip-runner/dist-sdk",
"packages/paperclip-runner/dist-standalone",
"packages/paperclip-runner/runner/target",
]));
for (const output of new Set(outputs)) expect(lastRuleFor(output)?.startsWith("!"), output).toBe(false);
});
});
});
/**
* The identity hashes the whole Dockerfile and the context inputs. That is
* the whole of what the build reads only while these three rules hold.
*/
describe("Runner image stages", () => {
// BuildKit accepts an instruction in any case and with any indentation, and
// FROM with flags. A stage starts wherever BuildKit starts one; the first
// rule below then allows only the spelling the other rules can read.
const stagesOf = (text: string) => [...text.matchAll(/^[ \t]*from[ \t].*$/gim)].map((match, index, all) => ({
from: /^FROM (\S+)(?: AS \S+)?$/.exec(match[0])?.[1] ?? "",
// The name BuildKit gives the stage, however the line is spelt.
name: (/\sas\s+(\S+)\s*$/i.exec(match[0])?.[1] ?? "").toLowerCase(),
lines: text.slice(match.index, all[index + 1]?.index ?? text.length).split("\n").filter((line) => !/^\s*#/.test(line)),
}));
type Stage = ReturnType<typeof stagesOf>[number];
/** The lines of a stage that read something and are not in the one spelling these rules read. */
const misspelt = (stage: Stage) => stage.lines.filter((line, index) => (index === 0
// No lowercase keyword, no indentation, no flag on FROM.
? !/^FROM \S+ AS runner-artifacts[a-z-]*$/.test(line)
// No ADD or ONBUILD, no other flag, and no source on a continuation line.
: /^\s*(?:from|copy|add|onbuild)(?:\s|$)/i.test(line)
&& !/^COPY (?:--link )?(?:--from=runner-artifacts[a-z-]* )?[^-\s]\S*(?: [^\s\\]+)+$/.test(line)));
const stages = stagesOf(dockerfile);
const names = stages.map((stage) => stage.name);
const runner = stages.filter((stage) => stage.name.startsWith("runner-artifacts"));
const copies = (stage: Stage) => stage.lines.filter((line) => /^COPY\s/.test(line));
it("spell every FROM and COPY the one way these rules read, and use no ADD or ONBUILD", () => {
expect(runner.length).toBeGreaterThan(3);
for (const stage of runner) expect(misspelt(stage), stage.name).toEqual([]);
});
it.each([
["a lowercase stage keyword", "FROM scratch as runner-artifacts-extra\nCOPY secrets /\n"],
["a lowercase from", "from scratch AS runner-artifacts-extra\n"],
["a flag on FROM", "FROM --platform=linux/amd64 node:24 AS runner-artifacts-extra\n"],
["a lowercase copy", "FROM scratch AS runner-artifacts-extra\ncopy secrets /\n"],
["an indented COPY", "FROM scratch AS runner-artifacts-extra\n COPY secrets /\n"],
["an ADD in any case", "FROM scratch AS runner-artifacts-extra\n add secrets /\n"],
["a COPY continued on the next line", "FROM scratch AS runner-artifacts-extra\nCOPY package.json \\\n secrets /\n"],
["a COPY with another flag", "FROM scratch AS runner-artifacts-extra\nCOPY --parents secrets /\n"],
["an ONBUILD trigger", "FROM scratch AS runner-artifacts-extra\nONBUILD COPY secrets /\n"],
])("would catch %s in a stage BuildKit reads as one of the Runner's", (_label, text) => {
const added = stagesOf(text).filter((stage) => stage.name.startsWith("runner-artifacts"));
expect(added).toHaveLength(1);
expect(misspelt(added[0])).not.toEqual([]);
});
it("start only from each other, from scratch, or from an image pinned by digest", () => {
for (const stage of runner) {
if (stage.from === "scratch" || names.includes(stage.from)) {
expect(stage.from === "scratch" || stage.from.startsWith("runner-artifacts"), stage.name).toBe(true);
} else {
expect(stage.from, stage.name).toMatch(/@sha256:[0-9a-f]{64}$/);
}
for (const line of copies(stage)) {
const source = /--from=(\S+)/.exec(line)?.[1];
if (source) expect(source, line).toMatch(/^runner-artifacts/);
}
}
});
it("read the build context only through runner-artifacts-context", () => {
for (const stage of runner) {
const text = stage.lines.join("\n");
expect(text, stage.name).not.toMatch(/^ADD\s/m);
expect(text, stage.name).not.toMatch(/--mount=/);
expect(text, stage.name).not.toMatch(/<</);
if (stage.name === "runner-artifacts-context") continue;
const fromContext = copies(stage).filter((line) => !line.includes("--from="));
// The identity stage copies the Dockerfile itself, whose text the identity covers.
expect(fromContext, stage.name).toEqual(stage.name === "runner-artifacts-identity-compute" ? ["COPY Dockerfile ./"] : []);
}
});
it("copy into runner-artifacts-context exactly the paths the identity hashes", () => {
const context = runner.find((stage) => stage.name === "runner-artifacts-context")!;
const sources = copies(context).flatMap((line) => line.split(/\s+/).slice(1, -1));
expect(sources.filter((source) => source.startsWith("--"))).toEqual([]);
expect([...sources].sort()).toEqual([...RUNNER_IMAGE_CONTEXT_INPUTS].sort());
});
it("end in two file trees, the full one being the slim one plus the transport layers", () => {
// Each COPY --link is one layer that does not depend on the layers under
// it. The slim variant is the first two; the full variant starts from it,
// so it has the same two layers and then the transport archives.
const slim = runner.find((stage) => stage.name === "runner-artifacts-slim")!;
const full = runner.find((stage) => stage.name === "runner-artifacts")!;
const instructions = (stage: typeof slim) => stage.lines.filter((line) => /^[A-Z]+\s/.test(line)).slice(1);
expect(slim.from).toBe("scratch");
expect(instructions(slim)).toEqual([
"COPY --link --from=runner-artifacts-build /out-large/ /",
"COPY --link --from=runner-artifacts-build /out/ /",
]);
expect(full.from).toBe("runner-artifacts-slim");
expect(instructions(full)).toEqual([
"COPY --link --from=runner-artifacts-build /out-archives/ /",
"COPY --link --from=runner-artifacts-build /out-transport/ /",
]);
// `docker build .` builds the last stage.
expect(names.at(-1)).toBe("cloud");
const application = stages.filter((stage) => !stage.name.startsWith("runner-artifacts"));
for (const stage of application) {
expect(stage.from, stage.name).not.toMatch(/^runner-artifacts/);
const read = [...stage.lines.join("\n").matchAll(/from=(runner-artifacts[a-z-]*)/g)].map((match) => match[1]);
expect(read.filter((name) => name !== "runner-artifacts"), stage.name).toEqual([]);
}
expect(application.filter((stage) => /from=runner-artifacts/.test(stage.lines.join("\n"))).map((stage) => stage.name))
.toEqual(["build", "runner-provider-pack"]);
});
it("name the published image in the pack only for a publication build, which is off by default", () => {
const build = runner.find((stage) => stage.name === "runner-artifacts-build")!.lines.join("\n");
expect(build).toContain('ARG PAPERCLIP_RUNNER_IMAGE_PUBLICATION=""\n');
expect(build).toContain('publication="${PAPERCLIP_RUNNER_IMAGE_PUBLICATION}";');
// The pack builder reads the variable from the environment. It is set in
// one place, inside the branch that tests for a publication build.
expect(build.match(/PAPERCLIP_RUNNER_INPUTS_DIGEST=/g)).toHaveLength(1);
expect(build).toMatch(/if \[ -n "\$publication" \]; then \\\n\s+PAPERCLIP_RUNNER_INPUTS_DIGEST="\$\(node -p /);
expect(build).not.toMatch(/^ENV .*PAPERCLIP_RUNNER_INPUTS_DIGEST/m);
});
it("run every output step of the Runner's own build, and know every other step", () => {
// The image build runs the steps of the Runner's `build` script that write
// output, between the two that record the build stamp. This list is the
// whole script: a new step fails here until someone decides whether the
// image build must run it too.
const manifest = JSON.parse(readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8"));
expect((manifest.scripts.build as string).split(" && ")).toEqual([
"node scripts/record-build-inputs.mjs --begin",
"pnpm run check:protocol-manifest",
"pnpm run build:typescript",
"pnpm run check:capability-contract",
"pnpm run check:capability-inventory",
"pnpm run check:protocol-coverage",
"node scripts/generate-semantic-contracts.mjs --check",
"node scripts/check-runner-workflow-traceability.mjs",
"node scripts/build-runner-binary.mjs",
"node scripts/generate-replay-goldens.mjs --check",
"node scripts/generate-semantic-action-catalog.mjs --check",
"node scripts/record-build-inputs.mjs --finish",
]);
const build = runner.find((stage) => stage.name === "runner-artifacts-build")!.lines.join("\n");
for (const step of ["node scripts/record-build-inputs.mjs --begin;", "pnpm run build:typescript;", "pnpm run build:binary;",
"node scripts/record-build-inputs.mjs --finish;", "pnpm install --frozen-lockfile --filter '@paperclipai/paperclip-runner...'"]) {
expect(build).toContain(step);
}
});
});