From 84a4d345b3218e5611da6af7c8eb083ae47b436e Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 13 Nov 2026 09:48:49 -0800 Subject: [PATCH] refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- .github/workflows/docker-app-check.yml | 1 + Dockerfile | 215 ++++++------------ doc/DEVELOPING.md | 8 +- doc/DOCKER.md | 70 ++++-- doc/runner-image-contract.md | 49 +++- .../paperclip-runner/scripts/build-inputs.mjs | 2 +- .../runner-release-crate-cache.test.mjs | 4 +- scripts/check-runner-image-expectation.mjs | 37 +++ server/scripts/prepare-runner-vendor.mjs | 2 +- .../src/__tests__/docker-build-stamp.test.ts | 141 +++++++++--- .../docker-image-layer-reuse.test.ts | 8 +- .../__tests__/runner-image-identity.test.ts | 9 +- ui/src/lib/vite-build-commit.test.ts | 2 +- 13 files changed, 335 insertions(+), 213 deletions(-) create mode 100644 scripts/check-runner-image-expectation.mjs diff --git a/.github/workflows/docker-app-check.yml b/.github/workflows/docker-app-check.yml index 7a5c62618d1..8d7e2a290fb 100644 --- a/.github/workflows/docker-app-check.yml +++ b/.github/workflows/docker-app-check.yml @@ -7,6 +7,7 @@ on: - Dockerfile - .dockerignore - server/scripts/prepare-runner-vendor.mjs + - scripts/check-runner-image-expectation.mjs - packages/paperclip-runner/scripts/build-inputs.mjs permissions: {} diff --git a/Dockerfile b/Dockerfile index d30284ec19f..7f6f1ea5904 100644 --- a/Dockerfile +++ b/Dockerfile @@ -57,139 +57,6 @@ COPY server/src/services/copilot-connection-probe.ts server/src/services/ # private build tree, then install from that exact generated lock. RUN pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile && pnpm install --frozen-lockfile -FROM base AS rust-toolchain -WORKDIR /app -# Debian's packaged rust lags the ecosystem (trixie ships 1.85) and the -# runner's dependency tree now requires a newer rustc. Install rustup from a -# version-pinned, checksum-verified installer and let the runner's own -# rust-toolchain.toml choose the compiler — one pin, owned by the runner -# package, shared by CI and image builds alike. -# -# The C toolchain is explicit: apt's cargo used to pull gcc in as a -# dependency, and rustup does not — without it every build script dies on -# "linker `cc` not found". -RUN apt-get update \ - && apt-get install -y --no-install-recommends gcc libc6-dev pkg-config \ - && rm -rf /var/lib/apt/lists/* -ENV RUSTUP_HOME=/usr/local/rustup \ - CARGO_HOME=/usr/local/cargo \ - PATH=/usr/local/cargo/bin:$PATH -ARG RUSTUP_VERSION=1.29.0 -ARG RUSTUP_SHA256_AMD64=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 -ARG RUSTUP_SHA256_ARM64=9732d6c5e2a098d3521fca8145d826ae0aaa067ef2385ead08e6feac88fa5792 -RUN set -eux; \ - arch="$(dpkg --print-architecture)"; \ - case "$arch" in \ - amd64) rustTarget="x86_64-unknown-linux-gnu"; sha256="$RUSTUP_SHA256_AMD64" ;; \ - arm64) rustTarget="aarch64-unknown-linux-gnu"; sha256="$RUSTUP_SHA256_ARM64" ;; \ - *) echo "unsupported architecture: $arch" >&2; exit 1 ;; \ - esac; \ - curl -fsSLo /tmp/rustup-init "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/${rustTarget}/rustup-init"; \ - echo "${sha256} /tmp/rustup-init" | sha256sum -c -; \ - chmod +x /tmp/rustup-init; \ - /tmp/rustup-init -y --no-modify-path --profile minimal --default-toolchain none; \ - rm /tmp/rustup-init -# Install the package-owned compiler before any application source enters the -# stage. rustup-init above installs rustup itself, not the selected compiler. -COPY packages/paperclip-runner/rust-toolchain.toml /tmp/runner-toolchain/rust-toolchain.toml -RUN cd /tmp/runner-toolchain && rustup show - -# Pin the recipe generator and its dependency lockfile. It is a build-only tool -# and uses the same package-owned compiler as both native build stages. -FROM rust-toolchain AS rust-chef -RUN cd /tmp/runner-toolchain && cargo install cargo-chef --version 0.1.73 --locked - -FROM rust-chef AS runner-plan -WORKDIR /app/packages/paperclip-runner -COPY packages/paperclip-runner/rust-toolchain.toml ./ -COPY packages/paperclip-runner/runner ./runner -RUN cd runner && cargo chef prepare --recipe-path /tmp/runner-recipe.json - -FROM rust-chef AS runner-deps -WORKDIR /app/packages/paperclip-runner/runner -COPY packages/paperclip-runner/rust-toolchain.toml ../ -# The recipe changes only when dependency manifests, the lockfile, or target -# metadata change. Source edits can reuse this compiled dependency layer. -COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json -RUN cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd --recipe-path /tmp/runner-recipe.json \ - && find . -mindepth 1 -maxdepth 1 ! -name target -exec rm -rf {} + - -FROM runner-deps AS runner-build -WORKDIR /app/packages/paperclip-runner -# Rust embeds protocol schemas and fixtures with include_str!. Keep those -# alongside the complete Cargo workspace so every compile-time input keys -# this layer. Ordinary server/UI edits can then reuse the native build. -COPY packages/paperclip-runner/rust-toolchain.toml ./ -COPY packages/paperclip-runner/runner ./runner -COPY packages/paperclip-runner/protocol ./protocol -COPY packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json ./ -# Cargo fingerprints source mtimes. Normalize them here and after the full -# source copy below so a fresh checkout cannot invalidate unchanged inputs. -RUN find runner protocol -type f -exec touch -d @0 {} + \ - && touch -d @0 rust-toolchain.toml acpx-profiles.json antigravity-distributions.json \ - && cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd - -FROM runner-build AS build -WORKDIR /app -COPY --from=deps /app /app -COPY . . -RUN find packages/paperclip-runner/runner packages/paperclip-runner/protocol -type f -exec touch -d @0 {} + \ - && touch -d @0 packages/paperclip-runner/rust-toolchain.toml packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json -# Both the browser bundle and server stamp need the source commit. Declare it -# after the stable dependency layers, before either application build. -ARG PAPERCLIP_BUILD_COMMIT="" -RUN pnpm --filter @paperclipai/ui build -RUN pnpm --filter @paperclipai/plugin-sdk build -# The server build runs scripts/write-build-stamp.mjs, which stamps the built -# commit into dist/build-info.json. The build context has no .git, so the -# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the -# next RUN as an environment variable. The production stage below declares the -# same ARG again for the runtime fallback; an ARG goes out of scope at the -# end of its stage. Empty for local `docker build`, which then writes no stamp. -# NODE_OPTIONS covers the Node build steps. tsc is a native binary and ignores it. -ENV NODE_OPTIONS=--max-old-space-size=4096 -# The image needs no declarations, and CI type-checks this source separately. -# Emitting without either cuts the tsc step from more than 10 GiB to about 2.5 GiB. -RUN PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build -RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1) -RUN pnpm --filter @paperclipai/server build:runtime \ - && test -f server/dist-runtime/index.mjs || (echo "ERROR: server runtime bundle missing" && exit 1) -RUN rm -rf packages/paperclip-runner/runner/target - -# Remote OpenCode and ACPX runs require a controller-owned provider pack to -# verify the sandbox installation or stage matching assets. Ship it in the -# standard image so downstream Cloud compositions inherit the same artifacts. -# Grok's native executable stays an external sandbox prerequisite; this pack -# contains only its launcher. -FROM build AS runner-provider-pack -# Unstamped local builds remain usable, but cannot qualify a remote pack. -# Never invent a source revision to make an unqualified pack look verified. -# Managed deployments can remap node's UID at startup, so the pack must stay -# readable by any UID. Grant that here, in the same layer that writes the -# files: COPY keeps these modes. A `chmod -R` after the COPY into production -# rewrote every changed file into a second copy of the pack (187 MB compressed -# and ~50 s of build time on amd64), and the production stage still checks -# readability as an unrelated UID below. -RUN mkdir -p /provider-pack /provider-pack-transport \ - && if [ -n "${PAPERCLIP_BUILD_COMMIT}" ]; then \ - PAPERCLIP_RUNNER_SOURCE_REVISION="${PAPERCLIP_BUILD_COMMIT}" node packages/paperclip-runner/scripts/build-provider-pack.mjs /provider-pack \ - && node scripts/create-provider-pack-archive.mjs /provider-pack \ - && mv /provider-pack.archives /provider-pack.archive.json /provider-pack-transport/; \ - else \ - echo "Skipping remote provider pack: supply a full PAPERCLIP_BUILD_COMMIT to enable remote OpenCode/ACPX execution"; \ - fi \ - && chmod -R a+rX /provider-pack -# The pack is ~2.5 GB of provider binaries that change only when a provider -# version does, next to small files that change on every commit (the manifest -# carries the source revision). Move the large files to their own tree, after -# the transport archive has read the whole pack, so production can copy them -# as a layer that stays identical between commits. Normalized mtimes keep that -# layer's bytes identical when this stage reruns. -RUN mkdir -p /provider-pack-large \ - && cd /provider-pack \ - && find . -type f -size +1048575c -exec sh -c 'mkdir -p "/provider-pack-large/$(dirname "$1")" && mv "$1" "/provider-pack-large/$1"' _ {} \; \ - && find /provider-pack-large -exec touch -h -d @0 {} + - # Runner image (`--target runner-artifacts`). # # The stages below build the Runner once, from its own inputs only, as a plain @@ -203,11 +70,10 @@ RUN mkdir -p /provider-pack-large \ # # These stages are independent of every stage above. They install with a # filtered, frozen lockfile and never read the server, the UI or the `deps` -# stage, so a change outside the Runner's inputs leaves them cached, and the -# default targets never build them. +# stage, so a change outside the Runner's inputs leaves them cached. # -# Consumers must only `COPY --from=runner-artifacts `, or from -# `runner-artifacts-slim`, the same tree without the pack's transport +# Consumers must only read `runner-artifacts`, with `COPY --from` or a bind +# mount, or `runner-artifacts-slim`, the same tree without the pack's transport # archives. A build can then replace the stage with an image built earlier: # --build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256: # @@ -251,9 +117,9 @@ RUN node scripts/runner-image-identity.mjs --json > /runner-image-identity.json FROM scratch AS runner-artifacts-identity COPY --from=runner-artifacts-identity-compute /runner-image-identity.json /runner-image-identity.json -# Same checksum-verified rustup as the `rust-toolchain` stage above, on the -# bookworm base. The compiler is the one packages/paperclip-runner/rust-toolchain.toml -# names. The base image supplies the C toolchain. +# A version-pinned, checksum-verified rustup on the bookworm base. The +# compiler is the one packages/paperclip-runner/rust-toolchain.toml names. The +# base image supplies the C toolchain. FROM runner-artifacts-node AS runner-artifacts-toolchain # cargo strips the release binary itself. The Runner's build stamp records the # size of every file under dist, so the binary must not change after the build. @@ -288,7 +154,8 @@ COPY --from=runner-artifacts-context /packages/paperclip-runner/rust-toolchain.t COPY --from=runner-artifacts-context /packages/paperclip-runner/runner ./runner COPY --from=runner-artifacts-context /packages/paperclip-runner/protocol ./protocol COPY --from=runner-artifacts-context /packages/paperclip-runner/acpx-profiles.json /packages/paperclip-runner/antigravity-distributions.json ./ -# Cargo fingerprints source mtimes; see the `runner-build` stage above. +# Cargo fingerprints source mtimes. Normalize them here and in the build +# below so a fresh checkout cannot invalidate unchanged inputs. RUN find runner protocol -type f -exec touch -d @0 {} + \ && touch -d @0 rust-toolchain.toml acpx-profiles.json antigravity-distributions.json \ && cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd @@ -300,7 +167,7 @@ COPY --from=runner-artifacts-identity /runner-image-identity.json /runner-image- # The source revision is provenance, not identity: the commit whose build # produced this image. PAPERCLIP_BUILD_COMMIT is the fallback so one argument # stamps a whole `docker build`. Without either, the tree has the daemon and -# the Runner build but no provider pack, as in the `runner-provider-pack` stage. +# the Runner build but no provider pack. ARG PAPERCLIP_RUNNER_SOURCE_REVISION="" ARG PAPERCLIP_BUILD_COMMIT="" # Set to 1 only by .github/workflows/runner-image.yml, which builds the @@ -310,8 +177,8 @@ ARG PAPERCLIP_BUILD_COMMIT="" # but must not claim to be this image. ARG PAPERCLIP_RUNNER_IMAGE_PUBLICATION="" # One layer: the install tree and the pnpm store are removed before the layer -# is committed, and the pack is granted a+rX in the layer that writes it (see -# the `runner-provider-pack` stage above for why). +# is committed, and the pack is granted a+rX in the layer that writes it. A +# `chmod -R` in a later layer stores every changed file a second time. # # The Runner is built with its own scripts, in the order its `build` script # runs them: the two steps that write build output, between the two that record @@ -411,6 +278,62 @@ FROM runner-artifacts-slim AS runner-artifacts COPY --link --from=runner-artifacts-build /out-archives/ / COPY --link --from=runner-artifacts-build /out-transport/ / +FROM base AS build +WORKDIR /app +COPY --from=deps /app /app +COPY . . +COPY --from=runner-artifacts /runner-package/dist /app/packages/paperclip-runner/dist +COPY --from=runner-artifacts /runner-package/node_modules/.cache/paperclip-runner-build-inputs.json /app/packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json +ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST="" +RUN --mount=type=bind,from=runner-artifacts,source=/runner-image.json,target=/tmp/runner-image.json \ + node scripts/check-runner-image-expectation.mjs /tmp/runner-image.json +ENV PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 +RUN node server/scripts/prepare-runner-vendor.mjs +# Both the browser bundle and server stamp need the source commit. Declare it +# after the stable dependency layers, before either application build. +ARG PAPERCLIP_BUILD_COMMIT="" +RUN pnpm --filter @paperclipai/ui build +RUN pnpm --filter @paperclipai/plugin-sdk build +# The server build runs scripts/write-build-stamp.mjs, which stamps the built +# commit into dist/build-info.json. The build context has no .git, so the +# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the +# next RUN as an environment variable. The production stage below declares the +# same ARG again for the runtime fallback; an ARG goes out of scope at the +# end of its stage. Empty for local `docker build`, which then writes no stamp. +# NODE_OPTIONS covers the Node build steps. tsc is a native binary and ignores it. +ENV NODE_OPTIONS=--max-old-space-size=4096 +# The image needs no declarations, and CI type-checks this source separately. +# Emitting without either cuts the tsc step from more than 10 GiB to about 2.5 GiB. +RUN PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build +RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1) +RUN pnpm --filter @paperclipai/server build:runtime \ + && test -f server/dist-runtime/index.mjs || (echo "ERROR: server runtime bundle missing" && exit 1) + +# Remote OpenCode and ACPX runs require a controller-owned provider pack to +# verify the sandbox installation or stage matching assets. Ship it in the +# standard image so downstream Cloud compositions inherit the same artifacts. +# Grok's native executable stays an external sandbox prerequisite; this pack +# contains only its launcher. +FROM base AS runner-provider-pack +# Unstamped local builds remain usable, but cannot qualify a remote pack. +# Never invent a source revision to make an unqualified pack look verified. +# The pack is ~2.5 GB of provider binaries that change only when a provider +# version does, next to small files that change on every commit (the manifest +# carries the source revision). Move the large files to their own tree so +# production can copy them as a layer that stays identical between commits. +# Normalized mtimes keep that layer's bytes identical when this stage reruns. +RUN --mount=type=bind,from=runner-artifacts,target=/runner-artifacts \ + mkdir -p /provider-pack /provider-pack-large /provider-pack-transport \ + && if [ -f /runner-artifacts/provider-pack/provider-pack.json ]; then \ + cp -a /runner-artifacts/provider-pack/. /provider-pack/ \ + && cp -a /runner-artifacts/provider-pack.archives /runner-artifacts/provider-pack.archive.json /provider-pack-transport/; \ + else \ + echo "Skipping remote provider pack: supply a full PAPERCLIP_BUILD_COMMIT to enable remote OpenCode/ACPX execution"; \ + fi \ + && cd /provider-pack \ + && find . -type f -size +1048575c -exec sh -c 'mkdir -p "/provider-pack-large/$(dirname "$1")" && mv "$1" "/provider-pack-large/$1"' _ {} \; \ + && find /provider-pack-large -exec touch -h -d @0 {} + + FROM base AS production ARG USER_UID=1000 ARG USER_GID=1000 @@ -448,8 +371,8 @@ COPY --chown=0:0 --from=runner-provider-pack /provider-pack /opt/paperclip-runne COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/ # Managed deployments can remap node's UID at startup. This immutable pack # contains public code and integrity metadata, never credentials; it must remain -# readable afterward. The runner-provider-pack stage grants a+rX before this -# COPY, which preserves modes. +# readable afterward. The Runner build grants a+rX, and every copy up to this +# COPY preserves modes. # Keep it root-owned and verify access as an unrelated unprivileged UID. RUN if [ -f /opt/paperclip-runner/provider-pack/provider-pack.json ]; then \ gosu 65534:65534 node -e 'const fs = require("node:fs"); const path = require("node:path"); const root = "/opt/paperclip-runner/provider-pack"; const manifest = JSON.parse(fs.readFileSync(path.join(root, "provider-pack.json"), "utf8")); for (const artifact of Object.values(manifest.payload.artifacts)) fs.readFileSync(path.join(root, artifact.path)); fs.accessSync(path.join(root, manifest.payload.artifacts.nodeCommand.path), fs.constants.X_OK);'; \ diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index a33077a74c7..8ebe43e238d 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -148,6 +148,10 @@ GitHub Actions owns every managed pnpm lockfile: each lockfile with `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` and opens one `chore/refresh-lockfile` pull request for all of them. See `doc/RELEASE-AUTOMATION-SETUP.md` for when the bot merges it. +- A local `docker build` installs the Runner with a frozen lockfile. When the + root `pnpm-lock.yaml` is behind the Runner's manifests, run that same command + in the checkout before the build, and do not commit the result. See "Runner + image" in `doc/DOCKER.md`. The Pi and Kimi distribution lockfiles under `packages/paperclip-runner/scripts/` are not on this list. They hold attested provider pins, so change them by hand @@ -1642,7 +1646,9 @@ Set `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` when the Runner under build it again, for example because it has no Rust toolchain. The step then reuses a current build as usual. When the build is not current it prints the reason and fails, without starting a rebuild and without removing the stamp. -Setting both variables fails. +Setting both variables fails. The `build` stage of the `Dockerfile` sets it: +that stage copies `dist` and the stamp out of the `runner-artifacts` stage +(see "Runner image" in `doc/DOCKER.md`). ### Task search evaluation diff --git a/doc/DOCKER.md b/doc/DOCKER.md index 7585abe97e2..b6c857d519f 100644 --- a/doc/DOCKER.md +++ b/doc/DOCKER.md @@ -21,6 +21,7 @@ Build arguments: | `CLI_TOOLS_CACHE_EPOCH` | empty | Refresh the CLI-install layer; CI supplies the current ISO week | | `PAPERCLIP_BUILD_VERSION` | empty | Runtime version when Git metadata is unavailable | | `PAPERCLIP_BUILD_COMMIT` | empty | Source commit written into the server build stamp and runtime environment | +| `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` | empty | Inputs digest of the published Runner image a build was given as `runner-artifacts`; the build fails on any other Runner. Leave it empty for a build that makes its own Runner | Changing the build version or commit preserves the CLI-install cache. The tool layer refreshes when its weekly epoch, base image, installation command, @@ -82,13 +83,42 @@ image before rollout. Resolve immutable digests instead of deploying mutable tag The `runner-artifacts` target builds the Runner alone: the daemon, the Runner package's build output, the remote provider pack and its transport archives. The `runner-artifacts-slim` target is the same image without the transport -archives. Neither depends on the server or UI build, and no other target -depends on them yet. `.github/workflows/runner-image.yml` builds and checks +archives. Neither depends on the server or UI build. +`.github/workflows/runner-image.yml` builds and checks both on pull requests, for Linux AMD64 and ARM64, and publishes them from `master` as `ghcr.io/paperclipai/paperclip-runner`. See [`doc/runner-image-contract.md`](runner-image-contract.md) for the layout, the identity rules and how to consume the image. +The application image takes its Runner from the `runner-artifacts` stage. It +has no Runner build of its own, no Rust toolchain and no C compiler. Every +build of the `production` or `cloud` target builds that stage itself, once: + +- The `build` stage copies the Runner's build output and its build stamp into + `packages/paperclip-runner`. It sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, + so the server build vendors that output and fails when the stamp does not + match the source tree. It never builds the Runner again. +- The `runner-provider-pack` stage copies the pack and its transport archives + out of `runner-artifacts` and moves the pack's large files to a tree of their + own. The image keeps its paths under `/opt/paperclip-runner/` and its layers. + +The daemon in the application image is therefore the stripped binary compiled +on Debian bookworm, and the pack's Node is the pinned bookworm build. The +build runs the steps of the Runner's `build` script that write output, not its +drift checks; repository CI runs those. See "The application image" in the +contract for the rules a consumer of the application image can rely on. + +The Runner stages install with a frozen lockfile, so `docker build` fails when +`pnpm-lock.yaml` is behind the manifests they read: the root `package.json`, +`packages/paperclip-runner` and `packages/paperclip-eval-kernel`. The `deps` +stage still resolves every other manifest change inside the image. Refresh the +lockfile in the checkout, then build again. Do not commit the result (see +"Dependency Lockfile Policy" in `doc/DEVELOPING.md`): + +```sh +pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile +``` + `.github/workflows/docker.yml` builds the application image on `master` and on tags only. `.github/workflows/docker-app-check.yml` builds the `build` and `runner-provider-pack` stages on a pull request that changes the `Dockerfile`, @@ -394,21 +424,20 @@ Notes: ## Native Runner build cache -The image compiles the native Runner in `runner-build`, before copying the -application source. A pinned `cargo-chef` generates a dependency recipe in -`runner-plan`. The separate `runner-deps` stage compiles that recipe with the -package-owned Rust compiler. Both the dependency build and the real binary use -the release profile and locked Cargo dependencies. The recipe stage never -modifies source in the checkout. +The image compiles the native Runner once, in the `runner-artifacts-runnerd` +stage, with the package-owned Rust compiler, the release profile and locked +Cargo dependencies. That stage reads only the files Cargo reads. -Changes to Rust source or embedded protocol inputs rebuild the real binary but -can reuse compiled dependencies when the recipe is unchanged. Dependency -manifests, the Cargo lockfile, target metadata, or compiler changes invalidate -the relevant cache. Ordinary server or UI changes can reuse the entire native -build through the existing registry cache (`mode=max`). Each platform gets its -own native build; no cross-architecture binary is reused. No additional GitHub -Actions cache is created. A cold build also installs the recipe generator and -compiles dependencies, so the savings apply after those layers are available. +Ordinary server, UI or Runner TypeScript changes can reuse the entire native +build through the existing registry cache (`mode=max`). A change to Rust +source, embedded protocol inputs, a dependency manifest, the Cargo lockfile or +the compiler compiles the daemon and its dependencies again. Each platform +gets its own native build; no cross-architecture binary is reused. No +additional GitHub Actions cache is created. + +The `runner-artifacts-build` stage takes the build commit as an argument, so +every stamped build runs it: the frozen install, the Runner's TypeScript +build and the provider pack. Cargo finds the compiled daemon current there. Cloud builds import one registry cache: the first available full-SHA cache in the current commit's ten-entry first-parent ancestry, with the legacy cache @@ -418,9 +447,6 @@ missed native layers that a single matching manifest reused. The selector inspects metadata after Docker login, stops at the first available cache, and permits a cold build if no cache can be read. -The application build inherits that stage and still runs the normal server -build, including Cargo, binary staging, and generated-contract checks. Rust -input file times are normalized in both stages so fresh checkouts do not force -Cargo to rebuild unchanged source. Changes made by build scripts still reach -Cargo's normal validation. The final application copy excludes Cargo's target -directory as before. Cache misses only cost compilation time. +Rust input file times are normalized in both stages so fresh checkouts do not +force Cargo to rebuild unchanged source. The application build runs no Cargo +command. Cache misses only cost compilation time. diff --git a/doc/runner-image-contract.md b/doc/runner-image-contract.md index fe81f28ce36..7178e9efafe 100644 --- a/doc/runner-image-contract.md +++ b/doc/runner-image-contract.md @@ -2,9 +2,10 @@ The Runner image is the Runner built alone, as a plain file tree with no entrypoint and no operating system. An image build copies paths out of it -instead of compiling the Runner again. Nothing in this repository consumes it -yet. Its name is `ghcr.io/paperclipai/paperclip-runner`, and one build makes -two variants: +instead of compiling the Runner again. The application image of this +repository takes its Runner from the stage that builds it (see "The +application image"); no build reads the published image yet. Its name is +`ghcr.io/paperclipai/paperclip-runner`, and one build makes two variants: - full, the `runner-artifacts` target of the `Dockerfile`: every path below; - slim, the `runner-artifacts-slim` target: the same without the two paths @@ -131,7 +132,8 @@ uses it. ## Consuming the image -Read the image only with `COPY --from`, and pin it by digest: +Read the image only with `COPY --from` or a read-only bind mount +(`RUN --mount=type=bind,from=`), and pin it by digest: ```dockerfile FROM ghcr.io/paperclipai/paperclip-runner@sha256: AS runner-artifacts @@ -144,6 +146,45 @@ its own, as in this repository, can take a built image or tree in its place with `--build-context =`. The stage is then not built. +### The application image + +The `production` and `cloud` targets of the `Dockerfile` read only the +`runner-artifacts` stage, in two stages: + +- `build` copies `/runner-package/dist` and the build stamp into + `packages/paperclip-runner` and sets + `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`. The server build then vendors + that output, and fails when the stamp does not match the Runner inputs of + the tree it builds. The lockfile is one of those inputs. +- `runner-provider-pack` copies `/provider-pack`, the receipt and the + archives, and moves each pack file of 1 MiB or more to a tree of its own. + `production` copies the three trees with the same three instructions as + before, so the paths under `/opt/paperclip-runner/` and the layers are the + same. A deployed server finds the transport layer of an application image + by the text of the instruction + `COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/` + in the image history (`retained-provider-pack-oci.ts`). Do not change it. + +No workflow replaces the stage yet, so every application image build runs +the Runner stages itself: an inline Runner. An inline Runner is not this +image, and an application image says so in three ways: + +- its pack names the build commit of the application image as + `payload.runnerSourceRevision`, the value of `PAPERCLIP_BUILD_COMMIT`; +- its pack has no `payload.runnerInputsDigest`; +- the image has neither label of "Choosing the Runner for an image build". + +A build without `PAPERCLIP_BUILD_COMMIT` has no pack, as before. + +`build` also declares the build argument +`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, which no build sets yet. It is for +a build that replaces the stage with a published image. When the argument is +set, `scripts/check-runner-image-expectation.mjs` reads `/runner-image.json` +of the `runner-artifacts` the build was given, and the build fails unless +that file has `publication: true` and names exactly those inputs. An inline +Runner is never a publication, so the argument without the image fails too. +The check runs before the server build vendors the Runner. + ### Choosing the Runner for an image build `scripts/runner-image-resolve.mjs` decides whether a build of a checkout can diff --git a/packages/paperclip-runner/scripts/build-inputs.mjs b/packages/paperclip-runner/scripts/build-inputs.mjs index 8d1874d21cf..0882a174694 100644 --- a/packages/paperclip-runner/scripts/build-inputs.mjs +++ b/packages/paperclip-runner/scripts/build-inputs.mjs @@ -3,7 +3,7 @@ // nothing it depends on has changed. // // server's `prepare:runner-vendor` runs the Runner's full `build`, because a -// standalone `pnpm --filter @paperclipai/server build` (the Dockerfile's path) +// standalone `pnpm --filter @paperclipai/server build` // must produce a current vendored Runner on its own. Inside `pnpm -r build` // the same command has just built the Runner a few seconds earlier, and the // rebuild cost 12-16s on the CI Build job's critical path (six master runs, diff --git a/scripts/__tests__/runner-release-crate-cache.test.mjs b/scripts/__tests__/runner-release-crate-cache.test.mjs index 1be0097a6c5..377eac9518c 100644 --- a/scripts/__tests__/runner-release-crate-cache.test.mjs +++ b/scripts/__tests__/runner-release-crate-cache.test.mjs @@ -294,13 +294,13 @@ test("isolated Docker Rust stages preserve the layout of embedded schema and ide else if (entry.name.endsWith(".rs")) sources.push(full); } })(crateRoot); - for (const [dockerfile, stage] of [["Dockerfile", "runner-build"], ["docker/daytona-runner/Dockerfile", "runnerd-build"]]) { + for (const [dockerfile, stage] of [["Dockerfile", "runner-artifacts-runnerd"], ["docker/daytona-runner/Dockerfile", "runnerd-build"]]) { const docker = readFileSync(path.join(repoRoot, dockerfile), "utf8"); const body = docker.split(/^FROM .+ AS /m).find(part => part.startsWith(`${stage}\n`)); assert.ok(body, `${dockerfile}: missing ${stage}`); const workdir = /^WORKDIR (.+)$/m.exec(body)?.[1]; assert.ok(workdir?.startsWith("/"), `${dockerfile}: an absolute stage working directory is required`); - const copies = [...body.matchAll(/^COPY (?!-)(.+)$/gm)].flatMap(([, instruction]) => { + const copies = [...body.matchAll(/^COPY (?:--from=runner-artifacts-context )?(?!-)(.+)$/gm)].flatMap(([, instruction]) => { const args = instruction.trim().split(/\s+/); const destination = path.posix.resolve(workdir, args.pop()); return args.map(source => { diff --git a/scripts/check-runner-image-expectation.mjs b/scripts/check-runner-image-expectation.mjs new file mode 100644 index 00000000000..02a74bc0eb3 --- /dev/null +++ b/scripts/check-runner-image-expectation.mjs @@ -0,0 +1,37 @@ +#!/usr/bin/env node + +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; + +export const EXPECT_INPUTS_DIGEST_VARIABLE = "PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST"; + +export function runnerImageExpectationProblem(image, expected) { + if (expected === "") return null; + if (!/^[0-9a-f]{64}$/.test(expected)) { + return `${EXPECT_INPUTS_DIGEST_VARIABLE} must be the 64 hexadecimal characters of a Runner inputs digest, not ${JSON.stringify(expected)}.`; + } + const found = image?.inputsDigest?.digest; + if (image?.publication === true && found === expected) return null; + const actual = image?.publication === true ? `have the inputs ${found ?? "(none)"}` : "are not a published Runner image"; + return `${EXPECT_INPUTS_DIGEST_VARIABLE} is ${expected}, but the runner-artifacts of this build ${actual}. ` + + "Pass the image of those inputs as --build-context runner-artifacts=..., or unset the argument."; +} + +export function checkRunnerImageExpectation(file, { env = process.env, read = readFileSync, log = console.error } = {}) { + const expected = env[EXPECT_INPUTS_DIGEST_VARIABLE] ?? ""; + if (expected === "") return 0; + const problem = runnerImageExpectationProblem(JSON.parse(read(file, "utf8")), expected); + if (problem === null) return 0; + log(`ERROR: ${problem}`); + return 1; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + const file = process.argv[2]; + if (!file || process.argv.length !== 3) { + console.error("Usage: check-runner-image-expectation.mjs "); + process.exit(2); + } + process.exit(checkRunnerImageExpectation(file)); +} diff --git a/server/scripts/prepare-runner-vendor.mjs b/server/scripts/prepare-runner-vendor.mjs index d6e48fc20f0..ed3e091c138 100644 --- a/server/scripts/prepare-runner-vendor.mjs +++ b/server/scripts/prepare-runner-vendor.mjs @@ -4,7 +4,7 @@ // The server build used to run the Runner's full `build` // unconditionally here. That is what lets a standalone // `pnpm --filter @paperclipai/server build` produce a current vendored Runner -// on its own, which the Dockerfile relies on. But inside `pnpm -r build` the +// on its own. But inside `pnpm -r build` the // Runner has just been built by the same command, and repeating it cost // 12-16s on the CI Build job's critical path (six master runs, 2026-10-11) // and the same again in every other `pnpm build`. diff --git a/server/src/__tests__/docker-build-stamp.test.ts b/server/src/__tests__/docker-build-stamp.test.ts index 3abff18739b..cbf3e7e5f6e 100644 --- a/server/src/__tests__/docker-build-stamp.test.ts +++ b/server/src/__tests__/docker-build-stamp.test.ts @@ -1,7 +1,10 @@ -import { readFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { describe, expect, it } from "vitest"; +import { runnerImageExpectationProblem } from "../../../scripts/check-runner-image-expectation.mjs"; /** * Drift guard for the Docker build-stamp wiring. @@ -22,6 +25,8 @@ const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".." const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8"); const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8"); const previewWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "release.yml"), "utf8"); +const mergeGroupWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-merge-group.yml"), "utf8"); +const appCheckWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-app-check.yml"), "utf8"); /** * Return the text of the Dockerfile stage that starts at the named target. @@ -77,31 +82,101 @@ describe("docker build-stamp wiring", () => { }); -describe("Docker Rust dependency cache", () => { - it("caches the locked dependency recipe separately from source and per-build metadata", () => { - const chef = stageBody(dockerfile, "rust-chef"); - const planner = stageBody(dockerfile, "runner-plan"); - const dependencies = stageBody(dockerfile, "runner-deps"); - expect(chef).toContain("FROM rust-toolchain AS rust-chef"); - expect(chef).toMatch(/cargo install cargo-chef --version \d+\.\d+\.\d+ --locked/); - expect(planner).toContain("COPY packages/paperclip-runner/runner ./runner"); - expect(planner).toContain("cargo chef prepare --recipe-path /tmp/runner-recipe.json"); - expect(dependencies).toContain("FROM rust-chef AS runner-deps"); - expect(dependencies).toContain("COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json"); - expect(dependencies).toContain("cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd"); - expect(dependencies).not.toMatch(/COPY .*\.\/runner|COPY .*\.\/protocol|COPY \. \.|PAPERCLIP_BUILD_COMMIT/); +describe("Application image Runner", () => { + it("takes the Runner build and its stamp from runner-artifacts, and checks them before any application build", () => { + const build = stageBody(dockerfile, "build"); + expect(build).toContain("FROM base AS build"); + const source = build.search(/^COPY \. \.$/m); + const output = build.search(/^COPY --from=runner-artifacts \/runner-package\/dist \/app\/packages\/paperclip-runner\/dist$/m); + const stamp = build.search( + /^COPY --from=runner-artifacts \/runner-package\/node_modules\/\.cache\/paperclip-runner-build-inputs\.json \/app\/packages\/paperclip-runner\/node_modules\/\.cache\/paperclip-runner-build-inputs\.json$/m, + ); + const required = build.search(/^ENV PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1$/m); + const check = build.search(/^RUN node server\/scripts\/prepare-runner-vendor\.mjs$/m); + const firstBuild = build.search(/^RUN pnpm /m); + expect(source).toBeGreaterThanOrEqual(0); + expect(output).toBeGreaterThan(source); + expect(stamp).toBeGreaterThan(source); + expect(required).toBeGreaterThan(Math.max(output, stamp)); + expect(check).toBeGreaterThan(required); + expect(firstBuild).toBeGreaterThan(check); }); - it("rebuilds real workspace code and embedded protocol inputs after cooking dependencies", () => { - const native = stageBody(dockerfile, "runner-build"); - expect(native).toContain("FROM runner-deps AS runner-build"); - for (const source of ["runner", "protocol"]) { - expect(native.indexOf(`COPY packages/paperclip-runner/${source} ./${source}`)) - .toBeLessThan(native.indexOf("cargo build --release")); - expect(native).toContain(`COPY packages/paperclip-runner/${source} ./${source}`); + it("checks the Runner against the inputs digest a build names, before it vendors the Runner", () => { + const build = stageBody(dockerfile, "build"); + const stamp = build.search(/^COPY --from=runner-artifacts \/runner-package\/node_modules\//m); + const argument = build.search(/^ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=""$/m); + const check = build.indexOf( + "RUN --mount=type=bind,from=runner-artifacts,source=/runner-image.json,target=/tmp/runner-image.json \\\n" + + " node scripts/check-runner-image-expectation.mjs /tmp/runner-image.json\n", + ); + const vendor = build.search(/^ENV PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1$/m); + expect(stamp).toBeGreaterThanOrEqual(0); + expect(argument).toBeGreaterThan(stamp); + expect(check).toBeGreaterThan(argument); + expect(vendor).toBeGreaterThan(check); + expect(dockerfile.match(/PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST/g)).toHaveLength(1); + expect(dockerfile).toMatch(/^ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST(?:=.*)?$/m); + }); + + const inputs = "1".repeat(64); + const other = "2".repeat(64); + const published = { publication: true, inputsDigest: { digest: inputs } }; + const inline = { publication: false, inputsDigest: { digest: inputs } }; + + it.each<[string, object | null, string, RegExp | null]>([ + ["no expectation and an inline Runner", inline, "", null], + ["no expectation and a published Runner", published, "", null], + ["the published Runner of the expected inputs", published, inputs, null], + ["an inline Runner, even of the expected inputs", inline, inputs, /^PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST is 1{64}, but the runner-artifacts of this build are not a published Runner image\. Pass /], + ["a published Runner of other inputs", published, other, /^PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST is 2{64}, but the runner-artifacts of this build have the inputs 1{64}\. Pass /], + ["a published Runner that names no inputs", { publication: true }, inputs, /have the inputs \(none\)/], + ["a record that is not an object", null, inputs, /are not a published Runner image/], + ["an expectation that is not a digest", published, "inputs-latest", /must be the 64 hexadecimal characters of a Runner inputs digest, not "inputs-latest"/], + ])("answers for %s", (_label, image, expected, problem) => { + const answer = runnerImageExpectationProblem(image, expected); + if (problem === null) expect(answer).toBeNull(); + else expect(answer).toMatch(problem); + }); + + it("runs as the build runs it: nothing is read without an expectation, and a mismatch fails with the reason", () => { + const directory = mkdtempSync(path.join(tmpdir(), "runner-image-expectation-")); + try { + const file = path.join(directory, "runner-image.json"); + const run = (expected: string | null, target = file) => spawnSync( + process.execPath, + [path.join(repoRoot, "scripts", "check-runner-image-expectation.mjs"), target], + { encoding: "utf8", env: expected === null ? {} : { PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST: expected } }, + ); + expect(run(null, path.join(directory, "absent.json")).status).toBe(0); + expect(run("", path.join(directory, "absent.json")).status).toBe(0); + writeFileSync(file, JSON.stringify(inline)); + expect(run(inputs)).toMatchObject({ status: 1, stderr: expect.stringMatching(/^ERROR: PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST is 1{64}, but .* are not a published Runner image/) }); + writeFileSync(file, JSON.stringify(published)); + expect(run(inputs)).toMatchObject({ status: 0, stderr: "" }); + expect(run(other).status).toBe(1); + } finally { + rmSync(directory, { recursive: true, force: true }); + } + }); + + it("compiles the Runner only in the runner-artifacts stages", () => { + const names = [...dockerfile.matchAll(/^FROM \S+ AS (\S+)$/gm)].map((match) => match[1]); + const application = names.filter((name) => !name.startsWith("runner-artifacts")); + expect(application).toEqual(["base", "deps", "build", "runner-provider-pack", "production", "cloud-plugins", "cloud-server-deps", "cloud"]); + for (const name of application) { + expect(stageBody(dockerfile, name), name).not.toMatch(/\b(?:cargo|rustup|rustc|gcc)\b/); + } + }); + + it("gives an inline Runner the build commit as its revision and no publication argument", () => { + const runner = stageBody(dockerfile, "runner-artifacts-build"); + expect(runner).toContain('revision="${PAPERCLIP_RUNNER_SOURCE_REVISION:-${PAPERCLIP_BUILD_COMMIT}}";'); + for (const [name, source] of [["standard", workflow], ["preview", previewWorkflow], ["merge group", mergeGroupWorkflow], ["pull request", appCheckWorkflow]]) { + expect(source, name).toContain("PAPERCLIP_BUILD_COMMIT="); + expect(source, name).not.toMatch(/PAPERCLIP_RUNNER_(?:SOURCE_REVISION|IMAGE_PUBLICATION)/); + expect(source, name).not.toMatch(/runner-artifacts|build-contexts?\b/); } - expect(native).toContain("cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd"); - expect(stageBody(dockerfile, "build")).toContain("FROM runner-build AS build"); }); }); @@ -110,17 +185,25 @@ describe("Standard image remote provider pack", () => { it("ships the build-owned pack in the production base used by downstream images", () => { const pack = stageBody(dockerfile, "runner-provider-pack"); const production = stageBody(dockerfile, "production"); - expect(pack).toContain("FROM build AS runner-provider-pack"); - expect(pack).toContain('PAPERCLIP_RUNNER_SOURCE_REVISION="${PAPERCLIP_BUILD_COMMIT}"'); - expect(pack).toContain("build-provider-pack.mjs /provider-pack"); - expect(pack).toContain('if [ -n "${PAPERCLIP_BUILD_COMMIT}" ]; then'); + const runner = stageBody(dockerfile, "runner-artifacts-build"); + expect(pack).toContain("FROM base AS runner-provider-pack"); + expect(pack).toContain("RUN --mount=type=bind,from=runner-artifacts,target=/runner-artifacts \\\n"); + expect(pack).toContain("if [ -f /runner-artifacts/provider-pack/provider-pack.json ]; then"); + expect(pack).toContain("cp -a /runner-artifacts/provider-pack/. /provider-pack/"); + expect(pack).toContain("cp -a /runner-artifacts/provider-pack.archives /runner-artifacts/provider-pack.archive.json /provider-pack-transport/;"); expect(pack).toContain("mkdir -p /provider-pack"); expect(pack).toContain("Skipping remote provider pack"); + expect(production.match(/^COPY .*--from=runner-provider-pack .*$/gm)).toEqual([ + "COPY --link --chown=0:0 --from=runner-provider-pack /provider-pack-large/ /opt/paperclip-runner/provider-pack/", + "COPY --chown=0:0 --from=runner-provider-pack /provider-pack /opt/paperclip-runner/provider-pack", + "COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/", + ]); expect(production).toContain("--from=runner-provider-pack /provider-pack /opt/paperclip-runner/provider-pack"); expect(production).not.toContain("--chown=node:node --from=runner-provider-pack"); // Modes are granted in the layer that writes the pack. A recursive chmod // after the COPY duplicates every changed file into another image layer. - expect(pack).toMatch(/build-provider-pack\.mjs[\s\S]*&& chmod -R a\+rX \/provider-pack$/m); + expect(runner).toMatch(/build-provider-pack\.mjs \/out\/provider-pack;[\s\S]*chmod -R a\+rX \/out;/); + expect(pack).not.toContain("chmod"); expect(production).not.toMatch(/chmod -R [^\n]*\/opt\/paperclip-runner\/provider-pack/); expect(production).toContain("gosu 65534:65534 node"); expect(production).toContain("Object.values(manifest.payload.artifacts)"); @@ -134,7 +217,7 @@ describe("Standard image remote provider pack", () => { expect(cloud).toContain("FROM production AS cloud"); expect(cloud).not.toContain("/opt/paperclip-runner/provider-pack"); expect(cloud).not.toContain("PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH"); - expect(dockerfile.match(/build-provider-pack\.mjs \/provider-pack/g)).toHaveLength(1); + expect(dockerfile.match(/build-provider-pack\.mjs/g)).toHaveLength(1); expect(previewWorkflow).toMatch(/^\s*target: cloud$/m); }); }); diff --git a/server/src/__tests__/docker-image-layer-reuse.test.ts b/server/src/__tests__/docker-image-layer-reuse.test.ts index 898919295b2..afd50612616 100644 --- a/server/src/__tests__/docker-image-layer-reuse.test.ts +++ b/server/src/__tests__/docker-image-layer-reuse.test.ts @@ -36,13 +36,13 @@ it("copies installed dependencies as their own layer, before the per-commit app it("copies the large provider-pack files as a linked layer with fixed timestamps", () => { const pack = stageBody("runner-provider-pack"); - const archive = pack.indexOf("create-provider-pack-archive.mjs"); + const copy = pack.indexOf("cp -a /runner-artifacts/provider-pack/. /provider-pack/"); const split = pack.indexOf("mv \"$1\" \"/provider-pack-large/$1\""); const timestamps = pack.indexOf("find /provider-pack-large -exec touch -h -d @0 {} +"); - // The transport archive reads the whole pack, so the split must follow it. - expect(archive).toBeGreaterThanOrEqual(0); - expect(split).toBeGreaterThan(archive); + expect(copy).toBeGreaterThanOrEqual(0); + expect(split).toBeGreaterThan(copy); expect(timestamps).toBeGreaterThan(split); + expect(pack.match(/^RUN /gm)).toHaveLength(1); const production = stageBody("production"); const large = production.search( diff --git a/server/src/__tests__/runner-image-identity.test.ts b/server/src/__tests__/runner-image-identity.test.ts index 2048d2d87f8..43f497eac14 100644 --- a/server/src/__tests__/runner-image-identity.test.ts +++ b/server/src/__tests__/runner-image-identity.test.ts @@ -312,9 +312,14 @@ describe("Runner image stages", () => { ]); // `docker build .` builds the last stage. expect(names.at(-1)).toBe("cloud"); - for (const stage of stages) { - if (!stage.name.startsWith("runner-artifacts")) expect(stage.lines.join("\n"), stage.name).not.toMatch(/runner-artifacts/); + const application = stages.filter((stage) => !stage.name.startsWith("runner-artifacts")); + for (const stage of application) { + expect(stage.from, stage.name).not.toMatch(/^runner-artifacts/); + const read = [...stage.lines.join("\n").matchAll(/from=(runner-artifacts[a-z-]*)/g)].map((match) => match[1]); + expect(read.filter((name) => name !== "runner-artifacts"), stage.name).toEqual([]); } + expect(application.filter((stage) => /from=runner-artifacts/.test(stage.lines.join("\n"))).map((stage) => stage.name)) + .toEqual(["build", "runner-provider-pack"]); }); it("name the published image in the pack only for a publication build, which is off by default", () => { diff --git a/ui/src/lib/vite-build-commit.test.ts b/ui/src/lib/vite-build-commit.test.ts index ea764732ab5..4fec93f1258 100644 --- a/ui/src/lib/vite-build-commit.test.ts +++ b/ui/src/lib/vite-build-commit.test.ts @@ -34,7 +34,7 @@ describe("browser build attribution", () => { it("makes the Docker build commit available before building the browser", () => { const dockerfile = readFileSync(fileURLToPath(new URL("../../../Dockerfile", import.meta.url)), "utf8"); - const stage = dockerfile.split("FROM runner-build AS build")[1]?.split("FROM base AS production")[0]; + const stage = dockerfile.split("FROM base AS build")[1]?.split("FROM base AS production")[0]; expect(stage).toBeDefined(); const arg = stage!.indexOf('ARG PAPERCLIP_BUILD_COMMIT=""'); const browserBuild = stage!.indexOf("RUN pnpm --filter @paperclipai/ui build");