## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The application image now takes its Runner from the `runner-artifacts` stages, which install with a frozen lockfile > - A contributor cannot commit `pnpm-lock.yaml`. After a change to a manifest of the Runner, the lockfile on `master` is behind until the refresh merges > - In that time a plain `docker build` fails late, inside a long step, with a message that is easy to miss. And a beta or a stable release can be cut from such a commit, because the release workflows refresh the lockfile for their own build > - This pull request adds an early check with a clear instruction to the image build, and the same check to the beta and the stable release > - The benefit is a failure in seconds that says what to run, and beta and stable tags that always build from source ## Linked Issues or Issue Description This is the third part of a chain. The first part is #21331. The second part takes the Runner of the application image from the `runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and its lockfile rule). No issue exists for this change. **What happened?** A tree with a lockfile behind the manifests of the Runner fails `docker build` after the daemon compile (169 s in the test). The error is `ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the `Dockerfile`. **Expected behavior** The build fails within seconds of the check running. The message names the files and the command to run. The application's dependency install starts from the check, so it cannot start before the check passes. Stages that do not depend on it (the Runner's toolchain download and daemon compile, and the tool layer of `production`) can already be running, and the failed check cancels them. A beta or a stable release from such a commit does not start. How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a commit whose lockfile is behind the manifests of the Runner (2026-09-01, 09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta is the promotion of a nightly, so a beta can be cut from such a commit. **Steps to reproduce** 1. Change a dependency range in `packages/paperclip-runner/package.json`. Do not refresh `pnpm-lock.yaml`. 2. Run `docker build .`. **Paperclip version or commit** `master` with the previous pull request of this chain. **Deployment mode** Not applicable. This is the image build and the release tooling. ## What Changed - `Dockerfile`: a `runner-lockfile` stage copies only the manifests the Runner build installs from, the lockfile and the patches. It runs the frozen, filtered, offline lockfile check, with the arguments of `runnerLockState`. `deps` starts from it. For a stale lockfile it prints the files and `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. For any other failure it prints the output of pnpm and says so. - `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs of a commit with `git archive` and gives them to `runnerLockState` (`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind, 2 for an error. - `scripts/release.sh` and `scripts/release-lib.sh`: `require_runner_lockfile_current` runs for the `beta` and the `stable` channel, after `--print-version` exits and before any build. It runs in a dry run too. It does not run for `canary` or `nightly`. - Both beta paths end in the `publish_beta` job: the promotion of a nightly, and a `candidate/beta-*` branch. That job checks out the commit that `select_beta` chose and runs `release.sh beta` there. So the check reads the committed files of the commit the beta is built from. A beta is refused before its soak starts. The stable that promotes it is checked again. - Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18 tests). Three of them pin the checkout of `publish_beta`, `preview_stable` and `publish_stable` to the selected commit. Two tests plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs` cover the beta, the candidate beta and the stable path. - Docs: `doc/DOCKER.md`, `doc/RELEASING.md`, `doc/runner-image-contract.md`. ## Verification Builds: `linux/arm64`, local. They ran before the rebase onto `master`, except the last one. `master` changed neither the `Dockerfile` nor a manifest in between. - A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest of the Runner: `docker buildx build --target build` fails after 6 s at `runner-lockfile`, with the message. - The same tree after `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` on the host, which is what the image workflows run: `--target build` builds. The check passes, the Runner stages install, and the vendor step logs `reusing the current Runner build` in both of its runs. - After the rebase: `build` and `runner-provider-pack` built from this branch without a build commit. The check passed. - The release check with the real pnpm: `behind` for the first tree, `current` for the second, exit 0 for this checkout. - `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with the other changed server tests, on macOS and in `node:24.22.3-bookworm` (249 pass in 11 files on Linux). The test file typechecks. - `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14 pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of its tests fail on `master` too. `release-verify-workflow.test.mjs` and `acpx-patch-packaging.test.mjs`, which read the release scripts: pass. - `docker buildx build --check`: no warning. - The Docker builds above ran before the second part of the chain got its expectation step and before the beta guard. The expectation step was built on the second part alone. This branch as it is now was not built again; its tests ran. - Not run: the release workflow, any `linux/amd64` build, the full test suite. Only CI and a release dry run can prove the release path. ## Risks - **A beta can be refused.** When the newest nightly is on a commit whose lockfile is behind, `release.sh beta` stops, in the dry run and in the real run. The operator must promote a nightly whose commit includes the lockfile refresh, or cut a candidate branch that includes it. `node scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the answer before the dispatch. - **A stable can be refused** only when its beta was cut before this change, or with release tooling that does not have the check. Promotions run the release tooling of the source commit. - The check runs in `release.sh`. In the beta workflow it runs in `publish_beta`, after the approval of the `npm-beta` environment and after the candidate verification. In the stable workflow it runs after verification, not in the `preflight_stable` job. The selection jobs have no pnpm. - `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before its first build step. The workflow has them. - `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI command today. The Vitest file carries the enforced tests. - The image build has one more small stage. It reads only manifests, so a source change does not run it again. - The digest of the Runner image changes again, because its identity covers the whole `Dockerfile`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
20 KiB
Runner image contract
The Runner image is the Runner built alone, as a plain file tree with no
entrypoint and no operating system. An image build copies paths out of it
instead of compiling the Runner again. The application image of this
repository takes its Runner from the stage that builds it (see "The
application image"); no build reads the published image yet. Its name is
ghcr.io/paperclipai/paperclip-runner, and one build makes two variants:
- full, the
runner-artifactstarget of theDockerfile: every path below; - slim, the
runner-artifacts-slimtarget: the same without the two paths marked "full only", the provider pack's transport archives.
Layout
Every path a consumer may copy (RUNNER_IMAGE_PATHS in
scripts/runner-image-layout.mjs). Every file is readable by every user.
| Path | Content |
|---|---|
/runner-image.json |
Identity and description of this image. |
/paperclip-runnerd |
The daemon. A stripped release binary, mode 0755. |
/provider-pack/ |
The remote provider pack, with provider-pack.json. |
/provider-pack.archive.json |
Full only. Receipt of the pack's transport archives. |
/provider-pack.archives/ |
Full only. The transport archives the receipt lists. |
/runner-package/package.json |
The Runner package manifest. |
/runner-package/dist/ |
The Runner's complete build output. dist/bin/paperclip-runnerd is the same file as /paperclip-runnerd. |
/runner-package/node_modules/.cache/paperclip-runner-build-inputs.json |
The build stamp the Runner's build recorded for that output. |
/runner-package/scripts/provision-grok.mjs |
Provisions the native Grok executable. Reads platforms.json by relative path. |
/runner-package/src/providers/grok/platforms.json |
The Grok release pins. |
/runner-package/src/live/runnerd-codex-transport.ts |
The transport source, as a contract file. It defines RUNNER_CLIENT_VERSION. |
/bundles/runnerd-codex-transport.mjs |
The pack's dist/live/runnerd-codex-transport.js as one module. It imports without node_modules. |
/pnpm-lock.yaml |
The lockfile the image was installed from. |
Paths under /runner-package/ keep their place inside
packages/paperclip-runner. runner-image.json describes one platform, and
is the same file in both variants:
inputsDigest(the identity below) andbuildInputsDigest(the digest in the Runner's build stamp);sourceRevision: the commit that built the image, also in the pack aspayload.runnerSourceRevision. An image is published once for its inputs, so later commits with the same inputs keep this revision;publication:trueonly for a build with the build argumentPAPERCLIP_RUNNER_IMAGE_PUBLICATION=1. Only then does the pack carry the identity, aspayload.runnerInputsDigest. Another image build that runs the stage inline builds the same Runner but is not this image;platform(linux,amd64orarm64) andlibc: the glibc the daemon was compiled on (Debian bookworm, the oldest a consumer runs) and the newest glibc version it needs;providerPack: the pack's digest and target;pathsandsha256: the table above, and the digest of every single file;fullImageOnly: the names inpathsthat only the full image has.
Layers
- The pack's dependency files of 1 MiB or more: provider binaries and Node.
- Everything else, with
runner-image.json. - Full only: the transport archive of each file in layer 1.
- Full only: the other transport archives, the overlay archive, the receipt.
The slim image is layers 1 and 2. The full image is those same two layer
blobs, then layers 3 and 4, which hold transport content and nothing else.
Layers 1 and 3 have the same bytes in every build with the same dependency
files, so a new image shares them with the one before it. A large file under
the pack's dist is the Runner's own output: it and its archive are in
layers 2 and 4. An archive is named by its file's place in the receipt, which
lists the dependency files first, so their names do not depend on dist.
Identity
Both variants are named by one digest: the variant is not part of the
identity. The full image has the tag inputs-<digest> and the slim image
inputs-<digest>-slim. scripts/runner-image-identity.mjs computes the
digest from:
- every file under the paths the build copies from the build context
(
RUNNER_IMAGE_CONTEXT_INPUTS, which includepnpm-lock.yaml): path, executable bit and content, and each symbolic link by its target; - the whole
Dockerfile: the recipe, the base image pinned by digest and the toolchain pins. Any edit to it makes a new image; - the two platforms,
linux/amd64andlinux/arm64.
The source revision is not part of it, and neither is the text of
.dockerignore: a rule changes the digest only when it changes which of
those files the build receives. The build computes the digest for itself:
docker buildx build --target runner-artifacts-identity --output type=local,dest=/tmp/identity .
jq -r .digest /tmp/identity/runner-image-identity.json
The install is frozen. It fails when the lockfile is behind the manifests the
build reads (the root one, packages/paperclip-runner and
packages/paperclip-eval-kernel). A contributor cannot edit the lockfile, so
pull-request runs resolve it first. For a local build, run
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile.
Three places ask pnpm the same question, with the same arguments
(runnerLockState in scripts/runner-image-publish.mjs):
- the publication workflow, which publishes nothing for a lockfile that is behind;
- the
runner-lockfilestage of theDockerfile, which fails an application image build and prints the command above. The application's dependency install starts from that stage, so it cannot start before the check passes. The Runner's toolchain and daemon stages do not depend on it, so they can already be running, and the failed check cancels them. scripts/check-release-runner-lockfile.mjs, whichscripts/release.shruns for a beta and for a stable release. It checks the committed files of the release commit, so a beta or stable tag always builds with a plaindocker build. A canary or a nightly is not checked.
Checks
CI builds each variant as an OCI archive and nothing else.
node scripts/runner-image-oci.mjs unpack <oci archive> --out <tree> writes
the tree of an image from its layers, so the tree CI checks is the bytes of
the image. It refuses a whiteout, an absolute or .. path, a path written
twice and an entry behind a link.
node scripts/runner-image-layout.mjs check <tree> --variant slim|full runs
in the build and again on each unpacked tree. The variant has no default: a
slim tree with the transport paths fails, and so does a full tree without
them. The check fails unless:
- every documented path exists, matches its digest in
runner-image.json, and resolves inside the tree; - the build stamp names the inputs the identity names;
- every artifact in
provider-pack.jsonhas its digest, thedisttree and each provider asset tree match their tree digests, and the pack names this image's revision and platform; - the pack names the inputs digest in a publication build and in no other.
--expect-inputs-digestfails on any other build; - in a full image, every transport archive matches its receipt, and the
archives extract to the pack. The check extracts them as a sandbox does (GNU tar
with the options in
provider-pack-layers.ts) and compares paths, content, link targets and all twelve permission bits, after thea+rXgrant the build makes once the archives are written.
node scripts/runner-image-oci.mjs check-layers --slim <oci archive> --full <oci archive>
checks "Layers" on the two archives, and that no file is in two layers.
The build also runs paperclip-runnerd --build-metadata on every base image
in docker/runner-image/Dockerfile.smoke. Add a base there before a consumer
uses it.
Consuming the image
Read the image only with COPY --from or a read-only bind mount
(RUN --mount=type=bind,from=), and pin it by digest:
FROM ghcr.io/paperclipai/paperclip-runner@sha256:<digest> AS runner-artifacts
FROM debian:trixie-slim
COPY --from=runner-artifacts /paperclip-runnerd /usr/local/bin/paperclip-runnerd
A Dockerfile with a runner-artifacts or runner-artifacts-slim stage of
its own, as in this repository, can take a built image or tree in its place
with --build-context <stage>=<image or directory>. The stage is then not
built.
The application image
The production and cloud targets of the Dockerfile read only the
runner-artifacts stage, in two stages:
buildcopies/runner-package/distand the build stamp intopackages/paperclip-runnerand setsPAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1. The server build then vendors that output, and fails when the stamp does not match the Runner inputs of the tree it builds. The lockfile is one of those inputs.runner-provider-packcopies/provider-pack, the receipt and the archives, and moves each pack file of 1 MiB or more to a tree of its own.productioncopies the three trees with the same three instructions as before, so the paths under/opt/paperclip-runner/and the layers are the same. A deployed server finds the transport layer of an application image by the text of the instructionCOPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/in the image history (retained-provider-pack-oci.ts). Do not change it.
No workflow replaces the stage yet, so every application image build runs the Runner stages itself: an inline Runner. An inline Runner is not this image, and an application image says so in three ways:
- its pack names the build commit of the application image as
payload.runnerSourceRevision, the value ofPAPERCLIP_BUILD_COMMIT; - its pack has no
payload.runnerInputsDigest; - the image has neither label of "Choosing the Runner for an image build".
A build without PAPERCLIP_BUILD_COMMIT has no pack, as before.
build also declares the build argument
PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST, which no build sets yet. It is for
a build that replaces the stage with a published image. When the argument is
set, scripts/check-runner-image-expectation.mjs reads /runner-image.json
of the runner-artifacts the build was given, and the build fails unless
that file has publication: true and names exactly those inputs. An inline
Runner is never a publication, so the argument without the image fails too.
The check runs before the server build vendors the Runner.
Choosing the Runner for an image build
scripts/runner-image-resolve.mjs decides whether a build of a checkout can
copy the published image. No build uses its answer yet: on a master push,
the runner_image_report job of .github/workflows/docker.yml runs it and
writes the answer to the job summary, and nothing reads that job. The job
runs on the committed files, holds no credential (the Runner package is
public, so the lookup is anonymous) and cannot fail the run. The script has
two commands, so that the one that builds reads no registry credential:
node scripts/runner-image-resolve.mjs inputs --dir "$dir"
node scripts/runner-image-resolve.mjs resolve --dir "$dir"
inputs exports the Runner build context, checks the lockfile and computes
the inputs digest in the build, as the publication workflow does. Run it on
the committed files, before any step that rewrites pnpm-lock.yaml.
resolve looks up inputs-<digest> and reports:
mode:publishedorinline, andreason;imageandslim_image: both variants, pinned by index digest. The slim digest is the one the full index names (see "Pairing");build_contexts: one--build-contextvalue on each line, which replace therunner-artifactsandrunner-artifacts-slimstages;labels:io.paperclip.runner.imageandio.paperclip.runner.inputs-digest, for the image that copies the Runner.
published needs a complete publication whose full index names these inputs
and both platforms. Every other answer is inline, with the last four values
empty: the lockfile is behind the Runner's manifests (lock-behind), the
image does not exist (absent), a publication is incomplete or broken,
the full index does not match (mismatch), or the registry refused, failed
or could not be reached (unknown). An inline build runs the Runner stages
itself and carries neither label. The command does not wait for a publication
and does not verify the attestation of the image.
Publication
.github/workflows/runner-image.yml publishes on a master push in this
repository and nowhere else. Other runs build and check, and publish nothing.
- Lockfile. A
masterpush builds from the committedpnpm-lock.yaml, so the digest is a function of the commit. When that file is behind the Runner's manifests, the run writes a notice, publishes nothing and stays green. The commit that merges the lockfile refresh publishes the image. - Once. The build runs only when
inputs-<digest>does not exist. - Verified bytes. The build legs cannot write to the registry. The
publish job pushes the images they checked by digest and joins each
variant into a multi-platform index under a
candidate-tag. It attests both indexes, and only then writes the twoinputs-tags.candidate-tags are internal: one can name an index that is not attested, and the next run overwrites it. Do not consume them. - Order. It writes
inputs-<digest>-slimfirst andinputs-<digest>last, so the full tag marks a complete publication. The full index names its slim index in the annotationio.paperclip.runner.slim-digest. - Pairing. Resolve the full tag first. Take the slim digest from that
annotation of the full index, and pull the slim image by that digest. Never
resolve
inputs-<digest>-slimby tag on its own: while a publication is incomplete, that tag can name the slim image of another run. - Recovery. A run that stops between the two tags leaves a slim tag
alone. Those inputs are not published. The next run for them builds again,
replaces that tag with its own slim image and then writes the full tag. A
full tag whose slim tag is missing or names another image fails every run
for those inputs. An operator must then delete both package versions and
re-run the workflow run of a
masterpush with those inputs. - Create-only. The workflow never moves a tag of a complete publication,
but the registry does not enforce that. Pin the index digest and verify it:
gh attestation verify oci://ghcr.io/paperclipai/paperclip-runner@sha256:<digest> --repo paperclipai/paperclip --signer-workflow paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref refs/heads/master - First publication. GitHub creates the package private. A maintainer must make it public once.
Build cache
The build legs read a registry build cache,
ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>. It is a package of
its own and holds no image, so the image package holds only attested images.
It covers the stages that do not take the commit: the Rust toolchain and the
compiled daemon. The last stage takes the source revision as a build
argument, so every build runs it: the frozen install, the Runner's build and
the provider pack. Of the two build commands of a leg, only the first reads
the registry cache; the second reads what the first left in the job's builder.
- Effect. BuildKit reuses a cached step only when the step and the content of every file it reads are the same. The cache changes how long a build takes, not what it builds, and the identity does not cover it.
- Writer. Only the
cachejob of the workflow writes it: on amasterpush in this repository, after the publish job, for inputs that had no image. When the image exists, nothing was built, and the run that published it wrote the cache for those inputs. That job cannot sign, builds no further than the daemon and never runs the install. Its token reaches the registry login and no other step. It reads the cache it writes and keeps every step of the cached stages (mode=max). Every other run reads the cache without a credential and cannot write it. - Not on the way to an image. No job waits for the
cachejob. When it fails, the image is published and the next build is slower. - One write at a time. The job holds one concurrency slot for each architecture. GitHub keeps one waiting job in a group and cancels the one that waited before it, so the newest write waits. A write from an older commit that lands last costs time and nothing else.
- No Actions cache. The workflow never uses it: workflows that handle pull requests can write it, and what they write must not reach a published image.
- Trust. The daemon in a published image can come from a cached step
that an earlier
masterrun compiled. Whoever can write the cache package can decide what a later build reuses, so write access to it must stay as narrow as write access to the image package. - Fetches. A cached step does not run again. A download inside one (the toolchain installer, the toolchain, the crates) is not repeated until the inputs of that step change.
- Weekly cold build. A scheduled run builds both variants once a week
with no cache: it passes
--no-cacheand reads no registry cache. Like a manual run, it resolves the lockfile, always builds, runs every check and publishes nothing. A fork's schedule runs nothing. It fails when a download is gone or when a step only works from the cache. - Missing cache. A cache that does not exist or cannot be read is a line in the build log, not a failure. The build then runs every step.
- First write. GitHub creates the cache package private. A maintainer must make it public once. Until then every build runs cold.
Reading the transport archives alone
The server needs the transport form of an older pack when a retained session
names a pack that is not the server's own. A pack with
payload.runnerInputsDigest came out of this image, so the server reads it
from inputs-<digest> (retained-provider-pack-oci.ts). It does not count
layers. It uses the rule of "Layers": the full image is the slim image, then
layers that hold transport content and nothing else.
- The full index must carry the annotations
io.paperclip.runner.inputs-digest, equal to the pack's digest of inputs,org.opencontainers.image.revision, equal to the pack'srunnerSourceRevision, andio.paperclip.runner.slim-digest. - The server reads the slim index by that digest, never by the slim tag, and checks the bytes against the digest.
- For the pack's platform, the layers of the slim image must be the first layers of the full image, as the same blobs in the same order. The layers after them are the transport layers: at least one and at most eight.
- The server downloads only those layers. It refuses any entry in them that is not a transport archive or the receipt, and any file that is in two of them.
- The importer then checks every archive against the pack's manifest, as for any other source.
- The record the server keeps of the source names the digest of inputs and
every transport layer under
runnerImage. ItstransportLayerDigestis then the last of those layers andtransportLayerBytestheir total.
When the image cannot be read (a tag or the slim index does not exist, or the
registry refuses a request), the server reads the application image tagged
sha-<runnerSourceRevision>, as it does for a pack without
runnerInputsDigest. An image that can be read and does not match fails.
Known limits
- A build without
PAPERCLIP_RUNNER_SOURCE_REVISIONorPAPERCLIP_BUILD_COMMIThas no pack. It is for local use. - The build runs the steps of the Runner's
buildscript that write output, not its drift checks; repository CI runs those. - A build that vendors
/runner-package/reusesdistonly when its own Runner build-inputs digest equals the stamp's. The image computes that digest in the.dockerignore-filtered context; a checkout gives another. # syntax=docker/dockerfile:1.20is a tag. A new frontend release under it does not change the identity.- The archive check does not tell a directory with its own archive member from one created only as a file's parent.
- A lockfile that the lockfile refresh cannot bring up to the Runner's
manifests leaves every
masterrun green with only a notice, for as long as it lasts. No image is published in that time, and nothing fails.