## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work, and `master` now publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner` > - The application image will soon copy that Runner instead of compiling its own. Part 1 of this stack adds the resolver that picks the Runner for a build > - Two gaps make that next change risky. The resolver has never run in CI on `master`, so its lookup time and how often a commit has a published Runner are unknown. And `docker.yml` has no pull-request trigger, so a change to how the application image gets its Runner first runs after it merged > - This pull request adds a report-only job that runs the resolver on every `master` push, and a pull-request check that builds the Runner path of the application image > - The benefit is real data from `master` before any build depends on the resolver, and a failed check on the pull request instead of a failed image build on `master` ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** The resolver decides whether a build copies the published Runner. On `master`, a commit that changes the Runner's inputs is ahead of its publication for some minutes, and nobody knows yet how often an application build will meet that. The application image itself is built only on `master` and on tags. The change that removes the Runner's compile stages from that image will edit the `Dockerfile`, and today nothing builds those stages on a pull request. **Proposed solution** Add one job to `docker.yml` that only reports, and one new workflow that only checks. Neither changes what an image contains, and no existing job reads either of them. **Alternatives considered** - Run the resolver inside the build legs. A failure there could fail the image build. A job of its own, with `continue-on-error`, cannot. - Give the lookup the job token. The Runner package is public, so an anonymous lookup works and the job needs no credential. - Build `production` in the pull-request check. It adds three `COPY` instructions and a readability check to the two stages that this check builds, at the cost of the tool layer and two more copies of several gigabytes. - Trigger the check on every Runner source change. `runner-image.yml` already builds the Runner on those pull requests, and CI builds the server and the UI. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path (this pull request) 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `.github/workflows/docker.yml`: new job `runner_image_report`. - It runs only for a `master` push in this repository. - It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the result goes to the job summary with the line "Report only: no build read this result." - It has `continue-on-error: true`, declares no `outputs`, and no job needs it. Its failure does not fail the run. - It holds no credential. Its only permission is `contents: read`, the checkout does not keep its token, and the lookup is anonymous. It does not log in to a registry and writes to none. - Time limits: 8 minutes for the job, 5 for the step that builds, 2 for the lookup. - It is the first job in the file. Existing tests cut the other jobs out of the file by name, up to the next job or to the end of the file, so no other place leaves those tests unchanged. - `.github/workflows/docker-app-check.yml` (new): on a pull request, builds the `build` and `runner-provider-pack` targets with `PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only. - `permissions: {}` at the top, `contents: read` for the job, no credential, no `pull_request_target`. - It exports nothing (`--output type=cacheonly`) and pushes nothing. It reads the public build cache that `docker.yml` writes and never writes it. - It refreshes the lockfile in the build context first, as `docker.yml` does. - Time limits: 60 minutes for the job, 45 for the two builds. - Trigger paths, and why: | Path | Reason | | --- | --- | | `.github/workflows/docker-app-check.yml` | the check itself | | `Dockerfile`, `.dockerignore` | the recipe, and the files it receives | | `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that reuses or rebuilds the Runner | | `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp that step compares with the files the image build received | - `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90 lines): only security properties. The report job has no credential and one read permission, nothing can read it, and its two steps have time limits. The check has no credential, cannot publish, and exports nothing. In both, every action is pinned by commit and no shell script holds an expression. - `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and the check. ## Verification macOS (Node 26.12.0): - In `server/`: `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse` and `cloud-image-sentry` tests: 52 tests pass in 7 files. - `tests/runner-e2e` workflow tests (`workflow-security`, `private-trust-workflow-security`, `workflow-image-only`, `paid-runner-routing`): 27 tests pass in 4 files. - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass. - Both workflow files parse as YAML. - `docker buildx build --target runner-artifacts-context --cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64 --output type=cacheonly .` succeeds, so both flags are accepted and the cache is readable. Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same three sets pass: 52, 27 and 962 of 962. The two commands the report job runs were run by hand, with no credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca` the answer was `published`, with the two index digests that `master` published. For a commit whose image was not yet published the answer was `inline`, `absent`. Part 1 has the output. With the job at the end of `docker.yml`, two existing tests failed. They are the reason for its place: `docker-canary-promotion.test.mjs` reads the last job to the end of the file, and `release-verify-workflow.test.mjs` allows no BuildKit configuration after `merge-and-push`. Never run: - Either workflow. Only GitHub can run them. This pull request starts the check by its own path; the report job first runs on `master`. - The two builds of the check (`--target build`, `--target runner-provider-pack`) on a local host. Their time and disk use on a hosted runner are not known. - The full test suite and the full typecheck. CI must supply them. ## Risks - The check builds the whole `build` stage, with the UI and server builds. Its duration is not measured. It runs only when one of five paths changes, and its first real run is on this pull request. - `docker.yml` holds one concurrency slot for each ref until all its jobs end. The report job normally ends long before the build legs. A stalled job can hold the slot for at most 8 minutes when the legs end sooner. - `scripts/watch-release-source.mjs` warns about every job of the image run that waited more than 60 seconds for a runner. The report job is one more job that it can name. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - The lookup has 2 minutes. The registry client retries a failing request for longer than that, so a registry outage ends the step by its time limit, and the job then reports nothing. The run stays green. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
13 KiB
Runner image contract
The Runner image is the Runner built alone, as a plain file tree with no
entrypoint and no operating system. An image build copies paths out of it
instead of compiling the Runner again. Nothing in this repository consumes it
yet. Its name is ghcr.io/paperclipai/paperclip-runner, and one build makes
two variants:
- full, the
runner-artifactstarget of theDockerfile: every path below; - slim, the
runner-artifacts-slimtarget: the same without the two paths marked "full only", the provider pack's transport archives.
Layout
Every path a consumer may copy (RUNNER_IMAGE_PATHS in
scripts/runner-image-layout.mjs). Every file is readable by every user.
| Path | Content |
|---|---|
/runner-image.json |
Identity and description of this image. |
/paperclip-runnerd |
The daemon. A stripped release binary, mode 0755. |
/provider-pack/ |
The remote provider pack, with provider-pack.json. |
/provider-pack.archive.json |
Full only. Receipt of the pack's transport archives. |
/provider-pack.archives/ |
Full only. The transport archives the receipt lists. |
/runner-package/package.json |
The Runner package manifest. |
/runner-package/dist/ |
The Runner's complete build output. dist/bin/paperclip-runnerd is the same file as /paperclip-runnerd. |
/runner-package/node_modules/.cache/paperclip-runner-build-inputs.json |
The build stamp the Runner's build recorded for that output. |
/runner-package/scripts/provision-grok.mjs |
Provisions the native Grok executable. Reads platforms.json by relative path. |
/runner-package/src/providers/grok/platforms.json |
The Grok release pins. |
/runner-package/src/live/runnerd-codex-transport.ts |
The transport source, as a contract file. It defines RUNNER_CLIENT_VERSION. |
/bundles/runnerd-codex-transport.mjs |
The pack's dist/live/runnerd-codex-transport.js as one module. It imports without node_modules. |
/pnpm-lock.yaml |
The lockfile the image was installed from. |
Paths under /runner-package/ keep their place inside
packages/paperclip-runner. runner-image.json describes one platform, and
is the same file in both variants:
inputsDigest(the identity below) andbuildInputsDigest(the digest in the Runner's build stamp);sourceRevision: the commit that built the image, also in the pack aspayload.runnerSourceRevision. An image is published once for its inputs, so later commits with the same inputs keep this revision;publication:trueonly for a build with the build argumentPAPERCLIP_RUNNER_IMAGE_PUBLICATION=1. Only then does the pack carry the identity, aspayload.runnerInputsDigest. Another image build that runs the stage inline builds the same Runner but is not this image;platform(linux,amd64orarm64) andlibc: the glibc the daemon was compiled on (Debian bookworm, the oldest a consumer runs) and the newest glibc version it needs;providerPack: the pack's digest and target;pathsandsha256: the table above, and the digest of every single file;fullImageOnly: the names inpathsthat only the full image has.
Layers
- The pack's dependency files of 1 MiB or more: provider binaries and Node.
- Everything else, with
runner-image.json. - Full only: the transport archive of each file in layer 1.
- Full only: the other transport archives, the overlay archive, the receipt.
The slim image is layers 1 and 2. The full image is those same two layer
blobs, then layers 3 and 4, which hold transport content and nothing else.
Layers 1 and 3 have the same bytes in every build with the same dependency
files, so a new image shares them with the one before it. A large file under
the pack's dist is the Runner's own output: it and its archive are in
layers 2 and 4. An archive is named by its file's place in the receipt, which
lists the dependency files first, so their names do not depend on dist.
Identity
Both variants are named by one digest: the variant is not part of the
identity. The full image has the tag inputs-<digest> and the slim image
inputs-<digest>-slim. scripts/runner-image-identity.mjs computes the
digest from:
- every file under the paths the build copies from the build context
(
RUNNER_IMAGE_CONTEXT_INPUTS, which includepnpm-lock.yaml): path, executable bit and content, and each symbolic link by its target; - the whole
Dockerfile: the recipe, the base image pinned by digest and the toolchain pins. Any edit to it makes a new image; - the two platforms,
linux/amd64andlinux/arm64.
The source revision is not part of it, and neither is the text of
.dockerignore: a rule changes the digest only when it changes which of
those files the build receives. The build computes the digest for itself:
docker buildx build --target runner-artifacts-identity --output type=local,dest=/tmp/identity .
jq -r .digest /tmp/identity/runner-image-identity.json
The install is frozen. It fails when the lockfile is behind the manifests the
build reads (the root one, packages/paperclip-runner and
packages/paperclip-eval-kernel). A contributor cannot edit the lockfile, so
pull-request runs resolve it first. For a local build, run
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile.
Checks
CI builds each variant as an OCI archive and nothing else.
node scripts/runner-image-oci.mjs unpack <oci archive> --out <tree> writes
the tree of an image from its layers, so the tree CI checks is the bytes of
the image. It refuses a whiteout, an absolute or .. path, a path written
twice and an entry behind a link.
node scripts/runner-image-layout.mjs check <tree> --variant slim|full runs
in the build and again on each unpacked tree. The variant has no default: a
slim tree with the transport paths fails, and so does a full tree without
them. The check fails unless:
- every documented path exists, matches its digest in
runner-image.json, and resolves inside the tree; - the build stamp names the inputs the identity names;
- every artifact in
provider-pack.jsonhas its digest, thedisttree and each provider asset tree match their tree digests, and the pack names this image's revision and platform; - the pack names the inputs digest in a publication build and in no other.
--expect-inputs-digestfails on any other build; - in a full image, every transport archive matches its receipt, and the
archives extract to the pack. The check extracts them as a sandbox does (GNU tar
with the options in
provider-pack-layers.ts) and compares paths, content, link targets and all twelve permission bits, after thea+rXgrant the build makes once the archives are written.
node scripts/runner-image-oci.mjs check-layers --slim <oci archive> --full <oci archive>
checks "Layers" on the two archives, and that no file is in two layers.
The build also runs paperclip-runnerd --build-metadata on every base image
in docker/runner-image/Dockerfile.smoke. Add a base there before a consumer
uses it.
Consuming the image
Read the image only with COPY --from, and pin it by digest:
FROM ghcr.io/paperclipai/paperclip-runner@sha256:<digest> AS runner-artifacts
FROM debian:trixie-slim
COPY --from=runner-artifacts /paperclip-runnerd /usr/local/bin/paperclip-runnerd
A Dockerfile with a runner-artifacts or runner-artifacts-slim stage of
its own, as in this repository, can take a built image or tree in its place
with --build-context <stage>=<image or directory>. The stage is then not
built.
Choosing the Runner for an image build
scripts/runner-image-resolve.mjs decides whether a build of a checkout can
copy the published image. No build uses its answer yet: on a master push,
the runner_image_report job of .github/workflows/docker.yml runs it and
writes the answer to the job summary, and nothing reads that job. The job
runs on the committed files, holds no credential (the Runner package is
public, so the lookup is anonymous) and cannot fail the run. The script has
two commands, so that the one that builds reads no registry credential:
node scripts/runner-image-resolve.mjs inputs --dir "$dir"
node scripts/runner-image-resolve.mjs resolve --dir "$dir"
inputs exports the Runner build context, checks the lockfile and computes
the inputs digest in the build, as the publication workflow does. Run it on
the committed files, before any step that rewrites pnpm-lock.yaml.
resolve looks up inputs-<digest> and reports:
mode:publishedorinline, andreason;imageandslim_image: both variants, pinned by index digest. The slim digest is the one the full index names (see "Pairing");build_contexts: one--build-contextvalue on each line, which replace therunner-artifactsandrunner-artifacts-slimstages;labels:io.paperclip.runner.imageandio.paperclip.runner.inputs-digest, for the image that copies the Runner.
published needs a complete publication whose full index names these inputs
and both platforms. Every other answer is inline, with the last four values
empty: the lockfile is behind the Runner's manifests (lock-behind), the
image does not exist (absent), a publication is incomplete or broken,
the full index does not match (mismatch), or the registry refused, failed
or could not be reached (unknown). An inline build runs the Runner stages
itself and carries neither label. The command does not wait for a publication
and does not verify the attestation of the image.
Publication
.github/workflows/runner-image.yml publishes on a master push in this
repository and nowhere else. Other runs build and check, and publish nothing.
- Lockfile. A
masterpush builds from the committedpnpm-lock.yaml, so the digest is a function of the commit. When that file is behind the Runner's manifests, the run writes a notice, publishes nothing and stays green. The commit that merges the lockfile refresh publishes the image. - Once. The build runs only when
inputs-<digest>does not exist. - Verified bytes. The build legs cannot write to the registry. The
publish job pushes the images they checked by digest and joins each
variant into a multi-platform index under a
candidate-tag. It attests both indexes, and only then writes the twoinputs-tags.candidate-tags are internal: one can name an index that is not attested, and the next run overwrites it. Do not consume them. - Order. It writes
inputs-<digest>-slimfirst andinputs-<digest>last, so the full tag marks a complete publication. The full index names its slim index in the annotationio.paperclip.runner.slim-digest. - Pairing. Resolve the full tag first. Take the slim digest from that
annotation of the full index, and pull the slim image by that digest. Never
resolve
inputs-<digest>-slimby tag on its own: while a publication is incomplete, that tag can name the slim image of another run. - Recovery. A run that stops between the two tags leaves a slim tag
alone. Those inputs are not published. The next run for them builds again,
replaces that tag with its own slim image and then writes the full tag. A
full tag whose slim tag is missing or names another image fails every run
for those inputs. An operator must then delete both package versions and
re-run the workflow run of a
masterpush with those inputs. - Create-only. The workflow never moves a tag of a complete publication,
but the registry does not enforce that. Pin the index digest and verify it:
gh attestation verify oci://ghcr.io/paperclipai/paperclip-runner@sha256:<digest> --repo paperclipai/paperclip --signer-workflow paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref refs/heads/master - First publication. GitHub creates the package private. A maintainer must make it public once.
Known limits
- A build without
PAPERCLIP_RUNNER_SOURCE_REVISIONorPAPERCLIP_BUILD_COMMIThas no pack. It is for local use. - The build runs the steps of the Runner's
buildscript that write output, not its drift checks; repository CI runs those. - A build that vendors
/runner-package/reusesdistonly when its own Runner build-inputs digest equals the stamp's. The image computes that digest in the.dockerignore-filtered context; a checkout gives another. # syntax=docker/dockerfile:1.20is a tag. A new frontend release under it does not change the identity.- The archive check does not tell a directory with its own archive member from one created only as a file's parent.
- A lockfile that the lockfile refresh cannot bring up to the Runner's
manifests leaves every
masterrun green with only a notice, for as long as it lasts. No image is published in that time, and nothing fails.