mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Today the application image build compiles the Runner. Other images that need the Runner build it again, or pull the whole application image to copy it out > - Each of those builds can produce a slightly different Runner, and each one costs time > - The Runner should be built one time for each change to its inputs, and every image should copy the same bytes > - This pull request adds the build target `runner-artifacts`, a contract for its layout, and a workflow that builds and checks it. It publishes nothing > - Nothing consumes the target in this pull request. The application image build does not change > - The benefit is one checked Runner build that later changes can publish and copy by digest ## Linked Issues or Issue Description Refs #21096 (the merge-queue image build) and #21167 (provider pack layers). Those are earlier work on the same build. No issue exists for this change. **Problem or motivation** The Runner is compiled inside the application image build. The daemon comes from the `runner-build` stage. The provider pack is built on top of the full server and UI build. Downstream sandbox and VM images need the same Runner. They rebuild it from source, or they pull the application image and copy paths out of it. There is no contract for those paths and no single identity for a Runner build. **Proposed solution** Build the Runner alone, from its own inputs, as a plain file tree. Give each build an identity: a digest of its inputs. Document the layout as a contract, and check every built tree against it. Keep every check that proves the Runner is good in this one build. **Alternatives considered** - Keep the current builds and only document the paths in the application image. This does not remove the duplicate builds, and consumers still need the full application image. - Build the Runner in `docker.yml`. That workflow has no pull-request trigger, so a change to the Runner build would have no check before it merges. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4. Each part is on top of the one before it: 1. `runner/pack-inputs-digest`: an optional field in the provider pack manifest 2. `runner/runner-image-build` (#21204): build and check the Runner image, with no publication (this pull request) 3. `runner/runner-image-layers`: layers by change rate, and the slim variant 4. `runner/runner-image-publish`: publish both variants from `master` This pull request had one larger form before. Publication, the image layers and the second variant are now parts 3 and 4. ## What Changed - `Dockerfile`: eight new stages that end in the target `runner-artifacts` (`FROM scratch`). They install with a filtered, frozen lockfile from a context that holds only the Runner's inputs. The base is a Node image on Debian bookworm, pinned by digest. No existing stage changed. `cloud` is still the default target. - The tree holds: the stripped daemon, the Runner package's complete `dist` with the build stamp that its own build records, the provider pack with its transport archives, the Grok provisioning inputs, the transport contract source and a bundled transport module, the lockfile, and `runner-image.json`. - `scripts/runner-image-identity.mjs`: computes the image identity. It hashes every file under the paths that the build copies from the build context (path, executable bit, content; symbolic links by target), the whole `Dockerfile`, and the two platforms. The build runs it on the files that the build context supplies after `.dockerignore`. - `scripts/runner-image-layout.mjs`: the layout contract. `write` assembles the tree in the build. `check` verifies a built tree: every documented path, the build stamp, every pack artifact and tree digest in `provider-pack.json`, and every transport archive. It also extracts the transport archives the way a sandbox does and compares the result with the pack. No checked path may resolve outside the tree. - Publication form. The build argument `PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it does the pack manifest carry `payload.runnerInputsDigest` (part 1), and `runner-image.json` records `publication: true`. Another image build that runs the stage inline builds the same Runner, but its pack names no image. `check` requires the field in a publication build and refuses it in any other. `--expect-inputs-digest` passes only for a publication build. - `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd --build-metadata` on each base image that a consumer uses. Each base is pinned by digest. - `.github/workflows/runner-image.yml`: two jobs, on pull requests that touch an input and on manual dispatch. `identity` resolves the lockfile and computes the digest in the build context. `build` builds each architecture on a native runner, checks the layout, and runs the smoke matrix. The workflow has `contents: read` only. It holds no credential, reads no registry, and publishes nothing. It builds the publication form, so the check can bind the tree to the digest that the identity job resolved. The tree stays on the runner. - `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`: the tree digest function moves here from `build-provider-pack.mjs` without a change in behavior, so the layout check can use it. A test pins its value. - `.github/workflows/pr-merge-review.yml`: lists the new workflow, as `doc/PR-MERGE-REVIEW-WEBHOOK.md` requires. - `doc/runner-image-contract.md` (127 lines): the contract for consumers. `doc/DOCKER.md` links to it. Changes from the earlier form of this pull request: - The identity hashes the whole `Dockerfile`. The script no longer parses stages, and it no longer refuses recipes. `scripts/runner-image-identity.mjs` went from 397 lines to 166. A short test keeps the three rules that make the hash enough: the Runner stages start only from pinned images, only `runner-artifacts-context` reads the build context, and it copies only the paths that the identity hashes. - The contract document went from 384 lines to 127. - The workflow tests keep only the security properties: job permissions, no credential and no registry access, triggers, actions pinned by commit, and no expression inside a shell script. They no longer pin step names, order, or wording. ## Verification Verified on `linux/arm64` only, on a local Docker host: - `docker buildx build --target runner-artifacts --platform linux/arm64 --build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .` builds in 166 seconds without a cache. The image is one layer of 850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files. - `node scripts/runner-image-layout.mjs check <tree> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on the exported tree. - `docker buildx build --file docker/runner-image/Dockerfile.smoke --build-context runner-artifacts=<tree> docker/runner-image` passes on all five base images. - The digest from `--target runner-artifacts-identity` equals the digest in the built `runner-image.json`: `d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`. - The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication argument builds. Its pack has `runnerSourceRevision` and no `runnerInputsDigest`. `check` passes on it, and `check --expect-inputs-digest` fails with "this is not a publication build". - An edit to `ui/package.json` and to `server/src` leaves the digest unchanged. A comment added to an application stage of the `Dockerfile` changes it. - The two points above ran one commit state earlier. The only difference to this commit is one type annotation in a comment of `scripts/runner-image-layout.mjs`. - A named build context replaces the `runner-artifacts` stage, and no Runner stage runs. A build with the application build's view of the context accepts the tree's `dist` and build stamp as current (`assessRunnerBuild`). - The daemon has the same bytes in the five cold builds of this stack that were compared. - Tests: `cd server && node node_modules/vitest/vitest.mjs run` on `runner-image-identity.test.ts`, `runner-image-layout.test.ts` and `runner-image-workflow.test.ts` (132 tests), with `remote-provider-pack-verification.test.ts` and the existing Dockerfile tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. Never run: - The `linux/amd64` build. It is the leg that downloads the Cursor and Copilot assets for the pack. - The workflow. No job has run on GitHub. - The transport archive round trip with GNU tar outside the build. The build runs it with GNU tar. The local host ran it with bsdtar. - `actionlint` and `hadolint`. They are not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - Low risk to the application image. The change adds stages and files. No existing target reaches the new stages. - The first run of the workflow is the first test of the amd64 build. This pull request triggers it. - The `Dockerfile` is one file for all images. Any edit to it, also to an application stage, changes the Runner image identity. That is the price of not parsing it. - The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag, not a digest. A new frontend release under that tag does not change the identity. - The image does not run the drift checks of the Runner's `build` script. They need files outside the Runner's inputs, and repository CI runs them. A test fails when the `build` script gains a step. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
229 lines
8.7 KiB
YAML
229 lines
8.7 KiB
YAML
name: Runner image
|
|
|
|
# Builds and checks the Runner image: the `runner-artifacts` target of the
|
|
# Dockerfile, a plain file tree that other image builds copy the Runner from.
|
|
# doc/runner-image-contract.md is the contract.
|
|
#
|
|
# This workflow builds and checks only. It has no access to a registry and
|
|
# publishes nothing.
|
|
on:
|
|
pull_request:
|
|
# Every input of scripts/runner-image-identity.mjs, .dockerignore (it
|
|
# decides which of those files the build receives), and the pipeline's
|
|
# own files. A test checks that every identity input is covered.
|
|
paths:
|
|
- .github/workflows/runner-image.yml
|
|
- Dockerfile
|
|
- .dockerignore
|
|
- .npmrc
|
|
- package.json
|
|
- pnpm-lock.yaml
|
|
- pnpm-workspace.yaml
|
|
- tsconfig.base.json
|
|
- patches/**
|
|
- packages/paperclip-runner/**
|
|
- packages/paperclip-eval-kernel/**
|
|
- packages/shared/src/provider-failure.generated.ts
|
|
- scripts/create-provider-pack-archive.mjs
|
|
- scripts/link-plugin-dev-sdk.mjs
|
|
- scripts/runner-image-identity.mjs
|
|
- scripts/runner-image-layout.mjs
|
|
- docker/runner-image/**
|
|
workflow_dispatch:
|
|
|
|
# Least privilege: nothing at the workflow level, and read-only contents in
|
|
# each job.
|
|
permissions: {}
|
|
|
|
# A newer push to a pull request replaces its older run.
|
|
concurrency:
|
|
group: runner-image-${{ github.event.pull_request.number || github.sha }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
jobs:
|
|
identity:
|
|
name: Resolve the Runner image identity
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
digest: ${{ steps.identity.outputs.digest }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
run_install: false
|
|
|
|
# No dependency cache: a restored cache must not reach the image.
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
# A contributor cannot edit pnpm-lock.yaml, so a pull request that
|
|
# changes the Runner's dependencies is built against a lockfile resolved
|
|
# here. The image build installs from it frozen.
|
|
- name: Resolve the lockfile
|
|
run: |
|
|
set -euo pipefail
|
|
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
|
|
|
|
changed="$(git status --porcelain)"
|
|
if [ -n "$changed" ] && printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
|
|
echo "Unexpected files changed during lockfile resolution:"
|
|
echo "$changed"
|
|
exit 1
|
|
fi
|
|
echo "This run built from a \`pnpm-lock.yaml\` resolved for it${changed:+ (it differs from the committed file)}." >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# Computed inside the build, from the files the build context supplies
|
|
# after .dockerignore. A digest computed from the checkout would cover
|
|
# files the image build never receives.
|
|
- name: Compute the identity in the build context
|
|
id: identity
|
|
run: |
|
|
set -euo pipefail
|
|
out="$RUNNER_TEMP/runner-image-identity"
|
|
docker buildx build --target runner-artifacts-identity --output "type=local,dest=$out" .
|
|
digest="$(jq -er .digest "$out/runner-image-identity.json")"
|
|
[[ "$digest" =~ ^[0-9a-f]{64}$ ]]
|
|
echo "digest=$digest" >> "$GITHUB_OUTPUT"
|
|
{
|
|
echo "Runner image inputs digest: \`$digest\`"
|
|
echo
|
|
echo '```json'
|
|
cat "$out/runner-image-identity.json"
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# The legs must build from the file the identity was computed from, not
|
|
# from a second resolution.
|
|
- name: Keep the resolved lockfile for the build legs
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: runner-image-lockfile
|
|
path: pnpm-lock.yaml
|
|
if-no-files-found: error
|
|
retention-days: 1
|
|
|
|
# Multi-arch by native runner, not QEMU, as in docker.yml. GitHub-hosted
|
|
# runners: the protected release fleets admit only docker.yml on master.
|
|
#
|
|
# These legs hold no credential and use no registry build cache. The install
|
|
# runs dependency lifecycle scripts.
|
|
build:
|
|
name: Build and check (${{ matrix.arch }})
|
|
needs: identity
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- platform: linux/amd64
|
|
runner: ubuntu-latest
|
|
arch: amd64
|
|
- platform: linux/arm64
|
|
runner: ubuntu-24.04-arm
|
|
arch: arm64
|
|
runs-on: ${{ matrix.runner }}
|
|
timeout-minutes: 60
|
|
permissions:
|
|
contents: read
|
|
env:
|
|
DIGEST: ${{ needs.identity.outputs.digest }}
|
|
ARCH: ${{ matrix.arch }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
- name: Download the resolved lockfile
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
name: runner-image-lockfile
|
|
path: ${{ runner.temp }}/runner-image-lockfile
|
|
|
|
- name: Use the resolved lockfile
|
|
run: cp "$RUNNER_TEMP/runner-image-lockfile/pnpm-lock.yaml" pnpm-lock.yaml
|
|
|
|
- name: Free runner disk
|
|
run: |
|
|
set -euo pipefail
|
|
sudo apt-get clean || true
|
|
sudo rm -rf /usr/share/dotnet /usr/share/swift /usr/local/lib/android /usr/local/share/boost \
|
|
/usr/local/share/powershell /opt/ghc /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/PyPy /opt/hostedtoolcache/Ruby || true
|
|
docker system prune -af || true
|
|
df -h
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# The build itself runs the frozen install, the Runner's own build, the
|
|
# provider pack integrity check and the layout check. The tree is
|
|
# exported for the checks below.
|
|
#
|
|
# This is the publication form of the build, the only one whose pack
|
|
# names the image (`runnerInputsDigest`), so the check below can bind
|
|
# the tree to the digest the identity job resolved. Nothing here
|
|
# publishes it: the tree stays on this runner.
|
|
- name: Build the Runner image
|
|
run: |
|
|
set -euo pipefail
|
|
docker buildx build \
|
|
--target runner-artifacts \
|
|
--platform "$PLATFORM" \
|
|
--build-arg "PAPERCLIP_RUNNER_SOURCE_REVISION=$GITHUB_SHA" \
|
|
--build-arg PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 \
|
|
--provenance=false --sbom=false \
|
|
--output "type=local,dest=$RUNNER_TEMP/runner-image/tree" \
|
|
.
|
|
|
|
# Against the exported tree, bound to this run: the digest the identity
|
|
# job resolved, this commit and this leg's architecture. Checks every
|
|
# documented path, the build stamp, every pack artifact and tree digest
|
|
# in provider-pack.json, and every transport archive. It also extracts
|
|
# the archives as a sandbox does and compares the result with the pack;
|
|
# that needs about as much free disk as the pack is large.
|
|
- name: Check the layout contract and pack integrity
|
|
run: |
|
|
node scripts/runner-image-layout.mjs check "$RUNNER_TEMP/runner-image/tree" \
|
|
--expect-inputs-digest "$DIGEST" \
|
|
--expect-source-revision "$GITHUB_SHA" \
|
|
--expect-architecture "$ARCH"
|
|
|
|
- name: Run the daemon on every consumer base image
|
|
run: |
|
|
docker buildx build \
|
|
--file docker/runner-image/Dockerfile.smoke \
|
|
--platform "$PLATFORM" \
|
|
--build-context "runner-artifacts=$RUNNER_TEMP/runner-image/tree" \
|
|
--output "type=local,dest=$RUNNER_TEMP/runner-image/smoke" \
|
|
docker/runner-image
|
|
ls -l "$RUNNER_TEMP/runner-image/smoke"
|