mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
chore(ci): remove the Docker dependency cache check of the Runner stages (#21331)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. The application image contains it > - Today the application image compiles the Runner in its own stages, with a cargo-chef dependency layer > - A pull-request job proves that a fresh builder reuses that layer. The job runs a script that appends to the `Dockerfile` and names those stages > - The next pull request removes those stages. The application image then takes its Runner from the `runner-artifacts` stages. The job cannot run against that `Dockerfile` > - This pull request removes the job and its script > - The benefit is that the next pull request changes only the image build, and its checks stay green ## Linked Issues or Issue Description Refs #21204, #21229, #21230, #21306 and #21307 (the Runner image and the preparation for this change). No issue exists for this change. **Problem or motivation** `.github/workflows/docker-runner-check.yml` has a pull-request job, `Compile isolated native Runner`. It runs `scripts/check-docker-runner-cache.sh`. The script adds stages to a copy of the `Dockerfile` that start from `runner-build` and copy from `runner-plan`. It reads the build log for the `runner-deps` step. The job runs on every pull request that changes the `Dockerfile`. The pull request that removes those stages would fail this job. **Proposed solution** Remove the job, its trigger and its script first, in a pull request of its own. **Alternatives considered** - Remove the job in the same pull request as the stages. That pull request is already large. - Rewrite the script for the `runner-artifacts` stages. Those stages have no dependency layer, so the property does not exist. `runner-image.yml` already builds them on a pull request. This is the first of three pull requests that move the application image onto the `runner-artifacts` stages. It has to merge first: the removed job runs on every `Dockerfile` change and builds stages that the second pull request deletes. The other two follow on top of it. ## What Changed - Delete `scripts/check-docker-runner-cache.sh`. - Remove the `runner` job and the `pull_request` trigger from `docker-runner-check.yml`. Only the manual jobs remain. The concurrency group no longer reads a pull request number. - Remove `Docker Runner check` from the workflow list of `pr-merge-review.yml`. That notifier acts only on pull-request runs, and the workflow has none now. - Remove the paragraph of `doc/DOCKER.md` that describes the check. ## Verification - `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`: 22 pass. - `vitest run server/src/__tests__/runner-image-workflow.test.ts`: pass (1 test is skipped on macOS, where bash is 3.2). - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/workflow-target.test.ts`: 28 pass on macOS. These files read `docker-runner-check.yml`. - The same commands in a `node:24.22.3-bookworm` container: all pass, except one test of `workflow-target.test.ts`. It fails with `jq: command not found`. It reads another workflow, and the container has no `jq`. - Not run: the full test suite, and the workflows themselves. Only CI can run the workflows. ## Risks - Until the next pull request merges, a pull request that changes only the native inputs of the Runner (`packages/paperclip-runner/runner/**`, `protocol/**`, `rust-toolchain.toml`) does not build the `runner-build` stage of the application image. `runner-image.yml` builds the daemon from the same files in the `runner-artifacts` stages on such a pull request. `docker-app-check.yml` builds `runner-build` when the `Dockerfile` or `.dockerignore` changes. - No check proves the cargo-chef layer reuse after this change. The next pull request removes that layer. - The job was not a required check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
b0e99dfcfa
commit
bef2aac905
4 files changed
+1
-149
No files matched your search
@@ -35,48 +35,15 @@ on:
|
||||
description: "Build the canonical eval report viewer on hosted Linux"
|
||||
type: boolean
|
||||
default: false
|
||||
pull_request:
|
||||
paths:
|
||||
- .github/workflows/docker-runner-check.yml
|
||||
- Dockerfile
|
||||
- .dockerignore
|
||||
- scripts/check-docker-runner-cache.sh
|
||||
- packages/paperclip-runner/rust-toolchain.toml
|
||||
- packages/paperclip-runner/runner/**
|
||||
- packages/paperclip-runner/protocol/**
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
# Publishing a newer image must not discard an earlier verification's evidence.
|
||||
group: docker-runner-check-${{ github.event.pull_request.number || github.ref }}-${{ inputs.verify_source && 'verification' || inputs.verify_public_install && 'public-install' || inputs.build_eval_viewer && 'viewer' || 'image' }}
|
||||
group: docker-runner-check-${{ github.ref }}-${{ inputs.verify_source && 'verification' || inputs.verify_public_install && 'public-install' || inputs.build_eval_viewer && 'viewer' || 'image' }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
runner:
|
||||
if: github.event_name == 'pull_request'
|
||||
name: Compile isolated native Runner
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
||||
# Compile the real target, then change source in a disposable context.
|
||||
# A fresh builder must import dependencies and produce changed binary metadata.
|
||||
# The baseline build anonymously seeds from the public BuildKit cache at
|
||||
# ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}; the verification
|
||||
# build imports only this run's locally exported cache on a fresh builder.
|
||||
# No registry credentials or image publication.
|
||||
- name: Verify native build and dependency cache reuse
|
||||
run: bash scripts/check-docker-runner-cache.sh
|
||||
|
||||
|
||||
authorize_manual:
|
||||
if: github.event_name == 'workflow_dispatch'
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -6,7 +6,6 @@ on:
|
||||
- PR
|
||||
- Runner Windows
|
||||
- Sentry SDK contract
|
||||
- Docker Runner check
|
||||
- Runner image
|
||||
- commitperclip PR Review
|
||||
- Storybook Visual
|
||||
|
||||
@@ -424,18 +424,3 @@ input file times are normalized in both stages so fresh checkouts do not force
|
||||
Cargo to rebuild unchanged source. Changes made by build scripts still reach
|
||||
Cargo's normal validation. The final application copy excludes Cargo's target
|
||||
directory as before. Cache misses only cost compilation time.
|
||||
|
||||
Pull requests that change the Dockerfile, Docker ignore rules, or Runner native
|
||||
inputs also build the isolated `runner-build` target in `Docker Runner check`.
|
||||
The check runs `bash scripts/check-docker-runner-cache.sh` against a disposable
|
||||
copy of tracked source and the actual Docker ignore rules. It compiles a baseline
|
||||
and exports a local cache, removes that builder, changes a Rust metadata constant,
|
||||
and rebuilds on a fresh builder using only the exported cache. It requires a
|
||||
cached dependency build, an unchanged dependency recipe, and changed metadata
|
||||
from the real binary. It also verifies that a dependency declaration change
|
||||
alters the recipe. The probe exports small metadata results instead of importing
|
||||
a large test image into the Docker daemon. Temporary builders and cache files
|
||||
are removed afterward. It catches missing embedded inputs before the post-merge
|
||||
build. It uses a GitHub-hosted runner with read-only repository access and never
|
||||
publishes images or registry caches. Allow up to 20 minutes for its cold build and
|
||||
source rebuild.
|
||||
@@ -1,99 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the real Docker target on two fresh builders using an exported cache.
|
||||
# Export only metadata, avoiding a multi-gigabyte test image in the daemon.
|
||||
# External access: the baseline build anonymously reads the public BuildKit
|
||||
# cache at ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64} (see
|
||||
# RUNNER_CHECK_SEED_CACHE below). No credentials are used or required, and
|
||||
# nothing is pushed.
|
||||
set -euo pipefail
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")"
|
||||
baseline_builder="${probe_dir##*/}-baseline"
|
||||
rebuild_builder="${probe_dir##*/}-rebuild"
|
||||
cleanup() {
|
||||
for builder in "$baseline_builder" "$rebuild_builder"; do
|
||||
docker buildx rm "$builder" >/dev/null 2>&1 || true
|
||||
done
|
||||
rm -rf "$probe_dir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir "$probe_dir/context"
|
||||
cd "$repo_root"
|
||||
git ls-files -z | tar -cf - --null -T - | tar -xf - -C "$probe_dir/context"
|
||||
cd "$probe_dir/context"
|
||||
export PROBE_DIR="$probe_dir"
|
||||
cp Dockerfile "$probe_dir/cache-probe.Dockerfile"
|
||||
cat >> "$probe_dir/cache-probe.Dockerfile" <<'DOCKER'
|
||||
FROM runner-build AS cache-proof
|
||||
RUN ./runner/target/release/paperclip-runnerd --build-metadata > /metadata.json
|
||||
FROM scratch AS cache-proof-export
|
||||
COPY --from=cache-proof /metadata.json /metadata.json
|
||||
COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json
|
||||
FROM scratch AS recipe-proof-export
|
||||
COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json
|
||||
DOCKER
|
||||
build_proof() {
|
||||
local result="$1" builder="$2"
|
||||
shift 2
|
||||
docker buildx build --builder "$builder" --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$result" --progress plain "$@" . 2>&1 | tee "$probe_dir/$result.log"
|
||||
}
|
||||
docker buildx create --name "$baseline_builder" --driver docker-container
|
||||
# Seed only the baseline with the public BuildKit cache that docker.yml
|
||||
# refreshes on every master push. The rust stages consume none of that
|
||||
# build's args, so their layer keys match, and mode=max re-exports the
|
||||
# imported layers into $probe_dir/cache — the verification build below
|
||||
# still proves what it always proved from this run's exported cache
|
||||
# alone, on a fresh builder. Anonymous pull only, nothing is pushed; a
|
||||
# missing or unreachable ref is a BuildKit warning and the baseline
|
||||
# degrades to the previous cold compile. Set RUNNER_CHECK_SEED_CACHE to
|
||||
# another ref, or to the empty string to force the cold path.
|
||||
if [[ -z "${RUNNER_CHECK_SEED_CACHE+x}" ]]; then
|
||||
case "$(uname -m)" in
|
||||
x86_64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-amd64" ;;
|
||||
aarch64 | arm64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-arm64" ;;
|
||||
*) RUNNER_CHECK_SEED_CACHE="" ;;
|
||||
esac
|
||||
fi
|
||||
seed_args=()
|
||||
if [[ -n "$RUNNER_CHECK_SEED_CACHE" ]]; then
|
||||
seed_args=(--cache-from "type=registry,ref=${RUNNER_CHECK_SEED_CACHE}")
|
||||
fi
|
||||
build_proof baseline "$baseline_builder" ${seed_args[@]+"${seed_args[@]}"} --cache-to "type=local,dest=$probe_dir/cache,mode=max"
|
||||
# Removing the first builder proves the second build cannot use daemon-local
|
||||
# state, and releases its disk space before importing the exported cache.
|
||||
docker buildx rm "$baseline_builder"
|
||||
docker buildx create --name "$rebuild_builder" --driver docker-container
|
||||
python3 - <<'CHECK'
|
||||
from pathlib import Path
|
||||
p=Path('packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs')
|
||||
s=p.read_text(); needle='paperclip-runner/runnerd-build-metadata/v1'
|
||||
assert s.count(needle)==1
|
||||
p.write_text(s.replace(needle,needle+'-cache-probe'))
|
||||
CHECK
|
||||
build_proof source-change "$rebuild_builder" --cache-from "type=local,src=$probe_dir/cache"
|
||||
python3 - <<'CHECK'
|
||||
import os,json,re
|
||||
from pathlib import Path
|
||||
root=Path(os.environ['PROBE_DIR'])
|
||||
before=json.loads((root/'baseline/metadata.json').read_text())
|
||||
after=json.loads((root/'source-change/metadata.json').read_text())
|
||||
assert before['schema']=='paperclip-runner/runnerd-build-metadata/v1'
|
||||
assert after['schema']==before['schema']+'-cache-probe'
|
||||
assert (root/'baseline/recipe.json').read_bytes()==(root/'source-change/recipe.json').read_bytes()
|
||||
log=(root/'source-change.log').read_text()
|
||||
step=re.search(r'#(\d+) \[runner-deps[^\n]+ RUN cargo chef cook',log)[1]
|
||||
assert f'#{step} CACHED' in log
|
||||
assert 'Compiling paperclip-runner-core' in log
|
||||
print('PASS: fresh builder imported compiled dependencies; real binary changed.')
|
||||
p=Path('packages/paperclip-runner/runner/Cargo.toml')
|
||||
s=p.read_text(); assert 'serde_json = "1.0"' in s
|
||||
p.write_text(s.replace('serde_json = "1.0"','serde_json = ">=1.0.0, <2.0.0"'))
|
||||
CHECK
|
||||
docker buildx build --builder "$rebuild_builder" --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain .
|
||||
python3 - <<'CHECK'
|
||||
from pathlib import Path
|
||||
import os
|
||||
root=Path(os.environ['PROBE_DIR'])
|
||||
assert (root/'source-change/recipe.json').read_bytes()!=(root/'manifest-change/recipe.json').read_bytes()
|
||||
print('PASS: dependency declaration change invalidates the recipe.')
|
||||
CHECK
|
||||
Reference in new issue
Block a user