From 3c2bc4f5464856837ed3a5ee627c5c1cd511bb97 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 21 Sep 2026 20:22:48 -0700 Subject: [PATCH] ci: halve the isolated native Runner check by seeding it from the public master build cache (#13736) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Recurring CI health check (PAP-31): on the most recent fully-green PR run (`cb703ac`, run [35519542997](https://github.com/paperclipai/paperclip/actions/runs/35519542997)), the slowest check was **Compile isolated native Runner** at **452s** — ahead of the largest test shards (379s). Every run recompiled the full Rust dependency tree from zero, even though `docker.yml` already refreshes a **public** `mode=max` BuildKit cache (`ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}`) on every master push, containing exactly these layers. This PR seeds **only the baseline build** with that registry cache, via anonymous pull. **Measured on this PR's own CI (which exercises the seeded path): the check completed in 123s, down from 452s — a 73% reduction, ~5.5 minutes saved per run.** ## Thinking Path Cost breakdown of the 452s from the job log: `cargo chef cook` dependency compile 214.7s, local cache export 43.1s, runner-core build 37.0s, metadata proof layer 36.8s, `cargo install cargo-chef` 36.4s, rebuild-verification build ~65s, setup/teardown ~20s. The dependency compile and toolchain layers are identical to what the production `docker.yml` build already caches publicly on every master push, so recompiling them here bought no signal — the check's real assertions live in the *verification* build, not the baseline. A first attempt used `actions/cache` plus a master `push` trigger, but the CI bot's GitHub App lacks `workflows` permission; the registry-cache approach is strictly better anyway (shared across PRs immediately, no 10GB Actions-cache quota pressure, no workflow change). ## What Changed - `scripts/check-docker-runner-cache.sh`: the baseline build now adds `--cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-{amd64|arm64}` (selected by host arch). `RUNNER_CHECK_SEED_CACHE` overrides the ref, or set it empty to force the old cold path. The script header documents the anonymous external read. - `.github/workflows/docker-runner-check.yml` (comment-only): the stale "no external cache" note now describes the anonymous GHCR seed and the verification build's local-cache-only isolation. This was pushed in a follow-up commit with workflow-edit permissions; the original CI-bot token could not touch workflow files. No Dockerfile stages or verification assertions changed. ## Verification - This PR's own `Compile isolated native Runner` check runs the seeded path (the script is in the workflow's trigger paths): **passed in 123s** vs the 452s baseline. - The rebuild-verification semantics are untouched: it still runs on a **fresh builder** importing **only the local cache exported by this run's baseline**, so it proves exactly what it proved before — that the runner image rebuilds reproducibly from this run's own exported layers. - Verified `ghcr.io/paperclipai/paperclip:buildcache-amd64` is anonymously readable (unauthenticated manifest pull succeeds), so the check gains no credential or secret dependency. ## Risks - **Stale or missing seed cache:** if the GHCR ref is unreachable, private, or garbage-collected, BuildKit logs a warning and falls back to the pre-PR cold compile — the check gets slower, never wrong. `RUNNER_CHECK_SEED_CACHE=""` restores the cold path explicitly. - **Cache trust:** the seed only accelerates the *baseline* build; the verification build still runs on a fresh builder against only this run's locally exported cache, so a stale or poisoned registry cache cannot make verification pass spuriously. The ref lives under `ghcr.io/paperclipai/*`, written only by repo CI on master pushes. ## Model Used Claude Fable 5 (`claude-fable-5`) via Paperclip agent **Bender (Fable)**, issue PAP-31. --------- Co-authored-by: Bender (Fable) Co-authored-by: Claude Fable 5 --- .github/workflows/docker-runner-check.yml | 5 ++++- scripts/check-docker-runner-cache.sh | 26 ++++++++++++++++++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/.github/workflows/docker-runner-check.yml b/.github/workflows/docker-runner-check.yml index d595522292..1f6002b4cd 100644 --- a/.github/workflows/docker-runner-check.yml +++ b/.github/workflows/docker-runner-check.yml @@ -33,6 +33,9 @@ jobs: uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 # Compile the real target, then change source in a disposable context. # A fresh builder must import dependencies and produce changed binary metadata. - # No registry credentials, external cache, or image publication. + # The baseline build anonymously seeds from the public BuildKit cache at + # ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}; the verification + # build imports only this run's locally exported cache on a fresh builder. + # No registry credentials or image publication. - name: Verify native build and dependency cache reuse run: bash scripts/check-docker-runner-cache.sh diff --git a/scripts/check-docker-runner-cache.sh b/scripts/check-docker-runner-cache.sh index 3456948793..a4b2e0c17a 100644 --- a/scripts/check-docker-runner-cache.sh +++ b/scripts/check-docker-runner-cache.sh @@ -1,6 +1,10 @@ #!/usr/bin/env bash # Build the real Docker target on two fresh builders using an exported cache. # Export only metadata, avoiding a multi-gigabyte test image in the daemon. +# External access: the baseline build anonymously reads the public BuildKit +# cache at ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64} (see +# RUNNER_CHECK_SEED_CACHE below). No credentials are used or required, and +# nothing is pushed. set -euo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")" @@ -34,7 +38,27 @@ build_proof() { docker buildx build --builder "$builder" --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$result" --progress plain "$@" . 2>&1 | tee "$probe_dir/$result.log" } docker buildx create --name "$baseline_builder" --driver docker-container -build_proof baseline "$baseline_builder" --cache-to "type=local,dest=$probe_dir/cache,mode=max" +# Seed only the baseline with the public BuildKit cache that docker.yml +# refreshes on every master push. The rust stages consume none of that +# build's args, so their layer keys match, and mode=max re-exports the +# imported layers into $probe_dir/cache — the verification build below +# still proves what it always proved from this run's exported cache +# alone, on a fresh builder. Anonymous pull only, nothing is pushed; a +# missing or unreachable ref is a BuildKit warning and the baseline +# degrades to the previous cold compile. Set RUNNER_CHECK_SEED_CACHE to +# another ref, or to the empty string to force the cold path. +if [[ -z "${RUNNER_CHECK_SEED_CACHE+x}" ]]; then + case "$(uname -m)" in + x86_64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-amd64" ;; + aarch64 | arm64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-arm64" ;; + *) RUNNER_CHECK_SEED_CACHE="" ;; + esac +fi +seed_args=() +if [[ -n "$RUNNER_CHECK_SEED_CACHE" ]]; then + seed_args=(--cache-from "type=registry,ref=${RUNNER_CHECK_SEED_CACHE}") +fi +build_proof baseline "$baseline_builder" ${seed_args[@]+"${seed_args[@]}"} --cache-to "type=local,dest=$probe_dir/cache,mode=max" # Removing the first builder proves the second build cannot use daemon-local # state, and releases its disk space before importing the exported cache. docker buildx rm "$baseline_builder"