mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-12 08:19:54 +02:00
ci: use a Docker Hub mirror for image builds (#15756)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Image workflows build the exact source selected for a release. > - BuildKit fetches the Dockerfile frontend and base images from Docker Hub. > - Docker Hub token failures can stop these builds before application code runs. > - BuildKit supports a public image cache with canonical Docker Hub fallback. > - This pull request configures that cache in the two affected image builders. > - The build inputs and publication gates stay the same. ## Linked Issues or Issue Description **What happened?** [Docker run 37995539183](https://github.com/paperclipai/paperclip/actions/runs/37995539183) failed before application builds started. The AMD64 job received HTTP 504 from the Docker Hub token endpoint while it fetched `docker/dockerfile:1.20`. The ARM64 job fetched the frontend, then received the same error while it resolved `node:24-trixie-slim`. [Release preview run 37992058052](https://github.com/paperclipai/paperclip/actions/runs/37992058052) also failed while it fetched the frontend. **Expected behavior** Use the supported public cache when it has the requested Docker Hub image. Retain canonical Docker Hub fallback when the cache has no usable copy. Preserve build and publish failures. **Steps to reproduce** Run either affected image builder during a Docker Hub token failure. The unconfigured BuildKit container contacts Docker Hub for the frontend and base image. It cannot use a host Docker daemon mirror for those build requests. **Paperclip version or commit** The observed Docker run used `00d944f96c8c4aba8332ce224df2be73a601228a`. The preview used `b6b3cb3761dabd1bd668054e7a99a245d53707c4`. **Deployment mode** GitHub Actions image builds. No application runtime change. The public PR and issue search found no matching open registry-mirror change. #15743 addresses a separate npm canary dependency publication failure. ## What Changed - Add the official `docker.io` to `mirror.gcr.io` BuildKit configuration to the public Docker build matrix and the release preview image builder. - Keep all existing action references, image sources, targets, build arguments, platforms, credentials, cache settings, outputs, permissions, triggers, and publication gates. - Extend the existing dependency-free workflow tests to check mirror placement, verified HTTPS, both native platforms, preview isolation, and publication access. - Document cache eviction, tag freshness, canonical fallback, and builder bootstrap limits. ## Verification - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs`: 43 passed on the final source. No dependency install required. - Six offline controls removed each builder's mirror, changed the registry or cache host, enabled insecure transport, or added preview registry login. Each control failed the new check as expected. - `actionlint -shellcheck='' .github/workflows/docker.yml .github/workflows/release.yml`: workflow validation passed. With optional ShellCheck enabled, the ten diagnostics match the unchanged base exactly. This PR changes no shell scripts. - A real TOML parser accepted the inline configuration. Removing the new block from each workflow produced byte-identical base files. - Read-only registry metadata checks confirmed matching raw bytes and SHA256 values for the current Node24 index and both native manifests. The Dockerfile frontend index and both native manifests also match Docker Hub and Google. No image config blobs or layers were fetched. - Full application tests, typecheck, and build were not repeated locally because application source is unchanged. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/37998739440) passed the full required checks on `25fd57f`, including all test shards, build, typecheck, policy, E2E, and canary dry run. - The supply-chain advisory returned neutral because it cannot verify these infrastructure edits. It reported zero annotations. Independent review confirmed all action references, Dockerfiles, package files, and lockfiles are unchanged. The other 53 checks passed; two Storybook checks skipped as expected. - Independent review approved all four changed files and the workflow graph. Greptile scored the same source 5/5 with no unresolved comments. - No Docker build, workflow rerun, image publish, or deployment was started for verification. A normal image workflow after merge must prove full image acquisition. ## Risks Google is a cache, not an outage guarantee. It can evict images. Mutable tags can lag upstream changes or deletions. Matching manifests now does not prove future tag freshness or blob availability. A cache miss can still fail when Docker Hub is unavailable. The initial `moby/buildkit:buildx-stable-1` bootstrap uses the host Docker daemon before this configuration takes effect and remains dependent on Docker Hub. The registry mirror applies to image pull and resolution only. GHCR publication and cache writes keep their existing routes. See [Docker's supported configuration](https://docs.docker.com/build/ci/github-actions/configure-builder/#registry-mirror) and [Google's cache limits](https://docs.cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images). ## Model Used OpenAI GPT-6 through Codex, with reasoning and code execution tools. The exact service variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
935fb63ed5
commit
e9dae72c0c
4 files changed
+61
No files matched your search
@@ -183,6 +183,10 @@ jobs:
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v4
|
||||
with:
|
||||
buildkitd-config-inline: |
|
||||
[registry."docker.io"]
|
||||
mirrors = ["mirror.gcr.io"]
|
||||
|
||||
# Deployment tooling reads these labels from the registry to verify an
|
||||
# image's schema expectations against a migrator before deploying it,
|
||||
|
||||
@@ -224,6 +224,10 @@ jobs:
|
||||
pnpm install --resolution-only --ignore-scripts --ignore-pnpmfile --no-frozen-lockfile
|
||||
echo "TOOLS_EPOCH=$(date -u +%G-W%V)" >> "$GITHUB_ENV"
|
||||
- uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
||||
with:
|
||||
buildkitd-config-inline: |
|
||||
[registry."docker.io"]
|
||||
mirrors = ["mirror.gcr.io"]
|
||||
- name: Build the immutable cloud image without registry credentials
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
|
||||
with:
|
||||
|
||||
@@ -304,6 +304,20 @@ suppresses push-triggered workflow runs for those pushes. The release jobs
|
||||
therefore dispatch `docker.yml` explicitly at the new tag ref; the tag
|
||||
mapping keys off `github.ref` either way.
|
||||
|
||||
The Docker build matrix and the release preview image builder configure
|
||||
BuildKit to check `mirror.gcr.io` for Docker Hub images first. This covers
|
||||
both the Dockerfile frontend and base images. BuildKit keeps Docker Hub as
|
||||
the fallback when the cache has no usable copy. The image references,
|
||||
build arguments, GHCR publication, and release gates stay the same.
|
||||
|
||||
This is an availability improvement, not an outage guarantee. Google can
|
||||
evict cached images, and mutable tags can lag upstream changes or deletions.
|
||||
Matching manifests at one point does not guarantee later tag freshness or
|
||||
blob availability. The initial `moby/buildkit` bootstrap still uses the host
|
||||
Docker daemon and can fail on Docker Hub before this configuration takes
|
||||
effect. See the [BuildKit mirror configuration](https://docs.docker.com/build/ci/github-actions/configure-builder/#registry-mirror)
|
||||
and [Google cache limits](https://docs.cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images).
|
||||
|
||||
## Local Commands
|
||||
|
||||
### Preview a canary locally
|
||||
|
||||
@@ -17,6 +17,45 @@ function readWorkflow(name) {
|
||||
return readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8");
|
||||
}
|
||||
|
||||
test("Docker Hub mirrors apply to both image builders without widening publication access", () => {
|
||||
const dockerWorkflow = readWorkflow("docker.yml");
|
||||
const releaseWorkflow = readWorkflow("release.yml");
|
||||
const docker = dockerWorkflow.split(" build-and-push:\n")[1].split("\n merge-and-push:")[0];
|
||||
const preview = releaseWorkflow.split(" image_preview:\n")[1].split("\n publish_image_preview:")[0];
|
||||
|
||||
for (const job of [docker, preview]) {
|
||||
const steps = job.split(/\n(?= - )/);
|
||||
const builder = steps.find((step) => /uses: docker\/setup-buildx-action@/.test(step));
|
||||
const build = steps.find((step) => /uses: docker\/build-push-action@/.test(step));
|
||||
assert.ok(builder && build, "the mirror must configure the builder that runs the image build");
|
||||
assert.ok(steps.indexOf(builder) < steps.indexOf(build));
|
||||
const config = builder.match(/buildkitd-config-inline: \|\n((?: {12,}[^\n]*(?:\n|$))+)/)?.[1];
|
||||
assert.ok(config, "configure the BuildKit container, not only the host Docker daemon");
|
||||
assert.deepEqual([...config.matchAll(/^\s*\[registry\."([^"]+)"\]/gm)].map((match) => match[1]), ["docker.io"],
|
||||
"GHCR publishing and cache traffic must not use the pull mirror");
|
||||
const mirrors = config.match(/^\s*mirrors = (\[[^\n]+\])$/m)?.[1];
|
||||
assert.ok(mirrors);
|
||||
assert.deepEqual(JSON.parse(mirrors), ["mirror.gcr.io"]);
|
||||
assert.doesNotMatch(config, /^\s*(?:http|insecure|ca|keypair)\s*=/m,
|
||||
"keep verified HTTPS and anonymous public-image pulls");
|
||||
assert.doesNotMatch(builder, /driver-opts:|buildkitd-flags:|secrets\./,
|
||||
"do not change bootstrap images, entitlements, or credentials");
|
||||
}
|
||||
|
||||
assert.match(docker, /platform: linux\/amd64/);
|
||||
assert.match(docker, /platform: linux\/arm64/);
|
||||
assert.match(docker, /permissions:\n\s+contents: read\n\s+packages: write/);
|
||||
assert.match(docker, /outputs: type=image,name=ghcr\.io\/\$\{\{ github\.repository \}\},push-by-digest=true,name-canonical=true,push=true/);
|
||||
assert.match(docker, /cache-to: type=registry,ref=ghcr\.io\/\$\{\{ github\.repository \}\}:buildcache-\$\{\{ matrix\.arch \}\},mode=max/);
|
||||
assert.match(preview, /permissions:\n\s+contents: read\n\s+steps:/);
|
||||
assert.match(preview, /platforms: linux\/amd64\n\s+push: false/);
|
||||
assert.match(preview, /outputs: type=docker,dest=\$\{\{ runner\.temp \}\}\/preview-image\.tar/);
|
||||
assert.doesNotMatch(preview, /docker\/login-action|secrets\.|packages: write|id-token: write/);
|
||||
assert.match(releaseWorkflow, /publish_image_preview:\n[\s\S]*?needs: \[plan_preview, image_preview\]/);
|
||||
assert.doesNotMatch(dockerWorkflow.split(" merge-and-push:\n")[1], /buildkitd-config-inline/,
|
||||
"the registry-only manifest merge does not need a Docker Hub mirror");
|
||||
});
|
||||
|
||||
test("chaos verification isolates callers that verify the same source commit", () => {
|
||||
const chaosWorkflow = readWorkflow("runner-chaos-evals.yml");
|
||||
const group = chaosWorkflow.match(/^ group: (.+)$/m)?.[1];
|
||||
|
||||
Reference in new issue
Block a user