fix(ci): reuse one available Cloud registry cache (#13334)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud needs a verified image for each merged source
commit.
> - Fresh builders restore compiled native dependencies from registry
caches.
> - The current workflow imports up to eleven historical cache manifests
at once.
> - Live builds missed native layers that a fresh builder reused from
one manifest.
> - This PR selects the nearest available cache and tests reuse across
fresh builders.

## Linked Issues or Issue Description

Refs #13329 and #13330. A search of open cache PRs found no duplicate of
this change.

**What existing behavior does this improve?**

Remote Docker cache reuse on fresh Cloud image builders.

**Current behavior**

[Cloud run
34714483272](https://github.com/paperclipai/paperclip/actions/runs/34714483272/job/103609096836)
imported the previous cache manifest successfully but rebuilt
`cargo-chef` and Rust dependencies. The dependency compile took 3m43s.
The preceding image build had already exported those layers.

A controlled [fresh-builder
diagnostic](https://github.com/paperclipai/paperclip/actions/runs/34715336530)
used the same source and registry cache. The single-manifest job reused
both layers immediately. The multiple-manifest job rebuilt them and
failed the cache assertion. Both jobs used GitHub-hosted runners with
read-only access.

**Proposed behavior**

Inspect cache manifests in first-parent order and import only the
nearest available one. Keep full-SHA cache exports, the ten-commit
search bound, and the legacy fallback. If caches cannot be read, permit
a cold build.

**Reason and benefit**

Avoid the observed cache misses without changing image contents or
builder sizes. Expected savings include about four minutes of native
tool/dependency compilation when those inputs are unchanged. The final
merge-to-deployable gain still needs a post-merge measurement.

**Breaking changes**

No image, artifact, deployment, or runner-routing contract changes.

## What Changed

- Select one available ancestor cache after Docker login and Buildx
setup.
- Preserve separate writable cache tags for each full source SHA.
- Test cache ordering, missing caches, registry errors, and workflow
integration.
- Add the selector tests to the existing release-registry suite.
- Export a local test cache, remove the first builder, and verify a
source rebuild on a fresh builder.
- Document cache selection and the stronger Docker check.

## Verification

- Passed 456 focused workflow, routing, readiness, preview-artifact, and
cache-selector tests.
- Passed shell syntax, ShellCheck for the changed probe, actionlint
workflow validation, and `git diff --check`. actionlint's shell checks
were disabled for the workflow validation because unchanged
migration-label commands trigger existing SC2012 notes.
- The fresh-builder registry diagnostic proves the single-cache
behavior. The [permanent two-builder probe
passed](https://github.com/paperclipai/paperclip/actions/runs/34715771048/job/103612624090),
including a changed real binary and dependency-declaration invalidation.
- Passed all 35 latest-head checks (green or intentionally skipped),
including full typecheck, test, build, and browser suites in [PR CI run
34715771217](https://github.com/paperclipai/paperclip/actions/runs/34715771217).
- The real selector CLI inspected registry metadata and chose the
nearest available ancestor cache.
- Fresh Greptile review is 5/5 with no open findings. The PR title was
corrected to meet the source-change naming rule; the review check passed
after that correction.
- Local full-suite runs and Docker builds are unavailable because the
local Docker daemon is unresponsive after disk exhaustion. CI provides
the Linux verification.

## Risks

- Missing or unreadable caches cause a slower cold build. The selector
logs that condition and preserves image publication.
- Inspecting several missing ancestors adds lookup time. Each lookup has
a ten-second timeout and the search is bounded.
- The Docker test now exports a local cache. It removes the first
builder before starting the second to release disk space, then cleans up
its builders and files.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

scripts/check-docker-runner-cache.sh

@@ -1,10 +1,18 @@
 #!/usr/bin/env bash
-# Build the real Docker target twice in a disposable copy of tracked source.
+# Build the real Docker target on two fresh builders using an exported cache.
 # Export only metadata, avoiding a multi-gigabyte test image in the daemon.
 set -euo pipefail
 repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")"
-trap 'rm -rf "$probe_dir"' EXIT
+baseline_builder="${probe_dir##*/}-baseline"
+rebuild_builder="${probe_dir##*/}-rebuild"
+cleanup() {
+  for builder in "$baseline_builder" "$rebuild_builder"; do
+    docker buildx rm "$builder" >/dev/null 2>&1 || true
+  done
+  rm -rf "$probe_dir"
+}
+trap cleanup EXIT
 mkdir "$probe_dir/context"
 cd "$repo_root"
 git ls-files -z | tar -cf - --null -T - | tar -xf - -C "$probe_dir/context"
@@ -21,9 +29,16 @@ FROM scratch AS recipe-proof-export
 COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json
 DOCKER
 build_proof() {
-  docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$1" --progress plain . 2>&1 | tee "$probe_dir/$1.log"
+  local result="$1" builder="$2"
+  shift 2
+  docker buildx build --builder "$builder" --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$result" --progress plain "$@" . 2>&1 | tee "$probe_dir/$result.log"
 }
-build_proof baseline
+docker buildx create --name "$baseline_builder" --driver docker-container
+build_proof baseline "$baseline_builder" --cache-to "type=local,dest=$probe_dir/cache,mode=max"
+# Removing the first builder proves the second build cannot use daemon-local
+# state, and releases its disk space before importing the exported cache.
+docker buildx rm "$baseline_builder"
+docker buildx create --name "$rebuild_builder" --driver docker-container
 python3 - <<'CHECK'
 from pathlib import Path
 p=Path('packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs')
@@ -31,7 +46,7 @@ s=p.read_text(); needle='paperclip-runner/runnerd-build-metadata/v1'
 assert s.count(needle)==1
 p.write_text(s.replace(needle,needle+'-cache-probe'))
 CHECK
-build_proof source-change
+build_proof source-change "$rebuild_builder" --cache-from "type=local,src=$probe_dir/cache"
 python3 - <<'CHECK'
 import os,json,re
 from pathlib import Path
@@ -45,12 +60,12 @@ log=(root/'source-change.log').read_text()
 step=re.search(r'#(\d+) \[runner-deps[^\n]+ RUN cargo chef cook',log)[1]
 assert f'#{step} CACHED' in log
 assert 'Compiling paperclip-runner-core' in log
-print('PASS: unchanged dependency recipe and cached cook layer; real binary changed.')
+print('PASS: fresh builder imported compiled dependencies; real binary changed.')
 p=Path('packages/paperclip-runner/runner/Cargo.toml')
 s=p.read_text(); assert 'serde_json = "1.0"' in s
 p.write_text(s.replace('serde_json = "1.0"','serde_json = ">=1.0.0, <2.0.0"'))
 CHECK
-docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain .
+docker buildx build --builder "$rebuild_builder" --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain .
 python3 - <<'CHECK'
 from pathlib import Path
 import os

scripts/preview-artifacts.test.mjs

@@ -4,6 +4,7 @@ import { planArtifacts } from "./preview-artifacts.mjs";
 import { readFileSync, mkdtempSync, writeFileSync, rmSync } from "node:fs";
 import { tmpdir } from "node:os";
 import path from "node:path";
+import { fileURLToPath } from "node:url";
 import { gzipSync } from "node:zlib";
 import { execFileSync, spawnSync } from "node:child_process";
 import { previewManifest, assertMetadata, validateRequest, versionFor, tarManifest, packageExists, imageExists, publishPreview, publishImage } from "./preview-artifacts.mjs";
@@ -217,8 +218,13 @@ test("cloud builds bake the managed runtime identity and verify it before public
 
 test("cloud cache imports are bounded, follow master ancestry, and retain the legacy fallback", () => {
   const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
-  const step = workflow.split("      - name: Select cloud cache ancestry")[1].split("      - name: Setup pnpm")[0];
-  const script = step.split("        run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
+  const selector = workflow.indexOf("      - name: Select cloud cache ancestry");
+  assert.ok(selector > workflow.indexOf("      - name: Login to GitHub Container Registry"));
+  assert.ok(selector > workflow.indexOf("      - name: Set up Docker Buildx"));
+  assert.ok(selector < workflow.indexOf("      - name: Build and push (cloud)"));
+  assert.match(workflow, /run: node scripts\/select-cloud-cache.mjs/);
+  assert.match(workflow, /cache-from: \$\{\{ steps.cloud-cache.outputs.source \}\}/);
+  const script = fileURLToPath(new URL("./select-cloud-cache.mjs", import.meta.url));
   const dir = mkdtempSync(path.join(tmpdir(), "cloud-cache-test-"));
   const output = path.join(dir, "output");
   const env = { ...process.env, GIT_AUTHOR_NAME: "Test", GIT_AUTHOR_EMAIL: "test@example.test", GIT_COMMITTER_NAME: "Test", GIT_COMMITTER_EMAIL: "test@example.test" };
@@ -235,14 +241,25 @@ test("cloud cache imports are bounded, follow master ancestry, and retain the le
     git("checkout", "master");
     git("merge", "--no-ff", "topic", "-m", "merge topic");
     commits.unshift(git("rev-parse", "HEAD"));
-    const result = spawnSync("bash", ["-c", script], { cwd: dir, encoding: "utf8", env: { ...env, CACHE_IMAGE: "ghcr.io/paperclipai/paperclip", GITHUB_OUTPUT: output } });
+    const available = `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commits[2]}`;
+    const inspections = path.join(dir, "inspections");
+    writeFileSync(path.join(dir, "docker"), `#!/usr/bin/env node
+const fs = require("node:fs");
+fs.appendFileSync(process.env.CACHE_INSPECTIONS, process.argv.at(-1) + "\\n");
+if (process.argv.at(-1) !== process.env.AVAILABLE_CACHE) {
+  process.stderr.write("manifest unknown");
+  process.exit(1);
+}
+`, { mode: 0o755 });
+    const result = spawnSync(process.execPath, [script], {
+      cwd: dir, encoding: "utf8", env: {
+        ...env, PATH: `${dir}${path.delimiter}${env.PATH}`, CACHE_IMAGE: "ghcr.io/paperclipai/paperclip",
+        GITHUB_OUTPUT: output, AVAILABLE_CACHE: available, CACHE_INSPECTIONS: inspections,
+      },
+    });
     assert.equal(result.status, 0, result.stderr);
-    assert.deepEqual(readFileSync(output, "utf8").trim().split("\n"), [
-      "sources<<CACHE_SOURCES",
-      ...commits.slice(0, 10).map((commit) => `type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-cloud-${commit}`),
-      "type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-cloud",
-      "CACHE_SOURCES",
-    ]);
+    assert.equal(readFileSync(output, "utf8"), `source=type=registry,ref=${available}\n`);
+    assert.deepEqual(readFileSync(inspections, "utf8").trim().split("\n"), commits.slice(0, 3).map((commit) => `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commit}`));
   } finally { rmSync(dir, { recursive: true, force: true }); }
 });
 

scripts/select-cloud-cache.mjs

@@ -0,0 +1,66 @@
+#!/usr/bin/env node
+import { execFileSync } from "node:child_process";
+import { appendFileSync } from "node:fs";
+import { pathToFileURL } from "node:url";
+
+export function cloudCacheCandidates(image, commits) {
+  if (!/^ghcr\.io\/[a-z0-9._-]+\/[a-z0-9._-]+$/.test(image ?? "")) {
+    throw new Error("Expected a GHCR owner/repository cache image.");
+  }
+  if (!Array.isArray(commits) || commits.length === 0 || commits.some((sha) => !/^[a-f0-9]{40}$/.test(sha))) {
+    throw new Error("Cloud cache ancestry requires full commit SHAs.");
+  }
+  return [
+    ...[...new Set(commits)].slice(0, 10).map((sha) => `${image}:buildcache-cloud-${sha}`),
+    `${image}:buildcache-cloud`,
+  ];
+}
+
+export async function selectCloudCache(image, commits, {
+  exists = registryCacheExists,
+  log = console.log,
+} = {}) {
+  for (const ref of cloudCacheCandidates(image, commits)) {
+    try {
+      if (!await exists(ref)) continue;
+      log(`Using cloud cache: ${ref}`);
+      return `type=registry,ref=${ref}`;
+    } catch {
+      // Cache availability must not turn an otherwise valid build into a
+      // failure. A later ancestor may still be available during a rollout.
+      log(`Could not inspect cloud cache ${ref}; trying the next ancestor.`);
+    }
+  }
+  log("No cloud cache is available; this build will populate one.");
+  return "";
+}
+
+function registryCacheExists(ref) {
+  try {
+    // Use the preceding Docker login, including for private registry caches.
+    // Inspect metadata only: no layer download and no image execution.
+    execFileSync("docker", ["buildx", "imagetools", "inspect", "--raw", ref], {
+      timeout: 10_000,
+      maxBuffer: 1024 * 1024,
+      stdio: ["ignore", "pipe", "pipe"],
+    });
+    return true;
+  } catch (error) {
+    if (/manifest unknown|not found|NAME_UNKNOWN/i.test(String(error.stderr ?? ""))) return false;
+    throw error;
+  }
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  try {
+    if (!process.env.GITHUB_OUTPUT) throw new Error("GITHUB_OUTPUT is required.");
+    const commits = execFileSync("git", ["rev-list", "--first-parent", "--max-count=10", "HEAD"], {
+      encoding: "utf8",
+    }).trim().split("\n");
+    const source = await selectCloudCache(process.env.CACHE_IMAGE, commits, { exists: registryCacheExists });
+    appendFileSync(process.env.GITHUB_OUTPUT, `source=${source}\n`);
+  } catch (error) {
+    console.error(error.message);
+    process.exitCode = 1;
+  }
+}

scripts/select-cloud-cache.test.mjs

@@ -0,0 +1,65 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { cloudCacheCandidates, selectCloudCache } from "./select-cloud-cache.mjs";
+
+const image = "ghcr.io/paperclipai/paperclip";
+const commits = ["a".repeat(40), "b".repeat(40), "c".repeat(40)];
+const candidates = cloudCacheCandidates(image, commits);
+
+test("a same-SHA rerun imports only its existing cache", async () => {
+  const inspected = [];
+  const source = await selectCloudCache(image, commits, {
+    exists: async (ref) => { inspected.push(ref); return true; }, log() {},
+  });
+  assert.equal(source, `type=registry,ref=${candidates[0]}`);
+  assert.deepEqual(inspected, candidates.slice(0, 1));
+});
+
+test("a new merge imports only its nearest available ancestor", async () => {
+  const inspected = [];
+  const source = await selectCloudCache(image, commits, {
+    exists: async (ref) => { inspected.push(ref); return ref === candidates[1]; }, log() {},
+  });
+  assert.equal(source, `type=registry,ref=${candidates[1]}`);
+  assert.deepEqual(inspected, candidates.slice(0, 2));
+  assert.equal(source.includes("\n"), false);
+});
+
+test("a still-building parent falls back to an older completed cache", async () => {
+  assert.equal(await selectCloudCache(image, commits, {
+    exists: async (ref) => ref === candidates[2], log() {},
+  }), `type=registry,ref=${candidates[2]}`);
+});
+
+test("the legacy cache is used only if no SHA cache exists", async () => {
+  const inspected = [];
+  assert.equal(await selectCloudCache(image, commits, {
+    exists: async (ref) => { inspected.push(ref); return ref === candidates.at(-1); }, log() {},
+  }), `type=registry,ref=${candidates.at(-1)}`);
+  assert.deepEqual(inspected, candidates);
+});
+
+test("missing caches permit a cold build", async () => {
+  assert.equal(await selectCloudCache(image, commits, { exists: async () => false, log() {} }), "");
+});
+
+test("a failed lookup can fall back without failing image publication", async () => {
+  const messages = [];
+  assert.equal(await selectCloudCache(image, commits, {
+    exists: async (ref) => {
+      if (ref === candidates[0]) throw new Error("registry temporarily unavailable");
+      return true;
+    },
+    log: (message) => messages.push(message),
+  }), `type=registry,ref=${candidates[1]}`);
+  assert.match(messages[0], /Could not inspect cloud cache/);
+});
+
+test("ancestry is bounded, deduplicated, and rejects output injection", () => {
+  const many = Array.from({ length: 20 }, (_, i) => i.toString(16).padStart(40, "0"));
+  assert.equal(cloudCacheCandidates(image, many).length, 11);
+  assert.deepEqual(cloudCacheCandidates(image, [commits[0], commits[0]]), [candidates[0], candidates.at(-1)]);
+  assert.throws(() => cloudCacheCandidates(`${image}\nsource=untrusted`, commits));
+  assert.throws(() => cloudCacheCandidates(image, ["master"]));
+  assert.throws(() => cloudCacheCandidates(image, []));
+});