mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-15 02:47:03 +02:00
fix(ci): reuse one available Cloud registry cache (#13334)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud needs a verified image for each merged source commit. > - Fresh builders restore compiled native dependencies from registry caches. > - The current workflow imports up to eleven historical cache manifests at once. > - Live builds missed native layers that a fresh builder reused from one manifest. > - This PR selects the nearest available cache and tests reuse across fresh builders. ## Linked Issues or Issue Description Refs #13329 and #13330. A search of open cache PRs found no duplicate of this change. **What existing behavior does this improve?** Remote Docker cache reuse on fresh Cloud image builders. **Current behavior** [Cloud run 34714483272](https://github.com/paperclipai/paperclip/actions/runs/34714483272/job/103609096836) imported the previous cache manifest successfully but rebuilt `cargo-chef` and Rust dependencies. The dependency compile took 3m43s. The preceding image build had already exported those layers. A controlled [fresh-builder diagnostic](https://github.com/paperclipai/paperclip/actions/runs/34715336530) used the same source and registry cache. The single-manifest job reused both layers immediately. The multiple-manifest job rebuilt them and failed the cache assertion. Both jobs used GitHub-hosted runners with read-only access. **Proposed behavior** Inspect cache manifests in first-parent order and import only the nearest available one. Keep full-SHA cache exports, the ten-commit search bound, and the legacy fallback. If caches cannot be read, permit a cold build. **Reason and benefit** Avoid the observed cache misses without changing image contents or builder sizes. Expected savings include about four minutes of native tool/dependency compilation when those inputs are unchanged. The final merge-to-deployable gain still needs a post-merge measurement. **Breaking changes** No image, artifact, deployment, or runner-routing contract changes. ## What Changed - Select one available ancestor cache after Docker login and Buildx setup. - Preserve separate writable cache tags for each full source SHA. - Test cache ordering, missing caches, registry errors, and workflow integration. - Add the selector tests to the existing release-registry suite. - Export a local test cache, remove the first builder, and verify a source rebuild on a fresh builder. - Document cache selection and the stronger Docker check. ## Verification - Passed 456 focused workflow, routing, readiness, preview-artifact, and cache-selector tests. - Passed shell syntax, ShellCheck for the changed probe, actionlint workflow validation, and `git diff --check`. actionlint's shell checks were disabled for the workflow validation because unchanged migration-label commands trigger existing SC2012 notes. - The fresh-builder registry diagnostic proves the single-cache behavior. The [permanent two-builder probe passed](https://github.com/paperclipai/paperclip/actions/runs/34715771048/job/103612624090), including a changed real binary and dependency-declaration invalidation. - Passed all 35 latest-head checks (green or intentionally skipped), including full typecheck, test, build, and browser suites in [PR CI run 34715771217](https://github.com/paperclipai/paperclip/actions/runs/34715771217). - The real selector CLI inspected registry metadata and chose the nearest available ancestor cache. - Fresh Greptile review is 5/5 with no open findings. The PR title was corrected to meet the source-change naming rule; the review check passed after that correction. - Local full-suite runs and Docker builds are unavailable because the local Docker daemon is unresponsive after disk exhaustion. CI provides the Linux verification. ## Risks - Missing or unreadable caches cause a slower cold build. The selector logs that condition and preserves image publication. - Inspecting several missing ancestors adds lookup time. Each lookup has a ten-second timeout and the search is bounded. - The Docker test now exports a local cache. It removes the first builder before starting the second to release disk space, then cleans up its builders and files. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
0ce7df2648
commit
44f6312cd8
8 files changed
+208
-45
No files matched your search
@@ -65,24 +65,6 @@ jobs:
|
||||
id: tools-epoch
|
||||
run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Each SHA exports its own cache. Import recent first-parent caches so
|
||||
# a late older build cannot overwrite a newer build's cache manifest.
|
||||
# The legacy ref keeps the first builds warm during the transition.
|
||||
- name: Select cloud cache ancestry
|
||||
id: cloud-cache
|
||||
env:
|
||||
CACHE_IMAGE: ghcr.io/${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
{
|
||||
echo 'sources<<CACHE_SOURCES'
|
||||
for commit in $(git rev-list --first-parent --max-count=10 HEAD); do
|
||||
echo "type=registry,ref=$CACHE_IMAGE:buildcache-cloud-$commit"
|
||||
done
|
||||
echo "type=registry,ref=$CACHE_IMAGE:buildcache-cloud"
|
||||
echo 'CACHE_SOURCES'
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
||||
with:
|
||||
@@ -161,6 +143,14 @@ jobs:
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
||||
|
||||
# Mixing several historical manifests missed otherwise reusable native
|
||||
# layers on fresh builders. Import the nearest available complete cache.
|
||||
- name: Select cloud cache ancestry
|
||||
id: cloud-cache
|
||||
env:
|
||||
CACHE_IMAGE: ghcr.io/${{ github.repository }}
|
||||
run: node scripts/select-cloud-cache.mjs
|
||||
|
||||
# Deployment tooling reads these labels from the registry to verify an
|
||||
# image's schema expectations against a migrator before deploying it,
|
||||
# without pulling the image. The server refuses to start when the
|
||||
@@ -225,7 +215,7 @@ jobs:
|
||||
# Same-SHA builds serialize above; different SHAs never share a
|
||||
# writable cache ref. Registry layers are content-addressed and
|
||||
# shared even when cache manifests have separate tags.
|
||||
cache-from: ${{ steps.cloud-cache.outputs.sources }}
|
||||
cache-from: ${{ steps.cloud-cache.outputs.source }}
|
||||
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud-${{ github.sha }},mode=max
|
||||
tags: ${{ steps.meta-cloud.outputs.tags }}
|
||||
labels: ${{ steps.meta-cloud.outputs.labels }}
|
||||
|
||||
@@ -32,7 +32,7 @@ jobs:
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
||||
# Compile the real target, then change source in a disposable context.
|
||||
# Require dependency-layer reuse and changed metadata from the real binary.
|
||||
# A fresh builder must import dependencies and produce changed binary metadata.
|
||||
# No registry credentials, external cache, or image publication.
|
||||
- name: Verify native build and dependency cache reuse
|
||||
run: bash scripts/check-docker-runner-cache.sh
|
||||
Reference in new issue
Block a user