mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha42fd790e, total 999 s): - Push 01:13:47. The slowest exact-source job, `General tests (server (7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a runner and ends 01:27:44. The watcher sees it at its next 30 s poll, 01:27:58. - `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both have slack before the gate. Run 43134250912 shows the same shape: the gate opened 01:44:37 and amd64 ended 01:41:53. - `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34. Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image (01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs the same probe again on the same bytes (5 s). The attestation ends 01:30:22. Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43 cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to first published image that contains the commit: p50 29.4 min, p90 55.5 min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The median gap between pushes is 4.5 min. Where the amd64 "Build and push by digest" step spends its 582 s: | Time | Phase | What happens | | --- | --- | --- | | 0–8 s | Load | Dockerfile frontend, cache manifest import | | 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`, `runner-deps`, `runner-build` and `base` are **all cache hits**. The `build` stage starts `FROM runner-build` and copies `deps`, so BuildKit downloads and extracts them: a 932 MB node_modules layer (13 s download, 24.5 s extract) and about 650 MB of Rust `target/` | | 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime bundle 5.6 s | | 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9 s, **Kimi smoke 37 s (amd64 only)**, archive 28 s | | parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB (11 s). `COPY --from=build /app` 24 s | | 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX` plus UID check **71 s** (arm64: 20.5 s) | | 294–420 s | Image export (126 s) | 115 s compressing new layers: `/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB (compressed). Push 10 s | | 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing" (compressing the per-commit layers of `build` and `runner-provider-pack`, which no later commit can reuse), 20 s upload | What rebuilds every commit, and why: only the `build` stage and everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency install already copies manifests first. The commit ARG is used by the UI build, the server stamp, and the provider pack, so those must rebuild. A pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral, and cache mounts are not exported to the registry cache. Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The difference is content, not CPU. amd64 has a larger provider pack with the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a 4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s). **Proposed behavior** Run the PID 1 probe in each build leg, on its own architecture, against the digest that leg pushed. Before signing, require that every Linux image in the standard index is one of those probed digests. Set the provider pack modes in the stage that writes the pack. **Reason and benefit** The probe pull leaves the post-gate path. The leg has about 3 min of slack before the gate. The chmod change removes about 50 s and 187 MB from the amd64 leg, which offsets most of the probe pull that the leg now does. **Breaking changes** None. Tags, digests, attestation subject, and signer identity are unchanged. **Additional context** The concurrency proposal and the build-in-PR plan are below, in "Risks". They are not implemented. ## What Changed - `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned processes". It runs `scripts/assert-orphan-reaping.sh` in `ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform. It runs before "Export digest", so `merge-and-push` only receives probed digests. - `docker.yml` `merge-and-push`: removed the two `docker run` probes (tag and digest). "Resolve standard image digest" now runs `standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests /tmp/digests`. - `standard-image-contract.mjs`: `--verified-digests <dir>` reads each probed leg digest from the registry, checks that its bytes hash to that digest, and collects its Linux image digests. It fails unless every Linux image in the standard index is in that set. Registry content is addressed by digest, so this proves that the signed index names exactly the probed bytes, without a second pull. - `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same `RUN` that writes the pack. `COPY` keeps the modes. The production stage keeps the readability check as UID 65534. - Tests: new contract tests (membership, bare manifest legs, digest mismatch, empty set, lookup by digest). New workflow test: the probe runs between the build and "Export digest", and the publisher does not `docker run`. Updated the provider-pack Dockerfile test. - `doc/standard-image-contract.md`: describes the per-leg probe and the membership check. Expected savings, per master commit: | Change | Saving | Confidence | | --- | --- | --- | | Probe moved out of `merge-and-push` | 95–100 s from gate to attestation (measured: 95 s pull + 5 s second probe) | High | | Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg (71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high on size, medium on seconds | | Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s, minus about 50 s). Still before the gate | Medium | ## Verification - `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2 new). - `node --test` on `.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`, `scripts/__tests__/release-verify-workflow.test.mjs`, `scripts/reuse-canary-image.test.mjs`, `scripts/standard-image-contract.test.mjs`, `scripts/cloud-source-verification.test.mjs`: 484 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new or changed assertions fail. - Real registry, read only: `node scripts/standard-image-contract.mjs --resolve42fd790e80--verified-digests <dir>` with the two leg digests from run 43133035209 returns the published index digest. With only the arm64 leg digest, it fails: "linux/amd64 image sha256:0f0642e6… did not pass the build-leg orphan-reaping probe". - `actionlint .github/workflows/docker.yml`: the same 6 shellcheck findings as `origin/master`, none new. - Shell syntax check (`sh -n`) of the changed pack `RUN`. Not exercised: the workflow itself (it runs only on master pushes, and I did not dispatch it). No local `docker build`: the Docker daemon is not running on this machine, and a full build does not fit a few minutes. The leg probe pull time is an estimate from the publisher runner. ## Risks - The leg now pulls its own image (4.9 GB amd64). AWS legs have about 175 GB free. Hosted tag builds run "Free runner disk" first, and the old publisher pulled the same image on a hosted runner for tag builds. - If the registry returns a leg index in an unexpected form, the resolve step fails closed and no master image is signed. The real-registry check above used today's exact shapes (OCI index with one platform image and one attestation manifest per leg). - Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per leg. They have no standard-image signing step, and the manifest list is made only from probed digests. ### Deliberately not changed - **The exact-source gate.** It is a deliberate "never publish an unverified image" rule (`watch-release-source.mjs --require-proof` and `--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12 min, sets the gate), the 20 s runner wait for `Cloud source verified v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten the poll because the org API rate limit was exhausted today. - **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses about 2 GB of per-commit pack layers every build. A fix: a separate cache-only build of a small aggregate stage (`deps`, `runner-build`, the production tool layer) with `mode=max`, and the main build with `cache-from` only. Not done here: the legs are not on the master critical path today, and I cannot measure it without a master run. It is worth doing for tag builds or if the gate gets faster. - **Cached layer pull (46 s).** The `build` stage starts `FROM runner-build`, so it downloads about 650 MB of Rust `target/` that it deletes at the end. The server build's `prepare:runner-vendor` uses that target, so this needs a separate change. - **Provider pack per commit (98 s amd64).** It rebuilds because it is `FROM build` and stamps the commit. Its 1.06 GB transport archives also duplicate the 1.01 GB pack inside the image. Keying the pack on its own inputs is a product-contract change. ### Concurrency proposal (not implemented) Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of that is waiting for the gate, not building. A newer commit waits for that run before its build starts (run 43134250912 waited 6.4 min). Proposal: 1. Remove the workflow-level group. Add job-level groups: `build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch }}` with `cancel-in-progress: false`. At most one build per architecture runs and the newest commit replaces the pending one. So not every commit is built. 2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with `cancel-in-progress: false`. It skips publication when a newer master commit already has a signed image, so the image order stays monotonic. 3. A superseded run must not keep its source watcher alive, or the API load grows with each push. Fold the early watcher into the gate job, or end it when its build job is cancelled. Effect: the newest commit starts building at push, so its image publishes at about gate time plus 45 s, instead of after the previous run ends. That removes the 0–17 min queue wait, which is most of the gap between the p50 of 29 min and the gate time of about 14 min. ### Plan: build in PR CI and promote on merge (not in this PR) What must change: 1. Turn on the GitHub merge queue for `master`. The queue tests a `gh-readonly-queue/...` commit and then fast-forwards `master` to that exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack `runnerSourceRevision`, the UI bundle) is the final master SHA. 2. On `merge_group`: build both architectures with **no secrets and no registry credentials**, because the code under test is not yet on master. Write each image as an OCI layout artifact. A second job that runs no PR code pushes those artifacts by digest to a staging repository, keyed by the merge-group SHA. 3. Run `cloud-readiness` on `merge_group` for the same SHA, so exact-source proof exists before merge. 4. On the master push, `docker.yml` looks up the staged digests for `github.sha`, checks that the tree and SHA match, retags them as `sha-<full>`, and signs with the same identity (`docker.yml@refs/heads/master`). If no staged image exists, it falls back to the current build. Guarantees to keep: never publish or sign an image without exact-source proof for that SHA. The signer identity and the attested digest stay the same. No secrets in any job that runs non-CODEOWNER code. The PID 1 probe runs on both architectures before signing. Superseded queue entries do not publish. Expected saving: from master push to signed image, about 1–2 min (lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The work moves before the merge, so the merge itself waits for the queue (about the current gate time of 14 min). A staging fix then reaches Cloud about 25 min sooner at the median. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
545 lines
27 KiB
YAML
545 lines
27 KiB
YAML
name: Docker
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- "master"
|
|
tags:
|
|
- "v*"
|
|
- "nightly/v*"
|
|
- "beta/v*"
|
|
# Release workflows push lane tags with GITHUB_TOKEN, and GitHub suppresses
|
|
# push-triggered runs for those, so release.yml dispatches this workflow at
|
|
# the new tag ref instead. The tag mapping below keys off github.ref either
|
|
# way.
|
|
workflow_dispatch:
|
|
|
|
# Least privilege: nothing at the workflow level; each job declares exactly
|
|
# the token scopes it uses (checkout needs contents:read, GHCR pushes need
|
|
# packages:write).
|
|
permissions: {}
|
|
|
|
# Serialise builds per ref without killing an in-flight one: a newer push
|
|
# supersedes only the pending slot, so the image build that is already
|
|
# running always finishes and publishes. Canary TAG refs each get their
|
|
# own group on purpose: their builds run in parallel so every published
|
|
# canary gets its sha images regardless of merge cadence. The mutable
|
|
# `:canary` channel tags are NOT written by the build matrix (which
|
|
# would race across parallel runs) — each canary-tag run retags the
|
|
# channel afterwards, only if it still matches the npm `canary`
|
|
# dist-tag, so the channel moves monotonically and always mirrors npm.
|
|
concurrency:
|
|
group: docker-${{ github.ref }}
|
|
cancel-in-progress: false
|
|
|
|
jobs:
|
|
source_verification:
|
|
name: Watch exact-source verification
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt >= 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 44
|
|
permissions:
|
|
contents: read
|
|
actions: write
|
|
outputs:
|
|
verified: ${{ steps.watch.outputs.verified }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Stop images when source checks fail and report runner waits
|
|
id: watch
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs
|
|
|
|
reuse_canary_image:
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && startsWith(github.ref, 'refs/tags/canary/v')
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
attestations: read
|
|
outputs:
|
|
reused: ${{ steps.reuse.outputs.reused }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
- name: Reuse the verified master image when present
|
|
id: reuse
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: node scripts/reuse-canary-image.mjs
|
|
|
|
# Multi-arch by native runner, not QEMU.
|
|
#
|
|
# This was one job building linux/amd64,linux/arm64 together on an x86
|
|
# runner. The arm64 half is emulated there, and it did not merely run slow:
|
|
# it wedged, every time, in `RUN pnpm --filter @paperclipai/server build`,
|
|
# emitting nothing for 38-45 minutes until `timeout-minutes: 60` killed the
|
|
# job. Verified across three consecutive runs on 2026-09-04; the amd64 half
|
|
# reached `production 5/5` minutes earlier in every one.
|
|
#
|
|
# A timed-out job is reported as *cancelled*, not failed, so the run read
|
|
# "cancelled" and the production image simply stopped publishing without
|
|
# anything going red in an obvious way.
|
|
#
|
|
# It also starved the queue. A run that burns the full hour holds the
|
|
# top-level concurrency slot for that hour, and `cancel-in-progress: false`
|
|
# keeps exactly one pending slot — so with merges arriving faster than one
|
|
# an hour, most runs were superseded before they ever started a job. Five of
|
|
# ten master commits sampled that day never produced an image at all.
|
|
#
|
|
# Each platform now builds on a runner of its own architecture and pushes by
|
|
# digest; `merge` assembles the manifest list. Both architectures remain
|
|
# available to self-hosted installations and downstream image composers.
|
|
build-and-push:
|
|
needs: [reuse_canary_image]
|
|
if: ${{ !cancelled() && (needs.reuse_canary_image.result == 'skipped' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'false')) }}
|
|
strategy:
|
|
# Independent legs: one architecture failing should still publish
|
|
# nothing, but it must not also hide the other's logs behind a
|
|
# cancellation.
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- platform: linux/amd64
|
|
runner: ubuntu-latest
|
|
arch: amd64
|
|
- platform: linux/arm64
|
|
runner: ubuntu-24.04-arm
|
|
arch: arm64
|
|
runs-on: ${{ vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && format('runs-on/fleet=paperclip-core-image-{0}/env=public-ci', matrix.arch == 'amd64' && 'x64' || 'arm64') || matrix.runner }}
|
|
# Stop before the AWS worker's 45-minute maximum lifetime. Hosted tag
|
|
# builds retain their existing limit.
|
|
timeout-minutes: ${{ vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && 40 || 60 }}
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v7
|
|
with:
|
|
# Full history and tags so `git describe` below can compute the
|
|
# release version to stamp into the image.
|
|
fetch-depth: 0
|
|
# Keep the version's full commit graph without downloading blobs
|
|
# from every historical revision of this large repository.
|
|
filter: blob:none
|
|
|
|
# `.git` is dockerignored, so a running image cannot derive its own
|
|
# version and otherwise reports the source package.json placeholder in
|
|
# analytics and the debug panel. Compute it here from the pristine
|
|
# checkout (real CalVer drift from the nearest release tag) and pass it
|
|
# into both builds. Empty when no release tag is reachable — the server
|
|
# then keeps its existing fallbacks.
|
|
- name: Compute build version
|
|
id: build-version
|
|
run: |
|
|
set -euo pipefail
|
|
case "${GITHUB_REF}" in
|
|
refs/tags/nightly/v*)
|
|
# Lane tags carry the exact published version; stamp it verbatim
|
|
# instead of describing drift from the nearest stable tag.
|
|
version="${GITHUB_REF#refs/tags/nightly/v}"
|
|
;;
|
|
refs/tags/beta/v*)
|
|
version="${GITHUB_REF#refs/tags/beta/v}"
|
|
;;
|
|
*)
|
|
version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)"
|
|
;;
|
|
esac
|
|
echo "version=${version}" >> "$GITHUB_OUTPUT"
|
|
echo "Stamping build version: ${version:-<none>}"
|
|
|
|
# ISO week stamp for the Dockerfile's tool layer: the layer caches
|
|
# across commits and re-pulls the @latest CLI tools when the week rolls
|
|
# over, instead of on every build.
|
|
- name: Compute tool cache epoch
|
|
id: tools-epoch
|
|
run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
version: 9.15.4
|
|
run_install: false
|
|
|
|
# No dependency cache here: this workflow publishes release images, and
|
|
# restoring a shared Actions cache into the build inputs would let a
|
|
# poisoned cache entry reach the published artifact.
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v7
|
|
with:
|
|
node-version: 24
|
|
|
|
- name: Refresh lockfile for Docker build context
|
|
run: |
|
|
set -euo pipefail
|
|
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
|
|
|
|
changed="$(git status --porcelain)"
|
|
if [ -z "$changed" ]; then
|
|
echo "Lockfile already matches package metadata."
|
|
exit 0
|
|
fi
|
|
|
|
if printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
|
|
echo "Unexpected files changed during lockfile refresh:"
|
|
echo "$changed"
|
|
exit 1
|
|
fi
|
|
|
|
echo "Using refreshed pnpm-lock.yaml in the Docker build context."
|
|
|
|
- name: Free runner disk
|
|
if: ${{ !(vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master') }}
|
|
run: |
|
|
set -euo pipefail
|
|
echo "Disk before cleanup:"
|
|
df -h
|
|
|
|
pnpm store prune || true
|
|
sudo apt-get clean || true
|
|
sudo rm -rf \
|
|
/usr/share/dotnet \
|
|
/usr/share/swift \
|
|
/usr/local/lib/android \
|
|
/usr/local/share/boost \
|
|
/usr/local/share/powershell \
|
|
/opt/ghc \
|
|
/opt/hostedtoolcache/CodeQL \
|
|
/opt/hostedtoolcache/PyPy \
|
|
/opt/hostedtoolcache/Ruby || true
|
|
docker system prune -af || true
|
|
|
|
echo "Disk after cleanup:"
|
|
df -h
|
|
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# Deployment tooling reads these labels from the registry to verify an
|
|
# image's schema expectations against a migrator before deploying it,
|
|
# without pulling the image. The server refuses to start when the
|
|
# database is missing bundled migrations, so orchestrators need a cheap
|
|
# way to check image/migrator compatibility up front.
|
|
- name: Compute schema migration labels
|
|
id: schema
|
|
run: |
|
|
set -euo pipefail
|
|
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
|
|
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
|
|
echo "last=${last}" >> "$GITHUB_OUTPUT"
|
|
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
|
|
|
# Lane tag mapping: nightly/v* tags publish `:nightly`, and only
|
|
# stable v* tags move `:latest` and the versioned tags.
|
|
# `:sha-<short>` is published on every build. `:canary` is
|
|
# deliberately absent here — the channel tag is moved by the
|
|
# dist-tag-checked retag step below, never by the build matrix,
|
|
# so parallel canary builds cannot race it backwards.
|
|
- name: Docker meta
|
|
id: meta
|
|
uses: docker/metadata-action@v6
|
|
with:
|
|
images: ghcr.io/${{ github.repository }}
|
|
tags: |
|
|
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
|
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
|
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=sha
|
|
type=raw,value=sha-${{ github.sha }},enable=${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
|
labels: |
|
|
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
|
|
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
|
|
- name: Build and push by digest
|
|
id: build
|
|
uses: docker/build-push-action@v7
|
|
with:
|
|
context: .
|
|
# Pin the self-hosted image to the production stage explicitly:
|
|
# the Dockerfile now declares a later `cloud` stage, and without a
|
|
# target the default would silently become that stage.
|
|
target: production
|
|
build-args: |
|
|
PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }}
|
|
PAPERCLIP_BUILD_COMMIT=${{ github.sha }}
|
|
CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }}
|
|
platforms: ${{ matrix.platform }}
|
|
# By digest, not by tag: two runners cannot each push the same tag
|
|
# and end up with a manifest list. Each leg publishes an untagged
|
|
# image and `merge` names them together.
|
|
outputs: type=image,name=ghcr.io/${{ github.repository }},push-by-digest=true,name-canonical=true,push=true
|
|
# Registry-backed BuildKit cache instead of type=gha: the Actions
|
|
# cache is capped at 10GB per repo, and two multi-arch mode=max jobs
|
|
# evict each other, so most builds ran effectively cold. The cache
|
|
# ref lives in ghcr next to the image and is written only by this
|
|
# workflow (docker.yml runs on master/tag pushes, never on PRs).
|
|
#
|
|
# Per-arch refs now the legs are separate runners: a shared ref would
|
|
# have each leg overwrite the other's cache on every build.
|
|
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.arch }}
|
|
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.arch }},mode=max
|
|
labels: ${{ steps.meta.outputs.labels }}
|
|
|
|
# PID 1 must be an init that reaps adopted orphans. With node there, the
|
|
# orphans agent runs leave behind are never wait()ed and pin as zombies
|
|
# until the cgroup pid limit is exhausted and every fork() in the
|
|
# container fails.
|
|
#
|
|
# Probed here, on this leg's own architecture, against the exact digest
|
|
# just pushed. On master this runs while the exact-source gate is still
|
|
# pending, so the image pull (~95 s for amd64) is off the publication
|
|
# path, and arm64 is exercised too (the publisher runner is x86_64 only).
|
|
# Only a probed digest is exported, and merge-and-push signs an index
|
|
# only when its every Linux image is one of these probed digests.
|
|
- name: Verify PID 1 reaps orphaned processes
|
|
env:
|
|
# Through the environment, not interpolated into the script body.
|
|
IMAGE_DIGEST: ${{ steps.build.outputs.digest }}
|
|
IMAGE_NAME: ghcr.io/${{ github.repository }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
run: |
|
|
set -euo pipefail
|
|
test -n "$IMAGE_DIGEST"
|
|
echo "Verifying orphan reaping in $IMAGE_NAME@$IMAGE_DIGEST ($PLATFORM)"
|
|
docker run --rm -i --pull always --platform "$PLATFORM" "$IMAGE_NAME@$IMAGE_DIGEST" sh -s < scripts/assert-orphan-reaping.sh
|
|
|
|
# The digest is the only thing `merge` needs from this job. Carried as an
|
|
# empty file named for it, which is the upstream pattern — the name is
|
|
# the payload, so several legs can upload without colliding on content.
|
|
- name: Export digest
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p /tmp/digests
|
|
digest="${{ steps.build.outputs.digest }}"
|
|
test -n "$digest"
|
|
touch "/tmp/digests/${digest#sha256:}"
|
|
|
|
- name: Upload digest
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: digests-production-${{ matrix.arch }}
|
|
path: /tmp/digests/*
|
|
if-no-files-found: error
|
|
retention-days: 1
|
|
|
|
# Wait for proof on ephemeral verification capacity. Publication runners do
|
|
# only registry/signing work. Each Spot observer stays below its 45m lifetime.
|
|
publication_verification:
|
|
name: Require exact-source verification
|
|
if: ${{ !cancelled() && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && needs.build-and-push.result == 'success' && needs.source_verification.result == 'success' && needs.source_verification.outputs.verified != 'true' }}
|
|
needs: [build-and-push, source_verification]
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt >= 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 44
|
|
permissions:
|
|
contents: read
|
|
actions: write # restart only exact-source producers deferred before admission
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Require source proof and migrator admission before publication
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs --require-proof
|
|
|
|
# Names the per-architecture digests as one manifest list under the real
|
|
# tags. Nothing is publicly tagged until this runs, so a half-published
|
|
# multi-arch image is not a state anything can pull.
|
|
merge-and-push:
|
|
if: ${{ !cancelled() && needs.build-and-push.result == 'success' && (needs.source_verification.result == 'success' || needs.source_verification.result == 'skipped') && (needs.publication_verification.result == 'success' || (needs.publication_verification.result == 'skipped' && (github.repository != 'paperclipai/paperclip' || github.repository_id != '1170821064' || github.ref != 'refs/heads/master' || needs.source_verification.outputs.verified == 'true'))) }}
|
|
needs: [build-and-push, source_verification, publication_verification]
|
|
runs-on: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (vars.CORE_PUBLICATION_RUNNER || 'cloud-release-32') || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
actions: read
|
|
packages: write
|
|
id-token: write
|
|
attestations: write
|
|
steps:
|
|
# Checked out for `packages/db` (schema labels) and the source-proof and
|
|
# standard-image contract scripts.
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
|
|
- name: Download digests
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
path: /tmp/digests
|
|
pattern: digests-production-*
|
|
merge-multiple: true
|
|
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
|
|
# Repeated from the build job rather than passed between them: job
|
|
# outputs would have to survive a matrix, and this is two `ls` calls.
|
|
- name: Compute schema migration labels
|
|
id: schema
|
|
run: |
|
|
set -euo pipefail
|
|
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
|
|
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
|
|
echo "last=${last}" >> "$GITHUB_OUTPUT"
|
|
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
|
|
|
# Same lane mapping as the build job; this is where it is actually
|
|
# applied, since the legs push untagged. `:canary` is deliberately
|
|
# absent, exactly as in the build job's mapping: the channel tag is
|
|
# moved only by the dist-tag-checked promote_canary_channel job below,
|
|
# so parallel canary-tag builds can never race the channel backwards.
|
|
- name: Docker meta
|
|
id: meta
|
|
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
|
|
with:
|
|
images: ghcr.io/${{ github.repository }}
|
|
tags: |
|
|
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
|
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
|
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=sha
|
|
type=raw,value=sha-${{ github.sha }},enable=${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
|
labels: |
|
|
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
|
|
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
|
|
# Revalidate the latest attempt after runner assignment without waiting
|
|
# here. A concurrent source retry must invalidate the earlier proof.
|
|
- name: Revalidate exact-source proof before publication
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs --check-proof
|
|
|
|
- name: Create manifest list and push
|
|
working-directory: /tmp/digests
|
|
run: |
|
|
set -euo pipefail
|
|
docker buildx imagetools create \
|
|
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
$(printf 'ghcr.io/${{ github.repository }}@sha256:%s ' *)
|
|
|
|
# The orphan-reaping probe ran in each build leg against the digest that
|
|
# leg pushed; /tmp/digests holds only probed digests. Pulling the image
|
|
# again here cost ~95 s after the source gate, on the publication path.
|
|
|
|
# This generic contract is useful to any downstream image composer. Only
|
|
# canonical master pushes own the full-SHA tag and its signed provenance.
|
|
# Capture exact registry bytes and sign that digest, so later tag movement
|
|
# cannot change what was verified. Registry content is addressed by
|
|
# digest: the resolve step fails unless every Linux image in the index is
|
|
# one the build legs probed.
|
|
- name: Resolve standard image digest
|
|
id: standard
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'
|
|
run: node scripts/standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests /tmp/digests
|
|
- name: Attest the standard production image
|
|
if: steps.standard.outputs.digest != ''
|
|
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4
|
|
with:
|
|
subject-name: ghcr.io/paperclipai/paperclip
|
|
subject-digest: ${{ steps.standard.outputs.digest }}
|
|
push-to-registry: true
|
|
|
|
# Moves the mutable `:canary` channel tag. Kept OUT
|
|
# of the build jobs and serialized in its own lane, and — the load-
|
|
# bearing property — CONVERGENT rather than self-interested: a
|
|
# promotion does not promote "its own" canary, it retags the channel
|
|
# to whatever the npm `canary` dist-tag names at execution time,
|
|
# provided that version's sha images are published. GitHub's shared
|
|
# concurrency lane keeps one running and one pending promotion and
|
|
# REPLACES the pending slot with the latest enqueued — an older build
|
|
# finishing late can therefore evict the newest canary's pending
|
|
# promotion. With convergent promotion that eviction is harmless:
|
|
# whichever promotion survives resolves the current dist-tag fresh
|
|
# and lands the channel there (the current canary's images always
|
|
# exist by the time any later promotion runs, because per-tag build
|
|
# groups mean canary builds are never superseded and each run's
|
|
# promotion is gated on its own completed pushes). Every interleaving
|
|
# converges the Docker channel onto the npm channel.
|
|
promote_canary_channel:
|
|
if: ${{ !cancelled() && startsWith(github.ref, 'refs/tags/canary/v') && (needs.merge-and-push.result == 'success' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'true')) }}
|
|
needs: [reuse_canary_image, merge-and-push]
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
concurrency:
|
|
group: docker-canary-channel-promotion
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Converge the channel tags onto the current npm canary
|
|
env:
|
|
IMAGE: ghcr.io/${{ github.repository }}
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: |
|
|
current="$(curl -fsS "https://registry.npmjs.org/-/package/@paperclipai%2Fdb/dist-tags" | jq -er .canary)"
|
|
sha="$(gh api "repos/${GITHUB_REPOSITORY}/commits/$(printf 'canary/v%s' "$current" | jq -sRr @uri)" --jq .sha 2>/dev/null || true)"
|
|
if [ -z "$sha" ]; then
|
|
echo "canary/v${current} does not resolve yet; a later promotion converges the channel"
|
|
exit 0
|
|
fi
|
|
short="$(printf '%s' "$sha" | cut -c1-7)"
|
|
if ! docker buildx imagetools inspect "$IMAGE:sha-${short}" >/dev/null 2>&1; then
|
|
echo "images for ${current} (sha-${short}) not published yet; its own promotion converges the channel"
|
|
exit 0
|
|
fi
|
|
docker buildx imagetools create -t "$IMAGE:canary" "$IMAGE:sha-${short}"
|
|
echo "channel tags moved to canary ${current} (sha-${short})"
|