mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
677 lines
33 KiB
YAML
677 lines
33 KiB
YAML
name: Docker
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- "master"
|
|
tags:
|
|
- "v*"
|
|
- "nightly/v*"
|
|
- "beta/v*"
|
|
# Release workflows push lane tags with GITHUB_TOKEN, and GitHub suppresses
|
|
# push-triggered runs for those, so release.yml dispatches this workflow at
|
|
# the new tag ref instead. The tag mapping below keys off github.ref either
|
|
# way.
|
|
workflow_dispatch:
|
|
|
|
# Least privilege: nothing at the workflow level; each job declares exactly
|
|
# the token scopes it uses (checkout needs contents:read, GHCR pushes need
|
|
# packages:write).
|
|
permissions: {}
|
|
|
|
# Serialise builds per ref without killing an in-flight one: a newer push
|
|
# supersedes only the pending slot, so the image build that is already
|
|
# running always finishes and publishes. Canary TAG refs each get their
|
|
# own group on purpose: their builds run in parallel so every published
|
|
# canary gets its sha images regardless of merge cadence. The mutable
|
|
# `:canary` channel tags are NOT written by the build matrix (which
|
|
# would race across parallel runs) — each canary-tag run retags the
|
|
# channel afterwards, only if it still matches the npm `canary`
|
|
# dist-tag, so the channel moves monotonically and always mirrors npm.
|
|
concurrency:
|
|
group: docker-${{ github.ref }}
|
|
cancel-in-progress: false
|
|
|
|
jobs:
|
|
runner_image_report:
|
|
name: Report the Runner image of this commit
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'
|
|
continue-on-error: true
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 8
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
run_install: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
- name: Compute the Runner inputs of this commit
|
|
timeout-minutes: 5
|
|
run: node scripts/runner-image-resolve.mjs inputs --dir "$RUNNER_TEMP/runner-image-resolve"
|
|
|
|
- name: Report the Runner an image build would use
|
|
timeout-minutes: 2
|
|
run: |
|
|
set -euo pipefail
|
|
node scripts/runner-image-resolve.mjs resolve --dir "$RUNNER_TEMP/runner-image-resolve"
|
|
echo "Report only: no build read this result." >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
source_verification:
|
|
name: Watch exact-source verification
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt >= 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 44
|
|
permissions:
|
|
contents: read
|
|
actions: write
|
|
checks: read # runner-loss evidence (annotations) for readiness retries
|
|
outputs:
|
|
verified: ${{ steps.watch.outputs.verified }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Stop images when source checks fail and report runner waits
|
|
id: watch
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs
|
|
|
|
reuse_canary_image:
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && startsWith(github.ref, 'refs/tags/canary/v')
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
attestations: read
|
|
outputs:
|
|
reused: ${{ steps.reuse.outputs.reused }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
- name: Reuse the verified master image when present
|
|
id: reuse
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: node scripts/reuse-canary-image.mjs
|
|
|
|
# Multi-arch by native runner, not QEMU.
|
|
#
|
|
# This was one job building linux/amd64,linux/arm64 together on an x86
|
|
# runner. The arm64 half is emulated there, and it did not merely run slow:
|
|
# it wedged, every time, in `RUN pnpm --filter @paperclipai/server build`,
|
|
# emitting nothing for 38-45 minutes until `timeout-minutes: 60` killed the
|
|
# job. Verified across three consecutive runs on 2026-09-04; the amd64 half
|
|
# reached `production 5/5` minutes earlier in every one.
|
|
#
|
|
# A timed-out job is reported as *cancelled*, not failed, so the run read
|
|
# "cancelled" and the production image simply stopped publishing without
|
|
# anything going red in an obvious way.
|
|
#
|
|
# It also starved the queue. A run that burns the full hour holds the
|
|
# top-level concurrency slot for that hour, and `cancel-in-progress: false`
|
|
# keeps exactly one pending slot — so with merges arriving faster than one
|
|
# an hour, most runs were superseded before they ever started a job. Five of
|
|
# ten master commits sampled that day never produced an image at all.
|
|
#
|
|
# Each platform now builds on a runner of its own architecture and pushes by
|
|
# digest; `merge` assembles the manifest list. Both architectures remain
|
|
# available to self-hosted installations and downstream image composers.
|
|
build-and-push:
|
|
needs: [reuse_canary_image]
|
|
if: ${{ !cancelled() && (needs.reuse_canary_image.result == 'skipped' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'false')) }}
|
|
strategy:
|
|
# Independent legs: one architecture failing should still publish
|
|
# nothing, but it must not also hide the other's logs behind a
|
|
# cancellation.
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- platform: linux/amd64
|
|
runner: ubuntu-latest
|
|
arch: amd64
|
|
- platform: linux/arm64
|
|
runner: ubuntu-24.04-arm
|
|
arch: arm64
|
|
runs-on: ${{ vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && format('runs-on/fleet=paperclip-core-image-{0}/env=public-ci', matrix.arch == 'amd64' && 'x64' || 'arm64') || matrix.runner }}
|
|
# Stop before the AWS worker's 45-minute maximum lifetime. Hosted tag
|
|
# builds retain their existing limit.
|
|
timeout-minutes: ${{ vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && 40 || 60 }}
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
actions: read # download a verified merge-queue image artifact of this commit
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v7
|
|
with:
|
|
# Full history and tags so `git describe` below can compute the
|
|
# release version to stamp into the image.
|
|
fetch-depth: 0
|
|
# Keep the version's full commit graph without downloading blobs
|
|
# from every historical revision of this large repository.
|
|
filter: blob:none
|
|
|
|
# `.git` is dockerignored, so a running image cannot derive its own
|
|
# version and otherwise reports the source package.json placeholder in
|
|
# analytics and the debug panel. Compute it here from the pristine
|
|
# checkout (real CalVer drift from the nearest release tag) and pass it
|
|
# into both builds. Empty when no release tag is reachable — the server
|
|
# then keeps its existing fallbacks.
|
|
- name: Compute build version
|
|
id: build-version
|
|
run: |
|
|
set -euo pipefail
|
|
case "${GITHUB_REF}" in
|
|
refs/tags/nightly/v*)
|
|
# Lane tags carry the exact published version; stamp it verbatim
|
|
# instead of describing drift from the nearest stable tag.
|
|
version="${GITHUB_REF#refs/tags/nightly/v}"
|
|
;;
|
|
refs/tags/beta/v*)
|
|
version="${GITHUB_REF#refs/tags/beta/v}"
|
|
;;
|
|
*)
|
|
version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)"
|
|
;;
|
|
esac
|
|
echo "version=${version}" >> "$GITHUB_OUTPUT"
|
|
echo "Stamping build version: ${version:-<none>}"
|
|
|
|
# ISO week stamp for the Dockerfile's tool layer: the layer caches
|
|
# across commits and re-pulls the @latest CLI tools when the week rolls
|
|
# over, instead of on every build.
|
|
- name: Compute tool cache epoch
|
|
id: tools-epoch
|
|
run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
version: 9.15.4
|
|
run_install: false
|
|
|
|
# No dependency cache here: this workflow publishes release images, and
|
|
# restoring a shared Actions cache into the build inputs would let a
|
|
# poisoned cache entry reach the published artifact.
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v7
|
|
with:
|
|
node-version: 24
|
|
|
|
- name: Refresh lockfile for Docker build context
|
|
run: |
|
|
set -euo pipefail
|
|
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
|
|
|
|
changed="$(git status --porcelain)"
|
|
if [ -z "$changed" ]; then
|
|
echo "Lockfile already matches package metadata."
|
|
exit 0
|
|
fi
|
|
|
|
if printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
|
|
echo "Unexpected files changed during lockfile refresh:"
|
|
echo "$changed"
|
|
exit 1
|
|
fi
|
|
|
|
echo "Using refreshed pnpm-lock.yaml in the Docker build context."
|
|
|
|
- name: Free runner disk
|
|
if: ${{ !(vars.CORE_RELEASE_AWS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master') }}
|
|
run: |
|
|
set -euo pipefail
|
|
echo "Disk before cleanup:"
|
|
df -h
|
|
|
|
pnpm store prune || true
|
|
sudo apt-get clean || true
|
|
sudo rm -rf \
|
|
/usr/share/dotnet \
|
|
/usr/share/swift \
|
|
/usr/local/lib/android \
|
|
/usr/local/share/boost \
|
|
/usr/local/share/powershell \
|
|
/opt/ghc \
|
|
/opt/hostedtoolcache/CodeQL \
|
|
/opt/hostedtoolcache/PyPy \
|
|
/opt/hostedtoolcache/Ruby || true
|
|
docker system prune -af || true
|
|
|
|
echo "Disk after cleanup:"
|
|
df -h
|
|
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
# Build-before-merge. When the merge queue already built and probed this
|
|
# exact commit (docker-merge-group.yml, with no registry credentials),
|
|
# push those bytes instead of rebuilding. scripts/merge-group-image.mjs
|
|
# requires a successful merge-group run of this repository for this SHA
|
|
# and checks the record's commit, tree, architecture and archive digest.
|
|
# Any missing or mismatched proof, or any failure here, leaves
|
|
# `steps.promote.outputs.digest` empty and the build below runs as
|
|
# before. The probe below and the exact-source gate apply either way.
|
|
- name: Find the merge-queue build of this commit
|
|
id: find
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'
|
|
continue-on-error: true
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
BEFORE: ${{ github.event.before }}
|
|
ARCH: ${{ matrix.arch }}
|
|
# Master's copy from before this push selects the run; see the promote
|
|
# step below.
|
|
run: |
|
|
set -euo pipefail
|
|
trusted="$RUNNER_TEMP/merge-group-image-trusted.mjs"
|
|
git show "$BEFORE:scripts/merge-group-image.mjs" > "$trusted"
|
|
node "$trusted" find --arch "$ARCH"
|
|
|
|
- name: Download the merge-queue image
|
|
if: steps.find.outputs.run_id != ''
|
|
continue-on-error: true
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
name: merge-group-image-${{ matrix.arch }}
|
|
run-id: ${{ steps.find.outputs.run_id }}
|
|
github-token: ${{ github.token }}
|
|
path: ${{ runner.temp }}/merge-group-image
|
|
|
|
- name: Promote the merge-queue image
|
|
id: promote
|
|
if: steps.find.outputs.run_id != ''
|
|
continue-on-error: true
|
|
env:
|
|
ARCH: ${{ matrix.arch }}
|
|
DIRECTORY: ${{ runner.temp }}/merge-group-image
|
|
IMAGE_NAME: ghcr.io/${{ github.repository }}
|
|
BEFORE: ${{ github.event.before }}
|
|
run: |
|
|
set -euo pipefail
|
|
# Say plainly when this runner image cannot promote, instead of
|
|
# failing later inside the copy.
|
|
if ! command -v skopeo >/dev/null; then
|
|
echo "::notice title=Merge-queue promotion unavailable::skopeo is not installed on this runner; building from source."
|
|
exit 1
|
|
fi
|
|
echo "Promoting with $(skopeo --version)"
|
|
# Run the verifier as master had it before this push, not the copy
|
|
# this push brings, so a push cannot weaken the check it must pass.
|
|
# The verifier refuses any push that changes the pipeline since
|
|
# BEFORE. A push whose BEFORE has no verifier builds from source.
|
|
trusted="$RUNNER_TEMP/merge-group-image-trusted.mjs"
|
|
git show "$BEFORE:scripts/merge-group-image.mjs" > "$trusted"
|
|
# Verify without writing a step output: the digest is set only after
|
|
# the exact bytes are in the registry.
|
|
verified="$(GITHUB_OUTPUT='' node "$trusted" verify --arch "$ARCH" --dir "$DIRECTORY" --since "$BEFORE")"
|
|
digest="$(printf '%s\n' "$verified" | sed -n 's/^digest=//p')"
|
|
printf '%s' "$digest" | grep -Eq '^sha256:[a-f0-9]{64}$'
|
|
# --preserve-digests fails rather than re-encode, so the registry
|
|
# holds exactly the archived image under the recorded digest.
|
|
skopeo copy --all --preserve-digests "oci-archive:$DIRECTORY/image.tar" "docker://$IMAGE_NAME@$digest"
|
|
echo "Promoted the merge-queue image $IMAGE_NAME@$digest"
|
|
echo "digest=$digest" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Set up Docker Buildx
|
|
if: steps.promote.outputs.digest == ''
|
|
uses: docker/setup-buildx-action@v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# Deployment tooling reads these labels from the registry to verify an
|
|
# image's schema expectations against a migrator before deploying it,
|
|
# without pulling the image. The server refuses to start when the
|
|
# database is missing bundled migrations, so orchestrators need a cheap
|
|
# way to check image/migrator compatibility up front.
|
|
- name: Compute schema migration labels
|
|
id: schema
|
|
if: steps.promote.outputs.digest == ''
|
|
run: |
|
|
set -euo pipefail
|
|
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
|
|
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
|
|
echo "last=${last}" >> "$GITHUB_OUTPUT"
|
|
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
|
|
|
# Lane tag mapping: nightly/v* tags publish `:nightly`, and only
|
|
# stable v* tags move `:latest` and the versioned tags.
|
|
# `:sha-<short>` is published on every build. `:canary` is
|
|
# deliberately absent here — the channel tag is moved by the
|
|
# dist-tag-checked retag step below, never by the build matrix,
|
|
# so parallel canary builds cannot race it backwards.
|
|
- name: Docker meta
|
|
id: meta
|
|
if: steps.promote.outputs.digest == ''
|
|
uses: docker/metadata-action@v6
|
|
with:
|
|
images: ghcr.io/${{ github.repository }}
|
|
tags: |
|
|
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
|
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
|
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=sha
|
|
type=raw,value=sha-${{ github.sha }},enable=${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
|
labels: |
|
|
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
|
|
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
|
|
- name: Build and push by digest
|
|
id: build
|
|
if: steps.promote.outputs.digest == ''
|
|
uses: docker/build-push-action@v7
|
|
with:
|
|
context: .
|
|
# Pin the self-hosted image to the production stage explicitly:
|
|
# the Dockerfile now declares a later `cloud` stage, and without a
|
|
# target the default would silently become that stage.
|
|
target: production
|
|
build-args: |
|
|
PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }}
|
|
PAPERCLIP_BUILD_COMMIT=${{ github.sha }}
|
|
CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }}
|
|
platforms: ${{ matrix.platform }}
|
|
# By digest, not by tag: two runners cannot each push the same tag
|
|
# and end up with a manifest list. Each leg publishes an untagged
|
|
# image and `merge` names them together.
|
|
outputs: type=image,name=ghcr.io/${{ github.repository }},push-by-digest=true,name-canonical=true,push=true
|
|
# Registry-backed BuildKit cache instead of type=gha: the Actions
|
|
# cache is capped at 10GB per repo, and two multi-arch mode=max jobs
|
|
# evict each other, so most builds ran effectively cold. The cache
|
|
# ref lives in ghcr next to the image and is written only by this
|
|
# workflow (docker.yml runs on master/tag pushes, never on PRs).
|
|
#
|
|
# Per-arch refs now the legs are separate runners: a shared ref would
|
|
# have each leg overwrite the other's cache on every build.
|
|
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.arch }}
|
|
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.arch }},mode=max
|
|
labels: ${{ steps.meta.outputs.labels }}
|
|
|
|
# Exactly one of the two paths produced this leg's digest.
|
|
- name: Select image digest
|
|
id: image
|
|
env:
|
|
PROMOTED: ${{ steps.promote.outputs.digest }}
|
|
BUILT: ${{ steps.build.outputs.digest }}
|
|
run: |
|
|
set -euo pipefail
|
|
digest="${PROMOTED:-$BUILT}"
|
|
test -n "$digest"
|
|
echo "digest=$digest" >> "$GITHUB_OUTPUT"
|
|
|
|
# PID 1 must be an init that reaps adopted orphans. With node there, the
|
|
# orphans agent runs leave behind are never wait()ed and pin as zombies
|
|
# until the cgroup pid limit is exhausted and every fork() in the
|
|
# container fails.
|
|
#
|
|
# Probed here, on this leg's own architecture, against the exact digest
|
|
# just pushed. On master this runs while the exact-source gate is still
|
|
# pending, so the image pull (~95 s for amd64) is off the publication
|
|
# path, and arm64 is exercised too (the publisher runner is x86_64 only).
|
|
# Only a probed digest is exported, and merge-and-push signs an index
|
|
# only when its every Linux image is one of these probed digests.
|
|
- name: Verify PID 1 reaps orphaned processes
|
|
env:
|
|
# Through the environment, not interpolated into the script body.
|
|
IMAGE_DIGEST: ${{ steps.image.outputs.digest }}
|
|
IMAGE_NAME: ghcr.io/${{ github.repository }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
run: |
|
|
set -euo pipefail
|
|
test -n "$IMAGE_DIGEST"
|
|
echo "Verifying orphan reaping in $IMAGE_NAME@$IMAGE_DIGEST ($PLATFORM)"
|
|
docker run --rm -i --pull always --platform "$PLATFORM" "$IMAGE_NAME@$IMAGE_DIGEST" sh -s < scripts/assert-orphan-reaping.sh
|
|
|
|
# The digest is the only thing `merge` needs from this job. Carried as an
|
|
# empty file named for it, which is the upstream pattern — the name is
|
|
# the payload, so several legs can upload without colliding on content.
|
|
- name: Export digest
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p /tmp/digests
|
|
digest="${{ steps.image.outputs.digest }}"
|
|
test -n "$digest"
|
|
touch "/tmp/digests/${digest#sha256:}"
|
|
|
|
- name: Upload digest
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: digests-production-${{ matrix.arch }}
|
|
path: /tmp/digests/*
|
|
if-no-files-found: error
|
|
retention-days: 1
|
|
|
|
# Wait for proof on ephemeral verification capacity. Publication runners do
|
|
# only registry/signing work. Each Spot observer stays below its 45m lifetime.
|
|
publication_verification:
|
|
name: Require exact-source verification
|
|
if: ${{ !cancelled() && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && needs.build-and-push.result == 'success' && needs.source_verification.result == 'success' && needs.source_verification.outputs.verified != 'true' }}
|
|
needs: [build-and-push, source_verification]
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.run_attempt >= 1 && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 44
|
|
permissions:
|
|
contents: read
|
|
actions: write # restart only exact-source producers deferred before admission
|
|
checks: read # runner-loss evidence (annotations) for readiness retries
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Require source proof and migrator admission before publication
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs --require-proof
|
|
|
|
# Names the per-architecture digests as one manifest list under the real
|
|
# tags. Nothing is publicly tagged until this runs, so a half-published
|
|
# multi-arch image is not a state anything can pull.
|
|
merge-and-push:
|
|
if: ${{ !cancelled() && needs.build-and-push.result == 'success' && (needs.source_verification.result == 'success' || needs.source_verification.result == 'skipped') && (needs.publication_verification.result == 'success' || (needs.publication_verification.result == 'skipped' && (github.repository != 'paperclipai/paperclip' || github.repository_id != '1170821064' || github.ref != 'refs/heads/master' || needs.source_verification.outputs.verified == 'true'))) }}
|
|
needs: [build-and-push, source_verification, publication_verification]
|
|
runs-on: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (vars.CORE_PUBLICATION_RUNNER || 'cloud-release-32') || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
actions: read
|
|
packages: write
|
|
id-token: write
|
|
attestations: write
|
|
steps:
|
|
# Checked out for `packages/db` (schema labels) and the source-proof and
|
|
# standard-image contract scripts.
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
|
|
- name: Download digests
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
path: /tmp/digests
|
|
pattern: digests-production-*
|
|
merge-multiple: true
|
|
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
|
|
# Repeated from the build job rather than passed between them: job
|
|
# outputs would have to survive a matrix, and this is two `ls` calls.
|
|
- name: Compute schema migration labels
|
|
id: schema
|
|
run: |
|
|
set -euo pipefail
|
|
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
|
|
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
|
|
echo "last=${last}" >> "$GITHUB_OUTPUT"
|
|
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
|
|
|
# Same lane mapping as the build job; this is where it is actually
|
|
# applied, since the legs push untagged. `:canary` is deliberately
|
|
# absent, exactly as in the build job's mapping: the channel tag is
|
|
# moved only by the dist-tag-checked promote_canary_channel job below,
|
|
# so parallel canary-tag builds can never race the channel backwards.
|
|
- name: Docker meta
|
|
id: meta
|
|
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
|
|
with:
|
|
images: ghcr.io/${{ github.repository }}
|
|
tags: |
|
|
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
|
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
|
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
|
type=sha
|
|
type=raw,value=sha-${{ github.sha }},enable=${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
|
labels: |
|
|
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
|
|
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
|
|
# Revalidate the latest attempt after runner assignment without waiting
|
|
# here. A concurrent source retry must invalidate the earlier proof.
|
|
- name: Revalidate exact-source proof before publication
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: node scripts/watch-release-source.mjs --check-proof
|
|
|
|
- name: Create manifest list and push
|
|
working-directory: /tmp/digests
|
|
run: |
|
|
set -euo pipefail
|
|
docker buildx imagetools create \
|
|
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
$(printf 'ghcr.io/${{ github.repository }}@sha256:%s ' *)
|
|
|
|
# The orphan-reaping probe ran in each build leg against the digest that
|
|
# leg pushed; /tmp/digests holds only probed digests. Pulling the image
|
|
# again here cost ~95 s after the source gate, on the publication path.
|
|
|
|
# This generic contract is useful to any downstream image composer. Only
|
|
# canonical master pushes own the full-SHA tag and its signed provenance.
|
|
# Capture exact registry bytes and sign that digest, so later tag movement
|
|
# cannot change what was verified. Registry content is addressed by
|
|
# digest: the resolve step fails unless every Linux image in the index is
|
|
# one the build legs probed.
|
|
- name: Resolve standard image digest
|
|
id: standard
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'
|
|
run: node scripts/standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests /tmp/digests
|
|
- name: Attest the standard production image
|
|
if: steps.standard.outputs.digest != ''
|
|
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4
|
|
with:
|
|
subject-name: ghcr.io/paperclipai/paperclip
|
|
subject-digest: ${{ steps.standard.outputs.digest }}
|
|
push-to-registry: true
|
|
|
|
# Moves the mutable `:canary` channel tag. Kept OUT
|
|
# of the build jobs and serialized in its own lane, and — the load-
|
|
# bearing property — CONVERGENT rather than self-interested: a
|
|
# promotion does not promote "its own" canary, it retags the channel
|
|
# to whatever the npm `canary` dist-tag names at execution time,
|
|
# provided that version's sha images are published. GitHub's shared
|
|
# concurrency lane keeps one running and one pending promotion and
|
|
# REPLACES the pending slot with the latest enqueued — an older build
|
|
# finishing late can therefore evict the newest canary's pending
|
|
# promotion. With convergent promotion that eviction is harmless:
|
|
# whichever promotion survives resolves the current dist-tag fresh
|
|
# and lands the channel there (the current canary's images always
|
|
# exist by the time any later promotion runs, because per-tag build
|
|
# groups mean canary builds are never superseded and each run's
|
|
# promotion is gated on its own completed pushes). Every interleaving
|
|
# converges the Docker channel onto the npm channel.
|
|
promote_canary_channel:
|
|
if: ${{ !cancelled() && startsWith(github.ref, 'refs/tags/canary/v') && (needs.merge-and-push.result == 'success' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'true')) }}
|
|
needs: [reuse_canary_image, merge-and-push]
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
concurrency:
|
|
group: docker-canary-channel-promotion
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Login to GitHub Container Registry
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.repository_owner }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Converge the channel tags onto the current npm canary
|
|
env:
|
|
IMAGE: ghcr.io/${{ github.repository }}
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: |
|
|
current="$(curl -fsS "https://registry.npmjs.org/-/package/@paperclipai%2Fdb/dist-tags" | jq -er .canary)"
|
|
sha="$(gh api "repos/${GITHUB_REPOSITORY}/commits/$(printf 'canary/v%s' "$current" | jq -sRr @uri)" --jq .sha 2>/dev/null || true)"
|
|
if [ -z "$sha" ]; then
|
|
echo "canary/v${current} does not resolve yet; a later promotion converges the channel"
|
|
exit 0
|
|
fi
|
|
short="$(printf '%s' "$sha" | cut -c1-7)"
|
|
if ! docker buildx imagetools inspect "$IMAGE:sha-${short}" >/dev/null 2>&1; then
|
|
echo "images for ${current} (sha-${short}) not published yet; its own promotion converges the channel"
|
|
exit 0
|
|
fi
|
|
docker buildx imagetools create -t "$IMAGE:canary" "$IMAGE:sha-${short}"
|
|
echo "channel tags moved to canary ${current} (sha-${short})"
|