mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Parts 2 and 3 of this stack build the Runner alone as an image with two variants, and check it on pull requests. Nothing publishes it > - Downstream sandbox and VM images can copy the Runner only from a registry, and they must be able to verify where it came from > - A published image must be exactly the bytes that were checked, and its name must never point at something else later > - This pull request publishes both variants from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set of inputs, and attests them > - Nothing consumes the image in this pull request. The application image build does not change > - The benefit is one verified, attested Runner that later changes can copy by digest ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image build). No issue exists for this change. **Problem or motivation** The Runner image exists only inside a pull-request check. A consumer cannot pull it. There is no rule for which commit publishes an image, no protection for its tags, and no way to verify a copied Runner. **Proposed solution** Publish from a `master` push in this repository, and from nothing else. Build only when no image has the digest of the commit's inputs. Let the build legs hold no write access. Push the bytes that the legs checked by digest, attest each multi-platform index, and write the tags last. Write the full tag after the slim tag, so that the full tag means "both exist". **Alternatives considered** - Publish from `docker.yml`. That workflow holds one concurrency slot for each ref, and the Runner image needs its own signer identity, so that its attestation cannot stand in for the application image's. - Resolve the lockfile in the workflow before a `master` build, as `docker.yml` does. Then the published Runner depends on the state of the npm registry at run time, and the digest cannot be computed again from a commit. - Hand each variant to the publish job as its own OCI archive. The two archives hold the two shared layers two times: 1.27 GB for each architecture in place of 0.85 GB. - Write the full tag first. Then a consumer can see a full image whose slim image does not exist yet. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 4 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant 4. #21230: publish both variants from `master` (this pull request) ## What Changed - `.github/workflows/runner-image.yml`: adds the trigger `push` to `master` and the job `publish`. - `identity` checks the committed lockfile against the Runner's manifests, computes the digest in the build context, and asks the registry whether these inputs are published. - `build` is the job of part 3. It also records what it built, on every event. On a canonical `master` push it uploads that. It cannot write to the registry on any event. The only token in the job can read packages, and only the lookup step receives it. - `publish` runs only for a `master` push in this repository. It builds nothing and runs no dependency code. - Lockfile rule. A `master` push builds from the committed `pnpm-lock.yaml` and resolves nothing, so the digest is a function of the commit. When the committed lockfile is behind the manifests that the Runner build reads, the run writes a notice, publishes nothing, and stays green. The commit that merges the lockfile refresh publishes the image. Pull requests and manual runs resolve the lockfile first, always build, and never publish. - Hand-off. Each leg joins its two OCI archives into one OCI layout that names the images `slim` and `full`. The layout stores the two shared layers one time. `record` then reads the layout back, each image by its name, and checks every blob against its digest. The leg's record binds the layout to both image digests, the architecture, the inputs digest, and the commit. Those digests are the digests of the archives that part 3 unpacks and checks, so the tree that is checked and the image that is pushed are the same bytes. The publish job accepts only records for its own inputs and commit, and only a layout that still has the recorded bytes. - `record` runs on pull requests too. Only the upload is for a `master` push. A pull request therefore runs the hand-off on real build output. - Publication order. The publish job pushes the four images by digest (`skopeo copy --preserve-digests`). It makes a candidate index for the slim variant, then one for the full variant that names the slim index in the annotation `io.paperclip.runner.slim-digest`. It verifies that each index names exactly the recorded images, attests both, and then writes `inputs-<digest>-slim` and, last, `inputs-<digest>`. - The full tag is the completion marker. Every "is it published" lookup reads the full tag. `publication-state` reports one of: | State | Registry | What a run does | | --- | --- | --- | | `absent` | Neither tag | Builds and publishes | | `incomplete` | The slim tag alone | Builds and publishes. It replaces the slim tag with its own image, then writes the full tag | | `published` | The full tag, and the slim tag at the digest that the full index names | Nothing | | broken | The full tag, and a slim tag that is missing or points at another image | Fails, with the steps for an operator | | `unknown` | No clear answer | Builds. The publish job fails before it writes a final tag | - Recovery rule. A run that stops between the two tags leaves the state `incomplete`. It does not block those inputs. The next run never keeps the earlier slim image: it writes its own slim tag immediately before its full tag, verifies both against its own index digests, and then reads the pair back as every later run will see it. The broken state cannot be repaired by a run, because no run moves a full tag. An operator must delete the two package versions and re-run the workflow run of a `master` push with those inputs. - Pairing rule for consumers, in the contract: resolve the full tag first, take the slim digest from the annotation of the full index, and pull the slim image by that digest. - `verify-index` accepts `linux/arm64` with the variant `v8`, which some BuildKit versions record, and no other variant. - `scripts/runner-image-publish.mjs` (533 lines): the commands `lock-state`, `record`, `verify-legs`, `publication-state` and `verify-index`. It reads archives with the reader of `scripts/runner-image-oci.mjs` (607 lines here), which can now read one named image of a layout. `lock-state` refuses a checkout, because pnpm can rewrite the lockfile of the tree that it checks. The workflow gives it an export that no build reads. - The workflow test pins the build arguments of the image build. Build arguments are not part of the image identity. - `doc/runner-image-contract.md` (197 lines): the section "Publication": the lockfile rule, the order, the pairing rule, the recovery rule, the create-only tags, the internal `candidate-` tags, the command that verifies provenance, and the step to make the package public. `doc/DOCKER.md` says where the image is published. ## Verification Tests, and the build jobs of the workflow on GitHub. Nothing in this pull request ran against a registry, and no publish job has run. - Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd): `runner-image-publish.test.ts` (75), `runner-image-workflow.test.ts` (12), the three other Runner image test files, the provider pack archive tests and the existing Dockerfile tests: 372 tests pass in 10 files. The same files pass on macOS. - The tests of the leg record failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does, and the publish script uses the one reader of part 3. - Leg records: both variants are accepted for this run's inputs and commit; other inputs, another commit, a changed layout, a record without a variant, an archive that lacks a blob, a full image that does not start with the slim image's layers, and a build that is not a publication build are refused. - Publication state: all five states, and the half-finished publication end to end. A slim tag alone lets the next run publish. A full image that is left with another run's slim tag is reported as broken. - Index: exactly the recorded images of one variant; the full index must name its slim index, and the slim index must name none; `v8` is accepted on arm64 only. - Registry lookups: a missing tag only when the registry says so; every refused, failed, or unreachable request is `unknown`. - Workflow: job permissions; the publish job and the upload are behind the canonical `master` push; the build steps receive no credential; the full tag is written last, after a strict lookup and both attestations; one export and a time limit for each build command; the two build arguments; actions pinned by commit; no expression inside a shell script. - `record` on the two OCI archives of a real `linux/arm64` build from part 3: the joined layout is 850,149,376 bytes, and both images read back by name with every blob under its digest. The two archives together are 1,273,646,592 bytes. The image digests in the record are the digests that `unpack` reports for the trees that were checked. - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `docker buildx build --check` reports nothing. The image identity is the same as in part 3 (`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`), because this pull request changes no input of the image. GitHub, at this head: - The `Runner image` workflow passes with these changes: `Build and check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each leg builds both variants, checks them, and writes the leg record. `Publish and attest` is skipped, as it must be on a pull request. Never run: - The publish job. It runs only for a `master` push. - The publish path: the registry lookups with a workflow token, `skopeo copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx imagetools create` with index annotations, and the tag writes. - The attestations, and the `gh attestation verify` command in the document. - `actionlint`. It is not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - **Manual step.** GitHub creates the `paperclip-runner` package private on the first publication. A maintainer must make it public one time. Until then, consumers cannot pull the image without credentials. - The first `master` runs of the workflow are the first test of the publish path. A failure before the tags publishes nothing. A failure between the two tags leaves the `incomplete` state, and the next run with those inputs repairs it. - A slim tag can move while its publication is incomplete. A consumer must resolve the full tag first and take the slim digest from the annotation of the full index. It must never resolve `inputs-<digest>-slim` by tag on its own. The contract says so. - `candidate-` tags are written before the attestations and can name an index that is not attested. They are internal. - The broken state needs an operator, and it fails every `master` run with those inputs until then. It can arise only when someone changes or deletes a tag outside the workflow. - **Known residual.** Two `master` runs that both start before a new digest is published can each build the image. Each build leg and the publish job look again before they work, and the publish job runs one at a time for each digest. The cost is runner time, not correctness. - A commit whose lockfile is behind the Runner's manifests has no published Runner image until the lockfile refresh merges. A lockfile that the refresh cannot fix leaves every `master` run green with only a notice, for as long as it lasts. - The tags are create-only because the workflow never moves a tag of a complete publication. The registry does not enforce that. Consumers must pin the digest and verify the attestation. - `payload.runnerSourceRevision` in a pack from this image is the commit that published the image. Code that reads it as the application's commit must change before it consumes the image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
668 lines
31 KiB
YAML
668 lines
31 KiB
YAML
name: Runner image
|
|
|
|
# Builds the Runner once for each distinct set of its inputs and publishes it
|
|
# as ghcr.io/paperclipai/paperclip-runner, a multi-platform image that other
|
|
# image builds copy from, in two variants: `inputs-<digest>` (full) and
|
|
# `inputs-<digest>-slim` (without the pack's transport archives).
|
|
# doc/runner-image-contract.md is the contract.
|
|
#
|
|
# Every master push computes the digest from the commit's own build context,
|
|
# with the committed pnpm-lock.yaml. The build runs only when no image has
|
|
# that digest yet. When the committed lockfile is behind the manifests the
|
|
# Runner build reads, the run says so, publishes nothing and stays green: the
|
|
# repository's lockfile refresh brings the commit that publishes.
|
|
#
|
|
# Pull requests that touch the Runner's inputs, its recipe or this pipeline,
|
|
# and manual runs, always build and check the image and never publish it.
|
|
# They resolve the lockfile first, because a contributor cannot edit it.
|
|
#
|
|
# This is a workflow of its own, not a job in docker.yml: docker.yml has no
|
|
# pull-request trigger and holds one concurrency slot per ref, and the Runner
|
|
# image needs its own signer identity so its attestation cannot stand in for
|
|
# the application image's.
|
|
on:
|
|
push:
|
|
branches:
|
|
- master
|
|
pull_request:
|
|
# Every input of scripts/runner-image-identity.mjs, .dockerignore (it
|
|
# decides which of those files the build receives), and the pipeline's
|
|
# own files. A test checks that every identity input is covered.
|
|
paths:
|
|
- .github/workflows/runner-image.yml
|
|
- Dockerfile
|
|
- .dockerignore
|
|
- .npmrc
|
|
- package.json
|
|
- pnpm-lock.yaml
|
|
- pnpm-workspace.yaml
|
|
- tsconfig.base.json
|
|
- patches/**
|
|
- packages/paperclip-runner/**
|
|
- packages/paperclip-eval-kernel/**
|
|
- packages/shared/src/provider-failure.generated.ts
|
|
- scripts/create-provider-pack-archive.mjs
|
|
- scripts/link-plugin-dev-sdk.mjs
|
|
- scripts/runner-image-identity.mjs
|
|
- scripts/runner-image-layout.mjs
|
|
- scripts/runner-image-oci.mjs
|
|
- scripts/runner-image-publish.mjs
|
|
- scripts/standard-image-contract.mjs
|
|
- docker/runner-image/**
|
|
workflow_dispatch:
|
|
|
|
# Least privilege: nothing at the workflow level. Only the publish job, which
|
|
# runs no build, can write to the registry or sign.
|
|
permissions: {}
|
|
|
|
# A newer push to a pull request replaces its older run. A master run has a
|
|
# group of its own, so no other run cancels it here. The publish job has a
|
|
# second group, one for each digest; see there for what that group can cancel.
|
|
concurrency:
|
|
group: runner-image-${{ github.event.pull_request.number || github.sha }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
env:
|
|
IMAGE: ghcr.io/paperclipai/paperclip-runner
|
|
|
|
jobs:
|
|
identity:
|
|
name: Resolve the Runner image identity
|
|
# A fork's own pushes do not run this workflow. Pull requests and manual
|
|
# runs are not affected.
|
|
if: github.event_name != 'push' || (github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064')
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
permissions:
|
|
contents: read
|
|
packages: read
|
|
outputs:
|
|
lock: ${{ steps.lock.outputs.state }}
|
|
digest: ${{ steps.identity.outputs.digest }}
|
|
exists: ${{ steps.lookup.outputs.exists }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
run_install: false
|
|
|
|
# No dependency cache: a restored cache must not reach a published image.
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
# A contributor cannot edit pnpm-lock.yaml, so a pull request that
|
|
# changes the Runner's dependencies is built against a lockfile resolved
|
|
# here. The image build installs from it frozen.
|
|
#
|
|
# Never on a master push. The lockfile refresh owns pnpm-lock.yaml on
|
|
# master and holds new package versions until they are old enough, so
|
|
# the published image installs from the committed file and its digest
|
|
# is a function of the commit alone.
|
|
- name: Resolve the lockfile
|
|
if: github.event_name != 'push'
|
|
run: |
|
|
set -euo pipefail
|
|
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
|
|
|
|
changed="$(git status --porcelain)"
|
|
if [ -n "$changed" ] && printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
|
|
echo "Unexpected files changed during lockfile resolution:"
|
|
echo "$changed"
|
|
exit 1
|
|
fi
|
|
{
|
|
echo "This run built from a \`pnpm-lock.yaml\` resolved for it${changed:+ (it differs from the committed file)}."
|
|
echo "A master push publishes only from the committed lockfile."
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# The files the Runner build receives, after .dockerignore: the root
|
|
# manifest and the manifests of the Runner and the eval kernel are the
|
|
# only ones in it.
|
|
- name: Export the Runner build context
|
|
timeout-minutes: 10
|
|
run: docker buildx build --target runner-artifacts-context --output "type=local,dest=$RUNNER_TEMP/runner-image-context" .
|
|
|
|
# The image build installs with a frozen lockfile. This asks the same
|
|
# question first, offline and without installing, so that a lockfile
|
|
# that is merely behind is told apart from a broken build. pnpm can
|
|
# rewrite the lockfile it checks. It checks the exported copy above,
|
|
# which no build reads: the builds read the checkout.
|
|
- name: Check the lockfile against the Runner's manifests
|
|
id: lock
|
|
run: node scripts/runner-image-publish.mjs lock-state --context "$RUNNER_TEMP/runner-image-context"
|
|
|
|
# On master this is expected between a dependency change and the next
|
|
# lockfile refresh. It is not a failure, and there is nothing to build.
|
|
- name: Report a lockfile that is behind the Runner's manifests
|
|
if: steps.lock.outputs.state == 'behind'
|
|
run: |
|
|
set -euo pipefail
|
|
message="pnpm-lock.yaml is behind the Runner's manifests at $GITHUB_SHA; no Runner image for this commit; the next lockfile refresh will publish one."
|
|
echo "::notice title=No Runner image for this commit::$message"
|
|
echo "$message" >> "$GITHUB_STEP_SUMMARY"
|
|
# A run that resolved its own lockfile cannot be behind it.
|
|
test "$GITHUB_EVENT_NAME" = "push"
|
|
|
|
# Computed inside the build, from the files the build context supplies
|
|
# after .dockerignore. A digest computed from the checkout would cover
|
|
# files the image build never receives.
|
|
- name: Compute the identity in the build context
|
|
id: identity
|
|
if: steps.lock.outputs.state == 'current'
|
|
timeout-minutes: 10
|
|
run: |
|
|
set -euo pipefail
|
|
out="$RUNNER_TEMP/runner-image-identity"
|
|
docker buildx build --target runner-artifacts-identity --output "type=local,dest=$out" .
|
|
digest="$(jq -er .digest "$out/runner-image-identity.json")"
|
|
[[ "$digest" =~ ^[0-9a-f]{64}$ ]]
|
|
echo "digest=$digest" >> "$GITHUB_OUTPUT"
|
|
{
|
|
echo "Runner image inputs digest: \`$digest\`"
|
|
echo
|
|
echo '```json'
|
|
cat "$out/runner-image-identity.json"
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# The legs must build from the file the identity was computed from, not
|
|
# from a second resolution. Only for runs that resolved one: a master
|
|
# push builds from the committed file, which every job checks out.
|
|
- name: Keep the resolved lockfile for the build legs
|
|
if: steps.lock.outputs.state == 'current' && github.event_name != 'push'
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: runner-image-lockfile
|
|
path: pnpm-lock.yaml
|
|
if-no-files-found: error
|
|
retention-days: 1
|
|
|
|
# An image that exists is never rebuilt on master. It exists when its
|
|
# full tag does: that tag is written last. An inconclusive answer
|
|
# builds: the publish job decides again, strictly, before it writes a
|
|
# tag. Other events build whether or not the image exists.
|
|
- name: Look for the published image
|
|
id: lookup
|
|
if: steps.lock.outputs.state == 'current' && github.event_name == 'push'
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
DIGEST: ${{ steps.identity.outputs.digest }}
|
|
run: node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST"
|
|
|
|
# Multi-arch by native runner, not QEMU, as in docker.yml. GitHub-hosted
|
|
# runners: the protected release fleets admit only docker.yml on master.
|
|
#
|
|
# These legs cannot write to the registry on any event, and they read and
|
|
# write no registry build cache. The install runs dependency lifecycle
|
|
# scripts; the only token in the job can read packages, and only the lookup
|
|
# step receives it.
|
|
#
|
|
# A master push builds only when the image does not exist. A pull request
|
|
# and a manual run always build, so a change to the smoke matrix or to the
|
|
# pipeline is exercised even when the Runner's inputs did not change.
|
|
build:
|
|
name: Build and check (${{ matrix.arch }})
|
|
needs: identity
|
|
if: needs.identity.outputs.lock == 'current' && (github.event_name != 'push' || (github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && needs.identity.outputs.exists != 'true'))
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- platform: linux/amd64
|
|
runner: ubuntu-latest
|
|
arch: amd64
|
|
- platform: linux/arm64
|
|
runner: ubuntu-24.04-arm
|
|
arch: arm64
|
|
runs-on: ${{ matrix.runner }}
|
|
timeout-minutes: 60
|
|
permissions:
|
|
contents: read
|
|
packages: read
|
|
env:
|
|
# Only a canonical master push keeps its image for publication.
|
|
KEEP_IMAGE: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
|
DIGEST: ${{ needs.identity.outputs.digest }}
|
|
ARCH: ${{ matrix.arch }}
|
|
PLATFORM: ${{ matrix.platform }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
# Another master run can have published these inputs since the identity
|
|
# job looked, most often the run of the push just before this one. Then
|
|
# this leg has nothing to do. Master only: other events always build.
|
|
# An inconclusive answer builds.
|
|
- name: Look again for the published image
|
|
id: published
|
|
if: github.event_name == 'push'
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST"
|
|
|
|
- name: Download the resolved lockfile
|
|
if: github.event_name != 'push'
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
name: runner-image-lockfile
|
|
path: ${{ runner.temp }}/runner-image-lockfile
|
|
|
|
# Build from the lockfile the identity was computed from, not from a
|
|
# second resolution that a registry change could make different.
|
|
- name: Use the resolved lockfile
|
|
if: github.event_name != 'push'
|
|
run: cp "$RUNNER_TEMP/runner-image-lockfile/pnpm-lock.yaml" pnpm-lock.yaml
|
|
|
|
- name: Free runner disk
|
|
if: steps.published.outputs.exists != 'true'
|
|
run: |
|
|
set -euo pipefail
|
|
sudo apt-get clean || true
|
|
sudo rm -rf /usr/share/dotnet /usr/share/swift /usr/local/lib/android /usr/local/share/boost \
|
|
/usr/local/share/powershell /opt/ghc /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/PyPy /opt/hostedtoolcache/Ruby || true
|
|
docker system prune -af || true
|
|
df -h
|
|
|
|
- name: Set up Docker Buildx
|
|
if: steps.published.outputs.exists != 'true'
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
with:
|
|
buildkitd-config-inline: |
|
|
[registry."docker.io"]
|
|
mirrors = ["mirror.gcr.io"]
|
|
|
|
# The build itself runs the frozen install, the Runner's own build, the
|
|
# provider pack integrity check and the layout check.
|
|
#
|
|
# Both variants come from one build: the second command only reads the
|
|
# cache of the first. Each is exported once, as an OCI archive and
|
|
# nothing else. A command with two exports, an archive and a tree,
|
|
# stalled on these runners when every stage came from the cache, so a
|
|
# command has one export, and the step has a time limit: a cold build
|
|
# takes about ten minutes.
|
|
#
|
|
# This is the publication form of the build, the only one whose pack
|
|
# names the image (`runnerInputsDigest`), so the check below can bind
|
|
# the tree to the digest the identity job resolved. This job cannot
|
|
# publish it. On a canonical master push the archives are the images
|
|
# the publish job pushes, byte for byte.
|
|
- name: Build both variants of the Runner image
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 30
|
|
run: |
|
|
set -euo pipefail
|
|
for variant in full slim; do
|
|
target=runner-artifacts
|
|
if [ "$variant" = slim ]; then target=runner-artifacts-slim; fi
|
|
docker buildx build \
|
|
--target "$target" \
|
|
--platform "$PLATFORM" \
|
|
--build-arg "PAPERCLIP_RUNNER_SOURCE_REVISION=$GITHUB_SHA" \
|
|
--build-arg PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 \
|
|
--provenance=false --sbom=false \
|
|
--output "type=oci,dest=$RUNNER_TEMP/runner-image/$variant.oci.tar" \
|
|
--metadata-file "$RUNNER_TEMP/runner-image/$variant.metadata.json" \
|
|
.
|
|
done
|
|
|
|
# The trees the checks below read are the layers of those archives,
|
|
# written out in order. They are the bytes of the images themselves,
|
|
# not a second export of the build, and each blob is checked against
|
|
# its digest as it is read.
|
|
- name: Unpack both images
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 10
|
|
run: |
|
|
set -euo pipefail
|
|
for variant in full slim; do
|
|
node scripts/runner-image-oci.mjs unpack "$RUNNER_TEMP/runner-image/$variant.oci.tar" \
|
|
--out "$RUNNER_TEMP/runner-image/$variant"
|
|
done
|
|
|
|
# Against each unpacked tree, bound to this run: the digest the identity
|
|
# job resolved, this commit and this leg's architecture. Checks every
|
|
# documented path, the build stamp, and every pack artifact and tree
|
|
# digest in provider-pack.json. A slim tree must not hold the transport
|
|
# archives. A full tree must: the check verifies each one, extracts
|
|
# them as a sandbox does and compares the result with the pack; that
|
|
# needs about as much free disk as the pack is large.
|
|
- name: Check the layout contract and pack integrity
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 10
|
|
run: |
|
|
set -euo pipefail
|
|
for variant in full slim; do
|
|
node scripts/runner-image-layout.mjs check "$RUNNER_TEMP/runner-image/$variant" \
|
|
--variant "$variant" \
|
|
--expect-inputs-digest "$DIGEST" \
|
|
--expect-source-revision "$GITHUB_SHA" \
|
|
--expect-architecture "$ARCH"
|
|
done
|
|
|
|
# The full image must be the slim image's layers, as the same blobs,
|
|
# then the transport layers, with nothing else in them. No file may be
|
|
# in two layers.
|
|
- name: Check the layers of both variants
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 10
|
|
run: |
|
|
set -euo pipefail
|
|
{
|
|
echo '```'
|
|
node scripts/runner-image-oci.mjs check-layers \
|
|
--slim "$RUNNER_TEMP/runner-image/slim.oci.tar" \
|
|
--full "$RUNNER_TEMP/runner-image/full.oci.tar"
|
|
echo '```'
|
|
} | tee -a "$GITHUB_STEP_SUMMARY"
|
|
|
|
# On the slim tree: the daemon is in a layer both variants share, so
|
|
# one run covers both, and the slim tree is the smaller one to load.
|
|
# This command has one export, a few small files.
|
|
- name: Run the daemon on every consumer base image
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 20
|
|
run: |
|
|
docker buildx build \
|
|
--file docker/runner-image/Dockerfile.smoke \
|
|
--platform "$PLATFORM" \
|
|
--build-context "runner-artifacts=$RUNNER_TEMP/runner-image/slim" \
|
|
--output "type=local,dest=$RUNNER_TEMP/runner-image/smoke" \
|
|
docker/runner-image
|
|
ls -l "$RUNNER_TEMP/runner-image/smoke"
|
|
|
|
# Written only after every check passed. The publish job accepts nothing
|
|
# without it. The two archives become one OCI layout that holds both
|
|
# variants and stores the layers they share once, so the upload is the
|
|
# size of the full image, not of both. The command reads the layout
|
|
# back and checks every blob against its digest.
|
|
#
|
|
# On every event, so that a pull request exercises the hand-off on
|
|
# real build output. Only the upload below is for a master push.
|
|
- name: Record the verified images
|
|
if: steps.published.outputs.exists != 'true'
|
|
timeout-minutes: 10
|
|
run: |
|
|
node scripts/runner-image-publish.mjs record \
|
|
--arch "$ARCH" \
|
|
--slim-archive "$RUNNER_TEMP/runner-image/slim.oci.tar" \
|
|
--slim-metadata "$RUNNER_TEMP/runner-image/slim.metadata.json" \
|
|
--full-archive "$RUNNER_TEMP/runner-image/full.oci.tar" \
|
|
--full-metadata "$RUNNER_TEMP/runner-image/full.metadata.json" \
|
|
--tree "$RUNNER_TEMP/runner-image/full" \
|
|
--layout "$RUNNER_TEMP/runner-image/images.oci.tar" \
|
|
--out "$RUNNER_TEMP/runner-image/record.json"
|
|
|
|
- name: Upload the verified images
|
|
if: env.KEEP_IMAGE == 'true' && steps.published.outputs.exists != 'true'
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: runner-image-${{ matrix.arch }}
|
|
path: |
|
|
${{ runner.temp }}/runner-image/images.oci.tar
|
|
${{ runner.temp }}/runner-image/record.json
|
|
if-no-files-found: error
|
|
# Layers are already compressed.
|
|
compression-level: 0
|
|
retention-days: 1
|
|
|
|
- name: Report an image another run published first
|
|
if: steps.published.outputs.exists == 'true'
|
|
run: echo "\`$IMAGE:inputs-$DIGEST\` and its slim variant already exist. This leg built nothing." >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# Pushes the verified bytes, names them and attests them. Only a master push
|
|
# in the canonical repository reaches this job; it builds nothing and runs no
|
|
# dependency code.
|
|
#
|
|
# It writes two tags, `inputs-<digest>-slim` and then `inputs-<digest>`. The
|
|
# full tag is last, so it marks a complete publication: when it exists, both
|
|
# exist, and the full index names its slim index by digest. A run that stops
|
|
# between the two tags leaves a slim tag alone. The inputs are then not
|
|
# published; the next run for them builds again, replaces that slim tag with
|
|
# its own, and writes the full tag. It never keeps another run's slim image.
|
|
publish:
|
|
name: Publish and attest
|
|
needs: [identity, build]
|
|
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master' && needs.identity.outputs.lock == 'current' && needs.identity.outputs.exists != 'true'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
permissions:
|
|
contents: read
|
|
packages: write
|
|
id-token: write
|
|
attestations: write
|
|
# One publication per digest at a time, so the strict check below and the
|
|
# tag writes cannot interleave with another run for the same inputs.
|
|
# GitHub keeps one waiting job for a group. When three runs for one digest
|
|
# reach this job, the second is cancelled as the third arrives. That does
|
|
# no harm: the third publishes the digest if the first did not.
|
|
concurrency:
|
|
group: runner-image-publish-${{ needs.identity.outputs.digest }}
|
|
cancel-in-progress: false
|
|
env:
|
|
DIGEST: ${{ needs.identity.outputs.digest }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
# A build leg that found the image published uploaded nothing. Ask
|
|
# before the downloads, so that run ends here and not on a missing
|
|
# artifact.
|
|
- name: Look for an image another run published
|
|
id: early
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST"
|
|
|
|
- name: Download the verified amd64 images
|
|
if: steps.early.outputs.exists != 'true'
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
name: runner-image-amd64
|
|
path: ${{ runner.temp }}/runner-image/amd64
|
|
|
|
- name: Download the verified arm64 images
|
|
if: steps.early.outputs.exists != 'true'
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
|
with:
|
|
name: runner-image-arm64
|
|
path: ${{ runner.temp }}/runner-image/arm64
|
|
|
|
- name: Accept only this run's verified images
|
|
id: legs
|
|
if: steps.early.outputs.exists != 'true'
|
|
run: |
|
|
node scripts/runner-image-publish.mjs verify-legs \
|
|
--dir "$RUNNER_TEMP/runner-image" \
|
|
--inputs-digest "$DIGEST" \
|
|
--source-revision "$GITHUB_SHA"
|
|
|
|
- name: Login to GitHub Container Registry
|
|
if: steps.early.outputs.exists != 'true'
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.actor }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Set up Docker Buildx
|
|
if: steps.early.outputs.exists != 'true'
|
|
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
|
|
|
|
# The downloads take a while. Ask once more immediately before the
|
|
# pushes, so that a run that lost the race does not upload its images.
|
|
- name: Look again immediately before pushing
|
|
id: prepush
|
|
if: steps.early.outputs.exists != 'true'
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST"
|
|
|
|
# By digest and untagged. --preserve-digests fails rather than re-encode,
|
|
# so the registry holds exactly the images the build legs verified. Each
|
|
# leg's layout names its two images `slim` and `full`.
|
|
- name: Push the verified images by digest
|
|
if: steps.prepush.outputs.exists == 'false'
|
|
timeout-minutes: 15
|
|
env:
|
|
SLIM_AMD64: ${{ steps.legs.outputs.slim_amd64 }}
|
|
SLIM_ARM64: ${{ steps.legs.outputs.slim_arm64 }}
|
|
FULL_AMD64: ${{ steps.legs.outputs.full_amd64 }}
|
|
FULL_ARM64: ${{ steps.legs.outputs.full_arm64 }}
|
|
run: |
|
|
set -euo pipefail
|
|
command -v skopeo >/dev/null || { echo "skopeo is not installed on this runner"; exit 1; }
|
|
echo "Pushing with $(skopeo --version)"
|
|
layout="$RUNNER_TEMP/runner-image"
|
|
skopeo copy --preserve-digests "oci-archive:$layout/amd64/images.oci.tar:slim" "docker://$IMAGE@$SLIM_AMD64"
|
|
skopeo copy --preserve-digests "oci-archive:$layout/arm64/images.oci.tar:slim" "docker://$IMAGE@$SLIM_ARM64"
|
|
skopeo copy --preserve-digests "oci-archive:$layout/amd64/images.oci.tar:full" "docker://$IMAGE@$FULL_AMD64"
|
|
skopeo copy --preserve-digests "oci-archive:$layout/arm64/images.oci.tar:full" "docker://$IMAGE@$FULL_ARM64"
|
|
|
|
# The tags are create-only. Another run can have published these inputs
|
|
# since the lookups above. Only a clear answer goes on: `absent`, or
|
|
# `incomplete` (a slim tag alone, which this run replaces). `published`
|
|
# skips the rest. An inconclusive answer fails, and so does a full tag
|
|
# whose slim image is missing or is another image.
|
|
- name: Confirm the images are not published
|
|
id: gate
|
|
if: steps.prepush.outputs.exists == 'false'
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST" --strict
|
|
|
|
# Candidate tags first, so a final tag appears only after its digest is
|
|
# attested. Candidate tags are internal and may be overwritten.
|
|
- name: Create the slim multi-platform candidate
|
|
if: steps.gate.outputs.publish == 'true'
|
|
env:
|
|
AMD64: ${{ steps.legs.outputs.slim_amd64 }}
|
|
ARM64: ${{ steps.legs.outputs.slim_arm64 }}
|
|
run: |
|
|
set -euo pipefail
|
|
docker buildx imagetools create \
|
|
--annotation "index:org.opencontainers.image.source=https://github.com/paperclipai/paperclip" \
|
|
--annotation "index:org.opencontainers.image.revision=$GITHUB_SHA" \
|
|
--annotation "index:io.paperclip.runner.inputs-digest=$DIGEST" \
|
|
--annotation "index:io.paperclip.runner.variant=slim" \
|
|
-t "$IMAGE:candidate-$DIGEST-slim" \
|
|
"$IMAGE@$AMD64" "$IMAGE@$ARM64"
|
|
|
|
- name: Verify the slim candidate names exactly the verified images
|
|
id: slim
|
|
if: steps.gate.outputs.publish == 'true'
|
|
env:
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: node scripts/runner-image-publish.mjs verify-index "candidate-$DIGEST-slim" --variant slim --dir "$RUNNER_TEMP/runner-image"
|
|
|
|
# The full index names the slim index of this same run by digest, so a
|
|
# consumer can tell that the two belong together.
|
|
- name: Create the full multi-platform candidate
|
|
if: steps.gate.outputs.publish == 'true'
|
|
env:
|
|
AMD64: ${{ steps.legs.outputs.full_amd64 }}
|
|
ARM64: ${{ steps.legs.outputs.full_arm64 }}
|
|
SLIM_INDEX: ${{ steps.slim.outputs.digest }}
|
|
run: |
|
|
set -euo pipefail
|
|
docker buildx imagetools create \
|
|
--annotation "index:org.opencontainers.image.source=https://github.com/paperclipai/paperclip" \
|
|
--annotation "index:org.opencontainers.image.revision=$GITHUB_SHA" \
|
|
--annotation "index:io.paperclip.runner.inputs-digest=$DIGEST" \
|
|
--annotation "index:io.paperclip.runner.variant=full" \
|
|
--annotation "index:io.paperclip.runner.slim-digest=$SLIM_INDEX" \
|
|
-t "$IMAGE:candidate-$DIGEST" \
|
|
"$IMAGE@$AMD64" "$IMAGE@$ARM64"
|
|
|
|
- name: Verify the full candidate names exactly the verified images and its slim index
|
|
id: full
|
|
if: steps.gate.outputs.publish == 'true'
|
|
env:
|
|
SLIM_INDEX: ${{ steps.slim.outputs.digest }}
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: |
|
|
node scripts/runner-image-publish.mjs verify-index "candidate-$DIGEST" --variant full \
|
|
--slim-digest "$SLIM_INDEX" --dir "$RUNNER_TEMP/runner-image"
|
|
|
|
- name: Attest the slim Runner image
|
|
if: steps.gate.outputs.publish == 'true'
|
|
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4
|
|
with:
|
|
subject-name: ghcr.io/paperclipai/paperclip-runner
|
|
subject-digest: ${{ steps.slim.outputs.digest }}
|
|
push-to-registry: true
|
|
|
|
- name: Attest the full Runner image
|
|
if: steps.gate.outputs.publish == 'true'
|
|
uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4
|
|
with:
|
|
subject-name: ghcr.io/paperclipai/paperclip-runner
|
|
subject-digest: ${{ steps.full.outputs.digest }}
|
|
push-to-registry: true
|
|
|
|
# Slim first, full last. Writing the slim tag replaces the one a run
|
|
# that stopped here left behind. The last command checks the pair as
|
|
# every later run will see it.
|
|
- name: Publish the tags
|
|
if: steps.gate.outputs.publish == 'true'
|
|
env:
|
|
SLIM_INDEX: ${{ steps.slim.outputs.digest }}
|
|
FULL_INDEX: ${{ steps.full.outputs.digest }}
|
|
RUNNER_IMAGE_REGISTRY_USERNAME: ${{ github.actor }}
|
|
RUNNER_IMAGE_REGISTRY_PASSWORD: ${{ github.token }}
|
|
run: |
|
|
set -euo pipefail
|
|
layout="$RUNNER_TEMP/runner-image"
|
|
docker buildx imagetools create -t "$IMAGE:inputs-$DIGEST-slim" "$IMAGE@$SLIM_INDEX"
|
|
node scripts/runner-image-publish.mjs verify-index "inputs-$DIGEST-slim" --variant slim --dir "$layout" --expect "$SLIM_INDEX"
|
|
docker buildx imagetools create -t "$IMAGE:inputs-$DIGEST" "$IMAGE@$FULL_INDEX"
|
|
node scripts/runner-image-publish.mjs verify-index "inputs-$DIGEST" --variant full --slim-digest "$SLIM_INDEX" --dir "$layout" --expect "$FULL_INDEX"
|
|
node scripts/runner-image-publish.mjs publication-state --inputs-digest "$DIGEST" --strict | grep -Fx "state=published"
|
|
{
|
|
echo "Published from \`$GITHUB_SHA\`:"
|
|
echo "- full: \`$IMAGE:inputs-$DIGEST\` at \`$IMAGE@$FULL_INDEX\`"
|
|
echo "- slim: \`$IMAGE:inputs-$DIGEST-slim\` at \`$IMAGE@$SLIM_INDEX\`"
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- name: Report images another run published first
|
|
if: steps.early.outputs.exists == 'true' || steps.prepush.outputs.exists == 'true' || steps.gate.outputs.state == 'published'
|
|
run: echo "\`$IMAGE:inputs-$DIGEST\` and its slim variant already exist. Another run published first; this run published nothing." >> "$GITHUB_STEP_SUMMARY"
|