Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 6826800425 feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to `master` publishes the standard image, and the time
from merge to published image limits how fast a fix reaches users
> - #21096 lets `docker.yml` push an image that the merge queue already
built, but the merge queue is not on
> - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only
`pr.yml` reports them, and only for pull requests
> - A merge queue waits for the same two names on its merge-group
commit. No workflow reports them there, so every queued merge would
stall
> - This pull request adds a workflow that reports both names on
merge-group commits
> - The benefit is that the owner can turn the merge queue on with one
ruleset change

## Linked Issues or Issue Description

Refs #21096

**What existing behavior does this improve?**

The merge-queue image build from #21096. It cannot run until the merge
queue is on, and the merge queue cannot be turned on safely.

**Current behavior**

`ci / verify` and `ci / e2e` come from `pr.yml`, which runs on
`pull_request` only. With a merge queue, GitHub waits for both checks on
the merge-group commit. They never arrive. Each queued pull request
would time out and leave the queue.

**Proposed behavior**

`merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify`
and `ci / e2e`. It runs no tests. It does not check out the queued
commit. It has no token permissions and no secrets.

**Reason and benefit**

A pull request enters the queue only after both checks pass on the pull
request. A queued merge therefore has the same protection as a direct
merge today. The exact-source gate still tests each `master` commit
before publication. Real tests on the combined tree can replace these
jobs later under the same names.

## What Changed

- Added `.github/workflows/merge-queue-checks.yml`. It reports `ci /
verify` and `ci / e2e` on merge-group commits.
- Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It
fails if the names differ from the names that `pr.yml` reports, or if
the workflow gains a checkout, an action, a secret, a token permission
or a self-hosted runner.
- `pr-trusted.yml` now runs that test and
`scripts/merge-group-image.test.mjs` in a step that skips test files an
older checkout does not have. The second test came with #21096 but no CI
step ran it.
- `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a
merge-queue image build that is still queued or running. The queue can
land a commit before its image build ends. Without the wait, the
`master` push always builds from source.
- `doc/RELEASING.md` explains the workflow and lists the ruleset
settings that turn the queue on and off.

## Verification

- `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`:
2 pass.
- `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new
cases for unfinished builds.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`:
pass.
- `actionlint` on both changed workflows: no findings.
- Not verified: no `merge_group` event can fire before the queue is on.
The first queued merge is the live test. `pr.yml` calls
`pr-trusted.yml@master`, so the new CI step runs only after this merges.

Owner steps after merge:

1. In the `master` ruleset, add **Require merge queue**: Squash, build
concurrency 5, group size 1 to 5, status check timeout 60 minutes.
2. After the first queued merge, confirm that the `master` commit SHA
equals the SHA of its merge-group runs.

## Risks

- Low risk while the queue is off. The workflow has no trigger other
than `merge_group`.
- With the queue on, the queue does not test the combined tree. Two pull
requests that pass alone can still break `master` together. This is the
same as today.
- A merge-queue image build that hangs delays the source build by up to
15 minutes. A build that fails does not.
- If GitHub does not accept a job named `ci / verify` for the required
context, queued merges time out. Rollback: remove the merge queue rule.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:18:13 -06:00
Nicky LeachandClaude Fable 5.2 647641a99b feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as an attested image, once for each set of its inputs
> - Since #21332 the application image takes its Runner only from the
`runner-artifacts` stage, and a build can replace that stage with the
published image
> - No workflow does that yet, so every application image still builds
its own Runner, and nothing on an application image says which Runner it
has
> - A build that copies the published Runner must also build the
lockfile that Runner was built from. The workflow refreshes the lockfile
before every build, and the server build then refuses the published
Runner
> - This pull request lets `docker.yml` copy the published Runner image
and label the application image with it, behind a repository variable
that is off, and makes that build use the committed lockfile
> - The benefit is one Runner for all images of the same inputs, named
on the image, with a switch that goes back to today's builds without a
revert

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), #21308 (the server reads a pack from its
Runner image), and #21331, #21332, #21333 (the application image takes
its Runner from `runner-artifacts`). No issue exists for this change.

**Problem or motivation**

`docker.yml` builds the application image on `master` and on tags. Each
build makes its own Runner. The report-only job that runs the resolver
on `master` feeds no build. Downstream image composers cannot tell from
an application image which Runner it has.

There is a second problem, which only shows when a build copies the
published Runner. The Runner's build stamp covers the whole
`pnpm-lock.yaml`. The step that vendors the Runner into the server fails
when the stamp does not match the tree. `docker.yml` refreshes
`pnpm-lock.yaml` on the runner before every build. The refresh rewrites
the file whenever the manifest of any package is ahead of it, also a
package the Runner does not read. On the tree of this pull request it
rewrites the file even for a pristine checkout: the registry changed the
deprecation text of one package. The published Runner was built from the
committed file, so the image build would fail.

**Proposed solution**

One job resolves the Runner once for the run. Both architecture builds
need it and add its answer to their build: a build context that replaces
the `runner-artifacts` stage, two labels, and the build argument with
which the build checks the Runner. A build that copies the published
Runner does not refresh the lockfile: it builds the committed tree, the
tree the Runner was published for. The job runs only when the repository
variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull
request merges with the variable unset, so no image changes when it
merges.

**Alternatives considered**

- Resolve in each architecture build. The two builds could then
disagree, and one image index would hold two Runners.
- Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for
an inline build.
- For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over
`/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step
reads the file: the two vendor checks are its only readers in `build`,
`production` copies it and takes `node_modules` from `deps`, both
`cloud-*` stages install with `--ignore-workspace --no-lockfile`, and
the server reads only a plugin's own lockfile and the pack's own
lockfile at run time. But the image would then ship a lockfile that its
`node_modules` did not come from, the copy changes that file's layer for
every inline build, and each `Dockerfile` edit makes a new Runner image.
- For the lockfile, bind-mount the Runner image's lockfile over the file
for the two steps that run the vendor check. This also edits the
`Dockerfile`, and it makes the lockfile part of the check always true
for a published Runner.
- For the lockfile, let the resolver refresh and compare. The answer
would depend on the registry at two different moments.
- Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check
as strict as it is, and builds exactly the tree the inputs digest was
computed from.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on `runner/app-image-consume`. Merge in this
order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. `runner/app-image-consume`: the wait and the `unsupported` answer of
the resolver.
3. This pull request.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `.github/workflows/docker.yml`: the job `runner_image` replaces
`runner_image_report`.
- It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in
this repository. A fork never runs it.
- Steps: `inputs` with no credential; `resolve`; then `gh attestation
verify` of the full index against the signer `runner-image.yml` on
`refs/heads/master`.
- For a `master` push, `resolve` runs with `--wait` and the job token.
For a tag or a manual run it does not wait and receives no token.
- Only the verification step writes the job's outputs. An image that
does not verify fails that step with a warning and writes no output. The
job is `continue-on-error`: the job is red, the run is not, and the
builds are inline.
- Permissions: `contents: read`, `actions: read`, `attestations: read`.
No registry login, no write.
- Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the
verification, 70 for the job. The wait of 40 minutes ends inside them.
- `build-and-push` needs `runner_image` and adds its outputs to the
build: `build-contexts`, one more line of `build-args`
(`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`.
With empty outputs these are the values the build had before.
- The step "Refresh lockfile for Docker build context" runs only when
the answer is not `published`.
- The step that looks for a merge-queue image of the commit runs only
when the answer is not `published`. The two steps after it already
depend on its result.
- Labels on an image that copied the published Runner:
`io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full
index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An
image without them made its own Runner.
- The `Dockerfile` does not change. The Runner image digest does not
move.
- Tests that pin lines this changes follow:
`scripts/merge-group-image.test.mjs` (the condition of the merge-queue
lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build
job), and `docker-build-stamp.test.ts` (no workflow names a build
context; `docker.yml` now names exactly one).
- `server/src/__tests__/docker-runner-workflows.test.ts`: the
report-only tests give way to tests of the new job.
- `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of
the lockfile case.
- `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application
image gets its Runner, the labels and what their absence means, the
variable, the wait, the verification, the lockfile rule, the merge-queue
rule and the rollback.

## Verification

### Real builds

`docker buildx build --target build --platform linux/arm64 --output
type=cacheonly`, with a `docker-container` builder made for this test.
The Runner image is the publication form
(`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`)
built from a clean export of this branch, exported as an OCI layout:
inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application
builds receive `--build-context
runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command
is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm
install --resolution-only --ignore-scripts --no-frozen-lockfile`.

| Build context | Result |
| --- | --- |
| The committed tree | Passes in 78 s. Both vendor checks: `reusing the
current Runner build (inputs unchanged …; 1333 files and 1860 outputs
checked)` |
| `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh
command (what the workflow did before this change) | Fails at `RUN node
server/scripts/prepare-runner-vendor.mjs`:
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild
this step needs (inputs changed since the recorded build; 1333 files
checked)` |
| The committed tree, then only the refresh command | Fails the same
way. The refresh changed only the `deprecated` text of one package |
| The same `ui/package.json` change with no refresh (what the workflow
does now for a `published` answer) | Passes in 66 s. `deps` resolves the
manifest inside the image; both vendor checks reuse the Runner |
| One line added to `packages/paperclip-runner/src/index.ts` | Fails at
the vendor step, as before: a Runner of other inputs is still refused |

Lockfile behind the Runner's own manifests
(`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` →
`^1.9.0`), no build context:

| Step | Result |
| --- | --- |
| `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no
digest; `inline (lock-behind)` |
| Build without the refresh | Fails in 4 s at the `runner-lockfile`
stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is
installed from …` |
| Build after the refresh command (what the workflow does for an
`inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes,
the build makes its own Runner with its pack, and both vendor checks
reuse it |

### Tests

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch:

- `pnpm exec vitest run` on `docker-runner-workflows`,
`runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`,
`container-init-reaping`, `cloud-image-bundled-plugins`,
`docker-image-layer-reuse`, `cloud-image-sentry`,
`runner-lockfile-guards` and `runner-image-identity` in
`server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files
pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test files typecheck with the options of
`server/src/__tests__/tsconfig.json`.

What the workflow tests hold:

- Variable off: the job's condition is the variable and the repository,
with no alternative. With empty outputs, the build step has exactly the
3 build arguments, 0 build contexts and 1 label entry it had before, and
the refresh step runs.
- Published: both architecture builds are one job that reads only the
outputs of `runner_image`, with no architecture in those lines.
- The verification step runs as real shell against a stub `gh`. When
`gh` succeeds it writes the context, the labels and the build argument.
When `gh` fails the step exits 1, writes no output and prints the
warning. The step has no `continue-on-error`; the job has. The condition
of the build job does not name the resolver job and has no `success()`.
- The lookup step runs as real shell against a stub `node`. It passes
`--wait` only when the event is a `master` push.
- Credentials: only the lookup and the verification steps have a token.
The steps that build have no `env`.
- The refresh step has the condition `needs.runner_image.outputs.mode !=
'published'`, and it is the only `pnpm install` of the build job.
- The fixture test gives the vendor step a Runner that another tree
built for the committed lockfile. The step reuses it. After a change of
one specifier of another package in the lockfile the step fails and
names the lockfile. With the committed lockfile back it reuses the
Runner again.
- The merge-queue workflow and the pull-request check do not name the
resolver, the variable, a build context or a Runner label.

Read-only checks against the real services:

- `gh attestation verify
oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo
paperclipai/paperclip --signer-workflow
paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref
refs/heads/master`: exit 0 for the full index, and exit 0 for the slim
index `sha256:69ab2001…69b3`.
- The same command with `--signer-workflow …/docker.yml`: exit 1 for
both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo
cli/cli`: exit 1. For a digest that does not exist: exit 1. With no
token: exit 4.
- The parsed YAML of `docker.yml` differs from its base only in the
resolver job, the `needs` of the build job, the conditions of the
refresh step and of the merge-queue lookup, and the three build inputs.

Never run:

- `docker.yml`. Only GitHub can run it, and it has no pull-request
trigger.
- A build with the variable on. The first one is the first run after a
maintainer sets the variable.
- A build that pulls the published Runner from the registry as
`docker-image://`. The real builds used the same image form from a local
OCI layout.
- An `amd64` build.
- The `production` and `cloud` targets. The real builds stopped at
`build`, the last stage that checks the Runner.

## Risks

- With the variable on, an application build on `master` starts after
the resolver job. That is 2 to 3 minutes when the image exists, and up
to 40 minutes from the push when the commit's own Runner publication is
in progress. `docker.yml` holds one concurrency slot for each ref, so a
long wait can make more `master` commits lose their image build to a
newer push.
- A build that copies the published Runner ships the committed
`pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of
other packages inside the image. A local `docker build` has always done
this; a CI image did not. Nothing in the image reads that file.
- Any other build that gives the `Dockerfile` a published Runner and a
refreshed lockfile fails at the vendor step. That is the intended answer
to a tree the Runner was not built for, and the message names the
lockfile.
- The pack of an image that copied the published Runner names the commit
that published the Runner (`runnerSourceRevision`), which can be older
than the commit of the image. A consumer that compares the two must
accept that before the variable is set.
- A failed verification makes the resolver job red on a green run. That
is on purpose: it means a published tag whose attestation does not
verify.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- With the variable off there is no report of the resolver on `master`
any more: the job that replaced the report-only job does not run.
- Rollback: set the variable to another value or delete it. The next run
builds inline, with the refresh. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds, read-only
calls to the GitHub API and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:47:20 -08:00
DottaandClaude Opus 5.5 1d153f62b6 feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds, probes, gates and signs the standard Core image
for every master push. Paperclip Cloud deploys a Core commit only after
that image is published.
> - Each master push rebuilds the image from scratch: about 10.6 min for
amd64 and 7.8 min for arm64.
> - The owner approved a merge queue for Core. A merge queue tests the
exact commit that master will become, before the merge.
> - This pull request builds and probes the image for that exact commit
in the queue, with no registry credentials. The later master push pushes
those exact bytes instead of rebuilding. When no matching verified build
exists, master builds from source exactly as today.
> - The benefit is that the image legs on master drop from about 8–11
min to about 3–4 min, and nothing changes until the owner turns the
queue on.

## Linked Issues or Issue Description

Refs #21013 (it measured the image path and proposed this plan).

**What existing behavior does this improve?**
Every master push rebuilds the standard image, even when the same commit
could already have been built while it waited to merge.

**Subsystem affected**
`.github/workflows/docker.yml` (`build-and-push` only), new
`.github/workflows/docker-merge-group.yml`, new
`scripts/merge-group-image.mjs`.

**Current behavior**
`build-and-push` always runs the full BuildKit build on master: 639 s
(amd64) and 468 s (arm64) in run 43133035209.

**Proposed behavior**
On a `merge_group` event, build both architectures for the merge-group
commit, probe PID 1, and upload an OCI archive plus a record of commit,
tree, architecture and digest. On the master push for the same commit,
each leg pushes that archive by digest instead of building. Everything
after the leg (probe, exact-source gate, signed-index check,
attestation) is unchanged.

**Reason and benefit**
The build moves before the merge, where it runs in parallel with the
queue's other checks. The master leg only downloads and pushes.

**Breaking changes**
None. Without `merge_group` events, `find` returns no run and the leg
builds as before.

**Additional context**
This PR does not touch the exact-source gate jobs
(`source_verification`, `publication_verification`, `merge-and-push`),
`scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another
change owns them.

## What Changed

- **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs
only for this repository's master queue. Workflow permissions are `{}`
and the job has `contents: read` only. There are no secrets, no registry
login, no `cache-to`, no `push: true`, no environments, and checkout
uses `persist-credentials: false`. It runs on GitHub-hosted runners, not
the paid AWS fleets. Steps: build `target: production` with the same
build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA,
which becomes the master SHA). Read the public registry cache. Write
`type=oci` to a tar. `docker load` it and run
`scripts/assert-orphan-reaping.sh`. Only then write `record.json` and
upload `merge-group-image-<arch>` (3-day retention).
- **`scripts/merge-group-image.mjs` (new).**
- `find` (master push only): selects a completed, successful run of this
workflow, with event `merge_group`, the same `head_sha`, repository and
head repository both `1170821064`, and a `gh-readonly-queue/master/`
head branch. It needs exactly one unexpired artifact for the
architecture.
- `record` (merge group only): reads the archive's `index.json`, checks
each blob's sha256 and that the image is `linux/<arch>`, and writes
`{sha, tree, arch, digest}`.
- `verify` (master push only): re-reads the archive the same way and
requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse
HEAD^{tree}` of the master checkout, same architecture, and archive
digest = recorded digest.
- **`docker.yml` `build-and-push`.**
  - The leg gets `actions: read` so it can download the artifact.
- New steps run after the registry login: find, download, then promote
(`verify`, then `skopeo copy --all --preserve-digests oci-archive:…
docker://ghcr.io/…@<digest>`). The step sets its `digest` output only
after the push succeeds.
- All three new steps are `continue-on-error`. Buildx, labels, metadata
and "Build and push by digest" run when `steps.promote.outputs.digest ==
''`, so any failure builds from source.
- "Select image digest" feeds the existing PID 1 probe and digest
export.
- **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):**
- Run selection rejects another commit, another event, another workflow,
a fork head, another repository, another queue base, and failed,
cancelled or running builds.
  - Artifact selection.
- Archive digest with and without provenance, a tampered blob, the wrong
architecture, and a real tar read through the system `tar`.
  - Record checks for commit, tree, architecture and digest mismatch.
- The CLI refuses `record` outside `merge_group` and `find`/`verify`
outside a canonical master push.
- Static checks of the merge-group workflow: credentials, triggers,
permissions and runners.
- Static checks of the `docker.yml` wiring: promotion is gated and falls
back, the output is set only after the push, and every build step has
the fallback condition.

### How promotion proves it publishes the verified tree

1. `find` accepts only a run whose `head_sha` equals `github.sha` of the
push, meaning the queue landed the commit it tested (unverified: see
"Merge method" in Risks).
2. `verify` compares the record's tree with the tree of the master
checkout, and its commit with `GITHUB_SHA`.
3. The archive is content-addressed. The digest is recomputed from the
archive blobs and must equal the recorded digest. `skopeo
--preserve-digests` to that digest reference fails rather than produce
other bytes.
4. The leg then runs the PID 1 probe on that registry digest.
`merge-and-push` still signs only an index whose Linux images are the
probed digests (#21013), after the unchanged exact-source gate.
5. The pipeline itself is trusted only as master already ran it. `find`
and `verify` run the copy of `scripts/merge-group-image.mjs` from the
push's `before` commit (`git show "$BEFORE:…"`), not the copy the push
brings. `verify` refuses promotion if anything in `before..HEAD` (every
commit of a multi-PR group, not only the tip) changes
`docker-merge-group.yml`, `docker.yml`, the promotion script or the PID
1 probe. Such a push builds from source. So does the first push after
this PR merges, because its `before` has no verifier.

## Verification

- `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds
a real git history in which a two-PR push changes the pipeline in its
first commit only, and checks that the range check catches it.
- `node --test` on the workflow and contract suites (cloud-readiness,
docker-canary-promotion, post-merge-runner-routing,
lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image,
standard-image-contract, cloud-source-verification, merge-group-image):
492 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass.
- `actionlint` on both workflows: no new findings (the 6 remaining are
pre-existing `ls` and word-splitting notes in `docker.yml`).
- Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI
layouts kept the index digest. It accepted a `docker://…@sha256:`
destination and failed only at the network step.

Cannot be exercised before the queue exists, because no `merge_group`
event fires:
- the merge-group run itself: the hosted-runner build time, disk use
with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a
~5 GB artifact upload;
- the cross-run artifact download on master;
- whether `skopeo` exists on the runs-on AMI and accepts digest
destinations in its installed version.

Each master-side failure falls back to the full build. A merge-group
failure only blocks the queue if its check is required, which is why
that check is required last in the steps below.

## Risks

- **Inert until enabled.** No `merge_group` events fire before the owner
adds a merge queue. On master, `find` lists runs (one API call per leg,
about 1 s), finds none, and the leg builds as today. The only always-on
change is the leg's `actions: read` permission.
- **Build cache freshness.** A promoted master push writes no
`buildcache-<arch>`, and merge-group builds read but never write it.
With every push promoted, the cache ages, so dependency or weekly
tool-layer changes rebuild in each merge-group run until a fallback
build refreshes the cache. A follow-up can refresh the cache from master
without blocking publication.
- **Publication time is still gate-bound.** The exact-source gate (about
13–14 min after push) still decides when the image publishes. See
"Expected timing".
- **Merge method (unverified).** Promotion needs the master commit to be
exactly the commit the queue tested, because the image stamps the commit
SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack
revision). Matching only the tree would publish an image stamped with
another SHA, so this PR does not relax the check. I expect the queue to
fast-forward master to its tested commit. GitHub's merge-queue page does
not state this, and it cannot be tested before the queue exists. If the
queue lands a different SHA, promotion never matches and every push
builds from source. That is safe but inert. Step 4 below checks this.
- **skopeo on the AWS fleet.** The promote step checks for `skopeo`
first and logs "Merge-queue promotion unavailable" if it is missing,
then builds from source. If that notice appears, add skopeo to the fleet
image.
- **Hosted runner capacity.** amd64 merge-group builds run on 4-core
hosted runners, colder and slower than the AWS fleet. That adds queue
latency, not master latency.

### Settings the owner must enable, in order

1. **Prerequisite, a code change, not a setting.** The `master` ruleset
(id 13619726) requires `ci / verify` and `ci / e2e`. Those come from
`pr.yml`, which triggers only on `pull_request`. A merge queue waits for
required checks on the `merge_group` event too, so every entry would
stall until its timeout. First decide, and merge in a separate PR, how
`ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on
the merge-group commit, or a fast pass that relies on the PR-head
result. Today the ruleset does not require up-to-date branches either.
2. Merge this PR. Nothing changes yet.
3. In Settings → Rules → Rulesets → `master` (13619726), add **Require
merge queue**:
   - merge method Squash;
- build concurrency 5, minimum group size 1, maximum group size 5, wait
0 min;
   - "Require all queue entries to pass required checks";
   - status check timeout 60 min.
Do not add the image build to the required checks yet. Promotion is then
opportunistic: if the merge-group build has not finished when master is
pushed, master builds from source.
4. After the first queued merge, confirm promotion can match. The master
commit SHA must equal the `head_sha` of its `Docker merge-queue build`
run, and the next master `Docker` run's "Promote the merge-queue image"
step must succeed. If the SHAs differ, promotion stays inert under this
merge method. Stop here and report it.
5. After about a week of green `Docker merge-queue build` runs, add
`build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64,
ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's
required status checks. Then every queued commit has a verified image
ready to promote.
6. The existing bypass actors (org admins, repository admins,
integration 5534676) can still merge outside the queue. Those pushes
have no merge-group run and build from source.

### Rollback

- Remove the image checks from the required status checks (step 5), or
remove **Require merge queue** (step 3). Master pushes then find no
merge-group run and build from source. Nothing to migrate.
- Code: revert this PR. Old merge-group artifacts expire after 3 days.
Promoted images are ordinary digests under
`ghcr.io/paperclipai/paperclip` and stay valid.

### Expected timing

- Image leg on master: about 3–4 min (download about 5 GB, push the new
layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min
(arm64).
- Push to published: still bounded by the exact-source gate on master,
about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is
the same bound, but with the legs as a second 11-min path and the
serialized-run queue in front.
- If the gate later accepts proof produced in the merge queue (owned by
the gate change), push to published would be about 4–5 min (leg
promotion plus `merge-and-push`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only GitHub API and registry reads).
Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:10:11 -06:00