Files
PaperClipAI/scripts/merge-group-image.mjs
T
DottaandClaude Fable 5.2 6826800425 feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to `master` publishes the standard image, and the time
from merge to published image limits how fast a fix reaches users
> - #21096 lets `docker.yml` push an image that the merge queue already
built, but the merge queue is not on
> - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only
`pr.yml` reports them, and only for pull requests
> - A merge queue waits for the same two names on its merge-group
commit. No workflow reports them there, so every queued merge would
stall
> - This pull request adds a workflow that reports both names on
merge-group commits
> - The benefit is that the owner can turn the merge queue on with one
ruleset change

## Linked Issues or Issue Description

Refs #21096

**What existing behavior does this improve?**

The merge-queue image build from #21096. It cannot run until the merge
queue is on, and the merge queue cannot be turned on safely.

**Current behavior**

`ci / verify` and `ci / e2e` come from `pr.yml`, which runs on
`pull_request` only. With a merge queue, GitHub waits for both checks on
the merge-group commit. They never arrive. Each queued pull request
would time out and leave the queue.

**Proposed behavior**

`merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify`
and `ci / e2e`. It runs no tests. It does not check out the queued
commit. It has no token permissions and no secrets.

**Reason and benefit**

A pull request enters the queue only after both checks pass on the pull
request. A queued merge therefore has the same protection as a direct
merge today. The exact-source gate still tests each `master` commit
before publication. Real tests on the combined tree can replace these
jobs later under the same names.

## What Changed

- Added `.github/workflows/merge-queue-checks.yml`. It reports `ci /
verify` and `ci / e2e` on merge-group commits.
- Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It
fails if the names differ from the names that `pr.yml` reports, or if
the workflow gains a checkout, an action, a secret, a token permission
or a self-hosted runner.
- `pr-trusted.yml` now runs that test and
`scripts/merge-group-image.test.mjs` in a step that skips test files an
older checkout does not have. The second test came with #21096 but no CI
step ran it.
- `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a
merge-queue image build that is still queued or running. The queue can
land a commit before its image build ends. Without the wait, the
`master` push always builds from source.
- `doc/RELEASING.md` explains the workflow and lists the ruleset
settings that turn the queue on and off.

## Verification

- `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`:
2 pass.
- `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new
cases for unfinished builds.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`:
pass.
- `actionlint` on both changed workflows: no findings.
- Not verified: no `merge_group` event can fire before the queue is on.
The first queued merge is the live test. `pr.yml` calls
`pr-trusted.yml@master`, so the new CI step runs only after this merges.

Owner steps after merge:

1. In the `master` ruleset, add **Require merge queue**: Squash, build
concurrency 5, group size 1 to 5, status check timeout 60 minutes.
2. After the first queued merge, confirm that the `master` commit SHA
equals the SHA of its merge-group runs.

## Risks

- Low risk while the queue is off. The workflow has no trigger other
than `merge_group`.
- With the queue on, the queue does not test the combined tree. Two pull
requests that pass alone can still break `master` together. This is the
same as today.
- A merge-queue image build that hangs delays the source build by up to
15 minutes. A build that fails does not.
- If GitHub does not accept a job named `ci / verify` for the required
context, queued merges time out. Rollback: remove the merge queue rule.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:18:13 -06:00

204 lines
11 KiB
JavaScript

// Build-before-merge image handoff between the merge queue and master.
//
// A `merge_group` run of docker-merge-group.yml builds the production image for
// the exact merge-group commit with no registry credentials, probes it, and
// uploads an OCI archive plus a record of what it built. When the merge queue
// then fast-forwards master to that commit, docker.yml may push those exact
// bytes instead of rebuilding them. Anything that does not prove the match
// returns no promotion, and docker.yml falls back to its normal build.
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { appendFile, readFile, writeFile } from "node:fs/promises";
import { pathToFileURL } from "node:url";
export const repository = "paperclipai/paperclip";
export const repositoryId = 1170821064;
export const workflowPath = ".github/workflows/docker-merge-group.yml";
export const recordSchema = "paperclip/merge-group-image/v1";
export const artifactName = arch => `merge-group-image-${arch}`;
const architectures = ["amd64", "arm64"];
const indexTypes = ["application/vnd.oci.image.index.v1+json", "application/vnd.docker.distribution.manifest.list.v2+json"];
const manifestTypes = ["application/vnd.oci.image.manifest.v1+json", "application/vnd.docker.distribution.manifest.v2+json"];
// How long a master push waits for a merge-queue build that is still running.
// A build takes about 12 minutes and the exact-source gate about 14.
const findWaitMs = 15 * 60_000;
const findPollMs = 20_000;
const fullSha = /^[a-f0-9]{40}$/;
const digestPattern = /^sha256:[a-f0-9]{64}$/;
const sha256 = bytes => `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
/**
* Choose the merge-queue build a master push may promote. Only a completed,
* successful run of this repository's merge-group workflow, on a merge-queue
* branch for master, for exactly this commit, qualifies.
*
* The commit must match, not only the tree: the image stamps the commit SHA
* (PAPERCLIP_BUILD_COMMIT, build-info.json, the provider pack revision). The
* merge queue lands the commit it tested for every merge method, squash
* included, by fast-forwarding master to it; a push that did not come through
* the queue has no matching run and builds from source.
*/
export function selectVerifiedRun({ sha, runs }) {
assert.match(sha ?? "", fullSha);
const eligible = runs.filter(run => run.head_sha === sha && run.event === "merge_group" && run.path === workflowPath &&
run.repository?.id === repositoryId && run.head_repository?.id === repositoryId &&
typeof run.head_branch === "string" && run.head_branch.startsWith("gh-readonly-queue/master/"));
if (eligible.length === 0) return { reason: "no merge-queue image build exists for this commit" };
const passed = eligible.filter(run => run.status === "completed" && run.conclusion === "success").sort((a, b) => b.id - a.id);
// The queue can land a commit while its image is still building. `pending`
// tells the caller that waiting can still produce a usable run.
if (passed.length === 0 && eligible.some(run => run.status !== "completed")) {
return { pending: true, reason: "the merge-queue image build for this commit did not finish in time" };
}
if (passed.length === 0) return { reason: "the merge-queue image build for this commit did not succeed" };
return { runId: passed[0].id };
}
/** The run's unexpired image artifact for one architecture, if there is exactly one. */
export function selectArtifact({ artifacts, arch }) {
assert.ok(architectures.includes(arch));
const matches = artifacts.filter(item => item.name === artifactName(arch) && !item.expired);
return matches.length === 1 ? { artifactId: matches[0].id } : { reason: `expected one unexpired ${artifactName(arch)} artifact, found ${matches.length}` };
}
/**
* The digest of the image an OCI archive holds, checked against the archive's
* own content-addressed blobs. `readEntry` returns the bytes of a tar entry.
*/
export function archiveImageDigest({ readEntry, arch }) {
assert.ok(architectures.includes(arch));
const layout = JSON.parse(readEntry("index.json").toString("utf8"));
assert.equal(layout.manifests?.length, 1, "The OCI archive must hold exactly one image");
const top = layout.manifests[0];
assert.match(top.digest ?? "", digestPattern);
const blob = name => {
const bytes = readEntry(`blobs/sha256/${name.slice("sha256:".length)}`);
assert.equal(sha256(bytes), name, `Archive blob ${name} does not match its digest`);
return JSON.parse(bytes.toString("utf8"));
};
const image = blob(top.digest);
const platformImages = indexTypes.includes(image.mediaType)
? (image.manifests ?? []).filter(item => item.platform?.os === "linux")
: [{ digest: top.digest, platform: undefined }];
assert.equal(platformImages.length, 1, "The OCI archive must hold exactly one Linux image");
const manifest = platformImages[0].digest === top.digest ? image : blob(platformImages[0].digest);
assert.ok(manifestTypes.includes(manifest.mediaType), "The OCI archive does not hold an image manifest");
const config = blob(manifest.config.digest);
assert.equal(config.os, "linux");
assert.equal(config.architecture, arch, `The OCI archive holds a ${config.architecture} image, not ${arch}`);
return top.digest;
}
/** What the merge-group build attests about the archive it uploads. */
export function createRecord({ sha, tree, arch, digest }) {
assert.match(sha ?? "", fullSha);
assert.match(tree ?? "", fullSha);
assert.ok(architectures.includes(arch));
assert.match(digest ?? "", digestPattern);
return { schema: recordSchema, repository, sha, tree, arch, digest };
}
/**
* Prove the archive is the build of the tree this master push publishes. The
* commit SHA covers its tree; the tree is compared too, so a record can never
* vouch for different content under a reused name.
*/
export function verifyRecord({ record, sha, tree, arch, archiveDigest }) {
assert.equal(record?.schema, recordSchema, "Unknown merge-group image record");
assert.equal(record.repository, repository);
assert.equal(record.sha, sha, "The merge-group build is for a different commit");
assert.equal(record.tree, tree, "The merge-group build is for a different tree");
assert.equal(record.arch, arch, "The merge-group build is for a different architecture");
assert.match(record.digest ?? "", digestPattern);
assert.equal(archiveDigest, record.digest, "The archive is not the image the merge-group build recorded");
return record.digest;
}
/**
* Files that produce, promote or vouch for a merge-group image. The merge-group
* run uses the queued commits' own copies, so a push that changes any of them
* is built from source on master instead: promotion then trusts only a
* pipeline that master already ran before this push. docker.yml runs this
* check from the copy of this script in the push's `before` commit, so a push
* cannot remove the check it is subject to.
*/
export const pipelineFiles = [workflowPath, ".github/workflows/docker.yml", "scripts/merge-group-image.mjs", "scripts/assert-orphan-reaping.sh"];
export function assertPipelineUnchanged(changedFiles) {
const changed = changedFiles.filter(file => pipelineFiles.includes(file));
assert.equal(changed.length, 0, `This push changes the merge-group image pipeline (${changed.join(", ")})`);
}
/** Pipeline files changed by everything this push landed: `since` (the push's `before`) to HEAD. */
export function changedSince(since, cwd = process.cwd()) {
assert.match(since ?? "", fullSha, "--since must be the push's full `before` SHA");
// Throws unless `since` is an ancestor: a push is a fast-forward of master.
execFileSync("git", ["merge-base", "--is-ancestor", since, "HEAD"], { cwd, stdio: "ignore" });
return execFileSync("git", ["diff", "--name-only", since, "HEAD", "--", ...pipelineFiles], { cwd, encoding: "utf8" }).split("\n").filter(Boolean);
}
const tarReader = archive => entry => execFileSync("tar", ["-xOf", archive, entry], { maxBuffer: 16 * 1024 * 1024 });
const gitTree = () => execFileSync("git", ["rev-parse", "HEAD^{tree}"], { encoding: "utf8" }).trim();
const option = name => {
const index = process.argv.indexOf(`--${name}`);
const value = index > 0 ? process.argv[index + 1] : undefined;
assert.ok(value, `--${name} is required`);
return value;
};
const output = async (key, value) => {
if (process.env.GITHUB_OUTPUT) await appendFile(process.env.GITHUB_OUTPUT, `${key}=${value}\n`);
console.log(`${key}=${value}`);
};
async function main(command) {
const sha = process.env.GITHUB_SHA;
const arch = option("arch");
if (command === "record") {
// Runs in the credential-free merge-group build.
assert.equal(process.env.GITHUB_EVENT_NAME, "merge_group", "Records are written only by merge-group builds");
const digest = archiveImageDigest({ readEntry: tarReader(option("archive")), arch });
await writeFile(option("out"), `${JSON.stringify(createRecord({ sha, tree: gitTree(), arch, digest }), null, 2)}\n`);
return output("digest", digest);
}
assert.ok(process.env.GITHUB_REPOSITORY === repository && process.env.GITHUB_REF === "refs/heads/master" &&
process.env.GITHUB_EVENT_NAME === "push", "Promotion runs only for canonical master pushes");
if (command === "find") {
const token = process.env.GITHUB_TOKEN;
assert.ok(token, "GITHUB_TOKEN is required");
const api = async path => {
const response = await fetch(`https://api.github.com/repos/${repository}${path}`, { redirect: "error",
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });
assert.ok(response.ok, `GitHub API ${path}: HTTP ${response.status}`);
return response.json();
};
// Publication waits for the exact-source gate in any case, so waiting here
// for an unfinished build costs nothing while the build is the shorter of
// the two. Past the deadline the push builds from source.
const deadline = Date.now() + findWaitMs;
let run;
for (;;) {
const { workflow_runs: runs } = await api(`/actions/workflows/docker-merge-group.yml/runs?event=merge_group&head_sha=${sha}&per_page=20`);
run = selectVerifiedRun({ sha, runs });
if (!run.pending || Date.now() >= deadline) break;
console.log("The merge-queue image build for this commit is still running. Waiting.");
await new Promise(resolve => setTimeout(resolve, findPollMs));
}
if (!run.runId) { console.log(`No promotion: ${run.reason}. Building from source.`); return; }
const { artifacts } = await api(`/actions/runs/${run.runId}/artifacts?per_page=100`);
const artifact = selectArtifact({ artifacts, arch });
if (!artifact.artifactId) { console.log(`No promotion: ${artifact.reason}. Building from source.`); return; }
return output("run_id", run.runId);
}
if (command === "verify") {
const directory = option("dir");
assertPipelineUnchanged(changedSince(option("since")));
const record = JSON.parse(await readFile(`${directory}/record.json`, "utf8"));
const archiveDigest = archiveImageDigest({ readEntry: tarReader(`${directory}/image.tar`), arch });
return output("digest", verifyRecord({ record, sha, tree: gitTree(), arch, archiveDigest }));
}
throw new Error(`Unknown command: ${command}`);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) await main(process.argv[2]);