Files
PaperClipAI/server/src/__tests__/docker-runner-workflows.test.ts
T
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00

87 lines
4.5 KiB
TypeScript

import { readFileSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { describe, expect, it } from "vitest";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const read = (name: string) => readFileSync(path.join(repoRoot, ".github", "workflows", name), "utf8");
const jobsOf = (workflow: string) => Object.fromEntries(workflow.slice(workflow.indexOf("\njobs:\n")).split(/\n(?= {2}[a-z_-]+:\n)/).slice(1)
.map((text) => [/^ {2}([a-z_-]+):/.exec(text)![1], text]));
const permissions = (job: string) =>
[.../\n {4}permissions:\n((?: {6}.+\n)+)/.exec(job)![1].matchAll(/^ {6}([a-z-]+): (\w+)/gm)].map((match) => `${match[1]}: ${match[2]}`);
const steps = (job: string) => job.slice(job.indexOf("\n steps:\n")).split(/\n(?= {6}- )/).slice(1);
const scripts = (text: string) => [...text.matchAll(/^( +)run: (\|\n((?:\1 {2,}.*\n|\n)+)|.+\n)/gm)].map((match) => match[3] ?? match[2]);
const credential = /github\.token|secrets\.|GITHUB_TOKEN|GH_TOKEN|REGISTRY_PASSWORD/;
const registryWrite = /docker\/login-action|skopeo|imagetools create|actions\/attest|--push|push=true|cache-to/;
function expectPinnedAndExpressionFree(text: string) {
const uses = [...text.matchAll(/^\s*(?:- )?uses: (\S+)/gm)].map((match) => match[1]);
expect(uses.length).toBeGreaterThanOrEqual(4);
for (const action of uses) expect(action).toMatch(/^[\w.-]+\/[\w./-]+@[0-9a-f]{40}$/);
const shell = scripts(text);
expect(shell.length).toBeGreaterThanOrEqual(2);
for (const script of shell) expect(script).not.toContain("${{");
}
describe("the report-only Runner image job in docker.yml", () => {
const workflow = read("docker.yml");
const jobs = jobsOf(workflow);
const report = jobs.runner_image_report;
it("holds no credential, can only read the repository, and cannot write to a registry or sign", () => {
expect(workflow).toMatch(/^permissions: \{\}$/m);
expect(permissions(report)).toEqual(["contents: read"]);
expect(report).not.toMatch(credential);
expect(report).not.toMatch(/github\.actor|REGISTRY_USERNAME/);
expect(report).not.toMatch(registryWrite);
expect(report).toContain("persist-credentials: false");
expectPinnedAndExpressionFree(report);
});
it("bounds the step that builds and the lookup", () => {
for (const command of ["runner-image-resolve.mjs inputs", "runner-image-resolve.mjs resolve"]) {
const running = steps(report).filter((step) => step.includes(command));
expect(running).toHaveLength(1);
expect(running[0]).toMatch(/^ {8}timeout-minutes: \d+$/m);
}
});
it("cannot fail the run or feed another job", () => {
expect(report).toContain("\n continue-on-error: true\n");
expect(report).toMatch(/^ {4}timeout-minutes: \d+$/m);
expect(report).not.toMatch(/^ {4}(?:needs|outputs):/m);
expect(workflow).not.toContain("needs.runner_image_report");
for (const [name, job] of Object.entries(jobs)) {
if (name !== "runner_image_report") expect(/^ {4}needs: .*$/m.exec(job)?.[0] ?? "").not.toContain("runner_image_report");
}
expect(/^ {4}if: (.+)$/m.exec(report)![1]).toBe(
"github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'",
);
});
});
describe("the pull-request check of the application image's Runner path", () => {
const workflow = read("docker-app-check.yml");
const jobs = jobsOf(workflow);
it("runs a contributor's code with no credential and no way to publish", () => {
expect(workflow).toMatch(/^permissions: \{\}$/m);
expect(workflow).toMatch(/^on:\n {2}pull_request:\n/m);
expect(workflow).not.toMatch(/pull_request_target|workflow_run|workflow_dispatch|\n {2}push:/);
expect(Object.keys(jobs)).toEqual(["runner_path"]);
expect(permissions(jobs.runner_path)).toEqual(["contents: read"]);
expect(workflow).not.toMatch(credential);
expect(workflow).not.toMatch(registryWrite);
expect(workflow).toContain("persist-credentials: false");
expectPinnedAndExpressionFree(workflow);
});
it("bounds every image build and exports nothing from it", () => {
const building = steps(jobs.runner_path).filter((step) => step.includes("docker buildx build"));
expect(building).toHaveLength(1);
expect(building[0]).toMatch(/^ {8}timeout-minutes: \d+$/m);
expect(building[0].match(/--output /g)).toEqual(["--output "]);
expect(building[0]).toContain("--output type=cacheonly");
});
});