Commit Graph
11 Commits
Author SHA1 Message Date
Nicky LeachandClaude Fable 5.2 647641a99b feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as an attested image, once for each set of its inputs
> - Since #21332 the application image takes its Runner only from the
`runner-artifacts` stage, and a build can replace that stage with the
published image
> - No workflow does that yet, so every application image still builds
its own Runner, and nothing on an application image says which Runner it
has
> - A build that copies the published Runner must also build the
lockfile that Runner was built from. The workflow refreshes the lockfile
before every build, and the server build then refuses the published
Runner
> - This pull request lets `docker.yml` copy the published Runner image
and label the application image with it, behind a repository variable
that is off, and makes that build use the committed lockfile
> - The benefit is one Runner for all images of the same inputs, named
on the image, with a switch that goes back to today's builds without a
revert

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), #21308 (the server reads a pack from its
Runner image), and #21331, #21332, #21333 (the application image takes
its Runner from `runner-artifacts`). No issue exists for this change.

**Problem or motivation**

`docker.yml` builds the application image on `master` and on tags. Each
build makes its own Runner. The report-only job that runs the resolver
on `master` feeds no build. Downstream image composers cannot tell from
an application image which Runner it has.

There is a second problem, which only shows when a build copies the
published Runner. The Runner's build stamp covers the whole
`pnpm-lock.yaml`. The step that vendors the Runner into the server fails
when the stamp does not match the tree. `docker.yml` refreshes
`pnpm-lock.yaml` on the runner before every build. The refresh rewrites
the file whenever the manifest of any package is ahead of it, also a
package the Runner does not read. On the tree of this pull request it
rewrites the file even for a pristine checkout: the registry changed the
deprecation text of one package. The published Runner was built from the
committed file, so the image build would fail.

**Proposed solution**

One job resolves the Runner once for the run. Both architecture builds
need it and add its answer to their build: a build context that replaces
the `runner-artifacts` stage, two labels, and the build argument with
which the build checks the Runner. A build that copies the published
Runner does not refresh the lockfile: it builds the committed tree, the
tree the Runner was published for. The job runs only when the repository
variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull
request merges with the variable unset, so no image changes when it
merges.

**Alternatives considered**

- Resolve in each architecture build. The two builds could then
disagree, and one image index would hold two Runners.
- Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for
an inline build.
- For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over
`/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step
reads the file: the two vendor checks are its only readers in `build`,
`production` copies it and takes `node_modules` from `deps`, both
`cloud-*` stages install with `--ignore-workspace --no-lockfile`, and
the server reads only a plugin's own lockfile and the pack's own
lockfile at run time. But the image would then ship a lockfile that its
`node_modules` did not come from, the copy changes that file's layer for
every inline build, and each `Dockerfile` edit makes a new Runner image.
- For the lockfile, bind-mount the Runner image's lockfile over the file
for the two steps that run the vendor check. This also edits the
`Dockerfile`, and it makes the lockfile part of the check always true
for a published Runner.
- For the lockfile, let the resolver refresh and compare. The answer
would depend on the registry at two different moments.
- Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check
as strict as it is, and builds exactly the tree the inputs digest was
computed from.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on `runner/app-image-consume`. Merge in this
order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. `runner/app-image-consume`: the wait and the `unsupported` answer of
the resolver.
3. This pull request.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `.github/workflows/docker.yml`: the job `runner_image` replaces
`runner_image_report`.
- It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in
this repository. A fork never runs it.
- Steps: `inputs` with no credential; `resolve`; then `gh attestation
verify` of the full index against the signer `runner-image.yml` on
`refs/heads/master`.
- For a `master` push, `resolve` runs with `--wait` and the job token.
For a tag or a manual run it does not wait and receives no token.
- Only the verification step writes the job's outputs. An image that
does not verify fails that step with a warning and writes no output. The
job is `continue-on-error`: the job is red, the run is not, and the
builds are inline.
- Permissions: `contents: read`, `actions: read`, `attestations: read`.
No registry login, no write.
- Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the
verification, 70 for the job. The wait of 40 minutes ends inside them.
- `build-and-push` needs `runner_image` and adds its outputs to the
build: `build-contexts`, one more line of `build-args`
(`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`.
With empty outputs these are the values the build had before.
- The step "Refresh lockfile for Docker build context" runs only when
the answer is not `published`.
- The step that looks for a merge-queue image of the commit runs only
when the answer is not `published`. The two steps after it already
depend on its result.
- Labels on an image that copied the published Runner:
`io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full
index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An
image without them made its own Runner.
- The `Dockerfile` does not change. The Runner image digest does not
move.
- Tests that pin lines this changes follow:
`scripts/merge-group-image.test.mjs` (the condition of the merge-queue
lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build
job), and `docker-build-stamp.test.ts` (no workflow names a build
context; `docker.yml` now names exactly one).
- `server/src/__tests__/docker-runner-workflows.test.ts`: the
report-only tests give way to tests of the new job.
- `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of
the lockfile case.
- `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application
image gets its Runner, the labels and what their absence means, the
variable, the wait, the verification, the lockfile rule, the merge-queue
rule and the rollback.

## Verification

### Real builds

`docker buildx build --target build --platform linux/arm64 --output
type=cacheonly`, with a `docker-container` builder made for this test.
The Runner image is the publication form
(`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`)
built from a clean export of this branch, exported as an OCI layout:
inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application
builds receive `--build-context
runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command
is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm
install --resolution-only --ignore-scripts --no-frozen-lockfile`.

| Build context | Result |
| --- | --- |
| The committed tree | Passes in 78 s. Both vendor checks: `reusing the
current Runner build (inputs unchanged …; 1333 files and 1860 outputs
checked)` |
| `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh
command (what the workflow did before this change) | Fails at `RUN node
server/scripts/prepare-runner-vendor.mjs`:
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild
this step needs (inputs changed since the recorded build; 1333 files
checked)` |
| The committed tree, then only the refresh command | Fails the same
way. The refresh changed only the `deprecated` text of one package |
| The same `ui/package.json` change with no refresh (what the workflow
does now for a `published` answer) | Passes in 66 s. `deps` resolves the
manifest inside the image; both vendor checks reuse the Runner |
| One line added to `packages/paperclip-runner/src/index.ts` | Fails at
the vendor step, as before: a Runner of other inputs is still refused |

Lockfile behind the Runner's own manifests
(`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` →
`^1.9.0`), no build context:

| Step | Result |
| --- | --- |
| `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no
digest; `inline (lock-behind)` |
| Build without the refresh | Fails in 4 s at the `runner-lockfile`
stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is
installed from …` |
| Build after the refresh command (what the workflow does for an
`inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes,
the build makes its own Runner with its pack, and both vendor checks
reuse it |

### Tests

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch:

- `pnpm exec vitest run` on `docker-runner-workflows`,
`runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`,
`container-init-reaping`, `cloud-image-bundled-plugins`,
`docker-image-layer-reuse`, `cloud-image-sentry`,
`runner-lockfile-guards` and `runner-image-identity` in
`server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files
pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test files typecheck with the options of
`server/src/__tests__/tsconfig.json`.

What the workflow tests hold:

- Variable off: the job's condition is the variable and the repository,
with no alternative. With empty outputs, the build step has exactly the
3 build arguments, 0 build contexts and 1 label entry it had before, and
the refresh step runs.
- Published: both architecture builds are one job that reads only the
outputs of `runner_image`, with no architecture in those lines.
- The verification step runs as real shell against a stub `gh`. When
`gh` succeeds it writes the context, the labels and the build argument.
When `gh` fails the step exits 1, writes no output and prints the
warning. The step has no `continue-on-error`; the job has. The condition
of the build job does not name the resolver job and has no `success()`.
- The lookup step runs as real shell against a stub `node`. It passes
`--wait` only when the event is a `master` push.
- Credentials: only the lookup and the verification steps have a token.
The steps that build have no `env`.
- The refresh step has the condition `needs.runner_image.outputs.mode !=
'published'`, and it is the only `pnpm install` of the build job.
- The fixture test gives the vendor step a Runner that another tree
built for the committed lockfile. The step reuses it. After a change of
one specifier of another package in the lockfile the step fails and
names the lockfile. With the committed lockfile back it reuses the
Runner again.
- The merge-queue workflow and the pull-request check do not name the
resolver, the variable, a build context or a Runner label.

Read-only checks against the real services:

- `gh attestation verify
oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo
paperclipai/paperclip --signer-workflow
paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref
refs/heads/master`: exit 0 for the full index, and exit 0 for the slim
index `sha256:69ab2001…69b3`.
- The same command with `--signer-workflow …/docker.yml`: exit 1 for
both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo
cli/cli`: exit 1. For a digest that does not exist: exit 1. With no
token: exit 4.
- The parsed YAML of `docker.yml` differs from its base only in the
resolver job, the `needs` of the build job, the conditions of the
refresh step and of the merge-queue lookup, and the three build inputs.

Never run:

- `docker.yml`. Only GitHub can run it, and it has no pull-request
trigger.
- A build with the variable on. The first one is the first run after a
maintainer sets the variable.
- A build that pulls the published Runner from the registry as
`docker-image://`. The real builds used the same image form from a local
OCI layout.
- An `amd64` build.
- The `production` and `cloud` targets. The real builds stopped at
`build`, the last stage that checks the Runner.

## Risks

- With the variable on, an application build on `master` starts after
the resolver job. That is 2 to 3 minutes when the image exists, and up
to 40 minutes from the push when the commit's own Runner publication is
in progress. `docker.yml` holds one concurrency slot for each ref, so a
long wait can make more `master` commits lose their image build to a
newer push.
- A build that copies the published Runner ships the committed
`pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of
other packages inside the image. A local `docker build` has always done
this; a CI image did not. Nothing in the image reads that file.
- Any other build that gives the `Dockerfile` a published Runner and a
refreshed lockfile fails at the vendor step. That is the intended answer
to a tree the Runner was not built for, and the message names the
lockfile.
- The pack of an image that copied the published Runner names the commit
that published the Runner (`runnerSourceRevision`), which can be older
than the commit of the image. A consumer that compares the two must
accept that before the variable is set.
- A failed verification makes the resolver job red on a green run. That
is on purpose: it means a published tag whose attestation does not
verify.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- With the variable off there is no report of the resolver on `master`
any more: the job that replaced the report-only job does not run.
- Rollback: set the variable to another value or delete it. The next run
builds inline, with the refresh. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds, read-only
calls to the GitHub API and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:47:20 -08:00
Nicky LeachandClaude Fable 5.2 e824c5cc16 feat(ci): let the Runner image resolver wait for the publication of a master push (#21339)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, once for
each set of its inputs
> - The application image will copy that Runner instead of making its
own. `scripts/runner-image-resolve.mjs` tells a build whether the Runner
of its inputs is published
> - On a `master` push, the commit's own run of `runner-image.yml` is
often still publishing when the application build asks. The resolver
answers once, so that build is told `absent` for an image that exists
some minutes later
> - The resolver also cannot say that a tree is not safe to give a
published Runner to. A `Dockerfile` that does not check the Runner it
receives builds with any Runner
> - This pull request adds an opt-in wait for the commit's own
publication run, and an `unsupported` answer for a `Dockerfile` that
cannot check its Runner
> - The benefit is that the next pull request can let `docker.yml` copy
the published Runner without a blind timeout and without a label that is
not true

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), and #21331, #21332, #21333 (the
application image takes its Runner from `runner-artifacts`). No issue
exists for this change.

**Problem or motivation**

A `master` push starts `runner-image.yml` and `docker.yml` at the same
time. Publication takes about 30 minutes from the push, and several of
those minutes are runner queue time. An application build that asks the
resolver at once gets `absent` for every commit that changes the
Runner's inputs, and for commits that follow one closely. A fixed poll
of the tag has the opposite fault: when no image will ever come (the
lockfile is behind, or the run failed), every build waits the full
timeout.

**Proposed solution**

`resolve --wait` waits for a run, not for a time. It finds the runs of
`runner-image.yml` for the push of the commit to `master`, and it looks
up the image again when they end. No caller uses the flag in this pull
request, so no workflow changes behavior.

**Alternatives considered**

- Poll the tag until a timeout. Every build with no image to come waits
the full timeout.
- Call `runner-image.yml` from `docker.yml` as a reusable workflow.
Publication would then share the one concurrency slot of `docker.yml`.
- Wait until the run completes and only then look. The run goes on after
publication (it writes the build cache), so the wait would last longer
than needed. The command looks up the image while a run is in progress.
- Verify the attestation in the resolver. The workflow that uses the
answer does it, in a step of its own, in the next pull request.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on #21333. Merge in this order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. This pull request: the wait and the `unsupported` answer of the
resolver.
3. `runner/app-image-consume-docker`: `docker.yml` copies the published
Runner, behind a repository variable.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `scripts/runner-image-resolve.mjs`
- `resolve --dir <directory> --wait`. When the first answer is `absent`
or `incomplete`, the command lists the runs of `runner-image.yml` for
the commit (Actions API, `GITHUB_SHA`, `GITHUB_TOKEN`) and looks again
when they end.
- Only a run of the `push` event on `master` in this repository counts.
A pull-request, scheduled or manual run of the same commit is never
waited for. The request filters by event and branch, and the script
checks each run again.
- A run that fails, is cancelled or publishes nothing ends the wait when
it ends. With several runs for one commit, the command waits for all of
them.
- While a run is in progress, the command looks up the image every 30
seconds, so the wait ends at publication.
  - A run that is not listed yet is looked for during 2 minutes.
- The limit is 40 minutes from the push: `RUNNER_IMAGE_WAIT_MINUTES`,
the one constant. The push time is the creation time of the oldest run
of the commit, so a build that starts late waits less.
- Three failed API requests in a row end the wait. No token or no full
commit SHA means no wait.
- After every ending the command looks up the image one last time. No
ending is an error. Without an image the answer is `inline`, and the
command prints a notice with the reason.
- `inputs` reads the `Dockerfile` first. Without a line `ARG
PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` it runs no build, and `resolve`
answers `inline` with the reason `unsupported`.
- `resolve` reports `build_args`: the value of that argument for a
`published` answer.
- The wait and the list of runs take the clock, the sleep and `fetch` as
arguments, so the tests use no network and do not sleep.
- `server/src/__tests__/runner-image-resolve.test.ts`: every outcome of
the wait and the answer that follows it, the run filter, the limit
measured from the push, the API failures, and the command line.
- `doc/runner-image-contract.md`: a "Waiting for a publication" section,
`build_args`, and the `unsupported` answer.

## Verification

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch on top of #21333:

- `pnpm exec vitest run` on `runner-image-resolve`,
`docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse`,
`cloud-image-sentry`, `runner-lockfile-guards` and
`runner-image-identity` in `server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 218 tests in 11 files
pass on Linux; on macOS 217 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test file typechecks with the options of
`server/src/__tests__/tsconfig.json`.

Read-only checks against the real services:

- The lookup, with no credential, answered `published` for the inputs
`d4df566f…af00` and `0e6662ad…d1ca`, with the index digests that
`master` published, and `absent` for a digest that does not exist.
- The real answer of the Actions API for the `runner-image.yml` runs of
four `master` commits went through the wait, with a clock that does not
sleep. Three completed runs ended it at once. A queued run ended it at
the limit of 40 minutes from its creation time. A commit with no run
ended it after the 2 minutes for a run that is not listed.
- Of the 60 newest runs of `runner-image.yml`, the filter accepted the
19 `push` runs on `master` and none of the 41 pull-request runs.
- `runner-image-resolve.mjs inputs` ran on a clean export of the tree of
the next pull request, with a `docker-container` builder. The answer was
`lock: current` and the inputs digest `c4f9d089…34f0`. The `Dockerfile`
of #21332 declares the argument, so the answer was not `unsupported`.
The same command on a tree with one changed range in
`packages/paperclip-runner/package.json` answered `lock: behind`, and
`resolve` then answered `inline (lock-behind)`.

Never run:

- `resolve --wait` in a workflow. No workflow calls it in this pull
request.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- `resolve` now prints a notice for every `inline` answer. The
report-only job in `docker.yml` therefore shows one notice on each
`master` push whose Runner image is not published yet, until the next
pull request replaces the job. Nothing reads that job.
- `inputs` answers `unsupported` for a tree whose `Dockerfile` does not
declare `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` at the start of a
line. #21332 adds that line; a later edit that moves or renames it turns
every answer to `inline`.
- The wait is bounded by time from the push, not by its own duration. A
build that starts more than 40 minutes after the push looks once and
does not wait.
- A registry that does not answer adds at most about 3 minutes to one
lookup, because the registry client retries. A caller must give the step
more than 40 minutes. The next pull request gives it 55.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only calls to the GitHub API
and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:33:40 -08:00
Nicky LeachandClaude Fable 5.2 999709b0a6 fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The application image now takes its Runner from the
`runner-artifacts` stages, which install with a frozen lockfile
> - A contributor cannot commit `pnpm-lock.yaml`. After a change to a
manifest of the Runner, the lockfile on `master` is behind until the
refresh merges
> - In that time a plain `docker build` fails late, inside a long step,
with a message that is easy to miss. And a beta or a stable release can
be cut from such a commit, because the release workflows refresh the
lockfile for their own build
> - This pull request adds an early check with a clear instruction to
the image build, and the same check to the beta and the stable release
> - The benefit is a failure in seconds that says what to run, and beta
and stable tags that always build from source

## Linked Issues or Issue Description

This is the third part of a chain. The first part is #21331. The second
part takes the Runner of the application image from the
`runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and
its lockfile rule). No issue exists for this change.

**What happened?**

A tree with a lockfile behind the manifests of the Runner fails `docker
build` after the daemon compile (169 s in the test). The error is
`ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the
`Dockerfile`.

**Expected behavior**

The build fails within seconds of the check running. The message names
the files and the command to run. The application's dependency install
starts from the check, so it cannot start before the check passes.
Stages that do not depend on it (the Runner's toolchain download and
daemon compile, and the tool layer of `production`) can already be
running, and the failed check cancels them. A beta or a stable release
from such a commit does not start.

How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a
commit whose lockfile is behind the manifests of the Runner (2026-09-01,
09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta
is the promotion of a nightly, so a beta can be cut from such a commit.

**Steps to reproduce**

1. Change a dependency range in
`packages/paperclip-runner/package.json`. Do not refresh
`pnpm-lock.yaml`.
2. Run `docker build .`.

**Paperclip version or commit**

`master` with the previous pull request of this chain.

**Deployment mode**

Not applicable. This is the image build and the release tooling.

## What Changed

- `Dockerfile`: a `runner-lockfile` stage copies only the manifests the
Runner build installs from, the lockfile and the patches. It runs the
frozen, filtered, offline lockfile check, with the arguments of
`runnerLockState`. `deps` starts from it. For a stale lockfile it prints
the files and `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile`. For any other failure it prints the output of
pnpm and says so.
- `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs
of a commit with `git archive` and gives them to `runnerLockState`
(`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind,
2 for an error.
- `scripts/release.sh` and `scripts/release-lib.sh`:
`require_runner_lockfile_current` runs for the `beta` and the `stable`
channel, after `--print-version` exits and before any build. It runs in
a dry run too. It does not run for `canary` or `nightly`.
- Both beta paths end in the `publish_beta` job: the promotion of a
nightly, and a `candidate/beta-*` branch. That job checks out the commit
that `select_beta` chose and runs `release.sh beta` there. So the check
reads the committed files of the commit the beta is built from. A beta
is refused before its soak starts. The stable that promotes it is
checked again.
- Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18
tests). Three of them pin the checkout of `publish_beta`,
`preview_stable` and `publish_stable` to the selected commit. Two tests
plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs`
cover the beta, the candidate beta and the stable path.
- Docs: `doc/DOCKER.md`, `doc/RELEASING.md`,
`doc/runner-image-contract.md`.

## Verification

Builds: `linux/arm64`, local. They ran before the rebase onto `master`,
except the last one. `master` changed neither the `Dockerfile` nor a
manifest in between.

- A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest
of the Runner: `docker buildx build --target build` fails after 6 s at
`runner-lockfile`, with the message.
- The same tree after `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile` on the host, which is what the image workflows
run: `--target build` builds. The check passes, the Runner stages
install, and the vendor step logs `reusing the current Runner build` in
both of its runs.
- After the rebase: `build` and `runner-provider-pack` built from this
branch without a build commit. The check passed.
- The release check with the real pnpm: `behind` for the first tree,
`current` for the second, exit 0 for this checkout.
- `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with
the other changed server tests, on macOS and in `node:24.22.3-bookworm`
(249 pass in 11 files on Linux). The test file typechecks.
- `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14
pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of
its tests fail on `master` too. `release-verify-workflow.test.mjs` and
`acpx-patch-packaging.test.mjs`, which read the release scripts: pass.
- `docker buildx build --check`: no warning.
- The Docker builds above ran before the second part of the chain got
its expectation step and before the beta guard. The expectation step was
built on the second part alone. This branch as it is now was not built
again; its tests ran.
- Not run: the release workflow, any `linux/amd64` build, the full test
suite. Only CI and a release dry run can prove the release path.

## Risks

- **A beta can be refused.** When the newest nightly is on a commit
whose lockfile is behind, `release.sh beta` stops, in the dry run and in
the real run. The operator must promote a nightly whose commit includes
the lockfile refresh, or cut a candidate branch that includes it. `node
scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the
answer before the dispatch.
- **A stable can be refused** only when its beta was cut before this
change, or with release tooling that does not have the check. Promotions
run the release tooling of the source commit.
- The check runs in `release.sh`. In the beta workflow it runs in
`publish_beta`, after the approval of the `npm-beta` environment and
after the candidate verification. In the stable workflow it runs after
verification, not in the `preflight_stable` job. The selection jobs have
no pnpm.
- `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before
its first build step. The workflow has them.
- `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI
command today. The Vitest file carries the enforced tests.
- The image build has one more small stage. It reads only manifests, so
a source change does not run it again.
- The digest of the Runner image changes again, because its identity
covers the whole `Dockerfile`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:17:15 -08:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
Nicky LeachandClaude Fable 5.2 503a736195 feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `.github/workflows/runner-image.yml` builds the Runner alone and
publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`,
one time for each set of inputs
> - The build used no cache, so every run compiled the daemon again. The
first publication took 25m46s from push to tag
> - The toolchain and the daemon change only with the Rust sources. Most
commits that change the Runner's inputs do not touch them
> - This pull request lets the build legs read a registry build cache
for those stages. Only a new job can write the cache, and only for a
`master` push
> - A weekly run builds with no cache, so the cache cannot hide a broken
build for long
> - The benefit is about 3 minutes less on each build leg when the Rust
sources did not change, with no change to the published image

## Linked Issues or Issue Description

Refs #21230 (publication of the Runner image), #21229 and #21204 (the
build this change makes faster). No issue exists for this change.

**What existing behavior does this improve?**

The Runner image build in `.github/workflows/runner-image.yml`. It
builds the same image in less time.

**Current behavior**

Each build leg starts from an empty builder. It installs the Rust
toolchain and compiles the daemon in every run. In the first `master`
publication (run 43365670428) the amd64 leg spent 7m22s in the build
step. The toolchain and the cargo build took 3m26s of that.

**Proposed behavior**

The build legs read
`ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages
up to the compiled daemon. A new `cache` job writes that cache after a
publication. A scheduled run builds with no cache one time each week.

**Reason and benefit**

The stages before the last one read the Rust sources, the protocol files
and the toolchain pin. In the last 30 days, 438 `master` commits changed
an input of the image, and at most 181 of them changed a file those
stages read. For the other commits, and for each pull request that does
not touch those files, the build step is about 3 minutes shorter on each
leg.

**Breaking changes**

None. The `Dockerfile` and each file in the image identity are
unchanged. The published digest for a set of inputs does not change.

This is part 4 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build (this pull
request)

This pull request uses no code from parts 1 to 3. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- The build legs pass `--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>`
to the first build command. They do not log in. Their permissions stay
`contents: read` and `packages: read`.
- A new `cache` job writes the cache with `--cache-to
type=registry,...,mode=max` and `--output type=cacheonly`. It runs only
for a `master` push in the canonical repository, for inputs that had no
image. Its permissions are `contents: read` and `packages: write`. It
has no `id-token` and no `attestations` permission.
- The `cache` job needs the `publish` job, and no job needs the `cache`
job. A cache write cannot delay or fail a publication.
- The `cache` job builds the target `runner-artifacts-runnerd` and no
later stage. The last stage takes the commit as a build argument, so no
other commit can reuse it. The install and its dependency lifecycle
scripts are in that stage, so the job with the write token does not run
them.
- The `cache` job has one concurrency group for each architecture, a job
time limit and a step time limit.
- A `schedule` trigger starts a weekly build. It passes `--no-cache` to
the first command and reads no cache. It runs all checks and publishes
nothing. A fork's schedule runs nothing.
- `server/src/__tests__/runner-image-workflow.test.ts` pins the new
security properties: which jobs hold `packages: write`, that the build
legs have no credential and cannot write the cache, the conditions of
the `cache` job, that the `publish` job does not depend on it, and that
a scheduled run cannot publish or write the cache.
- `doc/runner-image-contract.md` has a "Build cache" section.

## Verification

Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm`
container (Linux arm64, Node 24.22.3):

- `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts
src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55
tests pass on both.
- `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass
on both. The container needs `jq`.
- `pnpm verify` exits 2 for this change, because a workflow file is a
global CI input. Required CI supplies the broad coverage.
- Each new assertion fails when the workflow breaks its rule. I checked
seven such edits, for example: the `publish` job needs the `cache` job,
the `cache` job gets `id-token: write`, a build leg gets `--cache-to`.

Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2
(the version the hosted runners use), and an empty builder for each
build:

| Build of both variants | Time | Result |
| --- | --- | --- |
| No cache, cache ref does not exist | 260 s | An importer error in the
log, exit 0 |
| One changed TypeScript file, with the cache | 149 s | Toolchain and
cargo steps cached |
| Unchanged inputs and a new commit, with the cache | 129 s | Toolchain
and cargo steps cached |
| Each stage in the cache (a whole-build cache, which this change does
not write) | 16 s | Both OCI archives complete |
| `--no-cache` on the first command | 272 s | No cached step in the
first command. The second command reuses all of it |

- The images built from the cache pass `runner-image-layout.mjs check`
and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256
as the daemon of a cold build.
- A cache ref that does not exist, a host that refuses the connection, a
name that does not resolve, and the real
`ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not
exist yet) each give an importer error in the log and a complete build
with exit 0.
- The `cache` job command on an empty builder with the cache present
takes 6 s. It downloads no layer.

Not tested before merge:

- The `cache` job. It runs only on `master`.
- The scheduled run. It starts only from the default branch.
- The cache package. It does not exist. I did not test a private
package, which answers an anonymous read with an authorization error. I
expect the same importer error and a cold build.
- A fully cached build on the hosted runners. The local test used the
same BuildKit version and a different buildx version (0.37.0, and the
runners have 0.38.1).

## Risks

- **A second job can write packages.** Before this change, only the
`publish` job held `packages: write`, and it runs no build. Now the
`cache` job holds it too, and it runs BuildKit on the repository's
`Dockerfile`. GitHub cannot limit that token to one package, so it can
also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are:
the job runs only for a `master` push in the canonical repository, the
token goes only to the registry login, the build steps run inside
BuildKit without it, the job does not run the install, and the job
cannot sign. An image that it wrote has no attestation.
- **Part of a published image can come from an earlier run.** The daemon
in a published image can come from a step that an earlier `master` run
compiled and cached. The attestation names the run that published, not
the run that compiled. BuildKit reuses a step only for the same
instruction and the same file content, but a party that can write the
cache package can replace what a later build reuses. Write access to the
cache package must stay as narrow as write access to the image package.
- **A cached download is not repeated.** The toolchain installer, the
toolchain and the crates are not downloaded again until the inputs of
their step change. The weekly cold build finds a download that is gone.
It does not compare its output with the cache.
- **A registry failure after the cache import fails the leg.** A cache
that cannot be read at the start is not a failure. A layer download that
fails later is a failure. Run the job again.
- **The cache package starts private.** GitHub creates it on the first
write. A maintainer must make it public one time. Until then each build
runs cold and nothing fails.
- **The first cache write waits for an input change.** The merge of this
pull request changes no input of the image, so its `master` run builds
nothing and writes no cache.
- **A cancelled cache job marks its run as cancelled.** When three cache
writes for one architecture wait at the same time, GitHub cancels the
middle one. The image of that run is published.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code.
Tool use and shell execution. It wrote the change, ran the local builds
and tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:46:15 -08:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
Nicky LeachandClaude Fable 5.2 f79e2c5d84 feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `master` now publishes the Runner alone as the image
`ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner
inputs (#21204, #21229, #21230)
> - The application image still compiles its own Runner on every build.
The next steps make it copy the published Runner instead
> - Before a build can do that, it must answer one question the same way
as every other build: "does a published Runner exist for exactly this
tree?" It must also fail clearly when the Runner it was given does not
match the tree
> - This pull request adds the script that answers the question, and a
mode of the Runner vendor step that fails instead of rebuilding
> - The benefit is that the later change to the image build is small,
and that a wrong or unreachable answer can only mean "build the Runner
yourself"

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

Three builders make images that contain the Runner: `docker.yml`, the
merge-queue build and the release preview build. Each one would have to
repeat three steps to find the published Runner of its tree: check the
lockfile, compute the inputs digest in the build, and look up
`inputs-<digest>`. Three copies can decide differently. A build that
copies a Runner built elsewhere also has no Rust toolchain. If the
copied Runner does not match the tree, `prepare-runner-vendor.mjs`
starts a rebuild, and the build fails late at `cargo` with a message
that hides the cause.

**Proposed solution**

Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to
`prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets
the variable in this pull request, so no workflow job changes.

**Alternatives considered**

- One command that builds and looks up. Two commands keep a registry
credential, when a caller has one, out of the step that builds.
- A wait for the publication of the commit. The decision whether the
application build waits is open, so the resolver has no wait mode. It
reports `absent` and stops.
- A build argument that names the expected inputs. No `Dockerfile`
declares one yet. The change that makes the application image copy the
Runner will add both ends together.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode (this pull request)
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands
that share a directory:
- `inputs --dir <dir>` exports the `runner-artifacts-context` target,
runs the lockfile check on that export, and builds the
`runner-artifacts-identity` target when the lockfile is current. It
reads no credential. A lockfile that is behind stops there: no image is
published for such a tree.
- `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`,
`reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and
`labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to
`$GITHUB_STEP_SUMMARY`.
- `mode` is `published` only for a complete publication whose full index
names these inputs and both platforms. Both images are pinned by index
digest. The slim digest is the one the full index names in
`io.paperclip.runner.slim-digest`; the slim tag is never read on its
own.
- Every other answer is `inline`, with the last four values empty:

  | `reason` | Meaning |
  | --- | --- |
  | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests |
  | `absent` | no tag exists |
| `incomplete`, `broken` | a publication stopped, or its two tags do not
belong together |
| `mismatch` | the full index names other inputs, or lacks a platform |
| `unknown` | the registry refused, failed or could not be reached, or
the lookup threw |

- The resolver reuses `runnerLockState`, `readRunnerImageManifest`,
`runnerPublicationState` and `runnerImageTags` from
`scripts/runner-image-publish.mjs`. That file now exports
`environmentCredentials`; it has no other change.
- `server/scripts/prepare-runner-vendor.mjs`: with
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current
build as before. When the build is not current, it prints the reason,
starts no rebuild, leaves the stamp in place and exits with status 1. It
also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same
time. Without the variable the step is the same as before.
- `doc/runner-image-contract.md` has a new section, "Choosing the Runner
for an image build". `doc/DEVELOPING.md` describes the new variable.
- The resolver is not an input of the Runner image, so the inputs digest
does not change.

## Verification

macOS (Node 26.12.0), in `server/`:

- `node node_modules/vitest/vitest.mjs run
scripts/prepare-runner-vendor.test.mjs
src/__tests__/runner-image-resolve.test.ts
src/__tests__/runner-image-publish.test.ts
src/__tests__/runner-image-identity.test.ts
src/__tests__/runner-image-workflow.test.ts
src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6
files. 27 of them are new resolver tests and 3 are new tests of the
vendor step, whose end-to-end test also has a new case.
- `tsc` on the new test file, with the settings of
`server/src/__tests__/tsconfig.json`: no error.

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted
read-only): the same files, 179 tests pass.

Real runs, on clean exports of the tree (`git archive`), with no
credential, on a `linux/arm64` Docker host:

- On `master` at `62d673cc3f`: `inputs` gave `lock=current` and
`inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
`resolve` gave `mode=published`,
`image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162`
and
`slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`.
These are the two indexes that `master` published for those inputs.
- The same tree with one line added to a Runner source file: another
digest, `mode=inline`, `reason=absent`.
- On `master` at `bc71c23e0a`, which changed the Runner's inputs, before
its image was published: digest
`88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`,
`mode=inline`, `reason=absent`.
- Four deliberate faults in the resolver each make a test fail: no index
check, only `absent` is inline, the identity is built for a lockfile
that is behind, and the slim image is taken by tag.

Never run:

- A lookup with a registry credential. The package is public, so no
caller needs one now.
- Any workflow. No workflow calls the resolver in this pull request.
- `linux/amd64`.
- The full test suite, the full typecheck and `pnpm build`. `pnpm verify
--explain` reports exit 2, "broader verification required", so CI must
supply them.

## Risks

- Low for existing behaviour: nothing calls the resolver, and the vendor
step is the same unless the new variable is set.
- The resolver does not verify the attestation of the image. The
workflow that consumes the image must do that from the digest the
resolver reports.
- The resolver reports both build contexts, `runner-artifacts` and
`runner-artifacts-slim`. A build that reads only one stage gets one
context it does not use.
- A `broken` publication is `inline`, not a failure. The Runner image
workflow fails for that state.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:03:15 -08:00
Nicky LeachandClaude Fable 5.2 25e9a9c2e9 feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Parts 2 and 3 of this stack build the Runner alone as an image with
two variants, and check it on pull requests. Nothing publishes it
> - Downstream sandbox and VM images can copy the Runner only from a
registry, and they must be able to verify where it came from
> - A published image must be exactly the bytes that were checked, and
its name must never point at something else later
> - This pull request publishes both variants from `master` as
`ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set
of inputs, and attests them
> - Nothing consumes the image in this pull request. The application
image build does not change
> - The benefit is one verified, attested Runner that later changes can
copy by digest

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image
build). No issue exists for this change.

**Problem or motivation**

The Runner image exists only inside a pull-request check. A consumer
cannot pull it. There is no rule for which commit publishes an image, no
protection for its tags, and no way to verify a copied Runner.

**Proposed solution**

Publish from a `master` push in this repository, and from nothing else.
Build only when no image has the digest of the commit's inputs. Let the
build legs hold no write access. Push the bytes that the legs checked by
digest, attest each multi-platform index, and write the tags last. Write
the full tag after the slim tag, so that the full tag means "both
exist".

**Alternatives considered**

- Publish from `docker.yml`. That workflow holds one concurrency slot
for each ref, and the Runner image needs its own signer identity, so
that its attestation cannot stand in for the application image's.
- Resolve the lockfile in the workflow before a `master` build, as
`docker.yml` does. Then the published Runner depends on the state of the
npm registry at run time, and the digest cannot be computed again from a
commit.
- Hand each variant to the publish job as its own OCI archive. The two
archives hold the two shared layers two times: 1.27 GB for each
architecture in place of 0.85 GB.
- Write the full tag first. Then a consumer can see a full image whose
slim image does not exist yet.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 4 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant
4. #21230: publish both variants from `master` (this pull request)

## What Changed

- `.github/workflows/runner-image.yml`: adds the trigger `push` to
`master` and the job `publish`.
- `identity` checks the committed lockfile against the Runner's
manifests, computes the digest in the build context, and asks the
registry whether these inputs are published.
- `build` is the job of part 3. It also records what it built, on every
event. On a canonical `master` push it uploads that. It cannot write to
the registry on any event. The only token in the job can read packages,
and only the lookup step receives it.
- `publish` runs only for a `master` push in this repository. It builds
nothing and runs no dependency code.
- Lockfile rule. A `master` push builds from the committed
`pnpm-lock.yaml` and resolves nothing, so the digest is a function of
the commit. When the committed lockfile is behind the manifests that the
Runner build reads, the run writes a notice, publishes nothing, and
stays green. The commit that merges the lockfile refresh publishes the
image. Pull requests and manual runs resolve the lockfile first, always
build, and never publish.
- Hand-off. Each leg joins its two OCI archives into one OCI layout that
names the images `slim` and `full`. The layout stores the two shared
layers one time. `record` then reads the layout back, each image by its
name, and checks every blob against its digest. The leg's record binds
the layout to both image digests, the architecture, the inputs digest,
and the commit. Those digests are the digests of the archives that part
3 unpacks and checks, so the tree that is checked and the image that is
pushed are the same bytes. The publish job accepts only records for its
own inputs and commit, and only a layout that still has the recorded
bytes.
- `record` runs on pull requests too. Only the upload is for a `master`
push. A pull request therefore runs the hand-off on real build output.
- Publication order. The publish job pushes the four images by digest
(`skopeo copy --preserve-digests`). It makes a candidate index for the
slim variant, then one for the full variant that names the slim index in
the annotation `io.paperclip.runner.slim-digest`. It verifies that each
index names exactly the recorded images, attests both, and then writes
`inputs-<digest>-slim` and, last, `inputs-<digest>`.
- The full tag is the completion marker. Every "is it published" lookup
reads the full tag. `publication-state` reports one of:

  | State | Registry | What a run does |
  | --- | --- | --- |
  | `absent` | Neither tag | Builds and publishes |
| `incomplete` | The slim tag alone | Builds and publishes. It replaces
the slim tag with its own image, then writes the full tag |
| `published` | The full tag, and the slim tag at the digest that the
full index names | Nothing |
| broken | The full tag, and a slim tag that is missing or points at
another image | Fails, with the steps for an operator |
| `unknown` | No clear answer | Builds. The publish job fails before it
writes a final tag |

- Recovery rule. A run that stops between the two tags leaves the state
`incomplete`. It does not block those inputs. The next run never keeps
the earlier slim image: it writes its own slim tag immediately before
its full tag, verifies both against its own index digests, and then
reads the pair back as every later run will see it. The broken state
cannot be repaired by a run, because no run moves a full tag. An
operator must delete the two package versions and re-run the workflow
run of a `master` push with those inputs.
- Pairing rule for consumers, in the contract: resolve the full tag
first, take the slim digest from the annotation of the full index, and
pull the slim image by that digest.
- `verify-index` accepts `linux/arm64` with the variant `v8`, which some
BuildKit versions record, and no other variant.
- `scripts/runner-image-publish.mjs` (533 lines): the commands
`lock-state`, `record`, `verify-legs`, `publication-state` and
`verify-index`. It reads archives with the reader of
`scripts/runner-image-oci.mjs` (607 lines here), which can now read one
named image of a layout. `lock-state` refuses a checkout, because pnpm
can rewrite the lockfile of the tree that it checks. The workflow gives
it an export that no build reads.
- The workflow test pins the build arguments of the image build. Build
arguments are not part of the image identity.
- `doc/runner-image-contract.md` (197 lines): the section "Publication":
the lockfile rule, the order, the pairing rule, the recovery rule, the
create-only tags, the internal `candidate-` tags, the command that
verifies provenance, and the step to make the package public.
`doc/DOCKER.md` says where the image is published.

## Verification

Tests, and the build jobs of the workflow on GitHub. Nothing in this
pull request ran against a registry, and no publish job has run.

- Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd): `runner-image-publish.test.ts` (75),
`runner-image-workflow.test.ts` (12), the three other Runner image test
files, the provider pack archive tests and the existing Dockerfile
tests: 372 tests pass in 10 files. The same files pass on macOS.
- The tests of the leg record failed on Linux in the first runs of this
pull request: their fixtures named the archive members `./index.json`,
which bsdtar finds and GNU tar does not. The fixtures now name the
members as buildx does, and the publish script uses the one reader of
part 3.
- Leg records: both variants are accepted for this run's inputs and
commit; other inputs, another commit, a changed layout, a record without
a variant, an archive that lacks a blob, a full image that does not
start with the slim image's layers, and a build that is not a
publication build are refused.
- Publication state: all five states, and the half-finished publication
end to end. A slim tag alone lets the next run publish. A full image
that is left with another run's slim tag is reported as broken.
- Index: exactly the recorded images of one variant; the full index must
name its slim index, and the slim index must name none; `v8` is accepted
on arm64 only.
- Registry lookups: a missing tag only when the registry says so; every
refused, failed, or unreachable request is `unknown`.
- Workflow: job permissions; the publish job and the upload are behind
the canonical `master` push; the build steps receive no credential; the
full tag is written last, after a strict lookup and both attestations;
one export and a time limit for each build command; the two build
arguments; actions pinned by commit; no expression inside a shell
script.
- `record` on the two OCI archives of a real `linux/arm64` build from
part 3: the joined layout is 850,149,376 bytes, and both images read
back by name with every blob under its digest. The two archives together
are 1,273,646,592 bytes. The image digests in the record are the digests
that `unpack` reports for the trees that were checked.
- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `docker buildx build --check` reports
nothing. The image identity is the same as in part 3
(`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`),
because this pull request changes no input of the image.

GitHub, at this head:

- The `Runner image` workflow passes with these changes: `Build and
check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each
leg builds both variants, checks them, and writes the leg record.
`Publish and attest` is skipped, as it must be on a pull request.

Never run:

- The publish job. It runs only for a `master` push.
- The publish path: the registry lookups with a workflow token, `skopeo
copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx
imagetools create` with index annotations, and the tag writes.
- The attestations, and the `gh attestation verify` command in the
document.
- `actionlint`. It is not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **Manual step.** GitHub creates the `paperclip-runner` package private
on the first publication. A maintainer must make it public one time.
Until then, consumers cannot pull the image without credentials.
- The first `master` runs of the workflow are the first test of the
publish path. A failure before the tags publishes nothing. A failure
between the two tags leaves the `incomplete` state, and the next run
with those inputs repairs it.
- A slim tag can move while its publication is incomplete. A consumer
must resolve the full tag first and take the slim digest from the
annotation of the full index. It must never resolve
`inputs-<digest>-slim` by tag on its own. The contract says so.
- `candidate-` tags are written before the attestations and can name an
index that is not attested. They are internal.
- The broken state needs an operator, and it fails every `master` run
with those inputs until then. It can arise only when someone changes or
deletes a tag outside the workflow.
- **Known residual.** Two `master` runs that both start before a new
digest is published can each build the image. Each build leg and the
publish job look again before they work, and the publish job runs one at
a time for each digest. The cost is runner time, not correctness.
- A commit whose lockfile is behind the Runner's manifests has no
published Runner image until the lockfile refresh merges. A lockfile
that the refresh cannot fix leaves every `master` run green with only a
notice, for as long as it lasts.
- The tags are create-only because the workflow never moves a tag of a
complete publication. The registry does not enforce that. Consumers must
pin the digest and verify the attestation.
- `payload.runnerSourceRevision` in a pack from this image is the commit
that published the image. Code that reads it as the application's commit
must change before it consumes the image.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 20:27:36 -08:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00