mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, once for each set of its inputs > - The application image will copy that Runner instead of making its own. `scripts/runner-image-resolve.mjs` tells a build whether the Runner of its inputs is published > - On a `master` push, the commit's own run of `runner-image.yml` is often still publishing when the application build asks. The resolver answers once, so that build is told `absent` for an image that exists some minutes later > - The resolver also cannot say that a tree is not safe to give a published Runner to. A `Dockerfile` that does not check the Runner it receives builds with any Runner > - This pull request adds an opt-in wait for the commit's own publication run, and an `unsupported` answer for a `Dockerfile` that cannot check its Runner > - The benefit is that the next pull request can let `docker.yml` copy the published Runner without a blind timeout and without a label that is not true ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** A `master` push starts `runner-image.yml` and `docker.yml` at the same time. Publication takes about 30 minutes from the push, and several of those minutes are runner queue time. An application build that asks the resolver at once gets `absent` for every commit that changes the Runner's inputs, and for commits that follow one closely. A fixed poll of the tag has the opposite fault: when no image will ever come (the lockfile is behind, or the run failed), every build waits the full timeout. **Proposed solution** `resolve --wait` waits for a run, not for a time. It finds the runs of `runner-image.yml` for the push of the commit to `master`, and it looks up the image again when they end. No caller uses the flag in this pull request, so no workflow changes behavior. **Alternatives considered** - Poll the tag until a timeout. Every build with no image to come waits the full timeout. - Call `runner-image.yml` from `docker.yml` as a reusable workflow. Publication would then share the one concurrency slot of `docker.yml`. - Wait until the run completes and only then look. The run goes on after publication (it writes the build cache), so the wait would last longer than needed. The command looks up the image while a run is in progress. - Verify the attestation in the resolver. The workflow that uses the answer does it, in a step of its own, in the next pull request. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on #21333. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. This pull request: the wait and the `unsupported` answer of the resolver. 3. `runner/app-image-consume-docker`: `docker.yml` copies the published Runner, behind a repository variable. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `scripts/runner-image-resolve.mjs` - `resolve --dir <directory> --wait`. When the first answer is `absent` or `incomplete`, the command lists the runs of `runner-image.yml` for the commit (Actions API, `GITHUB_SHA`, `GITHUB_TOKEN`) and looks again when they end. - Only a run of the `push` event on `master` in this repository counts. A pull-request, scheduled or manual run of the same commit is never waited for. The request filters by event and branch, and the script checks each run again. - A run that fails, is cancelled or publishes nothing ends the wait when it ends. With several runs for one commit, the command waits for all of them. - While a run is in progress, the command looks up the image every 30 seconds, so the wait ends at publication. - A run that is not listed yet is looked for during 2 minutes. - The limit is 40 minutes from the push: `RUNNER_IMAGE_WAIT_MINUTES`, the one constant. The push time is the creation time of the oldest run of the commit, so a build that starts late waits less. - Three failed API requests in a row end the wait. No token or no full commit SHA means no wait. - After every ending the command looks up the image one last time. No ending is an error. Without an image the answer is `inline`, and the command prints a notice with the reason. - `inputs` reads the `Dockerfile` first. Without a line `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` it runs no build, and `resolve` answers `inline` with the reason `unsupported`. - `resolve` reports `build_args`: the value of that argument for a `published` answer. - The wait and the list of runs take the clock, the sleep and `fetch` as arguments, so the tests use no network and do not sleep. - `server/src/__tests__/runner-image-resolve.test.ts`: every outcome of the wait and the answer that follows it, the run filter, the limit measured from the push, the API failures, and the command line. - `doc/runner-image-contract.md`: a "Waiting for a publication" section, `build_args`, and the `unsupported` answer. ## Verification macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch on top of #21333: - `pnpm exec vitest run` on `runner-image-resolve`, `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 218 tests in 11 files pass on Linux; on macOS 217 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test file typechecks with the options of `server/src/__tests__/tsconfig.json`. Read-only checks against the real services: - The lookup, with no credential, answered `published` for the inputs `d4df566f…af00` and `0e6662ad…d1ca`, with the index digests that `master` published, and `absent` for a digest that does not exist. - The real answer of the Actions API for the `runner-image.yml` runs of four `master` commits went through the wait, with a clock that does not sleep. Three completed runs ended it at once. A queued run ended it at the limit of 40 minutes from its creation time. A commit with no run ended it after the 2 minutes for a run that is not listed. - Of the 60 newest runs of `runner-image.yml`, the filter accepted the 19 `push` runs on `master` and none of the 41 pull-request runs. - `runner-image-resolve.mjs inputs` ran on a clean export of the tree of the next pull request, with a `docker-container` builder. The answer was `lock: current` and the inputs digest `c4f9d089…34f0`. The `Dockerfile` of #21332 declares the argument, so the answer was not `unsupported`. The same command on a tree with one changed range in `packages/paperclip-runner/package.json` answered `lock: behind`, and `resolve` then answered `inline (lock-behind)`. Never run: - `resolve --wait` in a workflow. No workflow calls it in this pull request. - The full test suite and the full typecheck. CI must supply them. ## Risks - `resolve` now prints a notice for every `inline` answer. The report-only job in `docker.yml` therefore shows one notice on each `master` push whose Runner image is not published yet, until the next pull request replaces the job. Nothing reads that job. - `inputs` answers `unsupported` for a tree whose `Dockerfile` does not declare `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` at the start of a line. #21332 adds that line; a later edit that moves or renames it turns every answer to `inline`. - The wait is bounded by time from the push, not by its own duration. A build that starts more than 40 minutes after the push looks once and does not wait. - A registry that does not answer adds at most about 3 minutes to one lookup, because the registry client retries. A caller must give the step more than 40 minutes. The next pull request gives it 55. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>