From 999709b0a6dd416af335b5debded13e1de7fcb7a Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 13 Nov 2026 14:17:15 -0800 Subject: [PATCH] fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The application image now takes its Runner from the `runner-artifacts` stages, which install with a frozen lockfile > - A contributor cannot commit `pnpm-lock.yaml`. After a change to a manifest of the Runner, the lockfile on `master` is behind until the refresh merges > - In that time a plain `docker build` fails late, inside a long step, with a message that is easy to miss. And a beta or a stable release can be cut from such a commit, because the release workflows refresh the lockfile for their own build > - This pull request adds an early check with a clear instruction to the image build, and the same check to the beta and the stable release > - The benefit is a failure in seconds that says what to run, and beta and stable tags that always build from source ## Linked Issues or Issue Description This is the third part of a chain. The first part is #21331. The second part takes the Runner of the application image from the `runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and its lockfile rule). No issue exists for this change. **What happened?** A tree with a lockfile behind the manifests of the Runner fails `docker build` after the daemon compile (169 s in the test). The error is `ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the `Dockerfile`. **Expected behavior** The build fails within seconds of the check running. The message names the files and the command to run. The application's dependency install starts from the check, so it cannot start before the check passes. Stages that do not depend on it (the Runner's toolchain download and daemon compile, and the tool layer of `production`) can already be running, and the failed check cancels them. A beta or a stable release from such a commit does not start. How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a commit whose lockfile is behind the manifests of the Runner (2026-09-01, 09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta is the promotion of a nightly, so a beta can be cut from such a commit. **Steps to reproduce** 1. Change a dependency range in `packages/paperclip-runner/package.json`. Do not refresh `pnpm-lock.yaml`. 2. Run `docker build .`. **Paperclip version or commit** `master` with the previous pull request of this chain. **Deployment mode** Not applicable. This is the image build and the release tooling. ## What Changed - `Dockerfile`: a `runner-lockfile` stage copies only the manifests the Runner build installs from, the lockfile and the patches. It runs the frozen, filtered, offline lockfile check, with the arguments of `runnerLockState`. `deps` starts from it. For a stale lockfile it prints the files and `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. For any other failure it prints the output of pnpm and says so. - `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs of a commit with `git archive` and gives them to `runnerLockState` (`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind, 2 for an error. - `scripts/release.sh` and `scripts/release-lib.sh`: `require_runner_lockfile_current` runs for the `beta` and the `stable` channel, after `--print-version` exits and before any build. It runs in a dry run too. It does not run for `canary` or `nightly`. - Both beta paths end in the `publish_beta` job: the promotion of a nightly, and a `candidate/beta-*` branch. That job checks out the commit that `select_beta` chose and runs `release.sh beta` there. So the check reads the committed files of the commit the beta is built from. A beta is refused before its soak starts. The stable that promotes it is checked again. - Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18 tests). Three of them pin the checkout of `publish_beta`, `preview_stable` and `publish_stable` to the selected commit. Two tests plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs` cover the beta, the candidate beta and the stable path. - Docs: `doc/DOCKER.md`, `doc/RELEASING.md`, `doc/runner-image-contract.md`. ## Verification Builds: `linux/arm64`, local. They ran before the rebase onto `master`, except the last one. `master` changed neither the `Dockerfile` nor a manifest in between. - A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest of the Runner: `docker buildx build --target build` fails after 6 s at `runner-lockfile`, with the message. - The same tree after `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` on the host, which is what the image workflows run: `--target build` builds. The check passes, the Runner stages install, and the vendor step logs `reusing the current Runner build` in both of its runs. - After the rebase: `build` and `runner-provider-pack` built from this branch without a build commit. The check passed. - The release check with the real pnpm: `behind` for the first tree, `current` for the second, exit 0 for this checkout. - `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with the other changed server tests, on macOS and in `node:24.22.3-bookworm` (249 pass in 11 files on Linux). The test file typechecks. - `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14 pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of its tests fail on `master` too. `release-verify-workflow.test.mjs` and `acpx-patch-packaging.test.mjs`, which read the release scripts: pass. - `docker buildx build --check`: no warning. - The Docker builds above ran before the second part of the chain got its expectation step and before the beta guard. The expectation step was built on the second part alone. This branch as it is now was not built again; its tests ran. - Not run: the release workflow, any `linux/amd64` build, the full test suite. Only CI and a release dry run can prove the release path. ## Risks - **A beta can be refused.** When the newest nightly is on a commit whose lockfile is behind, `release.sh beta` stops, in the dry run and in the real run. The operator must promote a nightly whose commit includes the lockfile refresh, or cut a candidate branch that includes it. `node scripts/check-release-runner-lockfile.mjs --commit ` gives the answer before the dispatch. - **A stable can be refused** only when its beta was cut before this change, or with release tooling that does not have the check. Promotions run the release tooling of the source commit. - The check runs in `release.sh`. In the beta workflow it runs in `publish_beta`, after the approval of the `npm-beta` environment and after the candidate verification. In the stable workflow it runs after verification, not in the `preflight_stable` job. The selection jobs have no pnpm. - `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before its first build step. The workflow has them. - `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI command today. The Vitest file carries the enforced tests. - The image build has one more small stage. It reads only manifests, so a source change does not run it again. - The digest of the Runner image changes again, because its identity covers the whole `Dockerfile`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- Dockerfile | 17 +- doc/DOCKER.md | 13 +- doc/RELEASING.md | 21 +- doc/runner-image-contract.md | 15 ++ .../__tests__/release-dry-run-notes.test.mjs | 33 +++ scripts/check-release-runner-lockfile.mjs | 64 ++++++ scripts/release-lib.sh | 6 + scripts/release.sh | 4 + .../src/__tests__/docker-build-stamp.test.ts | 2 +- .../__tests__/runner-lockfile-guards.test.ts | 204 ++++++++++++++++++ 10 files changed, 373 insertions(+), 6 deletions(-) create mode 100644 scripts/check-release-runner-lockfile.mjs create mode 100644 server/src/__tests__/runner-lockfile-guards.test.ts diff --git a/Dockerfile b/Dockerfile index 7f6f1ea5904..14bf3c7b5e3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,7 +12,22 @@ RUN usermod -u $USER_UID --non-unique node \ && groupmod -g $USER_GID --non-unique node \ && usermod -g $USER_GID -d /paperclip node -FROM base AS deps +FROM base AS runner-lockfile +WORKDIR /runner-lockfile +COPY package.json pnpm-workspace.yaml pnpm-lock.yaml .npmrc ./ +COPY packages/paperclip-eval-kernel/package.json packages/paperclip-eval-kernel/ +COPY packages/paperclip-runner/package.json packages/paperclip-runner/ +COPY patches/ patches/ +RUN pnpm install --frozen-lockfile --lockfile-only --offline --ignore-scripts --filter '@paperclipai/paperclip-runner...' > /tmp/runner-lockfile.log 2>&1 \ + || { cat /tmp/runner-lockfile.log >&2; \ + if grep -Eq 'ERR_PNPM_OUTDATED_LOCKFILE|ERR_PNPM_LOCKFILE_CONFIG_MISMATCH' /tmp/runner-lockfile.log; then \ + echo "ERROR: pnpm-lock.yaml is behind the manifests the Runner is installed from: package.json, pnpm-workspace.yaml, patches, packages/paperclip-runner/package.json or packages/paperclip-eval-kernel/package.json. The image build installs the Runner with a frozen lockfile, so it stops here. Run this command in the checkout, then build again: pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile" >&2; \ + else \ + echo "ERROR: pnpm could not check pnpm-lock.yaml against the manifests the Runner is installed from. Its output is above." >&2; \ + fi; \ + exit 1; } + +FROM runner-lockfile AS deps WORKDIR /app COPY package.json pnpm-workspace.yaml pnpm-lock.yaml .npmrc ./ COPY cli/package.json cli/ diff --git a/doc/DOCKER.md b/doc/DOCKER.md index b6c857d519f..cc46a428ffb 100644 --- a/doc/DOCKER.md +++ b/doc/DOCKER.md @@ -110,8 +110,17 @@ contract for the rules a consumer of the application image can rely on. The Runner stages install with a frozen lockfile, so `docker build` fails when `pnpm-lock.yaml` is behind the manifests they read: the root `package.json`, -`packages/paperclip-runner` and `packages/paperclip-eval-kernel`. The `deps` -stage still resolves every other manifest change inside the image. Refresh the +`packages/paperclip-runner` and `packages/paperclip-eval-kernel`. The +`runner-lockfile` stage makes that check from those manifests and the lockfile +alone, so it runs as soon as the `base` stage is built. The `deps` stage starts +from it, so the application's dependency install cannot start before the check +passes. A stale lockfile fails the build within seconds of the check running, +with a message that names the command below. The Runner's own frozen install +fails on the same lockfile, but it comes after the daemon compile. Stages that +do not depend on the check can already be running when it fails: the Runner's +toolchain download and daemon compile, and the tool layer of the `production` +stage. The failed check stops the build and cancels them. The `deps` stage +still resolves every other manifest change inside the image. Refresh the lockfile in the checkout, then build again. Do not commit the result (see "Dependency Lockfile Policy" in `doc/DEVELOPING.md`): diff --git a/doc/RELEASING.md b/doc/RELEASING.md index 58f38ff8d60..16fcb0f8b46 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -211,6 +211,14 @@ Betas are manual promotions. Dispatch it does not exist or already shipped as a beta - the publish waits for approval in the **`npm-beta` environment** — its required reviewers are the promotion gate +- the publish job checks out the source commit and runs + `scripts/release.sh beta`, which refuses a commit whose `pnpm-lock.yaml` is + behind the Runner's manifests, in a dry run too. An image build installs + the Runner with a frozen lockfile, so a plain `docker build` of such a + commit fails, and the beta would start a soak that the stable cannot + finish. Promote a nightly whose commit includes the lockfile refresh, or + cut a candidate branch that does. Check a commit before the dispatch with + `node scripts/check-release-runner-lockfile.mjs --commit ` - promotions run the release tooling of the source commit, so the source nightly must postdate the beta channel's introduction; the selection job rejects older sources with a clear error (in practice every nightly cut @@ -250,7 +258,8 @@ from a short-lived candidate branch: 4. selection validates the branch name, rejects heads that already shipped as a beta, records the cherry-picked commits in the job summary, and the head runs **full verification** before publishing (it never went through - a canary or nightly) + a canary or nightly). The lockfile check above runs against the head of + the candidate branch 5. after the beta ships, land the fixes on `master` normally and delete the candidate branch @@ -303,7 +312,9 @@ Before running stable: `releases/beta/v.md`) is merged on `master` — or, for candidate builds, that the candidate branch itself carries `releases/vYYYY.MDD.P.md` -5. run the stable workflow from that source ref +5. check that the lockfile of the source commit matches the Runner's + manifests: `node scripts/check-release-runner-lockfile.mjs --commit ` +6. run the stable workflow from that source ref Example: @@ -320,6 +331,12 @@ The workflow: `releases/beta/v.md` on `master` (a promoted beta). When neither exists the run fails before the `npm-stable` approval gate with the missing path named +- refuses a source commit whose `pnpm-lock.yaml` is behind the Runner's + manifests, in a dry run too. `scripts/release.sh stable` makes this check + before it builds anything, as `scripts/release.sh beta` did when the beta + was cut. An image build installs the Runner with a frozen lockfile, so a + plain `docker build` of such a commit fails. Release a commit that + includes the lockfile refresh - publishes `YYYY.MDD.P` under npm dist-tag `latest` - creates git tag `vYYYY.MDD.P` - dispatches [`docker.yml`](../.github/workflows/docker.yml) at that tag to diff --git a/doc/runner-image-contract.md b/doc/runner-image-contract.md index 7178e9efafe..621ed998bd9 100644 --- a/doc/runner-image-contract.md +++ b/doc/runner-image-contract.md @@ -96,6 +96,21 @@ build reads (the root one, `packages/paperclip-runner` and pull-request runs resolve it first. For a local build, run `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. +Three places ask pnpm the same question, with the same arguments +(`runnerLockState` in `scripts/runner-image-publish.mjs`): + +- the publication workflow, which publishes nothing for a lockfile that is + behind; +- the `runner-lockfile` stage of the `Dockerfile`, which fails an application + image build and prints the command above. The application's dependency + install starts from that stage, so it cannot start before the check passes. + The Runner's toolchain and daemon stages do not depend on it, so they can + already be running, and the failed check cancels them. +- `scripts/check-release-runner-lockfile.mjs`, which `scripts/release.sh` + runs for a beta and for a stable release. It checks the committed files of + the release commit, so a beta or stable tag always builds with a plain + `docker build`. A canary or a nightly is not checked. + ## Checks CI builds each variant as an OCI archive and nothing else. diff --git a/scripts/__tests__/release-dry-run-notes.test.mjs b/scripts/__tests__/release-dry-run-notes.test.mjs index 02648b61849..d42b302d3b0 100644 --- a/scripts/__tests__/release-dry-run-notes.test.mjs +++ b/scripts/__tests__/release-dry-run-notes.test.mjs @@ -65,6 +65,13 @@ require_channel_tag_absent_at_head() { } require_on_master_branch() { :; } require_clean_worktree() { :; } +require_runner_lockfile_current() { + if [ "\${FAKE_RUNNER_LOCKFILE_BEHIND:-}" = "1" ]; then + echo "Error: a beta or stable release needs a pnpm-lock.yaml that matches the Runner's manifests at HEAD." >&2 + exit 1 + fi + echo "[fixture] require_runner_lockfile_current" +} require_npm_publish_auth() { :; } git_local_tag_exists() { return 1; } git_remote_tag_exists() { return 1; } @@ -187,6 +194,32 @@ test("stable publish still requires release notes before publish work starts", ( assert.doesNotMatch(result.calls, /^pnpm /m); }); +test("beta and stable check the lockfile against the Runner's manifests before any build, in a dry run too", () => { + for (const args of [["stable"], ["beta"], ["beta", "--from-candidate"]]) { + const current = runRelease([...args, "--skip-verify", "--dry-run"]); + assert.match(current.output, /\[fixture\] require_runner_lockfile_current/, args.join(" ")); + + const behind = runRelease([...args, "--skip-verify", "--dry-run"], { FAKE_RUNNER_LOCKFILE_BEHIND: "1" }); + assert.equal(behind.status, 1, args.join(" ")); + assert.match(behind.output, /needs a pnpm-lock\.yaml that matches the Runner's manifests at HEAD/); + assert.doesNotMatch(behind.output, /==> Release plan/); + assert.doesNotMatch(behind.calls, /^pnpm /m); + } +}); + +test("canary and nightly do not check the lockfile, and printing a version does not", () => { + for (const args of [["canary", "--skip-verify", "--dry-run"], ["nightly", "--skip-verify", "--dry-run"]]) { + const result = runRelease(args, { FAKE_RUNNER_LOCKFILE_BEHIND: "1" }); + assert.doesNotMatch(result.output, /Runner's manifests/, args[0]); + assert.match(result.output, /==> Release plan/, args[0]); + } + for (const channel of ["stable", "beta"]) { + const version = runRelease([channel, "--print-version"], { FAKE_RUNNER_LOCKFILE_BEHIND: "1" }); + assert.equal(version.status, 0, channel); + assert.doesNotMatch(version.output, /Runner's manifests/, channel); + } +}); + test("nightly dry-run publishes under the nightly identity without release notes", () => { const result = runRelease(["nightly", "--skip-verify", "--dry-run"]); diff --git a/scripts/check-release-runner-lockfile.mjs b/scripts/check-release-runner-lockfile.mjs new file mode 100644 index 00000000000..cce6978bd95 --- /dev/null +++ b/scripts/check-release-runner-lockfile.mjs @@ -0,0 +1,64 @@ +#!/usr/bin/env node + +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { RUNNER_IMAGE_CONTEXT_INPUTS } from "./runner-image-identity.mjs"; +import { runnerLockState } from "./runner-image-publish.mjs"; + +export const LOCKFILE_REFRESH_COMMAND = "pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile"; + +export function runnerLockStateAtCommit({ repoRoot, commit = "HEAD", run = spawnSync, lockState = runnerLockState }) { + const scratch = mkdtempSync(path.join(tmpdir(), "paperclip-release-runner-lockfile-")); + try { + const archive = path.join(scratch, "context.tar"); + const context = path.join(scratch, "context"); + mkdirSync(context); + const steps = [ + ["export", "git", ["-C", repoRoot, "archive", "--format=tar", "--output", archive, commit, "--", ...RUNNER_IMAGE_CONTEXT_INPUTS]], + ["unpack", "tar", ["-xf", archive, "-C", context]], + ]; + for (const [step, command, args] of steps) { + const result = run(command, args, { encoding: "utf8", stdio: ["ignore", "ignore", "pipe"] }); + if (result.error) throw result.error; + if (result.status !== 0) { + throw new Error(`Could not ${step} the Runner's inputs at ${commit} (${command} exit ${result.status}): ${(result.stderr ?? "").trim().slice(-1000)}`); + } + } + return lockState(context); + } finally { + rmSync(scratch, { recursive: true, force: true }); + } +} + +export function runReleaseRunnerLockfileCheck(argv, { + repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."), + check = runnerLockStateAtCommit, + log = console.error, +} = {}) { + if (argv.length > 2 || (argv.length > 0 && (argv[0] !== "--commit" || !argv[1]))) { + log("Usage: check-release-runner-lockfile.mjs [--commit ]"); + return 2; + } + const commit = argv[1] ?? "HEAD"; + const result = check({ repoRoot, commit }); + if (result.state === "current") { + log(`pnpm-lock.yaml at ${commit} matches the Runner's manifests.`); + return 0; + } + log(`pnpm-lock.yaml at ${commit} is behind the Runner's manifests (${result.detail}). ` + + "An image build from this commit installs the Runner with a frozen lockfile and fails. " + + `Release a commit that includes the lockfile refresh, which is the result of: ${LOCKFILE_REFRESH_COMMAND}`); + return 1; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + try { + process.exitCode = runReleaseRunnerLockfileCheck(process.argv.slice(2)); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 2; + } +} diff --git a/scripts/release-lib.sh b/scripts/release-lib.sh index da160af0041..7b4a890ccec 100644 --- a/scripts/release-lib.sh +++ b/scripts/release-lib.sh @@ -543,6 +543,12 @@ require_clean_worktree() { fi } +require_runner_lockfile_current() { + if ! node "$REPO_ROOT/scripts/check-release-runner-lockfile.mjs"; then + release_fail "a beta or stable release needs a pnpm-lock.yaml that matches the Runner's manifests at HEAD." + fi +} + require_on_master_branch() { local current_branch current_branch="$(git_current_branch)" diff --git a/scripts/release.sh b/scripts/release.sh index 6b154634fd6..ac354ddb1c4 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -223,6 +223,10 @@ fi require_clean_worktree require_npm_publish_auth "$dry_run" +if [ "$channel" = "beta" ] || [ "$channel" = "stable" ]; then + require_runner_lockfile_current +fi + if [ "$channel" = "stable" ] && [ "$dry_run" = false ] && [ ! -f "$NOTES_FILE" ]; then release_fail "stable release notes file is required at $NOTES_FILE before publishing stable." fi diff --git a/server/src/__tests__/docker-build-stamp.test.ts b/server/src/__tests__/docker-build-stamp.test.ts index cbf3e7e5f6e..2bb94d73700 100644 --- a/server/src/__tests__/docker-build-stamp.test.ts +++ b/server/src/__tests__/docker-build-stamp.test.ts @@ -163,7 +163,7 @@ describe("Application image Runner", () => { it("compiles the Runner only in the runner-artifacts stages", () => { const names = [...dockerfile.matchAll(/^FROM \S+ AS (\S+)$/gm)].map((match) => match[1]); const application = names.filter((name) => !name.startsWith("runner-artifacts")); - expect(application).toEqual(["base", "deps", "build", "runner-provider-pack", "production", "cloud-plugins", "cloud-server-deps", "cloud"]); + expect(application).toEqual(["base", "runner-lockfile", "deps", "build", "runner-provider-pack", "production", "cloud-plugins", "cloud-server-deps", "cloud"]); for (const name of application) { expect(stageBody(dockerfile, name), name).not.toMatch(/\b(?:cargo|rustup|rustc|gcc)\b/); } diff --git a/server/src/__tests__/runner-lockfile-guards.test.ts b/server/src/__tests__/runner-lockfile-guards.test.ts new file mode 100644 index 00000000000..5d14a1abd6e --- /dev/null +++ b/server/src/__tests__/runner-lockfile-guards.test.ts @@ -0,0 +1,204 @@ +import { execFileSync, spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, describe, expect, it } from "vitest"; +import { + LOCKFILE_REFRESH_COMMAND, + runReleaseRunnerLockfileCheck, + runnerLockStateAtCommit, +} from "../../../scripts/check-release-runner-lockfile.mjs"; +import { RUNNER_IMAGE_CONTEXT_INPUTS } from "../../../scripts/runner-image-identity.mjs"; +import { runnerLockState } from "../../../scripts/runner-image-publish.mjs"; + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", ".."); +const read = (...segments: string[]) => readFileSync(path.join(repoRoot, ...segments), "utf8"); +const dockerfile = read("Dockerfile"); +const scratch = mkdtempSync(path.join(tmpdir(), "runner-lockfile-guards-")); +afterAll(() => rmSync(scratch, { recursive: true, force: true })); + +function stageBody(name: string): string { + const froms = [...dockerfile.matchAll(/^FROM .*$/gm)]; + const start = froms.findIndex((match) => match[0].endsWith(` AS ${name}`)); + expect(start, name).toBeGreaterThanOrEqual(0); + return dockerfile.slice(froms[start].index, froms[start + 1]?.index ?? dockerfile.length); +} + +function pnpmArguments(): string[] { + const context = path.join(scratch, "arguments"); + mkdirSync(path.join(context, "packages", "paperclip-runner"), { recursive: true }); + writeFileSync(path.join(context, "pnpm-lock.yaml"), "lockfileVersion: '9.0'\n"); + writeFileSync(path.join(context, "packages", "paperclip-runner", "package.json"), "{}\n"); + let seen: string[] = []; + runnerLockState(context, { + run: (_command: string, args: string[]) => { + seen = args; + return { status: 0, stdout: "", stderr: "" }; + }, + }); + return seen; +} + +describe("application image lockfile check", () => { + const stage = stageBody("runner-lockfile"); + + it("runs before the dependency install, which starts from it", () => { + expect(stage).toContain("FROM base AS runner-lockfile\n"); + expect(stageBody("deps")).toContain("FROM runner-lockfile AS deps\n"); + }); + + it("asks pnpm the question the release and publication checks ask", () => { + const args = pnpmArguments(); + expect(args.at(-1)).toBe("@paperclipai/paperclip-runner..."); + const command = ["pnpm", ...args.slice(0, -1), `'${args.at(-1)}'`].join(" "); + expect(stage).toContain(`RUN ${command} > /tmp/runner-lockfile.log 2>&1 \\\n`); + expect(stageBody("runner-artifacts-build")).toContain(`pnpm install --frozen-lockfile --filter '${args.at(-1)}'`); + }); + + it("reads only manifests the Runner build reads", () => { + const sources = [...stage.matchAll(/^COPY (.+)$/gm)].flatMap((match) => match[1].split(/\s+/).slice(0, -1)); + expect([...sources].sort()).toEqual([ + ".npmrc", + "package.json", + "packages/paperclip-eval-kernel/package.json", + "packages/paperclip-runner/package.json", + "patches/", + "pnpm-lock.yaml", + "pnpm-workspace.yaml", + ]); + for (const source of sources) { + const file = source.replace(/\/$/, ""); + expect(RUNNER_IMAGE_CONTEXT_INPUTS.some((input: string) => file === input || file.startsWith(`${input}/`)), source).toBe(true); + } + const manifests = RUNNER_IMAGE_CONTEXT_INPUTS + .filter((input: string) => existsSync(path.join(repoRoot, input, "package.json"))) + .map((input: string) => `${input}/package.json`); + expect(sources.filter((source) => source.includes("/") && source.endsWith("package.json")).sort()).toEqual(manifests.sort()); + }); + + it("names the stale lockfile and the command that refreshes it", () => { + expect(stage).toContain("grep -Eq 'ERR_PNPM_OUTDATED_LOCKFILE|ERR_PNPM_LOCKFILE_CONFIG_MISMATCH' /tmp/runner-lockfile.log"); + expect(stage).toContain("ERROR: pnpm-lock.yaml is behind the manifests the Runner is installed from"); + expect(stage).toContain(`Run this command in the checkout, then build again: ${LOCKFILE_REFRESH_COMMAND}" >&2;`); + expect(stage).toMatch(/exit 1; \}\n/); + for (const workflow of ["docker.yml", "docker-merge-group.yml", "docker-app-check.yml"]) { + expect(read(".github", "workflows", workflow), workflow).toContain(` ${LOCKFILE_REFRESH_COMMAND}\n`); + } + }); +}); + +describe("beta and stable release lockfile check", () => { + const git = (cwd: string, ...args: string[]) => execFileSync("git", ["-C", cwd, ...args], { encoding: "utf8" }); + + function repository(lockfile = "lockfileVersion: '9.0'\n") { + const root = mkdtempSync(path.join(scratch, "repository-")); + for (const input of RUNNER_IMAGE_CONTEXT_INPUTS as readonly string[]) { + const file = input === "patches" ? "patches/a.patch" : input.startsWith("packages/paperclip-") ? `${input}/package.json` : input; + mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + writeFileSync(path.join(root, file), file === "pnpm-lock.yaml" ? lockfile : `${file}\n`); + } + mkdirSync(path.join(root, "server")); + writeFileSync(path.join(root, "server", "index.ts"), "export {};\n"); + git(root, "init", "--quiet"); + git(root, "add", "--all"); + git(root, "-c", "user.name=test", "-c", "user.email=test@example.invalid", "commit", "--quiet", "--message", "inputs"); + return root; + } + + it("checks the Runner inputs of the commit, not the working tree, and removes its copy", () => { + const root = repository("committed\n"); + writeFileSync(path.join(root, "pnpm-lock.yaml"), "edited after the commit\n"); + writeFileSync(path.join(root, "packages", "paperclip-runner", "untracked.txt"), "untracked\n"); + let context = ""; + const result = runnerLockStateAtCommit({ + repoRoot: root, + lockState: (directory: string) => { + context = directory; + expect(readFileSync(path.join(directory, "pnpm-lock.yaml"), "utf8")).toBe("committed\n"); + expect(existsSync(path.join(directory, "packages", "paperclip-runner", "package.json"))).toBe(true); + expect(existsSync(path.join(directory, "server"))).toBe(false); + expect(existsSync(path.join(directory, ".git"))).toBe(false); + expect(existsSync(path.join(directory, "packages", "paperclip-runner", "untracked.txt"))).toBe(false); + return runnerLockState(directory, { run: () => ({ status: 1, stdout: "", stderr: " ERR_PNPM_OUTDATED_LOCKFILE Cannot install" }) }); + }, + }); + expect(result.state).toBe("behind"); + expect(context).not.toBe(""); + expect(existsSync(context)).toBe(false); + }); + + it("fails when the commit lacks an input instead of calling the lockfile current", () => { + const root = repository(); + git(root, "rm", "--quiet", "pnpm-lock.yaml"); + git(root, "-c", "user.name=test", "-c", "user.email=test@example.invalid", "commit", "--quiet", "--message", "no lockfile"); + expect(() => runnerLockStateAtCommit({ repoRoot: root, lockState: () => ({ state: "current" as const, detail: "" }) })) + .toThrow(/Could not export the Runner's inputs at HEAD/); + expect(runnerLockStateAtCommit({ repoRoot: root, commit: "HEAD~1", lockState: () => ({ state: "current" as const, detail: "" }) }).state) + .toBe("current"); + }); + + it.each<["current" | "behind", number, RegExp]>([ + ["current", 0, /^pnpm-lock\.yaml at abc123 matches the Runner's manifests\.$/], + ["behind", 1, /^pnpm-lock\.yaml at abc123 is behind the Runner's manifests \(detail\)\. .*: pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile$/], + ])("exits for a %s lockfile with %i", (state, status, message) => { + const lines: string[] = []; + const checked: { repoRoot: string; commit?: string }[] = []; + expect(runReleaseRunnerLockfileCheck(["--commit", "abc123"], { + repoRoot: "/repo", + check: (input: { repoRoot: string; commit?: string }) => { + checked.push(input); + return { state, detail: "detail" }; + }, + log: (line: string) => lines.push(line), + })).toBe(status); + expect(checked).toEqual([{ repoRoot: "/repo", commit: "abc123" }]); + expect(lines).toHaveLength(1); + expect(lines[0]).toMatch(message); + }); + + it.each([[["--commit"]], [["--ref", "HEAD"]], [["--commit", "HEAD", "extra"]]])("refuses the arguments %j", (argv) => { + const check = () => { + throw new Error("must not run"); + }; + expect(runReleaseRunnerLockfileCheck(argv, { check, log: () => {} })).toBe(2); + }); + + it("stops a beta and a stable release, and no other, after the version is printed", () => { + const release = read("scripts", "release.sh"); + const guard = 'if [ "$channel" = "beta" ] || [ "$channel" = "stable" ]; then\n require_runner_lockfile_current\nfi\n'; + expect(release.split("require_runner_lockfile_current")).toHaveLength(2); + const at = release.indexOf(guard); + expect(at).toBeGreaterThan(release.indexOf('if [ "$print_version_only" = true ]; then')); + expect(at).toBeGreaterThan(release.indexOf("require_clean_worktree\n")); + expect(at).toBeLessThan(release.indexOf("==> Step 1/7")); + }); + + it.each([ + ["publish_beta", "needs.select_beta.outputs.sha", "args=(beta --skip-verify)"], + ["preview_stable", "needs.preflight_stable.outputs.sha", "args=(stable --skip-verify --dry-run)"], + ["publish_stable", "needs.preflight_stable.outputs.sha", "args=(stable --skip-verify)"], + ])("runs in %s at the commit the release is built from", (job, sha, args) => { + const workflow = read(".github", "workflows", "release.yml"); + const start = workflow.indexOf(`\n ${job}:\n`); + expect(start).toBeGreaterThanOrEqual(0); + const rest = workflow.slice(start + 1); + const body = rest.slice(0, /\n {2}[a-z_]+:\n/.exec(rest)?.index ?? rest.length); + expect(body.match(/^ {10}ref: .+$/gm)).toEqual([` ref: \${{ ${sha} }}`]); + expect(body).toContain(` ${args}\n`); + expect(body).toContain(' ./scripts/release.sh "${args[@]}"\n'); + }); + + it.each([[0, 0], [1, 1], [2, 1]])("turns exit %i of the check into exit %i of the release", (checkStatus, releaseStatus) => { + const root = mkdtempSync(path.join(scratch, "release-")); + mkdirSync(path.join(root, "scripts")); + writeFileSync(path.join(root, "scripts", "check-release-runner-lockfile.mjs"), `process.exit(${checkStatus});\n`); + const result = spawnSync("bash", ["-c", 'source "$1"; require_runner_lockfile_current; echo released', "guard", path.join(repoRoot, "scripts", "release-lib.sh")], { + encoding: "utf8", + env: { ...process.env, REPO_ROOT: root }, + }); + expect(result.status).toBe(releaseStatus); + expect(result.stdout.includes("released")).toBe(releaseStatus === 0); + if (releaseStatus !== 0) expect(result.stderr).toContain("Error: a beta or stable release needs a pnpm-lock.yaml that matches the Runner's manifests at HEAD."); + }); +});