mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as an attested image, once for each set of its inputs > - Since #21332 the application image takes its Runner only from the `runner-artifacts` stage, and a build can replace that stage with the published image > - No workflow does that yet, so every application image still builds its own Runner, and nothing on an application image says which Runner it has > - A build that copies the published Runner must also build the lockfile that Runner was built from. The workflow refreshes the lockfile before every build, and the server build then refuses the published Runner > - This pull request lets `docker.yml` copy the published Runner image and label the application image with it, behind a repository variable that is off, and makes that build use the committed lockfile > - The benefit is one Runner for all images of the same inputs, named on the image, with a switch that goes back to today's builds without a revert ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), #21308 (the server reads a pack from its Runner image), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** `docker.yml` builds the application image on `master` and on tags. Each build makes its own Runner. The report-only job that runs the resolver on `master` feeds no build. Downstream image composers cannot tell from an application image which Runner it has. There is a second problem, which only shows when a build copies the published Runner. The Runner's build stamp covers the whole `pnpm-lock.yaml`. The step that vendors the Runner into the server fails when the stamp does not match the tree. `docker.yml` refreshes `pnpm-lock.yaml` on the runner before every build. The refresh rewrites the file whenever the manifest of any package is ahead of it, also a package the Runner does not read. On the tree of this pull request it rewrites the file even for a pristine checkout: the registry changed the deprecation text of one package. The published Runner was built from the committed file, so the image build would fail. **Proposed solution** One job resolves the Runner once for the run. Both architecture builds need it and add its answer to their build: a build context that replaces the `runner-artifacts` stage, two labels, and the build argument with which the build checks the Runner. A build that copies the published Runner does not refresh the lockfile: it builds the committed tree, the tree the Runner was published for. The job runs only when the repository variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull request merges with the variable unset, so no image changes when it merges. **Alternatives considered** - Resolve in each architecture build. The two builds could then disagree, and one image index would hold two Runners. - Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for an inline build. - For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over `/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step reads the file: the two vendor checks are its only readers in `build`, `production` copies it and takes `node_modules` from `deps`, both `cloud-*` stages install with `--ignore-workspace --no-lockfile`, and the server reads only a plugin's own lockfile and the pack's own lockfile at run time. But the image would then ship a lockfile that its `node_modules` did not come from, the copy changes that file's layer for every inline build, and each `Dockerfile` edit makes a new Runner image. - For the lockfile, bind-mount the Runner image's lockfile over the file for the two steps that run the vendor check. This also edits the `Dockerfile`, and it makes the lockfile part of the check always true for a published Runner. - For the lockfile, let the resolver refresh and compare. The answer would depend on the registry at two different moments. - Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check as strict as it is, and builds exactly the tree the inputs digest was computed from. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on `runner/app-image-consume`. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. `runner/app-image-consume`: the wait and the `unsupported` answer of the resolver. 3. This pull request. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `.github/workflows/docker.yml`: the job `runner_image` replaces `runner_image_report`. - It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in this repository. A fork never runs it. - Steps: `inputs` with no credential; `resolve`; then `gh attestation verify` of the full index against the signer `runner-image.yml` on `refs/heads/master`. - For a `master` push, `resolve` runs with `--wait` and the job token. For a tag or a manual run it does not wait and receives no token. - Only the verification step writes the job's outputs. An image that does not verify fails that step with a warning and writes no output. The job is `continue-on-error`: the job is red, the run is not, and the builds are inline. - Permissions: `contents: read`, `actions: read`, `attestations: read`. No registry login, no write. - Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the verification, 70 for the job. The wait of 40 minutes ends inside them. - `build-and-push` needs `runner_image` and adds its outputs to the build: `build-contexts`, one more line of `build-args` (`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`. With empty outputs these are the values the build had before. - The step "Refresh lockfile for Docker build context" runs only when the answer is not `published`. - The step that looks for a merge-queue image of the commit runs only when the answer is not `published`. The two steps after it already depend on its result. - Labels on an image that copied the published Runner: `io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An image without them made its own Runner. - The `Dockerfile` does not change. The Runner image digest does not move. - Tests that pin lines this changes follow: `scripts/merge-group-image.test.mjs` (the condition of the merge-queue lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build job), and `docker-build-stamp.test.ts` (no workflow names a build context; `docker.yml` now names exactly one). - `server/src/__tests__/docker-runner-workflows.test.ts`: the report-only tests give way to tests of the new job. - `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of the lockfile case. - `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application image gets its Runner, the labels and what their absence means, the variable, the wait, the verification, the lockfile rule, the merge-queue rule and the rollback. ## Verification ### Real builds `docker buildx build --target build --platform linux/arm64 --output type=cacheonly`, with a `docker-container` builder made for this test. The Runner image is the publication form (`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`) built from a clean export of this branch, exported as an OCI layout: inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application builds receive `--build-context runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. | Build context | Result | | --- | --- | | The committed tree | Passes in 78 s. Both vendor checks: `reusing the current Runner build (inputs unchanged …; 1333 files and 1860 outputs checked)` | | `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh command (what the workflow did before this change) | Fails at `RUN node server/scripts/prepare-runner-vendor.mjs`: `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild this step needs (inputs changed since the recorded build; 1333 files checked)` | | The committed tree, then only the refresh command | Fails the same way. The refresh changed only the `deprecated` text of one package | | The same `ui/package.json` change with no refresh (what the workflow does now for a `published` answer) | Passes in 66 s. `deps` resolves the manifest inside the image; both vendor checks reuse the Runner | | One line added to `packages/paperclip-runner/src/index.ts` | Fails at the vendor step, as before: a Runner of other inputs is still refused | Lockfile behind the Runner's own manifests (`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` → `^1.9.0`), no build context: | Step | Result | | --- | --- | | `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no digest; `inline (lock-behind)` | | Build without the refresh | Fails in 4 s at the `runner-lockfile` stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is installed from …` | | Build after the refresh command (what the workflow does for an `inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes, the build makes its own Runner with its pack, and both vendor checks reuse it | ### Tests macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch: - `pnpm exec vitest run` on `docker-runner-workflows`, `runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test files typecheck with the options of `server/src/__tests__/tsconfig.json`. What the workflow tests hold: - Variable off: the job's condition is the variable and the repository, with no alternative. With empty outputs, the build step has exactly the 3 build arguments, 0 build contexts and 1 label entry it had before, and the refresh step runs. - Published: both architecture builds are one job that reads only the outputs of `runner_image`, with no architecture in those lines. - The verification step runs as real shell against a stub `gh`. When `gh` succeeds it writes the context, the labels and the build argument. When `gh` fails the step exits 1, writes no output and prints the warning. The step has no `continue-on-error`; the job has. The condition of the build job does not name the resolver job and has no `success()`. - The lookup step runs as real shell against a stub `node`. It passes `--wait` only when the event is a `master` push. - Credentials: only the lookup and the verification steps have a token. The steps that build have no `env`. - The refresh step has the condition `needs.runner_image.outputs.mode != 'published'`, and it is the only `pnpm install` of the build job. - The fixture test gives the vendor step a Runner that another tree built for the committed lockfile. The step reuses it. After a change of one specifier of another package in the lockfile the step fails and names the lockfile. With the committed lockfile back it reuses the Runner again. - The merge-queue workflow and the pull-request check do not name the resolver, the variable, a build context or a Runner label. Read-only checks against the real services: - `gh attestation verify oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo paperclipai/paperclip --signer-workflow paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref refs/heads/master`: exit 0 for the full index, and exit 0 for the slim index `sha256:69ab2001…69b3`. - The same command with `--signer-workflow …/docker.yml`: exit 1 for both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo cli/cli`: exit 1. For a digest that does not exist: exit 1. With no token: exit 4. - The parsed YAML of `docker.yml` differs from its base only in the resolver job, the `needs` of the build job, the conditions of the refresh step and of the merge-queue lookup, and the three build inputs. Never run: - `docker.yml`. Only GitHub can run it, and it has no pull-request trigger. - A build with the variable on. The first one is the first run after a maintainer sets the variable. - A build that pulls the published Runner from the registry as `docker-image://`. The real builds used the same image form from a local OCI layout. - An `amd64` build. - The `production` and `cloud` targets. The real builds stopped at `build`, the last stage that checks the Runner. ## Risks - With the variable on, an application build on `master` starts after the resolver job. That is 2 to 3 minutes when the image exists, and up to 40 minutes from the push when the commit's own Runner publication is in progress. `docker.yml` holds one concurrency slot for each ref, so a long wait can make more `master` commits lose their image build to a newer push. - A build that copies the published Runner ships the committed `pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of other packages inside the image. A local `docker build` has always done this; a CI image did not. Nothing in the image reads that file. - Any other build that gives the `Dockerfile` a published Runner and a refreshed lockfile fails at the vendor step. That is the intended answer to a tree the Runner was not built for, and the message names the lockfile. - The pack of an image that copied the published Runner names the commit that published the Runner (`runnerSourceRevision`), which can be older than the commit of the image. A consumer that compares the two must accept that before the variable is set. - A failed verification makes the resolver job red on a green run. That is on purpose: it means a published tag whose attestation does not verify. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - With the variable off there is no report of the resolver on `master` any more: the job that replaced the report-only job does not run. - Rollback: set the variable to another value or delete it. The next run builds inline, with the refresh. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
423 lines
17 KiB
JavaScript
423 lines
17 KiB
JavaScript
import { execFile } from "node:child_process";
|
|
import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { promisify } from "node:util";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
|
|
import {
|
|
assessRunnerBuild,
|
|
beginRunnerBuild,
|
|
buildInputsStampPath,
|
|
readBuildInputsStamp,
|
|
recordRunnerBuild,
|
|
} from "../../packages/paperclip-runner/scripts/build-inputs.mjs";
|
|
import {
|
|
FORCE_REBUILD_VARIABLE,
|
|
REQUIRE_CURRENT_VARIABLE,
|
|
RUNNER_VENDOR_BUILD_COMMAND,
|
|
TIMING_SCHEMA,
|
|
decideRunnerVendorAction,
|
|
prepareRunnerVendor,
|
|
resolvePnpmInvocation,
|
|
} from "./prepare-runner-vendor.mjs";
|
|
|
|
const execFileAsync = promisify(execFile);
|
|
const prepareScript = fileURLToPath(new URL("./prepare-runner-vendor.mjs", import.meta.url));
|
|
const recordScript = fileURLToPath(
|
|
new URL("../../packages/paperclip-runner/scripts/record-build-inputs.mjs", import.meta.url),
|
|
);
|
|
const executable = process.platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd";
|
|
|
|
const roots = [];
|
|
|
|
function writeTree(root, files) {
|
|
for (const [relative, content] of Object.entries(files)) {
|
|
const absolute = path.join(root, ...relative.split("/"));
|
|
mkdirSync(path.dirname(absolute), { recursive: true });
|
|
writeFileSync(absolute, content);
|
|
}
|
|
}
|
|
|
|
/** A throwaway workspace whose Runner is already built, with no stamp yet. */
|
|
function makeRepo() {
|
|
const repoRoot = mkdtempSync(path.join(tmpdir(), "prepare-runner-vendor-"));
|
|
roots.push(repoRoot);
|
|
writeTree(repoRoot, {
|
|
"package.json": "{}\n",
|
|
"pnpm-lock.yaml": "lockfileVersion: 9\n",
|
|
"pnpm-workspace.yaml": "packages:\n - packages/*\n - server\n",
|
|
"tsconfig.base.json": "{}\n",
|
|
"patches/acpx@0.22.0.patch": "--- a\n+++ b\n",
|
|
"server/package.json": '{"name":"@paperclipai/server"}\n',
|
|
"packages/paperclip-runner/package.json": '{"name":"@paperclipai/paperclip-runner"}\n',
|
|
"packages/paperclip-runner/src/index.ts": "export const one = 1;\n",
|
|
"packages/paperclip-runner/dist/index.js": "export const one = 1;\n",
|
|
"packages/paperclip-runner/dist/live/index.js": "export {};\n",
|
|
"packages/paperclip-runner/dist/testing.js": "export {};\n",
|
|
"packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "module.exports = {};\n",
|
|
[`packages/paperclip-runner/dist/bin/${executable}`]: "binary\n",
|
|
"packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 1;\n",
|
|
});
|
|
const packageRoot = path.join(repoRoot, "packages", "paperclip-runner");
|
|
return { repoRoot, packageRoot };
|
|
}
|
|
|
|
/** What the Runner's build does around its steps when nothing changes meanwhile. */
|
|
function recordCurrentBuild({ repoRoot, packageRoot }) {
|
|
beginRunnerBuild({ repoRoot, packageRoot });
|
|
return recordRunnerBuild({ repoRoot, packageRoot });
|
|
}
|
|
|
|
afterEach(() => {
|
|
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
|
});
|
|
|
|
describe("decideRunnerVendorAction", () => {
|
|
const current = { current: true, code: "inputs_unchanged", detail: "unchanged" };
|
|
const stale = { current: false, code: "inputs_changed", detail: "changed" };
|
|
|
|
it("reuses a current build and rebuilds a stale one, carrying the assessment's reason", () => {
|
|
expect(decideRunnerVendorAction({ env: {}, assessment: current })).toEqual({
|
|
action: "reuse",
|
|
code: "inputs_unchanged",
|
|
detail: "unchanged",
|
|
});
|
|
expect(decideRunnerVendorAction({ env: {}, assessment: stale })).toEqual({
|
|
action: "rebuild",
|
|
code: "inputs_changed",
|
|
detail: "changed",
|
|
});
|
|
});
|
|
|
|
it("rebuilds a current build when the operator forces it", () => {
|
|
expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "1" }, assessment: current })).toEqual({
|
|
action: "rebuild",
|
|
code: "forced",
|
|
detail: `${FORCE_REBUILD_VARIABLE}=1`,
|
|
});
|
|
// Only the documented value forces; an unrelated value does not.
|
|
expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "0" }, assessment: current }).action).toBe(
|
|
"reuse",
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("decideRunnerVendorAction when the build must be current", () => {
|
|
const current = { current: true, code: "inputs_unchanged", detail: "unchanged" };
|
|
const stale = { current: false, code: "inputs_changed", detail: "changed" };
|
|
const required = { [REQUIRE_CURRENT_VARIABLE]: "1" };
|
|
|
|
it("reuses a current build and fails on a stale one instead of rebuilding", () => {
|
|
expect(decideRunnerVendorAction({ env: required, assessment: current }).action).toBe("reuse");
|
|
expect(decideRunnerVendorAction({ env: required, assessment: stale })).toEqual({
|
|
action: "fail",
|
|
code: "inputs_changed",
|
|
detail: "changed",
|
|
});
|
|
expect(decideRunnerVendorAction({ env: { [REQUIRE_CURRENT_VARIABLE]: "0" }, assessment: stale }).action).toBe("rebuild");
|
|
});
|
|
|
|
it("fails when a rebuild is forced as well: the two cannot both hold", () => {
|
|
expect(decideRunnerVendorAction({ env: { ...required, [FORCE_REBUILD_VARIABLE]: "1" }, assessment: current })).toMatchObject({
|
|
action: "fail",
|
|
code: "conflict",
|
|
});
|
|
});
|
|
});
|
|
|
|
describe("resolvePnpmInvocation", () => {
|
|
it("runs the pnpm that is running the script, through Node for a JavaScript entry", () => {
|
|
expect(resolvePnpmInvocation({ npm_execpath: "/corepack/pnpm/9.15.4/bin/pnpm.cjs" }, "/usr/bin/node")).toEqual({
|
|
command: "/usr/bin/node",
|
|
argsPrefix: ["/corepack/pnpm/9.15.4/bin/pnpm.cjs"],
|
|
});
|
|
expect(resolvePnpmInvocation({ npm_execpath: "/opt/pnpm/pnpm" }, "/usr/bin/node")).toEqual({
|
|
command: "/opt/pnpm/pnpm",
|
|
argsPrefix: [],
|
|
});
|
|
expect(resolvePnpmInvocation({ npm_execpath: "C:\\pnpm\\pnpm.exe" }, "node.exe")).toEqual({
|
|
command: "C:\\pnpm\\pnpm.exe",
|
|
argsPrefix: [],
|
|
});
|
|
});
|
|
|
|
it("falls back to pnpm on PATH outside a pnpm script or under another package manager", () => {
|
|
expect(resolvePnpmInvocation({}, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] });
|
|
expect(resolvePnpmInvocation({ npm_execpath: "" }, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] });
|
|
expect(resolvePnpmInvocation({ npm_execpath: "/usr/lib/node_modules/npm/bin/npm-cli.js" }, "/usr/bin/node")).toEqual({
|
|
command: "pnpm",
|
|
argsPrefix: [],
|
|
});
|
|
});
|
|
});
|
|
|
|
describe("prepareRunnerVendor", () => {
|
|
it("still runs the Runner build the server build has always run", () => {
|
|
// The Dockerfile's standalone `pnpm --filter @paperclipai/server build`
|
|
// depends on this exact command producing a current vendored Runner.
|
|
expect(RUNNER_VENDOR_BUILD_COMMAND).toBe("pnpm --filter @paperclipai/paperclip-runner build");
|
|
});
|
|
|
|
it("skips the rebuild when the recorded build is current and reports the check time", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
recordCurrentBuild({ repoRoot, packageRoot });
|
|
const runs = [];
|
|
const lines = [];
|
|
|
|
const status = prepareRunnerVendor({
|
|
repoRoot,
|
|
env: {},
|
|
run: (...args) => {
|
|
runs.push(args);
|
|
return 0;
|
|
},
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
expect(status).toBe(0);
|
|
expect(runs).toEqual([]);
|
|
expect(lines[0]).toMatch(/^prepare:runner-vendor: reusing the current Runner build \(inputs unchanged/);
|
|
expect(lines[1]).toMatch(
|
|
new RegExp(`^${TIMING_SCHEMA},action:reuse,reason:inputs_unchanged,check_ms:\\d+,build_ms:0$`),
|
|
);
|
|
expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).not.toBeNull();
|
|
});
|
|
|
|
it("rebuilds when the inputs changed, removing the stale stamp before the build starts", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
recordCurrentBuild({ repoRoot, packageRoot });
|
|
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
|
|
const stampPath = buildInputsStampPath(packageRoot);
|
|
const observed = [];
|
|
const lines = [];
|
|
const env = { npm_execpath: "/corepack/pnpm.cjs" };
|
|
|
|
const status = prepareRunnerVendor({
|
|
repoRoot,
|
|
env,
|
|
run: (command, options) => {
|
|
observed.push({ command, ...options, stampPresent: readBuildInputsStamp(stampPath) !== null });
|
|
return 0;
|
|
},
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
expect(status).toBe(0);
|
|
expect(observed).toEqual([
|
|
{ command: RUNNER_VENDOR_BUILD_COMMAND, cwd: path.join(repoRoot, "server"), env, stampPresent: false },
|
|
]);
|
|
expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(inputs changed/);
|
|
expect(lines[1]).toMatch(
|
|
new RegExp(`^${TIMING_SCHEMA},action:rebuild,reason:inputs_changed,check_ms:\\d+,build_ms:\\d+,status:0$`),
|
|
);
|
|
});
|
|
|
|
it("rebuilds when a file the build left under dist is gone, even one no consumer imports", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
recordCurrentBuild({ repoRoot, packageRoot });
|
|
rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs"));
|
|
let runs = 0;
|
|
const lines = [];
|
|
|
|
const status = prepareRunnerVendor({
|
|
repoRoot,
|
|
env: {},
|
|
run: () => {
|
|
runs += 1;
|
|
return 0;
|
|
},
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
expect(status).toBe(0);
|
|
expect(runs).toBe(1);
|
|
expect(lines[0]).toMatch(
|
|
/^prepare:runner-vendor: rebuilding the Runner \(build output dist\/cli\/acpx-runtime-sidecar\.cjs is missing since /,
|
|
);
|
|
expect(lines[1]).toMatch(/,action:rebuild,reason:outputs_changed,/);
|
|
});
|
|
|
|
it("returns the build's failure status and leaves no stamp behind", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
const lines = [];
|
|
|
|
const status = prepareRunnerVendor({
|
|
repoRoot,
|
|
env: {},
|
|
run: () => 3,
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
expect(status).toBe(3);
|
|
expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(no build stamp/);
|
|
expect(lines[1]).toMatch(/,action:rebuild,reason:no_stamp,.*,status:3$/);
|
|
expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).toBeNull();
|
|
});
|
|
|
|
it("rebuilds a current build when PAPERCLIP_RUNNER_VENDOR_REBUILD=1", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
recordCurrentBuild({ repoRoot, packageRoot });
|
|
let runs = 0;
|
|
const lines = [];
|
|
|
|
const status = prepareRunnerVendor({
|
|
repoRoot,
|
|
env: { [FORCE_REBUILD_VARIABLE]: "1" },
|
|
run: () => {
|
|
runs += 1;
|
|
return 0;
|
|
},
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
expect(status).toBe(0);
|
|
expect(runs).toBe(1);
|
|
expect(lines[1]).toMatch(/,action:rebuild,reason:forced,/);
|
|
});
|
|
|
|
it("with PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1, reuses a current build and never starts a rebuild", () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
recordCurrentBuild({ repoRoot, packageRoot });
|
|
const stampPath = buildInputsStampPath(packageRoot);
|
|
const env = { [REQUIRE_CURRENT_VARIABLE]: "1" };
|
|
let runs = 0;
|
|
const prepare = (lines) =>
|
|
prepareRunnerVendor({
|
|
repoRoot,
|
|
env,
|
|
run: () => {
|
|
runs += 1;
|
|
return 0;
|
|
},
|
|
log: (line) => lines.push(line),
|
|
});
|
|
|
|
const reused = [];
|
|
expect(prepare(reused)).toBe(0);
|
|
expect(reused[1]).toMatch(/,action:reuse,reason:inputs_unchanged,/);
|
|
|
|
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
|
|
const failed = [];
|
|
expect(prepare(failed)).toBe(1);
|
|
expect(runs).toBe(0);
|
|
expect(failed[0]).toMatch(
|
|
new RegExp(`^prepare:runner-vendor: ${REQUIRE_CURRENT_VARIABLE}=1 forbids the Runner rebuild this step needs \\(inputs changed`),
|
|
);
|
|
expect(failed[1]).toMatch(
|
|
new RegExp(`^${TIMING_SCHEMA},action:fail,reason:inputs_changed,check_ms:\\d+,build_ms:0,status:1$`),
|
|
);
|
|
// The stamp still describes the build it was written for; only a rebuild removes it.
|
|
expect(readBuildInputsStamp(stampPath)).not.toBeNull();
|
|
});
|
|
|
|
it("with PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1, accepts a Runner built elsewhere for the committed lockfile and refuses it once the lockfile is refreshed for another package", () => {
|
|
const committed = [
|
|
"lockfileVersion: '9.0'",
|
|
"importers:",
|
|
" packages/paperclip-runner:",
|
|
" dependencies:",
|
|
" ajv: { specifier: ^8.20.0, version: 8.20.0 }",
|
|
" ui:",
|
|
" dependencies:",
|
|
" react: { specifier: ^19.2.7, version: 19.2.8 }",
|
|
"",
|
|
].join("\n");
|
|
const built = makeRepo();
|
|
writeFileSync(path.join(built.repoRoot, "pnpm-lock.yaml"), committed);
|
|
recordCurrentBuild(built);
|
|
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
writeFileSync(path.join(repoRoot, "pnpm-lock.yaml"), committed);
|
|
writeTree(repoRoot, {
|
|
"packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json": readFileSync(buildInputsStampPath(built.packageRoot), "utf8"),
|
|
});
|
|
const env = { [REQUIRE_CURRENT_VARIABLE]: "1" };
|
|
let runs = 0;
|
|
const prepare = (lines) => prepareRunnerVendor({ repoRoot, env, run: () => { runs += 1; return 0; }, log: (line) => lines.push(line) });
|
|
|
|
const reused = [];
|
|
expect(prepare(reused)).toBe(0);
|
|
expect(reused[1]).toMatch(/,action:reuse,reason:inputs_unchanged,/);
|
|
|
|
writeFileSync(path.join(repoRoot, "pnpm-lock.yaml"), committed.replace("specifier: ^19.2.7", "specifier: ^19.2.8"));
|
|
const refused = [];
|
|
expect(prepare(refused)).toBe(1);
|
|
expect(refused[0]).toContain("pnpm-lock.yaml included");
|
|
expect(refused[1]).toMatch(/,action:fail,reason:inputs_changed,/);
|
|
|
|
writeFileSync(path.join(repoRoot, "pnpm-lock.yaml"), committed);
|
|
expect(prepare([])).toBe(0);
|
|
expect(runs).toBe(0);
|
|
expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).not.toBeNull();
|
|
});
|
|
|
|
it.skipIf(process.platform === "win32")(
|
|
"end to end: the first server build rebuilds the Runner, the second reuses it, a change or a lost output rebuilds again",
|
|
async () => {
|
|
const { repoRoot, packageRoot } = makeRepo();
|
|
// A stand-in pnpm that behaves like the Runner build: it records how it
|
|
// was called, then brackets its (empty) step list with the same two
|
|
// record-build-inputs.mjs calls the real build makes.
|
|
const binDir = path.join(repoRoot, "bin");
|
|
const argvLog = path.join(repoRoot, "pnpm-argv.log");
|
|
mkdirSync(binDir);
|
|
const pnpm = path.join(binDir, "pnpm");
|
|
writeFileSync(
|
|
pnpm,
|
|
`${[
|
|
"#!/bin/sh",
|
|
"set -e",
|
|
`printf '%s\\n' "$*" >> "${argvLog}"`,
|
|
`"${process.execPath}" "${recordScript}" --begin --package-root "${packageRoot}" > /dev/null`,
|
|
`"${process.execPath}" "${recordScript}" --finish --package-root "${packageRoot}" > /dev/null`,
|
|
].join("\n")}\n`,
|
|
);
|
|
chmodSync(pnpm, 0o755);
|
|
const env = { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}` };
|
|
delete env[FORCE_REBUILD_VARIABLE];
|
|
delete env[REQUIRE_CURRENT_VARIABLE];
|
|
// The suite itself may run under pnpm; the script must then take the
|
|
// PATH fallback so the stand-in, not the real pnpm, receives the build.
|
|
delete env.npm_execpath;
|
|
const run = () => execFileAsync(process.execPath, [prepareScript, "--repo-root", repoRoot], { env });
|
|
const calls = () => readFileSync(argvLog, "utf8").split("\n").filter(Boolean);
|
|
|
|
const first = await run();
|
|
expect(calls()).toEqual(["--filter @paperclipai/paperclip-runner build"]);
|
|
expect(first.stdout).toMatch(/action:rebuild,reason:no_stamp,/);
|
|
expect(assessRunnerBuild({ repoRoot, packageRoot }).current).toBe(true);
|
|
|
|
const second = await run();
|
|
expect(calls()).toHaveLength(1);
|
|
expect(second.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/);
|
|
|
|
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n");
|
|
const third = await run();
|
|
expect(calls()).toHaveLength(2);
|
|
expect(third.stdout).toMatch(/action:rebuild,reason:inputs_changed,/);
|
|
|
|
rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs"));
|
|
const fourth = await run();
|
|
expect(calls()).toHaveLength(3);
|
|
expect(fourth.stdout).toMatch(/action:rebuild,reason:outputs_changed,/);
|
|
// The stand-in build does not recreate the helper, so the new stamp's
|
|
// manifest no longer lists it and the next run reuses the build.
|
|
const fifth = await run();
|
|
expect(calls()).toHaveLength(3);
|
|
expect(fifth.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/);
|
|
|
|
// Required to be current: a stale build fails the step and pnpm is not called.
|
|
writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 3;\n");
|
|
const required = { ...env, [REQUIRE_CURRENT_VARIABLE]: "1" };
|
|
await expect(execFileAsync(process.execPath, [prepareScript, "--repo-root", repoRoot], { env: required })).rejects.toMatchObject({
|
|
code: 1,
|
|
stdout: expect.stringMatching(/action:fail,reason:inputs_changed,/),
|
|
});
|
|
expect(calls()).toHaveLength(3);
|
|
},
|
|
);
|
|
});
|