mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-<digest>`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
625 lines
34 KiB
JavaScript
625 lines
34 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import { execFileSync, spawnSync } from "node:child_process";
|
|
import { chmodSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync } from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import test from "node:test";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
import { CACHED_PATHS, HASHED_INPUTS, REQUIRED, RUNNER_PACKAGE, computeScope, markCurrent } from "../runner-release-crate-cache.mjs";
|
|
|
|
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
|
|
const script = path.join(repoRoot, "scripts/runner-release-crate-cache.mjs");
|
|
const readWorkflow = (name) => readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8");
|
|
const prWorkflow = readWorkflow("pr-trusted.yml");
|
|
const releaseWorkflow = readWorkflow("release-verify.yml");
|
|
|
|
// Slice one job out of a workflow, from its two-space-indented key to the next.
|
|
function job(workflow, name) {
|
|
const start = workflow.indexOf(`\n ${name}:\n`);
|
|
assert.ok(start >= 0, `workflow is missing the ${name} job`);
|
|
const body = workflow.slice(start + 1);
|
|
const header = ` ${name}:\n`.length;
|
|
const next = body.slice(header).search(/\n [a-z0-9_-]+:\n/);
|
|
return next === -1 ? body : body.slice(0, header + next + 1);
|
|
}
|
|
|
|
// Slice one step out of a job by its name, up to the next step.
|
|
function step(body, name) {
|
|
const marker = ` - name: ${name}\n`;
|
|
const start = body.indexOf(marker);
|
|
assert.ok(start >= 0, `missing step: ${name}`);
|
|
const rest = body.slice(start + marker.length);
|
|
const next = rest.search(/\n - name: /);
|
|
return marker + (next === -1 ? rest : rest.slice(0, next + 1));
|
|
}
|
|
|
|
const RELEASE_DIR = `${RUNNER_PACKAGE}/runner/target/release`;
|
|
const KEY_STEP = "Compute the Runner release crate cache key";
|
|
const RESTORE_STEP = "Restore the Runner release crate (read only)";
|
|
const MARK_STEP = "Mark the restored Runner release crate current";
|
|
const LOOKUP_STEP = "Check for this push's Runner release crate";
|
|
const BUILD_STEP = "Build the Runner release crate for its cache";
|
|
const SAVE_STEP = "Save the Runner release crate";
|
|
const RUST_RESTORE_STEP = "Restore Runner Rust dependencies (read only)";
|
|
|
|
// Every PR job that builds the Runner release binary, and the command each
|
|
// one runs cargo through. The crate must be restored and marked before it.
|
|
const READERS = [
|
|
["typecheck_release_registry", "run: pnpm run typecheck:build-gaps"],
|
|
["verify_paperclip_runner", "run: pnpm --filter @paperclipai/paperclip-runner ${{ matrix.command }}"],
|
|
["build", "run: pnpm build"],
|
|
["canary_dry_run", "- name: Release canary dry run via release.sh internal build"],
|
|
];
|
|
|
|
const WRITER_GATE = "matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha";
|
|
|
|
const expectedPathBlock = ` path: |\n${CACHED_PATHS.map((entry) => ` ${entry}\n`).join("")}`;
|
|
const expectedKey = `key: runner-release-crate-v1-\${{ steps.runner_release_crate_key.outputs.scope }}-\${{ hashFiles(${HASHED_INPUTS.map((entry) => `'${entry}'`).join(", ")}) }}`;
|
|
|
|
function assertCacheStep(body, name, { lookupOnly = false } = {}) {
|
|
assert.match(body, /uses: actions\/cache\/(restore|save)@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6\.1\.0/, name);
|
|
assert.ok(body.includes(expectedPathBlock), `${name}: the cached path list must be exactly the script's CACHED_PATHS`);
|
|
assert.ok(body.includes(expectedKey), `${name}: the key must be built from the script's scope and HASHED_INPUTS`);
|
|
// An exact match or nothing: a prefix match would restore another
|
|
// revision's artifacts and the mark step would make cargo keep them.
|
|
assert.doesNotMatch(body, /restore-keys:/, name);
|
|
assert.equal(/lookup-only: true/.test(body), lookupOnly, name);
|
|
assert.doesNotMatch(body, /enableCrossOsArchive|fail-on-cache-miss: true/, name);
|
|
}
|
|
|
|
test("the script's path list and key inputs cover what the release build of the crate reads and writes", () => {
|
|
assert.deepEqual(CACHED_PATHS, [
|
|
`${RELEASE_DIR}/.fingerprint/paperclip-runner-core-*`,
|
|
`${RELEASE_DIR}/deps/libpaperclip_runner_core-*`,
|
|
`${RELEASE_DIR}/deps/paperclip_runner_core-*.d`,
|
|
`${RELEASE_DIR}/deps/paperclip_runnerd-*`,
|
|
`${RELEASE_DIR}/paperclip-runnerd`,
|
|
`${RELEASE_DIR}/paperclip-runnerd.d`,
|
|
`${RELEASE_DIR}/.fingerprint/paperclip-process-table-*`,
|
|
`${RELEASE_DIR}/deps/libpaperclip_process_table-*`,
|
|
`${RELEASE_DIR}/deps/paperclip_process_table-*.d`,
|
|
]);
|
|
assert.deepEqual(HASHED_INPUTS, [
|
|
`${RUNNER_PACKAGE}/rust-toolchain.toml`,
|
|
`${RUNNER_PACKAGE}/acpx-profiles.json`,
|
|
`${RUNNER_PACKAGE}/antigravity-distributions.json`,
|
|
`${RUNNER_PACKAGE}/runner/Cargo.toml`,
|
|
`${RUNNER_PACKAGE}/runner/Cargo.lock`,
|
|
`${RUNNER_PACKAGE}/runner/crates/**`,
|
|
`${RUNNER_PACKAGE}/protocol/**`,
|
|
`${RUNNER_PACKAGE}/test-fixtures/**`,
|
|
`${RUNNER_PACKAGE}/test/fixtures/**`,
|
|
"tests/runner-e2e/fixtures/**",
|
|
]);
|
|
for (const input of HASHED_INPUTS) {
|
|
const fixed = input.replace(/\/\*\*$/, "");
|
|
assert.ok(statSync(path.join(repoRoot, fixed)), `${input} must exist in the repository`);
|
|
}
|
|
// The stage step copies the uplifted binary; it has to be in the set.
|
|
const stage = readFileSync(path.join(repoRoot, RUNNER_PACKAGE, "scripts/stage-runner-binary.mjs"), "utf8");
|
|
assert.match(stage, /"runner", "target", "release", executable/);
|
|
assert.ok(CACHED_PATHS.includes(`${RELEASE_DIR}/paperclip-runnerd`));
|
|
});
|
|
|
|
// Every `include_str!` / `include_bytes!` in one Rust source, with the path
|
|
// each one names. Three argument forms are resolved: a string literal
|
|
// (relative to the source file), and `concat!(...)` of string literals,
|
|
// `env!("CARGO_MANIFEST_DIR")` (the directory of the nearest Cargo.toml) and
|
|
// macro metavariables (`$name`, which make the resolved path a directory
|
|
// prefix rather than a file). Anything else is reported as unsupported, so
|
|
// a new include form cannot slip past the guard by not matching a regex.
|
|
function crateIncludes(source, text) {
|
|
const includes = [];
|
|
for (const match of text.matchAll(/include_(?:str|bytes)!\s*\(/g)) {
|
|
const start = match.index + match[0].length;
|
|
let depth = 1;
|
|
let inString = false;
|
|
let end = start;
|
|
for (; end < text.length && depth > 0; end += 1) {
|
|
const ch = text[end];
|
|
if (inString) {
|
|
if (ch === "\\") end += 1;
|
|
else if (ch === '"') inString = false;
|
|
} else if (ch === '"') inString = true;
|
|
else if (ch === "(") depth += 1;
|
|
else if (ch === ")") depth -= 1;
|
|
}
|
|
assert.equal(depth, 0, `${source}: unbalanced include at offset ${match.index}`);
|
|
const argument = text.slice(start, end - 1).trim();
|
|
includes.push({ argument, ...resolveIncludeArgument(source, argument) });
|
|
}
|
|
return includes;
|
|
}
|
|
|
|
function manifestDir(source) {
|
|
let dir = path.dirname(source);
|
|
while (!statSync(path.join(dir, "Cargo.toml"), { throwIfNoEntry: false })?.isFile()) {
|
|
const parent = path.dirname(dir);
|
|
assert.notEqual(parent, dir, `${source}: no Cargo.toml above the source`);
|
|
dir = parent;
|
|
}
|
|
return dir;
|
|
}
|
|
|
|
function resolveIncludeArgument(source, argument) {
|
|
const literal = argument.match(/^"((?:[^"\\]|\\.)*)"$/);
|
|
if (literal) return { form: "literal", resolved: path.resolve(path.dirname(source), literal[1]), exact: true };
|
|
|
|
const concat = argument.match(/^concat!\s*\(([\s\S]*)\)$/);
|
|
assert.ok(concat, `${source}: unsupported include form: ${argument}`);
|
|
const parts = splitTopLevel(concat[1]);
|
|
// A relative path is joined to the source's directory; one that starts
|
|
// with the manifest directory is one string, since its next literal
|
|
// starts with the "/" that separates the two.
|
|
let base = `${path.dirname(source)}/`;
|
|
let text = "";
|
|
let exact = true;
|
|
let form = "concat";
|
|
for (const [index, part] of parts.entries()) {
|
|
const string = part.match(/^"((?:[^"\\]|\\.)*)"$/);
|
|
if (string) {
|
|
assert.ok(exact || !/(^|\/)\.\.(\/|$)/.test(string[1]), `${source}: a path segment after a metavariable must not climb out: ${argument}`);
|
|
if (exact) text += string[1];
|
|
continue;
|
|
}
|
|
if (part === 'env!("CARGO_MANIFEST_DIR")') {
|
|
assert.equal(index, 0, `${source}: env!("CARGO_MANIFEST_DIR") must be the first part of concat!: ${argument}`);
|
|
base = manifestDir(source);
|
|
form = "concat-manifest-dir";
|
|
continue;
|
|
}
|
|
if (/^\$[A-Za-z_][A-Za-z0-9_]*$/.test(part)) {
|
|
// The fixed prefix before the first metavariable is the directory the
|
|
// macro's callers can reach; the suffix cannot climb out of it.
|
|
exact = false;
|
|
form = form === "concat-manifest-dir" ? form : "concat-metavariable";
|
|
continue;
|
|
}
|
|
assert.fail(`${source}: unsupported concat! part in include: ${part}`);
|
|
}
|
|
const joined = base + text;
|
|
const resolved = path.resolve(exact || text.endsWith("/") ? joined : path.dirname(joined));
|
|
return { form, resolved, exact };
|
|
}
|
|
|
|
// Split concat!'s arguments on the commas outside string literals.
|
|
function splitTopLevel(text) {
|
|
const parts = [];
|
|
let current = "";
|
|
let inString = false;
|
|
for (let index = 0; index < text.length; index += 1) {
|
|
const ch = text[index];
|
|
if (inString) {
|
|
current += ch;
|
|
if (ch === "\\") current += text[++index] ?? "";
|
|
else if (ch === '"') inString = false;
|
|
} else if (ch === '"') {
|
|
inString = true;
|
|
current += ch;
|
|
} else if (ch === ",") {
|
|
parts.push(current);
|
|
current = "";
|
|
} else current += ch;
|
|
}
|
|
parts.push(current);
|
|
return parts.map((part) => part.trim()).filter(Boolean);
|
|
}
|
|
|
|
const underHashedInputs = (resolved) =>
|
|
HASHED_INPUTS.map((input) => path.join(repoRoot, input.replace(/\/\*\*$/, ""))).some((root) => resolved === root || resolved.startsWith(`${root}${path.sep}`));
|
|
|
|
test("the include parser resolves literals, concat! with the manifest directory and with metavariables, and rejects other forms", () => {
|
|
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-includes-"));
|
|
try {
|
|
const crate = path.join(root, "crates/demo");
|
|
mkdirSync(path.join(crate, "src/nested"), { recursive: true });
|
|
mkdirSync(path.join(root, "protocol/fixtures"), { recursive: true });
|
|
writeFileSync(path.join(crate, "Cargo.toml"), '[package]\nname = "demo"\n');
|
|
writeFileSync(path.join(root, "protocol/schema.json"), "{}");
|
|
const source = path.join(crate, "src/nested/mod.rs");
|
|
const text = [
|
|
'let a = include_str!("../../../../protocol/schema.json");',
|
|
'let b = include_bytes!(\n "../../../../protocol/schema.json"\n);',
|
|
'let c = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../protocol/", "schema.json"));',
|
|
'let d = include_str!(concat!("../../../../protocol/fixtures/", $fixture, ".json"));',
|
|
'let e = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../protocol/fixtures/", $golden, ".summary.json"));',
|
|
'let f = include_str!("a (paren) in the name.json");',
|
|
].join("\n");
|
|
const includes = crateIncludes(source, text);
|
|
assert.deepEqual(
|
|
includes.map(({ form, resolved, exact }) => [form, path.relative(root, resolved), exact]),
|
|
[
|
|
["literal", "protocol/schema.json", true],
|
|
["literal", "protocol/schema.json", true],
|
|
["concat-manifest-dir", "protocol/schema.json", true],
|
|
["concat-metavariable", "protocol/fixtures", false],
|
|
["concat-manifest-dir", "protocol/fixtures", false],
|
|
["literal", "crates/demo/src/nested/a (paren) in the name.json", true],
|
|
],
|
|
);
|
|
|
|
const rejected = [
|
|
'include_str!(concat!("../protocol/", $x, "/../../../outside.json"))',
|
|
'include_str!(concat!("../protocol/", env!("CARGO_MANIFEST_DIR")))',
|
|
'include_str!(concat!(env!("OTHER_DIR"), "/schema.json"))',
|
|
'include_str!(r"../protocol/schema.json")',
|
|
'include_str!(path_from_a_function())',
|
|
'include_str!(concat!("../protocol/", some_macro!("x")))',
|
|
];
|
|
for (const argument of rejected) {
|
|
assert.throws(() => crateIncludes(source, argument), /unsupported|must not climb out|must be the first part/, argument);
|
|
}
|
|
} finally {
|
|
rmSync(root, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
test("every file the crate includes by path sits under a hashed input, so an edit to it changes the key", () => {
|
|
const crateRoot = path.join(repoRoot, RUNNER_PACKAGE, "runner/crates");
|
|
const sources = [];
|
|
(function walk(dir) {
|
|
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
|
const full = path.join(dir, entry.name);
|
|
if (entry.isDirectory()) walk(full);
|
|
else if (entry.name.endsWith(".rs")) sources.push(full);
|
|
}
|
|
})(crateRoot);
|
|
assert.ok(sources.length > 10, "the crate sources must be found");
|
|
const forms = new Set();
|
|
let includes = 0;
|
|
for (const source of sources) {
|
|
for (const { argument, form, resolved, exact } of crateIncludes(source, readFileSync(source, "utf8"))) {
|
|
includes += 1;
|
|
forms.add(form);
|
|
const where = `${path.relative(repoRoot, source)} includes ${argument.replace(/\s+/g, " ")}`;
|
|
assert.ok(underHashedInputs(resolved), `${where}, which resolves to ${path.relative(repoRoot, resolved)}, outside every HASHED_INPUTS pattern`);
|
|
const info = statSync(resolved, { throwIfNoEntry: false });
|
|
assert.ok(exact ? info?.isFile() : info?.isDirectory(), `${where}: ${path.relative(repoRoot, resolved)} must exist`);
|
|
}
|
|
}
|
|
assert.ok(includes > 0, "the crate includes protocol files by path; the guard must see them");
|
|
// The crate uses every supported form today, so none of the parser's
|
|
// branches is exercised only by the synthetic test above.
|
|
assert.deepEqual([...forms].sort(), ["concat-manifest-dir", "concat-metavariable", "literal"]);
|
|
});
|
|
|
|
test("isolated Docker Rust stages preserve the layout of embedded schema and identity inputs", () => {
|
|
const crateRoot = path.join(repoRoot, RUNNER_PACKAGE, "runner/crates");
|
|
const sources = [];
|
|
(function walk(dir) {
|
|
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
|
const full = path.join(dir, entry.name);
|
|
if (entry.isDirectory()) walk(full);
|
|
else if (entry.name.endsWith(".rs")) sources.push(full);
|
|
}
|
|
})(crateRoot);
|
|
for (const [dockerfile, stage] of [["Dockerfile", "runner-artifacts-runnerd"], ["docker/daytona-runner/Dockerfile", "runnerd-build"]]) {
|
|
const docker = readFileSync(path.join(repoRoot, dockerfile), "utf8");
|
|
const body = docker.split(/^FROM .+ AS /m).find(part => part.startsWith(`${stage}\n`));
|
|
assert.ok(body, `${dockerfile}: missing ${stage}`);
|
|
const workdir = /^WORKDIR (.+)$/m.exec(body)?.[1];
|
|
assert.ok(workdir?.startsWith("/"), `${dockerfile}: an absolute stage working directory is required`);
|
|
const copies = [...body.matchAll(/^COPY (?:--from=runner-artifacts-context )?(?!-)(.+)$/gm)].flatMap(([, instruction]) => {
|
|
const args = instruction.trim().split(/\s+/);
|
|
const destination = path.posix.resolve(workdir, args.pop());
|
|
return args.map(source => {
|
|
const absolute = path.join(repoRoot, source);
|
|
return { source: absolute, destination: statSync(absolute).isDirectory() ? destination : path.posix.join(destination, path.basename(source)) };
|
|
});
|
|
});
|
|
const copiedPath = source => {
|
|
const copy = copies.find(copy => source === copy.source || source.startsWith(`${copy.source}${path.sep}`));
|
|
return copy ? path.posix.join(copy.destination, path.relative(copy.source, source)) : null;
|
|
};
|
|
let checked = 0;
|
|
for (const source of sources) {
|
|
for (const { resolved } of crateIncludes(source, readFileSync(source, "utf8"))) {
|
|
// Cargo test-only fixture trees are intentionally absent from production
|
|
// Docker stages. Their cache coverage is checked by the preceding test.
|
|
if (/\/(?:test-fixtures|test\/fixtures|tests\/runner-e2e\/fixtures)(?:\/|$)/.test(resolved)) continue;
|
|
const stagedSource = copiedPath(source);
|
|
assert.ok(stagedSource, `${dockerfile}: crate source was not copied: ${path.relative(repoRoot, source)}`);
|
|
const expected = path.posix.resolve(path.posix.dirname(stagedSource), path.relative(path.dirname(source), resolved));
|
|
assert.equal(copiedPath(resolved), expected, `${dockerfile}: include input ${path.relative(repoRoot, resolved)} must retain its crate-relative path`);
|
|
checked++;
|
|
}
|
|
}
|
|
assert.ok(checked > 0, `${dockerfile}: the guard must see real Rust include paths`);
|
|
}
|
|
});
|
|
|
|
test("every PR reader restores the crate after the dependency cache and marks it before building", () => {
|
|
for (const [name, buildCommand] of READERS) {
|
|
const body = job(prWorkflow, name);
|
|
const positions = [RUST_RESTORE_STEP, KEY_STEP, RESTORE_STEP, MARK_STEP].map((stepName) => body.indexOf(` - name: ${stepName}\n`));
|
|
assert.ok(positions.every((position) => position >= 0), `${name}: a crate cache step is missing`);
|
|
assert.deepEqual([...positions].sort((a, b) => a - b), positions, `${name}: the crate steps must follow the dependency restore in order`);
|
|
const build = body.indexOf(buildCommand);
|
|
assert.ok(build > positions.at(-1), `${name}: the crate must be marked before ${buildCommand}`);
|
|
|
|
const key = step(body, KEY_STEP);
|
|
assert.match(key, /id: runner_release_crate_key\n/, name);
|
|
assert.match(key, /run: echo "scope=\$\(node scripts\/runner-release-crate-cache\.mjs scope\)" >> "\$GITHUB_OUTPUT"/, name);
|
|
|
|
const restore = step(body, RESTORE_STEP);
|
|
assert.match(restore, /id: runner_release_crate\n/, name);
|
|
assertCacheStep(restore, `${name} restore`);
|
|
assert.doesNotMatch(restore, /actions\/cache\/save/, name);
|
|
|
|
const mark = step(body, MARK_STEP);
|
|
assert.match(mark, /if: \$\{\{ steps\.runner_release_crate\.outputs\.cache-hit == 'true' \}\}/, name);
|
|
assert.match(mark, /run: node scripts\/runner-release-crate-cache\.mjs mark-current/, name);
|
|
}
|
|
// No other job restores or saves the crate.
|
|
const total = prWorkflow.match(/- name: Restore the Runner release crate \(read only\)\n/g)?.length ?? 0;
|
|
assert.equal(total, READERS.length);
|
|
assert.doesNotMatch(prWorkflow, /actions\/cache\/save@/);
|
|
});
|
|
|
|
test("the Verify Paperclip Runner job restores the crate only in the lanes that build it", () => {
|
|
const body = job(prWorkflow, "verify_paperclip_runner");
|
|
const lanes = JSON.parse(readFileSync(path.join(repoRoot, ".github/ci-task-matrix.json"), "utf8"))
|
|
.verify_paperclip_runner.map((entry) => [entry.lane_label, entry.release_crate === true]);
|
|
assert.deepEqual(lanes, [
|
|
["static checks", false],
|
|
["rust", true],
|
|
["vitest package (1/3)", false],
|
|
["vitest package (2/3)", false],
|
|
["vitest package (3/3)", false],
|
|
["vitest native server", true],
|
|
]);
|
|
for (const stepName of [KEY_STEP, RESTORE_STEP]) {
|
|
assert.match(step(body, stepName), /if: \$\{\{ matrix\.release_crate == true \}\}/, stepName);
|
|
}
|
|
// A lane that skipped the restore has no cache-hit output, so the mark
|
|
// step's own condition is enough there.
|
|
assert.match(step(body, MARK_STEP), /if: \$\{\{ steps\.runner_release_crate\.outputs\.cache-hit == 'true' \}\}/);
|
|
for (const [name] of READERS.filter(([name]) => name !== "verify_paperclip_runner")) {
|
|
for (const stepName of [KEY_STEP, RESTORE_STEP]) {
|
|
assert.doesNotMatch(step(job(prWorkflow, name), stepName), /^\s+if:/m, `${name}: ${stepName} is unconditional`);
|
|
}
|
|
}
|
|
});
|
|
|
|
test("only the master Rust lane writes the crate, after a lookup, with the readers' exact path list and key", () => {
|
|
const body = job(releaseWorkflow, "verify_paperclip_runner");
|
|
const positions = ["Verify Paperclip Runner", "Warm debug dependencies for the shared Runner cache", KEY_STEP, LOOKUP_STEP, BUILD_STEP, SAVE_STEP].map((stepName) => body.indexOf(` - name: ${stepName}\n`));
|
|
assert.ok(positions.every((position) => position >= 0), "a writer step is missing");
|
|
assert.deepEqual([...positions].sort((a, b) => a - b), positions, "the writer steps must follow the checks and the debug warm-up, in order");
|
|
|
|
for (const stepName of [KEY_STEP, LOOKUP_STEP]) {
|
|
assert.match(step(body, stepName), new RegExp(`if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} \\}\\}`), stepName);
|
|
}
|
|
for (const stepName of [BUILD_STEP, SAVE_STEP]) {
|
|
assert.match(
|
|
step(body, stepName),
|
|
new RegExp(`if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} && steps\\.runner_release_crate\\.outputs\\.cache-hit != 'true' \\}\\}`),
|
|
stepName,
|
|
);
|
|
}
|
|
assert.match(step(body, KEY_STEP), /run: echo "scope=\$\(node scripts\/runner-release-crate-cache\.mjs scope\)" >> "\$GITHUB_OUTPUT"/);
|
|
|
|
const lookup = step(body, LOOKUP_STEP);
|
|
assert.match(lookup, /id: runner_release_crate\n/);
|
|
assert.match(lookup, /uses: actions\/cache\/restore@/);
|
|
assertCacheStep(lookup, "writer lookup", { lookupOnly: true });
|
|
|
|
// The saved artifacts come from the readers' exact command, on a crate
|
|
// cleaned of the test-profile variants the lane's checks built.
|
|
const build = step(body, BUILD_STEP);
|
|
assert.match(build, /working-directory: packages\/paperclip-runner\n/);
|
|
assert.match(build, /set -euo pipefail\n\s+cargo clean --release --manifest-path runner\/Cargo\.toml -p paperclip-runner-core\n\s+pnpm build:binary\n/);
|
|
const runnerScripts = JSON.parse(readFileSync(path.join(repoRoot, RUNNER_PACKAGE, "package.json"), "utf8")).scripts;
|
|
assert.match(runnerScripts["build:binary"], /^cargo build --release --manifest-path runner\/Cargo\.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && /);
|
|
|
|
const save = step(body, SAVE_STEP);
|
|
assert.match(save, /uses: actions\/cache\/save@/);
|
|
assertCacheStep(save, "writer save");
|
|
|
|
// The dependency cache's own writer gate is the same expression.
|
|
assert.match(body, new RegExp(`save-if: \\$\\{\\{ ${WRITER_GATE.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} \\}\\}`));
|
|
assert.equal((releaseWorkflow.match(/actions\/cache\/save@/g) ?? []).length, 2, "release and debug workspace caches have separate guarded writers");
|
|
});
|
|
|
|
test("the scope names the platform, the compiler and the checkout root, and survives a missing rustc", () => {
|
|
const a = computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" });
|
|
const b = computeScope({ workspace: "/home/runner/work/paperclip/paperclip", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" });
|
|
const c = computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.2-0123456ab-x86_64-unknown-linux-gnu" });
|
|
assert.notEqual(a, b, "a different checkout root must select a different entry");
|
|
assert.notEqual(a, c, "a different compiler must select a different entry");
|
|
assert.equal(a, computeScope({ workspace: "/home/runner/_work/paperclip/paperclip/", rustcVersion: "1.97.1-abcdef123-x86_64-unknown-linux-gnu" }));
|
|
assert.match(a, new RegExp(`^${os.platform()}-${os.arch()}-1\\.97\\.1-abcdef123-x86_64-unknown-linux-gnu-[0-9a-f]{12}$`));
|
|
assert.doesNotMatch(computeScope({ rustcVersion: "weird version/with spaces" }), /[^A-Za-z0-9._-]/);
|
|
|
|
// The CLI reads rustc from PATH; a stand-in proves which fields it reads,
|
|
// and an empty PATH proves a missing compiler still yields a usable key.
|
|
const bin = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-rustc-"));
|
|
try {
|
|
const stub = path.join(bin, "rustc");
|
|
writeFileSync(stub, '#!/bin/sh\nprintf "rustc 1.97.1 (8bab26f4f 2026-06-30)\\nbinary: rustc\\ncommit-hash: 8bab26f4f0123456789abcdef\\ncommit-date: 2026-06-30\\nhost: x86_64-unknown-linux-gnu\\nrelease: 1.97.1\\nLLVM version: 21.1.2\\n"\n');
|
|
chmodSync(stub, 0o755);
|
|
const withStub = execFileSync(process.execPath, [script, "scope"], { encoding: "utf8", env: { ...process.env, PATH: bin, GITHUB_WORKSPACE: "/home/runner/_work/paperclip/paperclip" } }).trim();
|
|
assert.equal(withStub, computeScope({ workspace: "/home/runner/_work/paperclip/paperclip", rustcVersion: "1.97.1-8bab26f4f-x86_64-unknown-linux-gnu" }));
|
|
const without = execFileSync(process.execPath, [script, "scope"], { encoding: "utf8", env: { ...process.env, PATH: mkdtempSync(path.join(os.tmpdir(), "empty-path-")), GITHUB_WORKSPACE: "/home/runner/_work/paperclip/paperclip" } }).trim();
|
|
assert.match(without, /-no-rustc-/);
|
|
} finally {
|
|
rmSync(bin, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
// The tree cargo leaves behind for the crate's release lib and bin units,
|
|
// with the mtimes GitHub's cache restores: older than a fresh checkout.
|
|
function layRestoredCrate(root) {
|
|
const release = path.join(root, RELEASE_DIR);
|
|
const lib = path.join(release, ".fingerprint/paperclip-runner-core-0b6c4654c2710743");
|
|
const bin = path.join(release, ".fingerprint/paperclip-runner-core-46cce78dbc4840c2");
|
|
const table = path.join(release, ".fingerprint/paperclip-process-table-5a1b2c3d4e5f6071");
|
|
mkdirSync(path.join(release, "deps"), { recursive: true });
|
|
mkdirSync(lib, { recursive: true });
|
|
mkdirSync(bin, { recursive: true });
|
|
mkdirSync(table, { recursive: true });
|
|
const files = {
|
|
[path.join(lib, "lib-paperclip_runner_core")]: "0123456789abcdef",
|
|
[path.join(lib, "lib-paperclip_runner_core.json")]: "{}",
|
|
[path.join(lib, "dep-lib-paperclip_runner_core")]: "dep",
|
|
[path.join(lib, "invoked.timestamp")]: "t",
|
|
[path.join(bin, "bin-paperclip-runnerd")]: "fedcba9876543210",
|
|
[path.join(bin, "bin-paperclip-runnerd.json")]: "{}",
|
|
[path.join(bin, "dep-bin-paperclip-runnerd")]: "dep",
|
|
[path.join(release, "deps/libpaperclip_runner_core-0b6c4654c2710743.rlib")]: "rlib",
|
|
[path.join(release, "deps/libpaperclip_runner_core-0b6c4654c2710743.rmeta")]: "rmeta",
|
|
[path.join(release, "deps/paperclip_runner_core-0b6c4654c2710743.d")]: "d",
|
|
[path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2")]: "ELF",
|
|
[path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2.d")]: "d",
|
|
[path.join(release, "paperclip-runnerd")]: "ELF",
|
|
[path.join(release, "paperclip-runnerd.d")]: "d",
|
|
// The process-table workspace library's lib unit.
|
|
[path.join(table, "lib-paperclip_process_table")]: "1111222233334444",
|
|
[path.join(table, "dep-lib-paperclip_process_table")]: "dep",
|
|
[path.join(release, "deps/libpaperclip_process_table-5a1b2c3d4e5f6071.rlib")]: "rlib",
|
|
[path.join(release, "deps/libpaperclip_process_table-5a1b2c3d4e5f6071.rmeta")]: "rmeta",
|
|
[path.join(release, "deps/paperclip_process_table-5a1b2c3d4e5f6071.d")]: "d",
|
|
// A dependency's artifacts sit beside the crate's and must not be touched.
|
|
[path.join(release, "deps/libserde-1234567890abcdef.rlib")]: "dep rlib",
|
|
};
|
|
const old = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
|
|
for (const [file, content] of Object.entries(files)) {
|
|
writeFileSync(file, content);
|
|
utimesSync(file, old, old);
|
|
}
|
|
for (const dir of [lib, bin, table, path.join(release, "deps"), path.join(release, ".fingerprint"), release]) utimesSync(dir, old, old);
|
|
return { release, files: Object.keys(files), old };
|
|
}
|
|
|
|
test("mark-current makes every restored artifact newer than the checkout and leaves dependencies alone", () => {
|
|
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
|
|
try {
|
|
const { release, files, old } = layRestoredCrate(root);
|
|
const checkout = path.join(root, RUNNER_PACKAGE, "runner/crates/runner-core/src/lib.rs");
|
|
mkdirSync(path.dirname(checkout), { recursive: true });
|
|
writeFileSync(checkout, "// checked out after the artifacts were built\n");
|
|
const checkedOutAt = statSync(checkout).mtimeMs;
|
|
|
|
const result = markCurrent({ root, now: new Date(checkedOutAt + 5000) });
|
|
assert.equal(result.action, "marked");
|
|
assert.equal(result.files, 19);
|
|
assert.equal(result.bytes, files.filter((file) => !file.includes("libserde")).reduce((sum, file) => sum + statSync(file).size, 0));
|
|
|
|
for (const file of files) {
|
|
const mtime = statSync(file).mtimeMs;
|
|
if (file.includes("libserde")) assert.equal(Math.floor(mtime / 1000), Math.floor(old.getTime() / 1000), "a dependency artifact must keep its mtime");
|
|
else assert.ok(mtime > checkedOutAt, `${path.relative(release, file)} must be newer than the checkout`);
|
|
}
|
|
for (const dir of [".fingerprint/paperclip-runner-core-0b6c4654c2710743", ".fingerprint/paperclip-runner-core-46cce78dbc4840c2"]) {
|
|
assert.ok(statSync(path.join(release, dir)).mtimeMs > checkedOutAt, `${dir} must be newer than the checkout`);
|
|
}
|
|
assert.ok(statSync(checkout).mtimeMs === checkedOutAt, "sources are never touched");
|
|
|
|
// The CLI prints the same result as one line of JSON for the job log.
|
|
const printed = JSON.parse(execFileSync(process.execPath, [script, "mark-current"], { encoding: "utf8", env: { ...process.env, GITHUB_WORKSPACE: root } }));
|
|
assert.equal(printed.schema, "paperclip.runner-release-crate.v1");
|
|
assert.equal(printed.action, "marked");
|
|
assert.equal(printed.files, 19);
|
|
} finally {
|
|
rmSync(root, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
// One file per REQUIRED entry, as cargo 1.97.1 lays them out for the crate's
|
|
// release lib and bin units. Measured: without any one of these cargo
|
|
// recompiles the unit (the whole crate for a lib file, a relink for a bin
|
|
// file); without any of the files in NOT_NEEDED it reports the crate fresh
|
|
// and recreates the uplifted binary from the one under deps.
|
|
const REQUIRED_FILES = {
|
|
"the lib fingerprint hash": ".fingerprint/paperclip-runner-core-0b6c4654c2710743/lib-paperclip_runner_core",
|
|
"the lib dep-info": ".fingerprint/paperclip-runner-core-0b6c4654c2710743/dep-lib-paperclip_runner_core",
|
|
"the lib rlib": "deps/libpaperclip_runner_core-0b6c4654c2710743.rlib",
|
|
"the lib rmeta": "deps/libpaperclip_runner_core-0b6c4654c2710743.rmeta",
|
|
"the bin fingerprint hash": ".fingerprint/paperclip-runner-core-46cce78dbc4840c2/bin-paperclip-runnerd",
|
|
"the bin dep-info": ".fingerprint/paperclip-runner-core-46cce78dbc4840c2/dep-bin-paperclip-runnerd",
|
|
"the bin under deps": "deps/paperclip_runnerd-46cce78dbc4840c2",
|
|
"the process-table lib fingerprint hash": ".fingerprint/paperclip-process-table-5a1b2c3d4e5f6071/lib-paperclip_process_table",
|
|
"the process-table lib dep-info": ".fingerprint/paperclip-process-table-5a1b2c3d4e5f6071/dep-lib-paperclip_process_table",
|
|
"the process-table lib rlib": "deps/libpaperclip_process_table-5a1b2c3d4e5f6071.rlib",
|
|
"the process-table lib rmeta": "deps/libpaperclip_process_table-5a1b2c3d4e5f6071.rmeta",
|
|
};
|
|
const NOT_NEEDED = [
|
|
".fingerprint/paperclip-runner-core-0b6c4654c2710743/lib-paperclip_runner_core.json",
|
|
".fingerprint/paperclip-runner-core-0b6c4654c2710743/invoked.timestamp",
|
|
".fingerprint/paperclip-runner-core-46cce78dbc4840c2/bin-paperclip-runnerd.json",
|
|
"deps/paperclip_runner_core-0b6c4654c2710743.d",
|
|
"deps/paperclip_runnerd-46cce78dbc4840c2.d",
|
|
"paperclip-runnerd",
|
|
"paperclip-runnerd.d",
|
|
"deps/paperclip_process_table-5a1b2c3d4e5f6071.d",
|
|
];
|
|
|
|
test("mark-current discards a restore that lacks any file cargo needs for either unit, naming the file, so cargo sees a plain miss", () => {
|
|
assert.deepEqual(REQUIRED.map(({ label }) => label), Object.keys(REQUIRED_FILES), "every REQUIRED entry has a missing-file case");
|
|
for (const [label, file] of Object.entries(REQUIRED_FILES)) {
|
|
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
|
|
try {
|
|
const { release } = layRestoredCrate(root);
|
|
rmSync(path.join(release, file));
|
|
const result = markCurrent({ root });
|
|
assert.equal(result.action, "discarded", `${file} missing must discard the restore`);
|
|
assert.equal(result.reason, `incomplete: ${label} is missing`);
|
|
assert.ok(result.removed > 0, label);
|
|
assert.deepEqual(readdirSync(path.join(release, ".fingerprint")), [], label);
|
|
assert.deepEqual(readdirSync(path.join(release, "deps")), ["libserde-1234567890abcdef.rlib"], `${label}: only the crate's own artifacts are removed`);
|
|
assert.deepEqual(readdirSync(release).sort(), [".fingerprint", "deps"], label);
|
|
} finally {
|
|
rmSync(root, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// The bin's dep-info alone does not stand in for the bin: the glob that
|
|
// finds the binary under deps must not be satisfied by the `.d` beside it.
|
|
const root = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
|
|
try {
|
|
const { release } = layRestoredCrate(root);
|
|
rmSync(path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2"));
|
|
assert.ok(statSync(path.join(release, "deps/paperclip_runnerd-46cce78dbc4840c2.d")).isFile());
|
|
assert.equal(markCurrent({ root }).reason, "incomplete: the bin under deps is missing");
|
|
} finally {
|
|
rmSync(root, { recursive: true, force: true });
|
|
}
|
|
|
|
// A file cargo neither reads nor checks for may be absent without costing
|
|
// a rebuild, so its absence must not cost one here either.
|
|
for (const file of NOT_NEEDED) {
|
|
const intact = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-"));
|
|
try {
|
|
const { release } = layRestoredCrate(intact);
|
|
rmSync(path.join(release, file));
|
|
const result = markCurrent({ root: intact });
|
|
assert.equal(result.action, "marked", `${file} missing must not discard the restore`);
|
|
assert.equal(result.files, 18, file);
|
|
} finally {
|
|
rmSync(intact, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Nothing restored at all is reported the same way and exits cleanly.
|
|
const empty = mkdtempSync(path.join(os.tmpdir(), "runner-release-crate-empty-"));
|
|
try {
|
|
const printed = JSON.parse(execFileSync(process.execPath, [script, "mark-current"], { encoding: "utf8", env: { ...process.env, GITHUB_WORKSPACE: empty } }));
|
|
assert.equal(printed.action, "discarded");
|
|
assert.equal(printed.reason, "incomplete: the lib fingerprint hash is missing");
|
|
assert.equal(printed.removed, 0);
|
|
} finally {
|
|
rmSync(empty, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
test("the CLI prints the lists the workflows embed", () => {
|
|
const paths = execFileSync(process.execPath, [script, "paths"], { encoding: "utf8" }).trim().split("\n");
|
|
assert.deepEqual(paths, CACHED_PATHS);
|
|
const inputs = execFileSync(process.execPath, [script, "hashed-inputs"], { encoding: "utf8" }).trim().split("\n");
|
|
assert.deepEqual(inputs, HASHED_INPUTS);
|
|
const usage = spawnSync(process.execPath, [script, "nonsense"], { encoding: "utf8" });
|
|
assert.equal(usage.status, 2);
|
|
assert.match(usage.stderr, /usage: runner-release-crate-cache\.mjs/);
|
|
});
|