mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-<digest>`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
452 lines
17 KiB
JavaScript
452 lines
17 KiB
JavaScript
// Records which inputs produced the Runner package's current build output, so
|
|
// a consumer that would otherwise rebuild the Runner can reuse the build when
|
|
// nothing it depends on has changed.
|
|
//
|
|
// server's `prepare:runner-vendor` runs the Runner's full `build`, because a
|
|
// standalone `pnpm --filter @paperclipai/server build`
|
|
// must produce a current vendored Runner on its own. Inside `pnpm -r build`
|
|
// the same command has just built the Runner a few seconds earlier, and the
|
|
// rebuild cost 12-16s on the CI Build job's critical path (six master runs,
|
|
// 2026-10-11), the same again in every other `pnpm build`.
|
|
//
|
|
// The Runner's `build` brackets itself with scripts/record-build-inputs.mjs:
|
|
// `--begin` removes any previous stamp and writes a pending record of the
|
|
// input digest before the first step reads anything; `--finish` recomputes
|
|
// the digest after the last check and writes the stamp under
|
|
// node_modules/.cache only if the inputs are still the ones the build started
|
|
// from, together with a manifest of every file the build left under dist. The
|
|
// server's prepare step (server/scripts/prepare-runner-vendor.mjs) reuses the
|
|
// build only when the stamp matches, every recorded output is still there at
|
|
// its recorded size, and the current input digest matches. Every other state
|
|
// rebuilds: a missing stamp, a changed input, a deleted or truncated output, a
|
|
// different platform, an edit that landed while the build was running. The
|
|
// slow direction is always the safe one.
|
|
//
|
|
// Each build chain has its own pending record, so two Runner builds running
|
|
// at once in one checkout (server `build` and `typecheck` side by side, say)
|
|
// do not consume each other's: `--begin` and `--finish` are separate
|
|
// processes, and what they share is the shell pnpm runs the chain in, so the
|
|
// record is keyed by that shell's pid. Both builds write the same dist, which
|
|
// the stamp cannot make safe; what it does guarantee is that a stamp is
|
|
// written only by a build whose own inputs did not move, and that the other
|
|
// build's `--finish` drops the stamp if its inputs did.
|
|
//
|
|
// Inputs: every file under packages/paperclip-runner except its build outputs
|
|
// (dist, dist-browser, runner/target, provider-assets), node_modules and dot-directories (tool
|
|
// caches); every file under packages/paperclip-eval-kernel except dist and
|
|
// node_modules, because build:typescript builds that package first; every
|
|
// file under the root patches directory, because check:acpx-profiles hashes
|
|
// the acpx patch and build-verified-provider-entrypoints.mjs copies the pi-acp
|
|
// and brace-expansion patches into dist; and the root files the Runner's
|
|
// tsconfig and dependency graph read. The toolchain pins live inside the
|
|
// package (rust-toolchain.toml, runner/Cargo.lock), and the TypeScript version
|
|
// comes from pnpm-lock.yaml, so both are covered.
|
|
//
|
|
// Outputs: the server build vendors packages/paperclip-runner/dist wholesale
|
|
// (`cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/`),
|
|
// so the stamp lists every file under dist with its size and reuse needs all
|
|
// of them present; a helper such as dist/cli/acpx-runtime-sidecar.cjs is
|
|
// opened by path at runtime, so a tree that lost it must not count as current.
|
|
|
|
import { createHash } from "node:crypto";
|
|
import {
|
|
existsSync,
|
|
lstatSync,
|
|
mkdirSync,
|
|
readdirSync,
|
|
readFileSync,
|
|
readlinkSync,
|
|
renameSync,
|
|
rmSync,
|
|
writeFileSync,
|
|
} from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
export const BUILD_INPUTS_SCHEMA = "paperclip.runner-build-inputs.v2";
|
|
|
|
const RUNNER_PACKAGE_RELATIVE = "packages/paperclip-runner";
|
|
const EVAL_KERNEL_PACKAGE_RELATIVE = "packages/paperclip-eval-kernel";
|
|
const ROOT_INPUT_DIRECTORIES = ["patches"];
|
|
const ROOT_INPUT_FILES = [
|
|
"package.json",
|
|
"pnpm-lock.yaml",
|
|
"pnpm-workspace.yaml",
|
|
"tsconfig.base.json",
|
|
];
|
|
const OUTPUT_DIRECTORY_RELATIVE = "dist";
|
|
const SKIPPED_DIRECTORY_NAMES = new Set(["node_modules", "dist", "dist-browser"]);
|
|
// provider-assets is written by the build itself (install-copilot-assets.mjs,
|
|
// the last step of build:typescript) from a lockfile-pinned package. Counting
|
|
// it as an input made the first build in every fresh checkout see its inputs
|
|
// change while it ran, so that build never recorded a stamp.
|
|
const RUNNER_SKIPPED_DIRECTORIES = new Set([
|
|
`${RUNNER_PACKAGE_RELATIVE}/runner/target`,
|
|
`${RUNNER_PACKAGE_RELATIVE}/provider-assets`,
|
|
]);
|
|
|
|
export function runnerPackageRootFromScripts() {
|
|
return path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
|
}
|
|
|
|
export function repoRootFromRunnerPackage(packageRoot) {
|
|
return path.resolve(packageRoot, "../..");
|
|
}
|
|
|
|
export function buildInputsStampPath(packageRoot) {
|
|
return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.json");
|
|
}
|
|
|
|
/** One record per build in progress, written by `--begin` and consumed by that build's `--finish`. */
|
|
export function buildInputsPendingDirectory(packageRoot) {
|
|
return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.pending");
|
|
}
|
|
|
|
export function buildInputsPendingPath(packageRoot, buildId) {
|
|
return path.join(buildInputsPendingDirectory(packageRoot), `${buildId}.json`);
|
|
}
|
|
|
|
/**
|
|
* A pending record older than this belongs to a build that never reached
|
|
* `--finish` (a failed step, an interrupted build); `--begin` removes such
|
|
* records so they cannot pile up. No Runner build runs anywhere near this long.
|
|
*/
|
|
export const PENDING_RECORD_MAX_AGE_MS = 24 * 60 * 60 * 1000;
|
|
|
|
/**
|
|
* Identifies the build chain a `--begin` or `--finish` process belongs to.
|
|
* The two are separate processes; what they share is the shell pnpm runs the
|
|
* chain in, so the chain is named after its pid, which `--begin` sees as its
|
|
* parent. A shell that execs the last command of a `-c` string (bash can)
|
|
* turns `--finish` into the shell itself, so `--finish` also accepts a record
|
|
* named after its own pid.
|
|
*/
|
|
export function currentBuildId() {
|
|
return String(process.ppid);
|
|
}
|
|
|
|
function finishingBuildIds() {
|
|
return [String(process.ppid), String(process.pid)];
|
|
}
|
|
|
|
function readPendingRecord(pendingPath) {
|
|
const pending = readBuildInputsStamp(pendingPath);
|
|
if (!pending || pending.schema !== BUILD_INPUTS_SCHEMA || typeof pending.digest !== "string") return null;
|
|
return pending;
|
|
}
|
|
|
|
/** Drops pending records of builds that never finished, leaving the one for `buildId`. */
|
|
function prunePendingRecords(packageRoot, buildId, now) {
|
|
const directory = buildInputsPendingDirectory(packageRoot);
|
|
let entries;
|
|
try {
|
|
entries = readdirSync(directory);
|
|
} catch {
|
|
return;
|
|
}
|
|
const cutoff = now().getTime() - PENDING_RECORD_MAX_AGE_MS;
|
|
for (const name of entries) {
|
|
if (name === `${buildId}.json`) continue;
|
|
const absolute = path.join(directory, name);
|
|
let stat;
|
|
try {
|
|
stat = lstatSync(absolute);
|
|
} catch {
|
|
continue;
|
|
}
|
|
if (stat.mtimeMs < cutoff) rmSync(absolute, { force: true });
|
|
}
|
|
}
|
|
|
|
/**
|
|
* The outputs every consumer of the vendored Runner reads; a build that did
|
|
* not produce them is not recorded, and a tree missing any of them is not
|
|
* current whatever the stamp says.
|
|
*/
|
|
export function expectedBuildOutputs(packageRoot, platform = process.platform) {
|
|
const executable = platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd";
|
|
return ["dist/index.js", "dist/live/index.js", "dist/testing.js", `dist/bin/${executable}`]
|
|
.map((relative) => path.join(packageRoot, ...relative.split("/")));
|
|
}
|
|
|
|
function walk(root, relativeDirectory, { skipNames, skippedDirectories, skipDotDirectories }, out) {
|
|
const absolute = path.join(root, ...relativeDirectory.split("/"));
|
|
if (!existsSync(absolute)) return;
|
|
const entries = readdirSync(absolute, { withFileTypes: true })
|
|
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
|
|
for (const entry of entries) {
|
|
const relative = `${relativeDirectory}/${entry.name}`;
|
|
if (entry.isDirectory()) {
|
|
if (skipDotDirectories && entry.name.startsWith(".")) continue;
|
|
if (skipNames.has(entry.name)) continue;
|
|
if (skippedDirectories.has(relative)) continue;
|
|
walk(root, relative, { skipNames, skippedDirectories, skipDotDirectories }, out);
|
|
continue;
|
|
}
|
|
// Files and symlinks; a symlinked directory is hashed as its link target
|
|
// rather than followed, so a link into node_modules cannot pull a
|
|
// dependency tree into the input set.
|
|
out.push(relative);
|
|
}
|
|
}
|
|
|
|
const INPUT_WALK = {
|
|
skipNames: SKIPPED_DIRECTORY_NAMES,
|
|
skippedDirectories: new Set(),
|
|
skipDotDirectories: true,
|
|
};
|
|
|
|
/** Repo-relative POSIX paths of every build input, sorted. */
|
|
export function collectBuildInputFiles(repoRoot) {
|
|
const files = [];
|
|
walk(repoRoot, RUNNER_PACKAGE_RELATIVE, { ...INPUT_WALK, skippedDirectories: RUNNER_SKIPPED_DIRECTORIES }, files);
|
|
walk(repoRoot, EVAL_KERNEL_PACKAGE_RELATIVE, INPUT_WALK, files);
|
|
for (const directory of ROOT_INPUT_DIRECTORIES) walk(repoRoot, directory, INPUT_WALK, files);
|
|
for (const name of ROOT_INPUT_FILES) {
|
|
if (existsSync(path.join(repoRoot, name))) files.push(name);
|
|
}
|
|
return files.sort();
|
|
}
|
|
|
|
export function computeBuildInputsDigest(repoRoot) {
|
|
const files = collectBuildInputFiles(repoRoot);
|
|
const hash = createHash("sha256");
|
|
let byteCount = 0;
|
|
for (const relative of files) {
|
|
const absolute = path.join(repoRoot, ...relative.split("/"));
|
|
hash.update(relative);
|
|
hash.update("\0");
|
|
if (lstatSync(absolute).isSymbolicLink()) {
|
|
hash.update("link\0");
|
|
hash.update(readlinkSync(absolute));
|
|
} else {
|
|
const content = readFileSync(absolute);
|
|
byteCount += content.length;
|
|
hash.update(`file ${content.length}\0`);
|
|
hash.update(content);
|
|
}
|
|
hash.update("\0");
|
|
}
|
|
return {
|
|
schema: BUILD_INPUTS_SCHEMA,
|
|
digest: hash.digest("hex"),
|
|
fileCount: files.length,
|
|
byteCount,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Every file the build left under dist (the tree the server vendors), as
|
|
* package-relative POSIX paths with the size lstat reports, sorted. Nothing
|
|
* under dist is skipped: the server copies all of it.
|
|
*/
|
|
export function collectBuildOutputFiles(packageRoot) {
|
|
const files = [];
|
|
walk(packageRoot, OUTPUT_DIRECTORY_RELATIVE, { skipNames: new Set(), skippedDirectories: new Set(), skipDotDirectories: false }, files);
|
|
return files.sort().map((relative) => ({
|
|
path: relative,
|
|
size: lstatSync(path.join(packageRoot, ...relative.split("/"))).size,
|
|
}));
|
|
}
|
|
|
|
/** The first recorded output that is gone or has changed size, or null when all are intact. */
|
|
export function findChangedBuildOutput(packageRoot, outputs) {
|
|
for (const output of outputs) {
|
|
const absolute = path.join(packageRoot, ...output.path.split("/"));
|
|
let stat;
|
|
try {
|
|
stat = lstatSync(absolute);
|
|
} catch {
|
|
return { ...output, state: "missing" };
|
|
}
|
|
if (stat.size !== output.size) return { ...output, state: "changed size", size: stat.size, recordedSize: output.size };
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function readBuildInputsStamp(stampPath) {
|
|
try {
|
|
const parsed = JSON.parse(readFileSync(stampPath, "utf8"));
|
|
return parsed && typeof parsed === "object" ? parsed : null;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
export function writeBuildInputsStamp(stampPath, record) {
|
|
mkdirSync(path.dirname(stampPath), { recursive: true });
|
|
const staged = `${stampPath}.${process.pid}.tmp`;
|
|
writeFileSync(staged, `${JSON.stringify(record, null, 2)}\n`);
|
|
renameSync(staged, stampPath);
|
|
}
|
|
|
|
export function removeBuildInputsStamp(stampPath) {
|
|
rmSync(stampPath, { force: true });
|
|
}
|
|
|
|
/**
|
|
* First step of the Runner's build: forgets the previous build (a build in
|
|
* progress can no longer vouch for its outputs) and records the inputs the
|
|
* build is about to read, so that `recordRunnerBuild` can tell whether they
|
|
* changed while it ran.
|
|
*/
|
|
export function beginRunnerBuild({
|
|
repoRoot,
|
|
packageRoot,
|
|
platform = process.platform,
|
|
arch = process.arch,
|
|
buildId = currentBuildId(),
|
|
now = () => new Date(),
|
|
}) {
|
|
removeBuildInputsStamp(buildInputsStampPath(packageRoot));
|
|
prunePendingRecords(packageRoot, buildId, now);
|
|
const inputs = computeBuildInputsDigest(repoRoot);
|
|
const pending = { ...inputs, platform, arch, buildId, startedAt: now().toISOString() };
|
|
const pendingPath = buildInputsPendingPath(packageRoot, buildId);
|
|
writeBuildInputsStamp(pendingPath, pending);
|
|
return { ...pending, pendingPath };
|
|
}
|
|
|
|
/**
|
|
* Last step of the Runner's build, after every step and check has passed:
|
|
* records the inputs captured by `beginRunnerBuild` as the ones that produced
|
|
* the outputs now under dist. An edit that landed while the build ran means
|
|
* the outputs may not reflect the current inputs, so nothing is recorded and
|
|
* the next consumer rebuilds. A build that did not begin with
|
|
* `beginRunnerBuild` (no pending record for this build chain), or that left
|
|
* a required output missing, is an error.
|
|
*/
|
|
export function recordRunnerBuild({
|
|
repoRoot,
|
|
packageRoot,
|
|
platform = process.platform,
|
|
arch = process.arch,
|
|
buildId = null,
|
|
now = () => new Date(),
|
|
}) {
|
|
const stampPath = buildInputsStampPath(packageRoot);
|
|
const candidates = buildId === null ? finishingBuildIds() : [String(buildId)];
|
|
let pending = null;
|
|
let pendingPath = null;
|
|
for (const candidate of candidates) {
|
|
pendingPath = buildInputsPendingPath(packageRoot, candidate);
|
|
pending = readPendingRecord(pendingPath);
|
|
if (pending) break;
|
|
}
|
|
if (!pending) {
|
|
removeBuildInputsStamp(stampPath);
|
|
throw new Error(
|
|
`no pending build inputs for build ${candidates.join(" or ")} under ` +
|
|
`${buildInputsPendingDirectory(packageRoot)}: the Runner build must start with ` +
|
|
"`node scripts/record-build-inputs.mjs --begin` before it can record its result",
|
|
);
|
|
}
|
|
rmSync(pendingPath, { force: true });
|
|
|
|
const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file));
|
|
if (missing.length > 0) {
|
|
removeBuildInputsStamp(stampPath);
|
|
throw new Error(`the Runner build finished without producing ${path.relative(packageRoot, missing[0])}`);
|
|
}
|
|
|
|
const inputs = computeBuildInputsDigest(repoRoot);
|
|
if (inputs.digest !== pending.digest) {
|
|
removeBuildInputsStamp(stampPath);
|
|
return {
|
|
recorded: false,
|
|
code: "inputs_changed_during_build",
|
|
detail:
|
|
`inputs changed while the build ran (digest ${pending.digest.slice(0, 12)} at start, ` +
|
|
`${inputs.digest.slice(0, 12)} now); the next consumer rebuilds`,
|
|
startedAt: pending.startedAt,
|
|
stampPath,
|
|
};
|
|
}
|
|
|
|
const outputs = collectBuildOutputFiles(packageRoot);
|
|
const record = {
|
|
...inputs,
|
|
platform,
|
|
arch,
|
|
startedAt: pending.startedAt,
|
|
recordedAt: now().toISOString(),
|
|
outputCount: outputs.length,
|
|
outputs,
|
|
};
|
|
writeBuildInputsStamp(stampPath, record);
|
|
return { recorded: true, code: "recorded", ...record, stampPath };
|
|
}
|
|
|
|
/**
|
|
* Decides whether the Runner's outputs are current for the present inputs.
|
|
* `code` is a stable slug for timing lines; `detail` is for humans.
|
|
*/
|
|
export function assessRunnerBuild({
|
|
repoRoot,
|
|
packageRoot,
|
|
platform = process.platform,
|
|
arch = process.arch,
|
|
}) {
|
|
const stampPath = buildInputsStampPath(packageRoot);
|
|
const stamp = readBuildInputsStamp(stampPath);
|
|
if (!stamp) {
|
|
return { current: false, code: "no_stamp", detail: `no build stamp at ${stampPath}`, stampPath };
|
|
}
|
|
if (stamp.schema !== BUILD_INPUTS_SCHEMA || !Array.isArray(stamp.outputs)) {
|
|
return {
|
|
current: false,
|
|
code: "schema_mismatch",
|
|
detail: `stamp schema ${String(stamp.schema)} is not ${BUILD_INPUTS_SCHEMA}`,
|
|
stampPath,
|
|
};
|
|
}
|
|
if (stamp.platform !== platform || stamp.arch !== arch) {
|
|
return {
|
|
current: false,
|
|
code: "platform_mismatch",
|
|
detail: `stamp recorded on ${stamp.platform}/${stamp.arch}, this is ${platform}/${arch}`,
|
|
stampPath,
|
|
};
|
|
}
|
|
const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file));
|
|
if (missing.length > 0) {
|
|
return {
|
|
current: false,
|
|
code: "missing_output",
|
|
detail: `missing build output ${path.relative(packageRoot, missing[0])}`,
|
|
stampPath,
|
|
};
|
|
}
|
|
const changedOutput = findChangedBuildOutput(packageRoot, stamp.outputs);
|
|
if (changedOutput) {
|
|
return {
|
|
current: false,
|
|
code: "outputs_changed",
|
|
detail:
|
|
`build output ${changedOutput.path} is ${changedOutput.state} since the build recorded at ` +
|
|
`${stamp.recordedAt}; ${stamp.outputs.length} outputs checked`,
|
|
stampPath,
|
|
};
|
|
}
|
|
const inputs = computeBuildInputsDigest(repoRoot);
|
|
if (inputs.digest !== stamp.digest) {
|
|
return {
|
|
current: false,
|
|
code: "inputs_changed",
|
|
detail: `inputs changed since the recorded build; ${inputs.fileCount} files checked`,
|
|
stampPath,
|
|
inputs,
|
|
};
|
|
}
|
|
return {
|
|
current: true,
|
|
code: "inputs_unchanged",
|
|
detail:
|
|
`inputs unchanged since the build recorded at ${stamp.recordedAt}; ${inputs.fileCount} files ` +
|
|
`and ${stamp.outputs.length} outputs checked`,
|
|
stampPath,
|
|
inputs,
|
|
stamp,
|
|
};
|
|
}
|