From 7df517d18fdb49fea608f6ec2f4aab482e86c023 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 12 Oct 2026 11:25:24 -0700 Subject: [PATCH] perf(build): reuse a current Runner build in the server's prepare:runner-vendor step (#15926) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its CI runs a `Build` job on every pull request. That job runs `pnpm build` and then verifies the public npm install. Since the server test shards got faster, `Build` is the slowest check in most runs (419-497s on the six fully green `master` runs created 2026-10-11 19:46Z to 20:17Z). > - Inside `pnpm build`, the server package is the last package to build. Its first step, `prepare:runner-vendor`, runs the full build of `@paperclipai/paperclip-runner` so the server can vendor `packages/paperclip-runner/dist`. In `pnpm -r build` that package was built seconds earlier by the same command. The second build took 12-16s on those six runs and sits on the critical path of the job. > - The step cannot just be removed. A standalone `pnpm --filter @paperclipai/server build` (the Dockerfile's path) must produce a current vendored Runner on its own. > - This pull request keeps that guarantee and skips only the build that is provably redundant. The Runner's `build` records a digest of its inputs, captured before its first step and confirmed after its last, together with a manifest of every file it left under `dist`. The server's `prepare:runner-vendor` reuses the build when the digest matches, the stamp is for this platform, and every recorded output is still there at its recorded size. Any other state runs the same full Runner build as before. > - The benefit is 12-16s off the `Build` job and off every other `pnpm build` (the Canary Dry Run's `release.sh` build, local builds), with no change to what a standalone server build produces. The PR's own first run measured it: the server step reused the build in 56ms where `master` runs rebuilt for 11-15s. ## Linked Issues or Issue Description No GitHub issue exists for this. The description below follows the enhancement template. **What existing behavior does this improve?** The server build's `prepare:runner-vendor` step, which builds `@paperclipai/paperclip-runner` so the server can vendor its `dist`. **Subsystem affected** Build and CI (`server/package.json`, `packages/paperclip-runner/package.json`, the `Build` job). **Current behavior** `prepare:runner-vendor` runs the Runner's full `build` every time. Inside `pnpm -r build`, the Runner was already built by the same command 20 to 25 seconds earlier, with every check passing. On six fully green `master` runs (2026-10-11, 19:46Z to 20:17Z) the repeated build took 11.8s to 15.9s, measured from the `server build$ pnpm run prepare:runner-vendor` log line to the Runner's `Finished release profile` line. The server package builds last, so that time is on the critical path of `Build`, which is the slowest check in most pull request runs. **Proposed behavior** - The Runner's `build` starts with `node scripts/record-build-inputs.mjs --begin` and ends with `node scripts/record-build-inputs.mjs --finish`. `--begin` removes the previous stamp and records a digest of every build input before the first step reads anything. `--finish`, after every step and check has passed, recomputes the digest and writes `packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json` only if the inputs are still the ones the build began with, together with the path and size of every file under `dist`. A stamp therefore exists only for a build that completed every step on inputs that did not move while it ran. - The server's `prepare:runner-vendor` becomes `node scripts/prepare-runner-vendor.mjs`. It reuses the build if the stamp is for this platform and architecture, every recorded output is still there at its recorded size, and the recomputed digest matches. In every other case it removes the stamp and runs `pnpm --filter @paperclipai/paperclip-runner build`, the exact command the step ran before. - `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` forces the rebuild. **Reason and benefit** 12-16s off the `Build` job and off every other `pnpm build` (the Canary Dry Run's `release.sh` build, local builds), with no change to what a standalone `pnpm --filter @paperclipai/server build` produces. **Breaking changes** None. The rebuild path runs the same command as before. A fresh checkout, a changed input, a missing or truncated output, an edit that landed during the build, or a stamp from another platform always rebuilds. ## What Changed - `packages/paperclip-runner/scripts/build-inputs.mjs` (new): defines the input set, the digest, the output manifest and the reuse decision. Inputs are every file under `packages/paperclip-runner` except `dist`, `dist-browser`, `runner/target`, `node_modules` and dot-directories; every file under `packages/paperclip-eval-kernel` except `dist` and `node_modules` (the Runner's `build:typescript` builds it first); every file under the root `patches` directory (`check:acpx-profiles` hashes `acpx@0.13.1.patch`, and `build-verified-provider-entrypoints.mjs` copies the `pi-acp` and `brace-expansion` patches into `dist/cli/pi-provision-inputs`); and the root `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and `tsconfig.base.json`. The toolchain pins are inside the package (`rust-toolchain.toml`, `runner/Cargo.lock`) and the TypeScript version is in the lockfile, so both are covered. A symlink is hashed as its target path and is not followed. Outputs are every file under `dist`, the tree the server copies wholesale, recorded with their sizes. Exports `beginRunnerBuild`, `recordRunnerBuild` and `assessRunnerBuild` (returns `current` plus a stable reason code: `no_stamp`, `schema_mismatch`, `platform_mismatch`, `missing_output`, `outputs_changed`, `inputs_changed`, `inputs_unchanged`). - `packages/paperclip-runner/scripts/record-build-inputs.mjs` (new): the two phases, `--begin` and `--finish`, added as the first and last step of the Runner's `build` in `packages/paperclip-runner/package.json`. Each build chain has its own pending record under `node_modules/.cache/paperclip-runner-build-inputs.pending/`, named after the shell pnpm runs the chain in (the parent of both phases; `--finish` also accepts a record named after its own pid, for a shell that execs the last command of the chain), so two Runner builds running at once in one checkout do not consume each other's record. `--finish` without a pending record for its chain is an error, so the chain cannot silently lose its first step; `--finish` after an edit that landed mid-build prints `not recorded: inputs changed while the build ran` and leaves no stamp. Records of builds that never finished are pruned by the next `--begin` after a day. - `server/scripts/prepare-runner-vendor.mjs` (new): the reuse-or-rebuild step. It resolves the pnpm that is running it through `npm_execpath` with the same rule as the CLI's `resolvePnpmInstallInvocation` (a JavaScript entry runs through this Node, a native executable runs directly, otherwise `pnpm` on PATH). It prints one `paperclip.runner-vendor.timing.v1` line with `action`, `reason`, `check_ms` and `build_ms`, so the saving is measured on this PR's own `Build` job. - `server/package.json`: `prepare:runner-vendor` now runs that script. The `build` and `typecheck` scripts are unchanged. - `server/src/__tests__/server-package-build-script.test.ts`: pins the new indirection and the command behind it (imported from the script, so the Dockerfile's standalone path stays pinned), and a new test asserts the Runner's `build` starts with `--begin`, ends with `--finish`, and has exactly those two record steps. - Tests: `packages/paperclip-runner/scripts/build-inputs.test.mjs` (node:test, added to `test:typescript:prep`) and `server/scripts/prepare-runner-vendor.test.mjs` (vitest, in the server lane). - `doc/DEVELOPING.md`: a "Reused Runner build in the server build" section under Test Commands. ## Review findings on the first head, all fixed in [2d57542](https://github.com/paperclipai/paperclip/pull/15926/commits/2d575425977f87b4612222d0a1961bc9cc5ba569) Greptile scored the first head 2/5 with three P1 findings. All three were valid. | Finding | Fix | | --- | --- | | **Patch edits leave stale output.** The root patch files the build reads were not in the digest. | Every file under the root `patches` directory is an input. Tests assert the patches are in the input list and that editing the pi-acp patch, the acpx patch, or adding a patch changes the digest. | | **Missing helpers pass the build.** Only four outputs were checked, so a `dist` that lost `dist/cli/acpx-runtime-sidecar.cjs` still counted as current. | `--finish` records the path and size of every file under `dist`; reuse requires all of them intact and reports `outputs_changed` naming the first missing or resized file. Schema bumped to `paperclip.runner-build-inputs.v2` so an older stamp is rejected. Tests cover a deleted helper, a truncated helper, and the server step rebuilding on `outputs_changed`. | | **New edits count as compiled.** The digest was hashed after the build, so a source edited while `build:binary` ran was recorded as compiled. | The build is bracketed: `--begin` captures the digest before the first step, `--finish` compares after the last and writes nothing if it moved. Tests cover the mid-build edit, the undisturbed build, and the CLI in both orders. | ## Review finding on the second head, fixed in the third commit Greptile scored the second head 4/5 with one P1, valid: two Runner builds running at once in one checkout (server `build` beside `typecheck`) shared one pending record, so the second `--begin` overwrote it, the first `--finish` consumed it, and the second `--finish` threw `no pending build inputs` after every step of its build had passed. Reproduced on the second head's module: two `beginRunnerBuild`, then the first `recordRunnerBuild` records and the second throws. | Finding | Fix | | --- | --- | | **Overlapping builds fail each other.** One pending file for every build chain. | One record per chain, named after the shell pnpm runs the chain in: `--begin` sees it as its parent, and so does `--finish`, unless the shell execs the last command of the chain (bash can), in which case `--finish` is that shell and finds the record under its own pid. Pending records of builds that never finished are pruned after a day. An edit between two overlapping starts leaves no stamp whichever build finishes last: the build that began before the edit reports `inputs_changed_during_build` and removes the stamp the other wrote, so a stamp always means every build that wrote `dist` read the inputs it names. Tests: both finishing orders with and without an edit, the pruning, a `--finish` that cannot use another chain's record, and two real `sh -c` chains (plus one under `bash`) held open at the same time, both recording. On the real repo, two `pnpm --filter @paperclipai/paperclip-runner build` chains started 8s apart with a cargo stand-in both recorded (`files=1223 outputs=1434`) and left the pending directory empty. | ## Measured on this PR's runs The `Build` job ran `pnpm build` from the branch, so both builds of the Runner in that job used the new chain. First head (c23faab, run 38175281058), against the four fully green `master` runs created 20:41Z to 21:37Z the same evening: | | PR run 38175281058 | `master`, four runs | | --- | --- | --- | | Runner build inside `pnpm -r build` | 55.3s | 47.3-62.0s | | Server step `prepare:runner-vendor` | `action:reuse,reason:inputs_unchanged,check_ms:56,build_ms:0` | full Runner rebuild, 11.0-14.8s | | Server package build | 29.7s | 32.3-43.5s | | `pnpm build` step | 114s | 102-135s | | `Build` job | 411s | 409-474s | Second head (2d57542, run 38178686999), against the two fully green `master` runs created in the same half hour (38178951607 and 38180574768): | | PR run 38178686999 | `master`, two runs | | --- | --- | --- | | Runner build inside `pnpm -r build` | 53.9s (`begin:` 22:03:04, `recorded:` 22:03:57, digest 460e02616081 over 1231 files and 1434 outputs) | 62.2s, 64.6s | | Server step `prepare:runner-vendor` | `action:reuse,reason:inputs_unchanged,check_ms:109,build_ms:0` | full Runner rebuild, 14.5s, 15.2s | | Server package build | 27.4s | 43.5s, 44.8s | | `pnpm build` step | 106s | 91s, 95s to the Runner's end plus the rest; 127s, 135s in all | | `Build` job | 405s | 461s, 479s | The rebuild is gone from the server build on both heads, and the check costs 56-109ms where `master` rebuilds for 11-15s; the rest of the spread between runs is runner speed on the `runs-on` fleet. The third commit changes nothing on the reuse path: it only changes where the pending record lives between `--begin` and `--finish`. ## Verification - `node --test packages/paperclip-runner/scripts/build-inputs.test.mjs`: 11 tests, three consecutive runs clean. Two builds at once in one checkout each keep their own pending record and both record, in either finishing order; with an edit between their starts the build that began before it leaves no stamp whichever finishes last. A build's start prunes records older than a day and no others. A `--finish` cannot use another chain's record. Two chains run by separate `sh -c` shells at the same time (and one under `bash`) both record and leave the pending directory empty. The input set is exactly the sources, the eval kernel, the root patches and the root config, and skips outputs and caches. The output manifest lists every file under `dist` with its size and nothing outside it. The digest is stable across reads, does not change when `dist`, `dist-browser`, `node_modules`, `.vite` or `runner/target` change, and does change for a Runner source, a Rust source, an eval-kernel source, the pi-acp patch, the acpx patch, a new patch, the lockfile, the base tsconfig, a new file and a rename. A symlink changes the digest but its target's files are not read. A build is current only with a matching v2 stamp, the same platform and architecture, the four named outputs present, and every recorded output intact (a deleted or truncated helper is `outputs_changed`; a file the build did not produce may appear). An input edited between `--begin` and `--finish` leaves no stamp and removes the pending record; the next build records normally. `--finish` without `--begin`, or after a build that left a required output missing, throws and leaves no stamp. The CLI brackets the build for the package it is pointed at, reports a mid-build edit, and rejects `--finish` first, both flags, neither flag, and unknown arguments. - `vitest run scripts/prepare-runner-vendor.test.mjs src/__tests__/server-package-build-script.test.ts` in `server/`: 21 tests. The decision reuses a current build, rebuilds a stale one, and rebuilds a current one only when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1`. The pnpm resolver covers a JavaScript entry, a native executable, a Windows executable, no `npm_execpath`, and npm's `npm_execpath`. With an injected build command: a current stamp runs nothing and prints `action:reuse`; changed inputs run the exact command from `server/` with the stale stamp already removed; a lost `dist/cli/acpx-runtime-sidecar.cjs` rebuilds with `reason:outputs_changed`; a failing build returns its status and leaves no stamp. End to end, the CLI with a stand-in `pnpm` on PATH that brackets its build the way the real one does rebuilds on the first call, reuses on the second, rebuilds after a source change, rebuilds after a lost output, and reuses again. The guard test asserts `prepare:runner-vendor` is the new script, the rebuild command is still `pnpm --filter @paperclipai/paperclip-runner build`, the server build still starts with the step, and the Runner's `build` starts with `--begin` and ends with `--finish` with exactly two record steps; it fails against the previous build script on its first assertion. - Real repo, this checkout, with `cargo` and `pnpm` stand-ins on PATH (no Rust toolchain here; the stand-in writes the one file `stage-runner-binary.mjs` copies): `pnpm --filter @paperclipai/paperclip-runner build` ran the complete chain through the real nested pnpm (every TypeScript step and check, the stand-in cargo, the binary staging, the two binary-reading checks) and recorded `files=1223 bytes=18735059 outputs=1434`; the stamp is 128 KB. `pnpm run prepare:runner-vendor` in `server/` then printed `action:reuse,reason:inputs_unchanged,check_ms:222,build_ms:0`. Appending a newline to `patches/pi-acp@0.0.33.patch` made the assessment `inputs_changed`; truncating or deleting `dist/cli/acpx-runtime-sidecar.cjs` or `dist/cli/opencode-app-server-proxy.cjs` made it `outputs_changed` in 1-5ms; restoring each made it current again. `--begin`, an edit to `src/index.ts`, then `--finish` printed `not recorded: inputs changed while the build ran` and left no stamp. - The whole check on the real set (1223 inputs, 18 MB; 1434 outputs) takes 165-227ms on this host and took 56ms on the CI runner; the output manifest part is 1-5ms. - Not run here: the real `cargo build`, and `pnpm build` end to end. This PR's `Build` job runs `pnpm build` from the branch, so its log carries the Runner's `begin:` and `recorded:` lines and the server build's `paperclip.runner-vendor.timing.v1` line with the measured result. ## Risks - A build input outside the digest would let a stale Runner be vendored. The digest covers the whole Runner package except its outputs, the eval kernel the Runner builds first, the root patches, and the root files the Runner's tsconfig and dependency graph read. The lockfile covers dependency versions, and the pins inside the package cover the Rust toolchain. An input missed here is a bug to fix by adding it to the set; the fallback direction is always a rebuild, never a skip. - The whole `patches` directory is an input, although the Runner build reads three of its fourteen files. An edit to an unrelated patch therefore costs one extra Runner rebuild, which is the safe direction and avoids a hand-kept list of file names that could drift from the scripts that read them. - The stamp lives in `node_modules/.cache`, which pnpm does not manage and `.gitignore` and `.dockerignore` already exclude. It is about 128 KB with the output manifest. The Docker build has no stamp and no prior Runner build, so it rebuilds exactly as before. - A stamp is written only after the Runner's `build` passed every step on inputs that did not change while it ran. The Runner's first step and the prepare step both remove any existing stamp before a rebuild starts, so a build that stops partway, or a build command that fails before its first step, leaves no stamp. - `node scripts/record-build-inputs.mjs --finish` on its own is an error; the script is only meant to run as the two ends of the `build` chain, and the guard test pins that. - The pending record is keyed by the pid of the shell that runs the chain. pnpm runs every script through a shell (`sh -c` on POSIX, `cmd` on Windows), and both phases are children of that shell; a shell that execs the last command of the chain is covered by matching the finishing process's own pid too. A chain run some other way, with the two phases under different parents, fails loudly at `--finish` rather than recording. Two builds running at once still write the same `dist`, which no stamp can make safe; what the record guarantees is that a stamp is written only by a build whose own inputs did not move. - The platform and architecture are part of the comparison, so a stamp from another machine is never trusted; an older v1 stamp is rejected by schema. - The Typecheck job builds the Runner once, through this step, with no prior `pnpm -r build`, so it gains nothing and loses nothing: there is no stamp and it rebuilds as before. - The step now spawns pnpm through `npm_execpath` instead of relying on `pnpm` being on PATH, because under corepack the script shell has no `pnpm`. Outside a pnpm script it still falls back to `pnpm` on PATH. ## Model Used - Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1), extended thinking, with tool use (shell, file editing, GitHub CLI) in a Paperclip agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 --- doc/DEVELOPING.md | 35 ++ packages/paperclip-runner/package.json | 4 +- .../paperclip-runner/scripts/build-inputs.mjs | 444 +++++++++++++++ .../scripts/build-inputs.test.mjs | 533 ++++++++++++++++++ .../scripts/record-build-inputs.mjs | 79 +++ server/package.json | 2 +- server/scripts/prepare-runner-vendor.mjs | 143 +++++ server/scripts/prepare-runner-vendor.test.mjs | 312 ++++++++++ .../server-package-build-script.test.ts | 32 +- 9 files changed, 1578 insertions(+), 6 deletions(-) create mode 100644 packages/paperclip-runner/scripts/build-inputs.mjs create mode 100644 packages/paperclip-runner/scripts/build-inputs.test.mjs create mode 100644 packages/paperclip-runner/scripts/record-build-inputs.mjs create mode 100644 server/scripts/prepare-runner-vendor.mjs create mode 100644 server/scripts/prepare-runner-vendor.test.mjs diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 441b13685a..feef188075 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -560,6 +560,41 @@ falls back to the fresh path automatically, so the cache can make a suite slower but never fail it or skip it: a migration that cannot apply still fails the suite on the fresh path. +#### Reused Runner build in the server build + +The server build vendors `packages/paperclip-runner/dist`, so its +`prepare:runner-vendor` step has to make sure that directory is a current +build. It used to run the Runner's full `build` every time. Now the Runner's +`build` brackets itself with `scripts/record-build-inputs.mjs`: `--begin`, its +first step, forgets the previous stamp and records a digest of the build's +inputs (every file under `packages/paperclip-runner` except `dist`, +`dist-browser`, `runner/target`, `node_modules` and dot-directories; every +file under `packages/paperclip-eval-kernel` except `dist` and `node_modules`; +every file under the root `patches` directory, which the acpx profile check +hashes and the provider entrypoint bundler copies into `dist`; and the root +`package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and +`tsconfig.base.json`). `--finish`, its last step, recomputes the digest and +writes the stamp +`packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json` +only if the inputs are still the ones the build began with, together with the +path and size of every file under `dist`. An edit that lands while the build +runs leaves no stamp, so the next consumer rebuilds. +`node server/scripts/prepare-runner-vendor.mjs` reuses the build when the +stamp is for this platform, every recorded output is still there at its +recorded size, and the current digest matches. Anything else (no stamp, a +changed input, a deleted or truncated output, a stamp from another platform) +runs `pnpm --filter @paperclipai/paperclip-runner build` exactly as before, so +a standalone `pnpm --filter @paperclipai/server build` on a fresh checkout +still builds the Runner itself. Inside `pnpm build` the Runner has just been +built by the same command, and the step skips a 12-16s rebuild. The step +prints one `paperclip.runner-vendor.timing.v1` line saying which it did. + +Set `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` to rebuild regardless of the stamp. A +stamp is only written after a Runner build that passed every step, and both +the Runner's first build step and the prepare step remove any existing stamp +before a rebuild starts, so a build that stops partway never leaves a stamp +that vouches for it. + ### Task search evaluation The task search relevance rubric and regression corpus are documented in diff --git a/packages/paperclip-runner/package.json b/packages/paperclip-runner/package.json index f962fe6c20..910453c327 100644 --- a/packages/paperclip-runner/package.json +++ b/packages/paperclip-runner/package.json @@ -63,7 +63,7 @@ }, "sideEffects": false, "scripts": { - "build": "pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check", + "build": "node scripts/record-build-inputs.mjs --begin && pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && node scripts/generate-semantic-contracts.mjs --check && node scripts/check-runner-workflow-traceability.mjs && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check && node scripts/record-build-inputs.mjs --finish", "build:typescript": "pnpm run check:acpx-profiles && pnpm run ensure:eval-build-deps && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json && node scripts/build-verified-provider-entrypoints.mjs", "build:rust": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins", "build:binary": "cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/stage-runner-binary.mjs", @@ -81,7 +81,7 @@ "typecheck:browser": "tsc -p tsconfig.browser.json --noEmit", "test": "pnpm run test:typescript && pnpm run test:rust", "test:typescript": "pnpm run test:typescript:prep && vitest run", - "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs", + "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs", "test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs", "test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace", "test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider", diff --git a/packages/paperclip-runner/scripts/build-inputs.mjs b/packages/paperclip-runner/scripts/build-inputs.mjs new file mode 100644 index 0000000000..746ea66c92 --- /dev/null +++ b/packages/paperclip-runner/scripts/build-inputs.mjs @@ -0,0 +1,444 @@ +// Records which inputs produced the Runner package's current build output, so +// a consumer that would otherwise rebuild the Runner can reuse the build when +// nothing it depends on has changed. +// +// server's `prepare:runner-vendor` runs the Runner's full `build`, because a +// standalone `pnpm --filter @paperclipai/server build` (the Dockerfile's path) +// must produce a current vendored Runner on its own. Inside `pnpm -r build` +// the same command has just built the Runner a few seconds earlier, and the +// rebuild cost 12-16s on the CI Build job's critical path (six master runs, +// 2026-10-11), the same again in every other `pnpm build`. +// +// The Runner's `build` brackets itself with scripts/record-build-inputs.mjs: +// `--begin` removes any previous stamp and writes a pending record of the +// input digest before the first step reads anything; `--finish` recomputes +// the digest after the last check and writes the stamp under +// node_modules/.cache only if the inputs are still the ones the build started +// from, together with a manifest of every file the build left under dist. The +// server's prepare step (server/scripts/prepare-runner-vendor.mjs) reuses the +// build only when the stamp matches, every recorded output is still there at +// its recorded size, and the current input digest matches. Every other state +// rebuilds: a missing stamp, a changed input, a deleted or truncated output, a +// different platform, an edit that landed while the build was running. The +// slow direction is always the safe one. +// +// Each build chain has its own pending record, so two Runner builds running +// at once in one checkout (server `build` and `typecheck` side by side, say) +// do not consume each other's: `--begin` and `--finish` are separate +// processes, and what they share is the shell pnpm runs the chain in, so the +// record is keyed by that shell's pid. Both builds write the same dist, which +// the stamp cannot make safe; what it does guarantee is that a stamp is +// written only by a build whose own inputs did not move, and that the other +// build's `--finish` drops the stamp if its inputs did. +// +// Inputs: every file under packages/paperclip-runner except its build outputs +// (dist, dist-browser, runner/target), node_modules and dot-directories (tool +// caches); every file under packages/paperclip-eval-kernel except dist and +// node_modules, because build:typescript builds that package first; every +// file under the root patches directory, because check:acpx-profiles hashes +// the acpx patch and build-verified-provider-entrypoints.mjs copies the pi-acp +// and brace-expansion patches into dist; and the root files the Runner's +// tsconfig and dependency graph read. The toolchain pins live inside the +// package (rust-toolchain.toml, runner/Cargo.lock), and the TypeScript version +// comes from pnpm-lock.yaml, so both are covered. +// +// Outputs: the server build vendors packages/paperclip-runner/dist wholesale +// (`cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/`), +// so the stamp lists every file under dist with its size and reuse needs all +// of them present; a helper such as dist/cli/acpx-runtime-sidecar.cjs is +// opened by path at runtime, so a tree that lost it must not count as current. + +import { createHash } from "node:crypto"; +import { + existsSync, + lstatSync, + mkdirSync, + readdirSync, + readFileSync, + readlinkSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +export const BUILD_INPUTS_SCHEMA = "paperclip.runner-build-inputs.v2"; + +const RUNNER_PACKAGE_RELATIVE = "packages/paperclip-runner"; +const EVAL_KERNEL_PACKAGE_RELATIVE = "packages/paperclip-eval-kernel"; +const ROOT_INPUT_DIRECTORIES = ["patches"]; +const ROOT_INPUT_FILES = [ + "package.json", + "pnpm-lock.yaml", + "pnpm-workspace.yaml", + "tsconfig.base.json", +]; +const OUTPUT_DIRECTORY_RELATIVE = "dist"; +const SKIPPED_DIRECTORY_NAMES = new Set(["node_modules", "dist", "dist-browser"]); +const RUNNER_SKIPPED_DIRECTORIES = new Set([`${RUNNER_PACKAGE_RELATIVE}/runner/target`]); + +export function runnerPackageRootFromScripts() { + return path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +} + +export function repoRootFromRunnerPackage(packageRoot) { + return path.resolve(packageRoot, "../.."); +} + +export function buildInputsStampPath(packageRoot) { + return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.json"); +} + +/** One record per build in progress, written by `--begin` and consumed by that build's `--finish`. */ +export function buildInputsPendingDirectory(packageRoot) { + return path.join(packageRoot, "node_modules", ".cache", "paperclip-runner-build-inputs.pending"); +} + +export function buildInputsPendingPath(packageRoot, buildId) { + return path.join(buildInputsPendingDirectory(packageRoot), `${buildId}.json`); +} + +/** + * A pending record older than this belongs to a build that never reached + * `--finish` (a failed step, an interrupted build); `--begin` removes such + * records so they cannot pile up. No Runner build runs anywhere near this long. + */ +export const PENDING_RECORD_MAX_AGE_MS = 24 * 60 * 60 * 1000; + +/** + * Identifies the build chain a `--begin` or `--finish` process belongs to. + * The two are separate processes; what they share is the shell pnpm runs the + * chain in, so the chain is named after its pid, which `--begin` sees as its + * parent. A shell that execs the last command of a `-c` string (bash can) + * turns `--finish` into the shell itself, so `--finish` also accepts a record + * named after its own pid. + */ +export function currentBuildId() { + return String(process.ppid); +} + +function finishingBuildIds() { + return [String(process.ppid), String(process.pid)]; +} + +function readPendingRecord(pendingPath) { + const pending = readBuildInputsStamp(pendingPath); + if (!pending || pending.schema !== BUILD_INPUTS_SCHEMA || typeof pending.digest !== "string") return null; + return pending; +} + +/** Drops pending records of builds that never finished, leaving the one for `buildId`. */ +function prunePendingRecords(packageRoot, buildId, now) { + const directory = buildInputsPendingDirectory(packageRoot); + let entries; + try { + entries = readdirSync(directory); + } catch { + return; + } + const cutoff = now().getTime() - PENDING_RECORD_MAX_AGE_MS; + for (const name of entries) { + if (name === `${buildId}.json`) continue; + const absolute = path.join(directory, name); + let stat; + try { + stat = lstatSync(absolute); + } catch { + continue; + } + if (stat.mtimeMs < cutoff) rmSync(absolute, { force: true }); + } +} + +/** + * The outputs every consumer of the vendored Runner reads; a build that did + * not produce them is not recorded, and a tree missing any of them is not + * current whatever the stamp says. + */ +export function expectedBuildOutputs(packageRoot, platform = process.platform) { + const executable = platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd"; + return ["dist/index.js", "dist/live/index.js", "dist/testing.js", `dist/bin/${executable}`] + .map((relative) => path.join(packageRoot, ...relative.split("/"))); +} + +function walk(root, relativeDirectory, { skipNames, skippedDirectories, skipDotDirectories }, out) { + const absolute = path.join(root, ...relativeDirectory.split("/")); + if (!existsSync(absolute)) return; + const entries = readdirSync(absolute, { withFileTypes: true }) + .sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)); + for (const entry of entries) { + const relative = `${relativeDirectory}/${entry.name}`; + if (entry.isDirectory()) { + if (skipDotDirectories && entry.name.startsWith(".")) continue; + if (skipNames.has(entry.name)) continue; + if (skippedDirectories.has(relative)) continue; + walk(root, relative, { skipNames, skippedDirectories, skipDotDirectories }, out); + continue; + } + // Files and symlinks; a symlinked directory is hashed as its link target + // rather than followed, so a link into node_modules cannot pull a + // dependency tree into the input set. + out.push(relative); + } +} + +const INPUT_WALK = { + skipNames: SKIPPED_DIRECTORY_NAMES, + skippedDirectories: new Set(), + skipDotDirectories: true, +}; + +/** Repo-relative POSIX paths of every build input, sorted. */ +export function collectBuildInputFiles(repoRoot) { + const files = []; + walk(repoRoot, RUNNER_PACKAGE_RELATIVE, { ...INPUT_WALK, skippedDirectories: RUNNER_SKIPPED_DIRECTORIES }, files); + walk(repoRoot, EVAL_KERNEL_PACKAGE_RELATIVE, INPUT_WALK, files); + for (const directory of ROOT_INPUT_DIRECTORIES) walk(repoRoot, directory, INPUT_WALK, files); + for (const name of ROOT_INPUT_FILES) { + if (existsSync(path.join(repoRoot, name))) files.push(name); + } + return files.sort(); +} + +export function computeBuildInputsDigest(repoRoot) { + const files = collectBuildInputFiles(repoRoot); + const hash = createHash("sha256"); + let byteCount = 0; + for (const relative of files) { + const absolute = path.join(repoRoot, ...relative.split("/")); + hash.update(relative); + hash.update("\0"); + if (lstatSync(absolute).isSymbolicLink()) { + hash.update("link\0"); + hash.update(readlinkSync(absolute)); + } else { + const content = readFileSync(absolute); + byteCount += content.length; + hash.update(`file ${content.length}\0`); + hash.update(content); + } + hash.update("\0"); + } + return { + schema: BUILD_INPUTS_SCHEMA, + digest: hash.digest("hex"), + fileCount: files.length, + byteCount, + }; +} + +/** + * Every file the build left under dist (the tree the server vendors), as + * package-relative POSIX paths with the size lstat reports, sorted. Nothing + * under dist is skipped: the server copies all of it. + */ +export function collectBuildOutputFiles(packageRoot) { + const files = []; + walk(packageRoot, OUTPUT_DIRECTORY_RELATIVE, { skipNames: new Set(), skippedDirectories: new Set(), skipDotDirectories: false }, files); + return files.sort().map((relative) => ({ + path: relative, + size: lstatSync(path.join(packageRoot, ...relative.split("/"))).size, + })); +} + +/** The first recorded output that is gone or has changed size, or null when all are intact. */ +export function findChangedBuildOutput(packageRoot, outputs) { + for (const output of outputs) { + const absolute = path.join(packageRoot, ...output.path.split("/")); + let stat; + try { + stat = lstatSync(absolute); + } catch { + return { ...output, state: "missing" }; + } + if (stat.size !== output.size) return { ...output, state: "changed size", size: stat.size, recordedSize: output.size }; + } + return null; +} + +export function readBuildInputsStamp(stampPath) { + try { + const parsed = JSON.parse(readFileSync(stampPath, "utf8")); + return parsed && typeof parsed === "object" ? parsed : null; + } catch { + return null; + } +} + +export function writeBuildInputsStamp(stampPath, record) { + mkdirSync(path.dirname(stampPath), { recursive: true }); + const staged = `${stampPath}.${process.pid}.tmp`; + writeFileSync(staged, `${JSON.stringify(record, null, 2)}\n`); + renameSync(staged, stampPath); +} + +export function removeBuildInputsStamp(stampPath) { + rmSync(stampPath, { force: true }); +} + +/** + * First step of the Runner's build: forgets the previous build (a build in + * progress can no longer vouch for its outputs) and records the inputs the + * build is about to read, so that `recordRunnerBuild` can tell whether they + * changed while it ran. + */ +export function beginRunnerBuild({ + repoRoot, + packageRoot, + platform = process.platform, + arch = process.arch, + buildId = currentBuildId(), + now = () => new Date(), +}) { + removeBuildInputsStamp(buildInputsStampPath(packageRoot)); + prunePendingRecords(packageRoot, buildId, now); + const inputs = computeBuildInputsDigest(repoRoot); + const pending = { ...inputs, platform, arch, buildId, startedAt: now().toISOString() }; + const pendingPath = buildInputsPendingPath(packageRoot, buildId); + writeBuildInputsStamp(pendingPath, pending); + return { ...pending, pendingPath }; +} + +/** + * Last step of the Runner's build, after every step and check has passed: + * records the inputs captured by `beginRunnerBuild` as the ones that produced + * the outputs now under dist. An edit that landed while the build ran means + * the outputs may not reflect the current inputs, so nothing is recorded and + * the next consumer rebuilds. A build that did not begin with + * `beginRunnerBuild` (no pending record for this build chain), or that left + * a required output missing, is an error. + */ +export function recordRunnerBuild({ + repoRoot, + packageRoot, + platform = process.platform, + arch = process.arch, + buildId = null, + now = () => new Date(), +}) { + const stampPath = buildInputsStampPath(packageRoot); + const candidates = buildId === null ? finishingBuildIds() : [String(buildId)]; + let pending = null; + let pendingPath = null; + for (const candidate of candidates) { + pendingPath = buildInputsPendingPath(packageRoot, candidate); + pending = readPendingRecord(pendingPath); + if (pending) break; + } + if (!pending) { + removeBuildInputsStamp(stampPath); + throw new Error( + `no pending build inputs for build ${candidates.join(" or ")} under ` + + `${buildInputsPendingDirectory(packageRoot)}: the Runner build must start with ` + + "`node scripts/record-build-inputs.mjs --begin` before it can record its result", + ); + } + rmSync(pendingPath, { force: true }); + + const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file)); + if (missing.length > 0) { + removeBuildInputsStamp(stampPath); + throw new Error(`the Runner build finished without producing ${path.relative(packageRoot, missing[0])}`); + } + + const inputs = computeBuildInputsDigest(repoRoot); + if (inputs.digest !== pending.digest) { + removeBuildInputsStamp(stampPath); + return { + recorded: false, + code: "inputs_changed_during_build", + detail: + `inputs changed while the build ran (digest ${pending.digest.slice(0, 12)} at start, ` + + `${inputs.digest.slice(0, 12)} now); the next consumer rebuilds`, + startedAt: pending.startedAt, + stampPath, + }; + } + + const outputs = collectBuildOutputFiles(packageRoot); + const record = { + ...inputs, + platform, + arch, + startedAt: pending.startedAt, + recordedAt: now().toISOString(), + outputCount: outputs.length, + outputs, + }; + writeBuildInputsStamp(stampPath, record); + return { recorded: true, code: "recorded", ...record, stampPath }; +} + +/** + * Decides whether the Runner's outputs are current for the present inputs. + * `code` is a stable slug for timing lines; `detail` is for humans. + */ +export function assessRunnerBuild({ + repoRoot, + packageRoot, + platform = process.platform, + arch = process.arch, +}) { + const stampPath = buildInputsStampPath(packageRoot); + const stamp = readBuildInputsStamp(stampPath); + if (!stamp) { + return { current: false, code: "no_stamp", detail: `no build stamp at ${stampPath}`, stampPath }; + } + if (stamp.schema !== BUILD_INPUTS_SCHEMA || !Array.isArray(stamp.outputs)) { + return { + current: false, + code: "schema_mismatch", + detail: `stamp schema ${String(stamp.schema)} is not ${BUILD_INPUTS_SCHEMA}`, + stampPath, + }; + } + if (stamp.platform !== platform || stamp.arch !== arch) { + return { + current: false, + code: "platform_mismatch", + detail: `stamp recorded on ${stamp.platform}/${stamp.arch}, this is ${platform}/${arch}`, + stampPath, + }; + } + const missing = expectedBuildOutputs(packageRoot, platform).filter((file) => !existsSync(file)); + if (missing.length > 0) { + return { + current: false, + code: "missing_output", + detail: `missing build output ${path.relative(packageRoot, missing[0])}`, + stampPath, + }; + } + const changedOutput = findChangedBuildOutput(packageRoot, stamp.outputs); + if (changedOutput) { + return { + current: false, + code: "outputs_changed", + detail: + `build output ${changedOutput.path} is ${changedOutput.state} since the build recorded at ` + + `${stamp.recordedAt}; ${stamp.outputs.length} outputs checked`, + stampPath, + }; + } + const inputs = computeBuildInputsDigest(repoRoot); + if (inputs.digest !== stamp.digest) { + return { + current: false, + code: "inputs_changed", + detail: `inputs changed since the recorded build; ${inputs.fileCount} files checked`, + stampPath, + inputs, + }; + } + return { + current: true, + code: "inputs_unchanged", + detail: + `inputs unchanged since the build recorded at ${stamp.recordedAt}; ${inputs.fileCount} files ` + + `and ${stamp.outputs.length} outputs checked`, + stampPath, + inputs, + stamp, + }; +} diff --git a/packages/paperclip-runner/scripts/build-inputs.test.mjs b/packages/paperclip-runner/scripts/build-inputs.test.mjs new file mode 100644 index 0000000000..ce650a871d --- /dev/null +++ b/packages/paperclip-runner/scripts/build-inputs.test.mjs @@ -0,0 +1,533 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { existsSync, readdirSync } from "node:fs"; +import { mkdtemp, mkdir, readdir, readFile, rm, stat, symlink, truncate, utimes, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { test } from "node:test"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; +import { + BUILD_INPUTS_SCHEMA, + PENDING_RECORD_MAX_AGE_MS, + assessRunnerBuild, + beginRunnerBuild, + buildInputsPendingDirectory, + buildInputsPendingPath, + buildInputsStampPath, + collectBuildInputFiles, + collectBuildOutputFiles, + computeBuildInputsDigest, + currentBuildId, + expectedBuildOutputs, + readBuildInputsStamp, + recordRunnerBuild, +} from "./build-inputs.mjs"; + +const execFileAsync = promisify(execFile); +const recordScript = fileURLToPath(new URL("./record-build-inputs.mjs", import.meta.url)); +/** A child of this process names its build after this process, as `--begin` names it after the shell. */ +const childBuildId = String(process.pid); + +async function waitFor(condition, label, timeoutMs = 10_000) { + const deadline = Date.now() + timeoutMs; + while (!condition()) { + if (Date.now() > deadline) throw new Error(`timed out waiting for ${label}`); + await new Promise((resolve) => setTimeout(resolve, 20)); + } +} + +/** + * The whole build chain the way pnpm runs it: one shell, `--begin` first, + * `--finish` last, with a stand-in for the build steps that waits for a + * marker file so two chains can be held open at the same time. + */ +function runChainInShell(shell, packageRoot, marker) { + const quote = (value) => `'${value.replace(/'/g, `'\\''`)}'`; + const wait = + `${quote(process.execPath)} -e ${quote( + "const fs=require('node:fs');const [marker]=process.argv.slice(1);const deadline=Date.now()+10000;" + + "(function poll(){if(fs.existsSync(marker))return;if(Date.now()>deadline)process.exit(3);setTimeout(poll,20)})()", + )} ${quote(marker)}`; + const phase = (flag) => `${quote(process.execPath)} ${quote(recordScript)} ${flag} --package-root ${quote(packageRoot)}`; + return execFileAsync(shell, ["-c", `${phase("--begin")} && ${wait} && ${phase("--finish")}`]); +} + +async function writeTree(root, files) { + for (const [relative, content] of Object.entries(files)) { + const absolute = path.join(root, ...relative.split("/")); + await mkdir(path.dirname(absolute), { recursive: true }); + await writeFile(absolute, content); + } +} + +/** + * A throwaway repo with one Runner source file, one Rust source, two root + * patches, built outputs (the four every consumer reads plus one helper that + * is only opened by path), and the usual noise. + */ +async function makeRepo() { + const repoRoot = await mkdtemp(path.join(tmpdir(), "runner-build-inputs-")); + await writeTree(repoRoot, { + "package.json": "{}\n", + "pnpm-lock.yaml": "lockfileVersion: 9\n", + "pnpm-workspace.yaml": "packages:\n - packages/*\n", + "tsconfig.base.json": "{}\n", + "patches/acpx@0.13.1.patch": "--- a\n+++ b\n", + "patches/pi-acp@0.0.33.patch": "--- c\n+++ d\n", + "packages/paperclip-runner/package.json": '{"name":"@paperclipai/paperclip-runner"}\n', + "packages/paperclip-runner/src/index.ts": "export const one = 1;\n", + "packages/paperclip-runner/runner/src/main.rs": "fn main() {}\n", + "packages/paperclip-runner/runner/target/release/paperclip-runnerd": "binary\n", + "packages/paperclip-runner/dist/index.js": "export const one = 1;\n", + "packages/paperclip-runner/dist/live/index.js": "export {};\n", + "packages/paperclip-runner/dist/testing.js": "export {};\n", + "packages/paperclip-runner/dist/bin/paperclip-runnerd": "binary\n", + "packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "module.exports = {};\n", + "packages/paperclip-runner/dist-browser/index.js": "browser\n", + "packages/paperclip-runner/node_modules/dep/index.js": "dep\n", + "packages/paperclip-runner/.vite/cache.json": "{}\n", + "packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 1;\n", + "packages/paperclip-eval-kernel/dist/index.js": "export const kernel = 1;\n", + }); + const packageRoot = path.join(repoRoot, "packages", "paperclip-runner"); + return { repoRoot, packageRoot }; +} + +const INPUT_FILE_COUNT = 10; +const OUTPUT_FILE_COUNT = 5; + +async function withRepo(action) { + const repo = await makeRepo(); + try { + await action(repo); + } finally { + await rm(repo.repoRoot, { recursive: true, force: true }); + } +} + +/** A complete, undisturbed build: begin, (nothing changes), finish. */ +function recordCurrentBuild({ repoRoot, packageRoot, platform, arch, now }) { + beginRunnerBuild({ repoRoot, packageRoot, platform, arch, now }); + return recordRunnerBuild({ repoRoot, packageRoot, platform, arch, now }); +} + +test("the input set covers sources, the eval kernel, root patches and root config, and skips outputs and caches", async () => { + await withRepo(async ({ repoRoot }) => { + const files = collectBuildInputFiles(repoRoot); + assert.deepEqual(files, [ + "package.json", + "packages/paperclip-eval-kernel/src/index.ts", + "packages/paperclip-runner/package.json", + "packages/paperclip-runner/runner/src/main.rs", + "packages/paperclip-runner/src/index.ts", + "patches/acpx@0.13.1.patch", + "patches/pi-acp@0.0.33.patch", + "pnpm-lock.yaml", + "pnpm-workspace.yaml", + "tsconfig.base.json", + ]); + assert.equal(files.length, INPUT_FILE_COUNT); + }); +}); + +test("the output manifest lists every file under dist with its size, and nothing outside it", async () => { + await withRepo(async ({ packageRoot }) => { + const outputs = collectBuildOutputFiles(packageRoot); + assert.deepEqual(outputs, [ + { path: "dist/bin/paperclip-runnerd", size: 7 }, + { path: "dist/cli/acpx-runtime-sidecar.cjs", size: 21 }, + { path: "dist/index.js", size: 22 }, + { path: "dist/live/index.js", size: 11 }, + { path: "dist/testing.js", size: 11 }, + ]); + assert.equal(outputs.length, OUTPUT_FILE_COUNT); + }); +}); + +test("the digest is stable, ignores outputs and caches, and tracks every kind of input", async () => { + await withRepo(async ({ repoRoot }) => { + const before = computeBuildInputsDigest(repoRoot); + assert.equal(before.schema, BUILD_INPUTS_SCHEMA); + assert.equal(before.fileCount, INPUT_FILE_COUNT); + assert.equal(computeBuildInputsDigest(repoRoot).digest, before.digest, "two reads disagree"); + + // Outputs, dependencies, caches and the Rust target directory may change freely. + await writeTree(repoRoot, { + "packages/paperclip-runner/dist/index.js": "export const one = 2;\n", + "packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "changed\n", + "packages/paperclip-runner/dist-browser/index.js": "changed\n", + "packages/paperclip-runner/node_modules/dep/index.js": "changed\n", + "packages/paperclip-runner/.vite/cache.json": '{"changed":true}\n', + "packages/paperclip-runner/runner/target/release/paperclip-runnerd": "rebuilt\n", + "packages/paperclip-eval-kernel/dist/index.js": "changed\n", + }); + assert.equal(computeBuildInputsDigest(repoRoot).digest, before.digest, "an output changed the digest"); + + // Every input kind changes it: TypeScript, Rust, the eval kernel, a root + // patch, root config, a new file, a rename. + const seen = new Set([before.digest]); + const expectChange = async (label, files) => { + await writeTree(repoRoot, files); + const next = computeBuildInputsDigest(repoRoot).digest; + assert.ok(!seen.has(next), `${label} did not change the digest`); + seen.add(next); + }; + await expectChange("a Runner source", { "packages/paperclip-runner/src/index.ts": "export const one = 2;\n" }); + await expectChange("a Rust source", { "packages/paperclip-runner/runner/src/main.rs": "fn main() { }\n" }); + await expectChange("an eval-kernel source", { "packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 2;\n" }); + await expectChange("the pi-acp patch the build copies into dist", { "patches/pi-acp@0.0.33.patch": "--- c\n+++ d\n+e\n" }); + await expectChange("the acpx patch the profile check hashes", { "patches/acpx@0.13.1.patch": "--- a\n+++ b\n+c\n" }); + await expectChange("a new patch", { "patches/brace-expansion@5.0.9.patch": "--- x\n+++ y\n" }); + await expectChange("the lockfile", { "pnpm-lock.yaml": "lockfileVersion: 9\nchanged: true\n" }); + await expectChange("the base tsconfig", { "tsconfig.base.json": '{"compilerOptions":{}}\n' }); + await expectChange("a new file", { "packages/paperclip-runner/protocol/new.json": "{}\n" }); + await rm(path.join(repoRoot, "packages/paperclip-runner/protocol/new.json")); + await expectChange("a renamed file", { "packages/paperclip-runner/protocol/renamed.json": "{}\n" }); + }); +}); + +test("a symlink is hashed as its target path, not followed", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const before = computeBuildInputsDigest(repoRoot); + await symlink("../node_modules/dep", path.join(packageRoot, "src", "linked"), "dir"); + const linked = computeBuildInputsDigest(repoRoot); + assert.notEqual(linked.digest, before.digest, "the link itself is an input"); + assert.equal(linked.fileCount, before.fileCount + 1, "the link target's files were followed"); + await writeFile(path.join(packageRoot, "node_modules/dep/index.js"), "changed\n"); + assert.equal(computeBuildInputsDigest(repoRoot).digest, linked.digest, "a change behind the link changed the digest"); + }); +}); + +test("a build is current only with a matching stamp and every recorded output intact", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const platform = "linux"; + const arch = "x64"; + const assess = () => assessRunnerBuild({ repoRoot, packageRoot, platform, arch }); + + assert.equal(assess().current, false); + assert.equal(assess().code, "no_stamp"); + + const record = recordCurrentBuild({ repoRoot, packageRoot, platform, arch, now: () => new Date(0) }); + assert.equal(record.recorded, true); + assert.equal(record.stampPath, buildInputsStampPath(packageRoot)); + const stamp = readBuildInputsStamp(record.stampPath); + assert.equal(stamp.schema, BUILD_INPUTS_SCHEMA); + assert.equal(stamp.digest, record.digest); + assert.equal(stamp.fileCount, INPUT_FILE_COUNT); + assert.equal(stamp.byteCount, record.byteCount); + assert.equal(stamp.platform, platform); + assert.equal(stamp.arch, arch); + assert.equal(stamp.startedAt, "1970-01-01T00:00:00.000Z"); + assert.equal(stamp.recordedAt, "1970-01-01T00:00:00.000Z"); + assert.equal(stamp.outputCount, OUTPUT_FILE_COUNT); + assert.deepEqual(stamp.outputs, collectBuildOutputFiles(packageRoot)); + const current = assess(); + assert.equal(current.current, true, current.detail); + assert.equal(current.code, "inputs_unchanged"); + assert.match(current.detail, /10 files and 5 outputs checked/); + + assert.equal(assessRunnerBuild({ repoRoot, packageRoot, platform: "win32", arch }).code, "platform_mismatch"); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot, platform, arch: "arm64" }).code, "platform_mismatch"); + + // The four outputs every consumer reads are checked by name. + for (const output of expectedBuildOutputs(packageRoot, platform)) { + const content = await readFile(output); + await rm(output); + const missing = assess(); + assert.equal(missing.current, false, `${output} missing still counted as current`); + assert.equal(missing.code, "missing_output"); + assert.match(missing.detail, new RegExp(path.basename(output))); + await writeFile(output, content); + } + assert.equal(assess().current, true); + + // So is every other file the build left under dist: a helper that is only + // opened by path must be there, at the size the build wrote. + const helper = path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs"); + const helperContent = await readFile(helper); + await rm(helper); + const gone = assess(); + assert.equal(gone.current, false, "a missing helper still counted as current"); + assert.equal(gone.code, "outputs_changed"); + assert.match(gone.detail, /^build output dist\/cli\/acpx-runtime-sidecar\.cjs is missing since the build recorded at /); + await writeFile(helper, helperContent); + assert.equal(assess().current, true); + + await truncate(helper, 3); + const truncated = assess(); + assert.equal(truncated.current, false, "a truncated helper still counted as current"); + assert.equal(truncated.code, "outputs_changed"); + assert.match(truncated.detail, /^build output dist\/cli\/acpx-runtime-sidecar\.cjs is changed size since /); + await writeFile(helper, helperContent); + assert.equal(assess().current, true); + + // A file the build did not produce may appear without invalidating it. + await writeFile(path.join(packageRoot, "dist", "extra.txt"), "later\n"); + assert.equal(assess().current, true); + + await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 3;\n"); + const changed = assess(); + assert.equal(changed.current, false); + assert.equal(changed.code, "inputs_changed"); + + await writeFile(record.stampPath, JSON.stringify({ ...readBuildInputsStamp(record.stampPath), schema: "other" })); + assert.equal(assess().code, "schema_mismatch"); + await writeFile(record.stampPath, JSON.stringify({ ...readBuildInputsStamp(record.stampPath), schema: BUILD_INPUTS_SCHEMA, outputs: undefined })); + assert.equal(assess().code, "schema_mismatch", "a stamp without an output manifest was trusted"); + await writeFile(record.stampPath, "not json"); + assert.equal(assess().code, "no_stamp"); + }); +}); + +test("an input edited while the build runs leaves no stamp, so the next consumer rebuilds", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const stampPath = buildInputsStampPath(packageRoot); + const pendingPath = buildInputsPendingPath(packageRoot, currentBuildId()); + const assess = () => assessRunnerBuild({ repoRoot, packageRoot }); + + // A previous good build, then a new build begins: the old stamp is gone at + // once, because the outputs are about to be replaced. + recordCurrentBuild({ repoRoot, packageRoot }); + assert.equal(assess().current, true); + const pending = beginRunnerBuild({ repoRoot, packageRoot, now: () => new Date(1000) }); + assert.equal(pending.pendingPath, pendingPath); + assert.equal(pending.buildId, String(process.ppid), "a build is named after the shell that runs its chain"); + assert.equal(readBuildInputsStamp(stampPath), null, "the old stamp survived the start of a new build"); + assert.equal(readBuildInputsStamp(pendingPath).digest, pending.digest); + assert.equal(readBuildInputsStamp(pendingPath).buildId, pending.buildId); + assert.equal(assess().code, "no_stamp"); + + // The build "compiles" the old source, then an edit lands before it ends. + await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n"); + const result = recordRunnerBuild({ repoRoot, packageRoot }); + assert.equal(result.recorded, false); + assert.equal(result.code, "inputs_changed_during_build"); + assert.match(result.detail, /^inputs changed while the build ran \(digest [0-9a-f]{12} at start, [0-9a-f]{12} now\); the next consumer rebuilds$/); + assert.equal(result.startedAt, "1970-01-01T00:00:01.000Z"); + assert.equal(readBuildInputsStamp(stampPath), null, "a stamp was written for a build whose inputs moved"); + assert.equal(readBuildInputsStamp(pendingPath), null, "the pending record was left behind"); + assert.equal(assess().code, "no_stamp"); + + // An undisturbed build records normally, and only the pending record's + // start time is kept. + const record = recordCurrentBuild({ repoRoot, packageRoot }); + assert.equal(record.recorded, true); + assert.equal(readBuildInputsStamp(pendingPath), null); + assert.equal(assess().current, true); + }); +}); + +test("a build that did not begin, or that left a required output missing, cannot be recorded", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const stampPath = buildInputsStampPath(packageRoot); + + recordCurrentBuild({ repoRoot, packageRoot }); + assert.throws( + () => recordRunnerBuild({ repoRoot, packageRoot }), + new RegExp( + `no pending build inputs for build ${process.ppid} or ${process.pid} under .*paperclip-runner-build-inputs\\.pending: ` + + "the Runner build must start with `node scripts/record-build-inputs\\.mjs --begin`", + ), + ); + assert.equal(readBuildInputsStamp(stampPath), null, "the previous stamp outlived a build that did not begin"); + + // Another build's record is not this build's: a chain that did not begin + // cannot finish on the strength of one that is still running. + beginRunnerBuild({ repoRoot, packageRoot, buildId: "other-shell" }); + assert.throws(() => recordRunnerBuild({ repoRoot, packageRoot }), /no pending build inputs for build /); + assert.equal( + readBuildInputsStamp(buildInputsPendingPath(packageRoot, "other-shell")).digest, + computeBuildInputsDigest(repoRoot).digest, + "a build that did not begin consumed another build's record", + ); + await rm(buildInputsPendingPath(packageRoot, "other-shell")); + + beginRunnerBuild({ repoRoot, packageRoot }); + await rm(path.join(packageRoot, "dist", "testing.js")); + assert.throws( + () => recordRunnerBuild({ repoRoot, packageRoot }), + /the Runner build finished without producing dist\/testing\.js/, + ); + assert.equal(readBuildInputsStamp(stampPath), null); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).code, "no_stamp"); + }); +}); + +test("record-build-inputs.mjs brackets the build for the package it is pointed at", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const run = (...args) => execFileAsync(process.execPath, [recordScript, ...args, "--package-root", packageRoot]); + const expected = computeBuildInputsDigest(repoRoot); + const short = expected.digest.slice(0, 12); + + // --finish before any --begin is a wiring error, not a silent stamp. + await assert.rejects(run("--finish"), /no pending build inputs/); + assert.equal(readBuildInputsStamp(buildInputsStampPath(packageRoot)), null); + + const begin = await run("--begin"); + assert.match(begin.stdout, new RegExp(`^${BUILD_INPUTS_SCHEMA} begin: digest=${short} files=${INPUT_FILE_COUNT} bytes=\\d+\n$`)); + assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, childBuildId)).digest, expected.digest); + + const finish = await run("--finish"); + assert.match( + finish.stdout, + new RegExp(`^${BUILD_INPUTS_SCHEMA} recorded: digest=${short} files=${INPUT_FILE_COUNT} bytes=\\d+ outputs=${OUTPUT_FILE_COUNT} stamp=node_modules/.cache/paperclip-runner-build-inputs.json\n$`), + ); + const stamp = readBuildInputsStamp(buildInputsStampPath(packageRoot)); + assert.equal(stamp.digest, expected.digest); + assert.equal(stamp.platform, process.platform); + assert.equal(stamp.arch, process.arch); + assert.equal(stamp.outputCount, OUTPUT_FILE_COUNT); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true); + assert.equal((await stat(buildInputsStampPath(packageRoot))).isFile(), true); + + // An edit between the two phases is reported and leaves nothing behind. + await run("--begin"); + await writeFile(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n"); + const moved = await run("--finish"); + assert.match(moved.stdout, /^paperclip\.runner-build-inputs not recorded: inputs changed while the build ran /); + assert.equal(readBuildInputsStamp(buildInputsStampPath(packageRoot)), null); + + await assert.rejects(run(), /one of --begin or --finish is required/); + await assert.rejects(run("--begin", "--finish"), /--finish cannot be combined with --begin/); + await assert.rejects( + execFileAsync(process.execPath, [recordScript, "--begin", "--bogus"]), + /unknown argument --bogus/, + ); + }); +}); + +test("two builds running at once in one checkout each keep their own pending record", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const stampPath = buildInputsStampPath(packageRoot); + const pendingDirectory = buildInputsPendingDirectory(packageRoot); + const first = { repoRoot, packageRoot, buildId: "shell-1" }; + const second = { repoRoot, packageRoot, buildId: "shell-2" }; + const digestNow = () => computeBuildInputsDigest(repoRoot).digest; + + // Nothing changes: both builds record, in either finishing order, and the + // second to finish does not fail for want of a record. + beginRunnerBuild(first); + beginRunnerBuild(second); + assert.deepEqual((await readdir(pendingDirectory)).sort(), ["shell-1.json", "shell-2.json"]); + assert.equal(recordRunnerBuild(first).recorded, true); + assert.deepEqual(await readdir(pendingDirectory), ["shell-2.json"], "the first finish consumed the wrong record"); + assert.equal(readBuildInputsStamp(stampPath).digest, digestNow()); + assert.equal(recordRunnerBuild(second).recorded, true); + assert.deepEqual(await readdir(pendingDirectory), []); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true); + + beginRunnerBuild(first); + beginRunnerBuild(second); + assert.equal(recordRunnerBuild(second).recorded, true); + assert.equal(recordRunnerBuild(first).recorded, true); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true); + + // An edit lands between the two starts. The build that began before it + // must not leave a stamp whichever of the two finishes last: a stamp means + // every build that wrote dist read the inputs it names. + const source = path.join(packageRoot, "src", "index.ts"); + beginRunnerBuild(first); + await writeFile(source, "export const one = 2;\n"); + beginRunnerBuild(second); + const staleFirst = recordRunnerBuild(first); + assert.equal(staleFirst.recorded, false); + assert.equal(staleFirst.code, "inputs_changed_during_build"); + assert.equal(readBuildInputsStamp(stampPath), null); + assert.equal(recordRunnerBuild(second).recorded, true, "the build that read the edited inputs was not recorded"); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true); + + beginRunnerBuild(first); + await writeFile(source, "export const one = 3;\n"); + beginRunnerBuild(second); + assert.equal(recordRunnerBuild(second).recorded, true); + assert.equal(readBuildInputsStamp(stampPath).digest, digestNow()); + const staleLast = recordRunnerBuild(first); + assert.equal(staleLast.recorded, false); + assert.equal(staleLast.code, "inputs_changed_during_build"); + assert.equal(readBuildInputsStamp(stampPath), null, "a build that began before the edit left the other build's stamp standing"); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).code, "no_stamp"); + assert.deepEqual(await readdir(pendingDirectory), []); + }); +}); + +test("a build's start drops the pending records of builds that never finished, and no others", async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const pendingDirectory = buildInputsPendingDirectory(packageRoot); + const now = new Date("2026-10-11T22:00:00.000Z"); + const age = async (buildId, ageMs) => { + const moment = new Date(now.getTime() - ageMs); + await utimes(buildInputsPendingPath(packageRoot, buildId), moment, moment); + }; + + // A build that failed a step yesterday, one that failed a minute ago, and + // one still running, all left records; a new build starts. + beginRunnerBuild({ repoRoot, packageRoot, buildId: "crashed-yesterday", now: () => now }); + await age("crashed-yesterday", PENDING_RECORD_MAX_AGE_MS + 60_000); + beginRunnerBuild({ repoRoot, packageRoot, buildId: "crashed-just-now", now: () => now }); + await age("crashed-just-now", 60_000); + beginRunnerBuild({ repoRoot, packageRoot, buildId: "still-running", now: () => now }); + await writeFile(path.join(pendingDirectory, "notes.txt"), "not a record\n"); + await age("still-running", 0); + + beginRunnerBuild({ repoRoot, packageRoot, buildId: "new-build", now: () => now }); + assert.deepEqual( + (await readdir(pendingDirectory)).sort(), + ["crashed-just-now.json", "new-build.json", "notes.txt", "still-running.json"], + "pruning removed the wrong records", + ); + + // The running build and the new one still finish on their own records. + assert.equal(recordRunnerBuild({ repoRoot, packageRoot, buildId: "still-running" }).recorded, true); + assert.equal(recordRunnerBuild({ repoRoot, packageRoot, buildId: "new-build" }).recorded, true); + // A record that fell out of the directory is simply absent. + await rm(pendingDirectory, { recursive: true, force: true }); + beginRunnerBuild({ repoRoot, packageRoot, buildId: "after-wipe" }); + assert.deepEqual(await readdir(pendingDirectory), ["after-wipe.json"]); + }); +}); + +test("two chains run by separate shells at the same time both record", { skip: process.platform === "win32" }, async () => { + await withRepo(async ({ repoRoot, packageRoot }) => { + const pendingDirectory = buildInputsPendingDirectory(packageRoot); + const marker = path.join(repoRoot, "release-the-chains"); + const records = () => (existsSync(pendingDirectory) ? readdirSync(pendingDirectory).sort() : []); + + const first = runChainInShell("sh", packageRoot, marker); + await waitFor(() => records().length === 1, "the first chain's record"); + const second = runChainInShell("sh", packageRoot, marker); + await waitFor(() => records().length === 2, "the second chain's record"); + const [firstId, secondId] = records().map((name) => name.replace(/\.json$/, "")); + assert.notEqual(firstId, secondId); + for (const id of [firstId, secondId]) { + assert.match(id, /^\d+$/, "a chain's record is named after its shell's pid"); + assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, id)).buildId, id); + } + + await writeFile(marker, "go\n"); + const results = await Promise.all([first, second]); + for (const result of results) { + assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} begin: `)); + assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `), "a chain finished without recording"); + } + assert.deepEqual(records(), []); + assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true); + + // The same chain under bash, which may exec its last command: `--finish` + // is then the shell itself and still finds the record `--begin` wrote. + let bash; + try { + bash = (await execFileAsync("bash", ["-c", "echo ok"])).stdout.trim() === "ok"; + } catch { + bash = false; + } + if (bash) { + await rm(marker); + const chain = runChainInShell("bash", packageRoot, marker); + await waitFor(() => records().length === 1, "the bash chain's record"); + await writeFile(marker, "go\n"); + assert.match((await chain).stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `)); + assert.deepEqual(records(), []); + } + }); +}); diff --git a/packages/paperclip-runner/scripts/record-build-inputs.mjs b/packages/paperclip-runner/scripts/record-build-inputs.mjs new file mode 100644 index 0000000000..6e0973dfb9 --- /dev/null +++ b/packages/paperclip-runner/scripts/record-build-inputs.mjs @@ -0,0 +1,79 @@ +// Brackets the Runner package's `build` so server's prepare:runner-vendor can +// reuse this build instead of repeating it when nothing has changed. See +// build-inputs.mjs for what counts as an input and an output. +// +// --begin first step of the build: forgets the previous stamp and records +// the inputs the build is about to read +// --finish last step of the build, after every check: writes the stamp if +// the inputs are still the ones the build began with, with a +// manifest of every file under dist; otherwise writes nothing, so +// the next consumer rebuilds +// +// Each build chain has its own pending record, named after the shell that +// runs the chain (the parent of both phases), so two Runner builds running at +// once in one checkout do not consume each other's record. +// +// Usage: node scripts/record-build-inputs.mjs (--begin | --finish) [--package-root ] +// The package root defaults to this package; the flag exists for tests that +// build a throwaway package layout. + +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { + beginRunnerBuild, + recordRunnerBuild, + repoRootFromRunnerPackage, + runnerPackageRootFromScripts, +} from "./build-inputs.mjs"; + +export function parseArguments(argv) { + let packageRoot = null; + let phase = null; + for (let index = 0; index < argv.length; index += 1) { + const argument = argv[index]; + if (argument === "--begin" || argument === "--finish") { + if (phase) throw new Error(`${argument} cannot be combined with --${phase}`); + phase = argument.slice(2); + continue; + } + if (argument === "--package-root") { + packageRoot = argv[index + 1]; + if (!packageRoot) throw new Error("--package-root needs a directory"); + index += 1; + continue; + } + throw new Error(`unknown argument ${argument}`); + } + if (!phase) throw new Error("one of --begin or --finish is required"); + return { phase, packageRoot }; +} + +export function main(argv = process.argv.slice(2), log = console.log) { + const { phase, packageRoot: requested } = parseArguments(argv); + const packageRoot = requested ? path.resolve(requested) : runnerPackageRootFromScripts(); + const repoRoot = repoRootFromRunnerPackage(packageRoot); + + if (phase === "begin") { + const pending = beginRunnerBuild({ repoRoot, packageRoot }); + log( + `${pending.schema} begin: digest=${pending.digest.slice(0, 12)} files=${pending.fileCount} ` + + `bytes=${pending.byteCount}`, + ); + return pending; + } + + const record = recordRunnerBuild({ repoRoot, packageRoot }); + if (!record.recorded) { + log(`${record.code === "inputs_changed_during_build" ? "paperclip.runner-build-inputs" : record.code} not recorded: ${record.detail}`); + return record; + } + log( + `${record.schema} recorded: digest=${record.digest.slice(0, 12)} files=${record.fileCount} ` + + `bytes=${record.byteCount} outputs=${record.outputCount} stamp=${path.relative(packageRoot, record.stampPath)}`, + ); + return record; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main(); +} diff --git a/server/package.json b/server/package.json index 657e961e72..82088f5548 100644 --- a/server/package.json +++ b/server/package.json @@ -40,7 +40,7 @@ "postpack": "rm -rf ui-dist", "clean": "rm -rf dist", "start": "node dist/index.js", - "prepare:runner-vendor": "pnpm --filter @paperclipai/paperclip-runner build", + "prepare:runner-vendor": "node scripts/prepare-runner-vendor.mjs", "typecheck": "pnpm run prepare:runner-vendor && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit" }, "dependencies": { diff --git a/server/scripts/prepare-runner-vendor.mjs b/server/scripts/prepare-runner-vendor.mjs new file mode 100644 index 0000000000..d41e86bd14 --- /dev/null +++ b/server/scripts/prepare-runner-vendor.mjs @@ -0,0 +1,143 @@ +// Makes sure packages/paperclip-runner/dist is a current build before the +// server build vendors it, rebuilding the Runner only when that is needed. +// +// The server build (and typecheck) used to run the Runner's full `build` +// unconditionally here. That is what lets a standalone +// `pnpm --filter @paperclipai/server build` produce a current vendored Runner +// on its own, which the Dockerfile relies on. But inside `pnpm -r build` the +// Runner has just been built by the same command, and repeating it cost +// 12-16s on the CI Build job's critical path (six master runs, 2026-10-11) +// and the same again in every other `pnpm build`. +// +// The Runner's `build` now records a digest of its inputs, captured before its +// first step and confirmed after its last, together with a manifest of every +// file it left under dist (packages/paperclip-runner/scripts/build-inputs.mjs). +// This step recomputes the digest and reuses the build when the stamp matches +// and every recorded output is still there at its recorded size; anything else +// runs the full Runner build exactly as before. The standalone guarantee +// therefore holds: a fresh checkout, a changed input, a deleted or truncated +// output, or an edit that landed while the build ran always rebuilds. Set +// PAPERCLIP_RUNNER_VENDOR_REBUILD=1 to rebuild regardless of the stamp. +// +// Usage: node scripts/prepare-runner-vendor.mjs [--repo-root ] +// The repo root defaults to this checkout; the flag exists for tests that +// build a throwaway workspace layout. + +import { spawnSync } from "node:child_process"; +import { performance } from "node:perf_hooks"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { + assessRunnerBuild, + removeBuildInputsStamp, +} from "../../packages/paperclip-runner/scripts/build-inputs.mjs"; + +/** The command the server build has always run here; the rebuild path is unchanged. */ +export const RUNNER_VENDOR_BUILD_COMMAND = "pnpm --filter @paperclipai/paperclip-runner build"; +export const FORCE_REBUILD_VARIABLE = "PAPERCLIP_RUNNER_VENDOR_REBUILD"; +export const TIMING_SCHEMA = "paperclip.runner-vendor.timing.v1"; + +export function decideRunnerVendorAction({ env, assessment }) { + if (env[FORCE_REBUILD_VARIABLE] === "1") { + return { action: "rebuild", code: "forced", detail: `${FORCE_REBUILD_VARIABLE}=1` }; + } + if (assessment.current) { + return { action: "reuse", code: assessment.code, detail: assessment.detail }; + } + return { action: "rebuild", code: assessment.code, detail: assessment.detail }; +} + +/** + * The pnpm that is running this script, the same rule as the CLI's + * resolvePnpmInstallInvocation: pnpm sets npm_execpath to its own entry, which + * is a JavaScript file for a normal install (run through this Node) or a native + * executable for a standalone pnpm (run directly). Outside a pnpm script, + * `pnpm` on PATH is the only option. The pnpm entry is not always on PATH in a + * script (corepack, for one), so this cannot just spawn `pnpm`. + */ +export function resolvePnpmInvocation(env = process.env, nodeExecPath = process.execPath) { + const npmExecPath = env.npm_execpath?.trim(); + if (npmExecPath && npmExecPath.toLowerCase().includes("pnpm")) { + if (/\.(cjs|mjs|js)$/i.test(npmExecPath)) { + return { command: nodeExecPath, argsPrefix: [npmExecPath] }; + } + return { command: npmExecPath, argsPrefix: [] }; + } + return { command: "pnpm", argsPrefix: [] }; +} + +export function runBuildCommand(command, { cwd, env = process.env }) { + const [tool, ...args] = command.split(" "); + if (tool !== "pnpm") throw new Error(`unexpected build command ${command}`); + const pnpm = resolvePnpmInvocation(env); + const result = spawnSync(pnpm.command, [...pnpm.argsPrefix, ...args], { + cwd, + env, + stdio: "inherit", + // A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve. + shell: pnpm.command === "pnpm" && process.platform === "win32", + }); + if (result.error) throw result.error; + return result.status ?? 1; +} + +/** + * Returns the exit status for the step. `run` is injectable so tests can + * observe the command and the stamp state at the moment it runs. + */ +export function prepareRunnerVendor({ + repoRoot, + env = process.env, + run = runBuildCommand, + log = console.log, + now = () => performance.now(), +}) { + const packageRoot = path.join(repoRoot, "packages", "paperclip-runner"); + const checkStart = now(); + const assessment = assessRunnerBuild({ repoRoot, packageRoot }); + const decision = decideRunnerVendorAction({ env, assessment }); + const checkMs = Math.round(now() - checkStart); + + if (decision.action === "reuse") { + log(`prepare:runner-vendor: reusing the current Runner build (${decision.detail})`); + log(`${TIMING_SCHEMA},action:reuse,reason:${decision.code},check_ms:${checkMs},build_ms:0`); + return 0; + } + + log(`prepare:runner-vendor: rebuilding the Runner (${decision.detail})`); + // A build that stops partway must not leave a stamp that vouches for it. The + // Runner's own build removes the stamp in its first step and writes a fresh + // one in its last; removing it here as well covers a build command that + // fails before its first step runs. + removeBuildInputsStamp(assessment.stampPath); + const buildStart = now(); + const status = run(RUNNER_VENDOR_BUILD_COMMAND, { cwd: path.join(repoRoot, "server"), env }); + const buildMs = Math.round(now() - buildStart); + log( + `${TIMING_SCHEMA},action:rebuild,reason:${decision.code},check_ms:${checkMs},build_ms:${buildMs},status:${status}`, + ); + return status; +} + +export function parseArguments(argv) { + let repoRoot = null; + for (let index = 0; index < argv.length; index += 1) { + const argument = argv[index]; + if (argument === "--repo-root") { + repoRoot = argv[index + 1]; + if (!repoRoot) throw new Error("--repo-root needs a directory"); + index += 1; + continue; + } + throw new Error(`unknown argument ${argument}`); + } + return { repoRoot }; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const { repoRoot: requested } = parseArguments(process.argv.slice(2)); + const repoRoot = requested + ? path.resolve(requested) + : path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); + process.exitCode = prepareRunnerVendor({ repoRoot }); +} diff --git a/server/scripts/prepare-runner-vendor.test.mjs b/server/scripts/prepare-runner-vendor.test.mjs new file mode 100644 index 0000000000..d42be80df1 --- /dev/null +++ b/server/scripts/prepare-runner-vendor.test.mjs @@ -0,0 +1,312 @@ +import { execFile } from "node:child_process"; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; + +import { afterEach, describe, expect, it } from "vitest"; + +import { + assessRunnerBuild, + beginRunnerBuild, + buildInputsStampPath, + readBuildInputsStamp, + recordRunnerBuild, +} from "../../packages/paperclip-runner/scripts/build-inputs.mjs"; +import { + FORCE_REBUILD_VARIABLE, + RUNNER_VENDOR_BUILD_COMMAND, + TIMING_SCHEMA, + decideRunnerVendorAction, + prepareRunnerVendor, + resolvePnpmInvocation, +} from "./prepare-runner-vendor.mjs"; + +const execFileAsync = promisify(execFile); +const prepareScript = fileURLToPath(new URL("./prepare-runner-vendor.mjs", import.meta.url)); +const recordScript = fileURLToPath( + new URL("../../packages/paperclip-runner/scripts/record-build-inputs.mjs", import.meta.url), +); +const executable = process.platform === "win32" ? "paperclip-runnerd.exe" : "paperclip-runnerd"; + +const roots = []; + +function writeTree(root, files) { + for (const [relative, content] of Object.entries(files)) { + const absolute = path.join(root, ...relative.split("/")); + mkdirSync(path.dirname(absolute), { recursive: true }); + writeFileSync(absolute, content); + } +} + +/** A throwaway workspace whose Runner is already built, with no stamp yet. */ +function makeRepo() { + const repoRoot = mkdtempSync(path.join(tmpdir(), "prepare-runner-vendor-")); + roots.push(repoRoot); + writeTree(repoRoot, { + "package.json": "{}\n", + "pnpm-lock.yaml": "lockfileVersion: 9\n", + "pnpm-workspace.yaml": "packages:\n - packages/*\n - server\n", + "tsconfig.base.json": "{}\n", + "patches/acpx@0.13.1.patch": "--- a\n+++ b\n", + "server/package.json": '{"name":"@paperclipai/server"}\n', + "packages/paperclip-runner/package.json": '{"name":"@paperclipai/paperclip-runner"}\n', + "packages/paperclip-runner/src/index.ts": "export const one = 1;\n", + "packages/paperclip-runner/dist/index.js": "export const one = 1;\n", + "packages/paperclip-runner/dist/live/index.js": "export {};\n", + "packages/paperclip-runner/dist/testing.js": "export {};\n", + "packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs": "module.exports = {};\n", + [`packages/paperclip-runner/dist/bin/${executable}`]: "binary\n", + "packages/paperclip-eval-kernel/src/index.ts": "export const kernel = 1;\n", + }); + const packageRoot = path.join(repoRoot, "packages", "paperclip-runner"); + return { repoRoot, packageRoot }; +} + +/** What the Runner's build does around its steps when nothing changes meanwhile. */ +function recordCurrentBuild({ repoRoot, packageRoot }) { + beginRunnerBuild({ repoRoot, packageRoot }); + return recordRunnerBuild({ repoRoot, packageRoot }); +} + +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +describe("decideRunnerVendorAction", () => { + const current = { current: true, code: "inputs_unchanged", detail: "unchanged" }; + const stale = { current: false, code: "inputs_changed", detail: "changed" }; + + it("reuses a current build and rebuilds a stale one, carrying the assessment's reason", () => { + expect(decideRunnerVendorAction({ env: {}, assessment: current })).toEqual({ + action: "reuse", + code: "inputs_unchanged", + detail: "unchanged", + }); + expect(decideRunnerVendorAction({ env: {}, assessment: stale })).toEqual({ + action: "rebuild", + code: "inputs_changed", + detail: "changed", + }); + }); + + it("rebuilds a current build when the operator forces it", () => { + expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "1" }, assessment: current })).toEqual({ + action: "rebuild", + code: "forced", + detail: `${FORCE_REBUILD_VARIABLE}=1`, + }); + // Only the documented value forces; an unrelated value does not. + expect(decideRunnerVendorAction({ env: { [FORCE_REBUILD_VARIABLE]: "0" }, assessment: current }).action).toBe( + "reuse", + ); + }); +}); + +describe("resolvePnpmInvocation", () => { + it("runs the pnpm that is running the script, through Node for a JavaScript entry", () => { + expect(resolvePnpmInvocation({ npm_execpath: "/corepack/pnpm/9.15.4/bin/pnpm.cjs" }, "/usr/bin/node")).toEqual({ + command: "/usr/bin/node", + argsPrefix: ["/corepack/pnpm/9.15.4/bin/pnpm.cjs"], + }); + expect(resolvePnpmInvocation({ npm_execpath: "/opt/pnpm/pnpm" }, "/usr/bin/node")).toEqual({ + command: "/opt/pnpm/pnpm", + argsPrefix: [], + }); + expect(resolvePnpmInvocation({ npm_execpath: "C:\\pnpm\\pnpm.exe" }, "node.exe")).toEqual({ + command: "C:\\pnpm\\pnpm.exe", + argsPrefix: [], + }); + }); + + it("falls back to pnpm on PATH outside a pnpm script or under another package manager", () => { + expect(resolvePnpmInvocation({}, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] }); + expect(resolvePnpmInvocation({ npm_execpath: "" }, "/usr/bin/node")).toEqual({ command: "pnpm", argsPrefix: [] }); + expect(resolvePnpmInvocation({ npm_execpath: "/usr/lib/node_modules/npm/bin/npm-cli.js" }, "/usr/bin/node")).toEqual({ + command: "pnpm", + argsPrefix: [], + }); + }); +}); + +describe("prepareRunnerVendor", () => { + it("still runs the Runner build the server build has always run", () => { + // The Dockerfile's standalone `pnpm --filter @paperclipai/server build` + // depends on this exact command producing a current vendored Runner. + expect(RUNNER_VENDOR_BUILD_COMMAND).toBe("pnpm --filter @paperclipai/paperclip-runner build"); + }); + + it("skips the rebuild when the recorded build is current and reports the check time", () => { + const { repoRoot, packageRoot } = makeRepo(); + recordCurrentBuild({ repoRoot, packageRoot }); + const runs = []; + const lines = []; + + const status = prepareRunnerVendor({ + repoRoot, + env: {}, + run: (...args) => { + runs.push(args); + return 0; + }, + log: (line) => lines.push(line), + }); + + expect(status).toBe(0); + expect(runs).toEqual([]); + expect(lines[0]).toMatch(/^prepare:runner-vendor: reusing the current Runner build \(inputs unchanged/); + expect(lines[1]).toMatch( + new RegExp(`^${TIMING_SCHEMA},action:reuse,reason:inputs_unchanged,check_ms:\\d+,build_ms:0$`), + ); + expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).not.toBeNull(); + }); + + it("rebuilds when the inputs changed, removing the stale stamp before the build starts", () => { + const { repoRoot, packageRoot } = makeRepo(); + recordCurrentBuild({ repoRoot, packageRoot }); + writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n"); + const stampPath = buildInputsStampPath(packageRoot); + const observed = []; + const lines = []; + const env = { npm_execpath: "/corepack/pnpm.cjs" }; + + const status = prepareRunnerVendor({ + repoRoot, + env, + run: (command, options) => { + observed.push({ command, ...options, stampPresent: readBuildInputsStamp(stampPath) !== null }); + return 0; + }, + log: (line) => lines.push(line), + }); + + expect(status).toBe(0); + expect(observed).toEqual([ + { command: RUNNER_VENDOR_BUILD_COMMAND, cwd: path.join(repoRoot, "server"), env, stampPresent: false }, + ]); + expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(inputs changed/); + expect(lines[1]).toMatch( + new RegExp(`^${TIMING_SCHEMA},action:rebuild,reason:inputs_changed,check_ms:\\d+,build_ms:\\d+,status:0$`), + ); + }); + + it("rebuilds when a file the build left under dist is gone, even one no consumer imports", () => { + const { repoRoot, packageRoot } = makeRepo(); + recordCurrentBuild({ repoRoot, packageRoot }); + rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs")); + let runs = 0; + const lines = []; + + const status = prepareRunnerVendor({ + repoRoot, + env: {}, + run: () => { + runs += 1; + return 0; + }, + log: (line) => lines.push(line), + }); + + expect(status).toBe(0); + expect(runs).toBe(1); + expect(lines[0]).toMatch( + /^prepare:runner-vendor: rebuilding the Runner \(build output dist\/cli\/acpx-runtime-sidecar\.cjs is missing since /, + ); + expect(lines[1]).toMatch(/,action:rebuild,reason:outputs_changed,/); + }); + + it("returns the build's failure status and leaves no stamp behind", () => { + const { repoRoot, packageRoot } = makeRepo(); + const lines = []; + + const status = prepareRunnerVendor({ + repoRoot, + env: {}, + run: () => 3, + log: (line) => lines.push(line), + }); + + expect(status).toBe(3); + expect(lines[0]).toMatch(/^prepare:runner-vendor: rebuilding the Runner \(no build stamp/); + expect(lines[1]).toMatch(/,action:rebuild,reason:no_stamp,.*,status:3$/); + expect(readBuildInputsStamp(buildInputsStampPath(packageRoot))).toBeNull(); + }); + + it("rebuilds a current build when PAPERCLIP_RUNNER_VENDOR_REBUILD=1", () => { + const { repoRoot, packageRoot } = makeRepo(); + recordCurrentBuild({ repoRoot, packageRoot }); + let runs = 0; + const lines = []; + + const status = prepareRunnerVendor({ + repoRoot, + env: { [FORCE_REBUILD_VARIABLE]: "1" }, + run: () => { + runs += 1; + return 0; + }, + log: (line) => lines.push(line), + }); + + expect(status).toBe(0); + expect(runs).toBe(1); + expect(lines[1]).toMatch(/,action:rebuild,reason:forced,/); + }); + + it.skipIf(process.platform === "win32")( + "end to end: the first server build rebuilds the Runner, the second reuses it, a change or a lost output rebuilds again", + async () => { + const { repoRoot, packageRoot } = makeRepo(); + // A stand-in pnpm that behaves like the Runner build: it records how it + // was called, then brackets its (empty) step list with the same two + // record-build-inputs.mjs calls the real build makes. + const binDir = path.join(repoRoot, "bin"); + const argvLog = path.join(repoRoot, "pnpm-argv.log"); + mkdirSync(binDir); + const pnpm = path.join(binDir, "pnpm"); + writeFileSync( + pnpm, + `${[ + "#!/bin/sh", + "set -e", + `printf '%s\\n' "$*" >> "${argvLog}"`, + `"${process.execPath}" "${recordScript}" --begin --package-root "${packageRoot}" > /dev/null`, + `"${process.execPath}" "${recordScript}" --finish --package-root "${packageRoot}" > /dev/null`, + ].join("\n")}\n`, + ); + chmodSync(pnpm, 0o755); + const env = { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}` }; + delete env[FORCE_REBUILD_VARIABLE]; + // The suite itself may run under pnpm; the script must then take the + // PATH fallback so the stand-in, not the real pnpm, receives the build. + delete env.npm_execpath; + const run = () => execFileAsync(process.execPath, [prepareScript, "--repo-root", repoRoot], { env }); + const calls = () => readFileSync(argvLog, "utf8").split("\n").filter(Boolean); + + const first = await run(); + expect(calls()).toEqual(["--filter @paperclipai/paperclip-runner build"]); + expect(first.stdout).toMatch(/action:rebuild,reason:no_stamp,/); + expect(assessRunnerBuild({ repoRoot, packageRoot }).current).toBe(true); + + const second = await run(); + expect(calls()).toHaveLength(1); + expect(second.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/); + + writeFileSync(path.join(packageRoot, "src", "index.ts"), "export const one = 2;\n"); + const third = await run(); + expect(calls()).toHaveLength(2); + expect(third.stdout).toMatch(/action:rebuild,reason:inputs_changed,/); + + rmSync(path.join(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs")); + const fourth = await run(); + expect(calls()).toHaveLength(3); + expect(fourth.stdout).toMatch(/action:rebuild,reason:outputs_changed,/); + // The stand-in build does not recreate the helper, so the new stamp's + // manifest no longer lists it and the next run reuses the build. + const fifth = await run(); + expect(calls()).toHaveLength(3); + expect(fifth.stdout).toMatch(/action:reuse,reason:inputs_unchanged,/); + }, + ); +}); diff --git a/server/src/__tests__/server-package-build-script.test.ts b/server/src/__tests__/server-package-build-script.test.ts index 0c97256947..52458efaa0 100644 --- a/server/src/__tests__/server-package-build-script.test.ts +++ b/server/src/__tests__/server-package-build-script.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { describe, expect, it } from "vitest"; +import { RUNNER_VENDOR_BUILD_COMMAND } from "../../scripts/prepare-runner-vendor.mjs"; const packageJsonPath = fileURLToPath( new URL("../../package.json", import.meta.url), @@ -57,14 +58,39 @@ describe("server package build script", () => { expect(packageJson.devDependencies?.["@paperclipai/paperclip-runner"]).toBe( "workspace:*", ); + // prepare-runner-vendor.mjs reuses a Runner build whose recorded inputs + // are unchanged (the `pnpm -r build` case) and otherwise runs the Runner's + // full build, which is what a standalone server build (the Dockerfile's + // path) relies on. Pin both the indirection and the command behind it. expect(packageJson.scripts?.["prepare:runner-vendor"]).toBe( + "node scripts/prepare-runner-vendor.mjs", + ); + expect(RUNNER_VENDOR_BUILD_COMMAND).toBe( "pnpm --filter @paperclipai/paperclip-runner build", ); + expect(packageJson.scripts?.build.startsWith("pnpm run prepare:runner-vendor && ")).toBe(true); expect(packageJson.scripts?.build).toContain( "cp -Rf ../packages/paperclip-runner/dist/. dist/vendor/paperclip-runner/", ); }); + it("brackets the Runner's build with its input recording", () => { + // The stamp prepare-runner-vendor.mjs trusts must only exist for a build + // that completed every step and check, so it has to be written last; and + // it must vouch for the inputs the build read, not for whatever the + // sources are when it ends, so the digest has to be captured before the + // first step reads anything (--begin) and compared again at the end + // (--finish). Nothing else in the chain may touch the stamp. + const runner = JSON.parse(readFileSync(new URL( + "../../../packages/paperclip-runner/package.json", import.meta.url, + ), "utf8")) as { scripts: Record }; + const steps = runner.scripts.build.split(" && "); + expect(steps.at(0)).toBe("node scripts/record-build-inputs.mjs --begin"); + expect(steps.at(-1)).toBe("node scripts/record-build-inputs.mjs --finish"); + expect(steps.filter((step) => step.includes("record-build-inputs"))).toHaveLength(2); + expect(steps.length).toBeGreaterThan(2); + }); + it("replaces an existing read-only image manifest with the build's copy step", () => { const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8")); const step = packageJson.scripts.build.split(" && ").find((command: string) => @@ -104,10 +130,10 @@ describe("server package build script", () => { }); it("compiles the vendored Runner's TypeScript surface once per build", () => { - // prepare:runner-vendor runs the Runner's full build, and `pnpm -r build` - // has already run it once by the time the server builds. Each extra + // prepare:runner-vendor runs the Runner's full build whenever its recorded + // inputs are stale or missing (every standalone server build). Each extra // build:typescript inside that chain (two tsc projects plus the eval-kernel - // rebuild) costs about 7s on the CI Build job's critical path, twice. + // rebuild) costs about 7s on the CI Build job's critical path. // Checks that read dist/ must call their script directly, the way // generate-replay-goldens.mjs already does, not through a `check:*` // wrapper that rebuilds first for standalone use.