From 1d23cb6962f57c2f21a76d3402cec293d1a4dc76 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Wed, 30 Sep 2026 16:04:23 -0700 Subject: [PATCH] ci: pin a checkout-independent Rust cache path so PR lanes hit master's cache (#14394) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip ships a native Runner binary, written in Rust, and seven CI lanes build it on every pull request > - `Canary Dry Run` is the slowest check on every green PR run, and most of its time is `cargo build --release` on third-party crates > - Master saves a Rust dependency cache for these lanes, but every PR lane logs `No cache found` and compiles every crate from zero > - The cache key matches, but GitHub also compares a hash of the absolute cache paths, and the master writer (RunsOn fleet, `/home/runner/_work/...`) and the PR readers (GitHub-hosted, `/home/runner/work/...`) hash different paths > - This pull request gives both sides a checkout-independent workspace path, so the hashes match and the PR lanes restore master's cache > - The benefit is about 2.5 minutes less wall clock per PR run and about 18 fewer runner-minutes per run ## Linked Issues or Issue Description No public issue exists for this problem. The description below follows the enhancement template. Related prior PRs on the same cache: Refs #13194, Refs #13259, Refs #13457, Refs #13459, Refs #13500, Refs #13586. None of them pins the workspace path, so none of them fixes this miss. **What existing behavior does this improve?** The `Swatinem/rust-cache` restore step in the PR workflow lanes that build the Runner: `Canary Dry Run`, `Build`, `Typecheck + Release Registry`, and the four `Verify Paperclip Runner` lanes. **Subsystem affected** CI workflows under `.github/workflows/`, their guard tests under `.github/scripts/tests/`, and `doc/RELEASE-AUTOMATION-SETUP.md`. **Current behavior** Every PR lane logs `No cache found` although master holds an entry with the exact key. Run 36424309181 computed `v0-rust-release-runner-v1-Linux-x64-c3a3ca66-a95b0328`, and master holds a 678 MB entry with that key. GitHub matches a cache entry on the key and on a version hash of the absolute paths in the cache. The master writer runs on the RunsOn fleet, where the checkout is `/home/runner/_work/paperclip/paperclip`. The PR readers run on GitHub-hosted `ubuntu-latest`, where the checkout is `/home/runner/work/paperclip/paperclip`. The stored version `5c40870d…` is the sha256 of the `_work` paths plus `zstd-without-long|1.0`. The `work` paths hash to `1656e9ee…`. The key can never match, so each lane compiles every third-party crate again. **Proposed behavior** The writer and the readers pass the same checkout-independent path to `rust-cache`. Both runner layouts then produce the same version hash, and the PR lanes restore master's cache. **Reason and benefit** `Canary Dry Run` takes 533s on a green run. 251s of that is dependency compilation that a warm cache removes. Seven lanes pay this cost in every PR run. **Breaking changes** None. This change affects CI only. ## What Changed - Add a `Pin the Runner Rust workspace path` step before `rust-cache` in the master writer (`release-verify.yml`, typecheck and runner lanes) and in all four PR readers (`pr-trusted.yml`). The step creates the symlink `$HOME/paperclip-runner-rust` → `$GITHUB_WORKSPACE/packages/paperclip-runner/runner` and passes that path to `rust-cache` as `workspaces: -> target`. `rust-cache` resolves the input with `path.resolve`, which does not follow symlinks, so both runner layouts now produce the same cache paths and the same version hash. `$HOME` is `/home/runner` on both images, which is why the `~/.cargo` paths already agreed. - Bump the shared keys `release-runner-v1` → `release-runner-v2` and `release-typecheck-v1` → `release-typecheck-v2`. The old, unreachable entries are then visibly orphaned instead of sharing a key with the new ones. - Extend the guard tests `pr-runner-rust-cache`, `release-runner-cache`, and `typecheck-rust-cache`. They now require the pin step in both workflows with identical text, placed before the cache step, and they reject a `workspaces:` value that resolves under the checkout. The `pr-runner-rust-cache` test checks all four PR reader jobs and fails if a `rust-cache` step appears in a PR job that is not in its reader list. - Update `doc/RELEASE-AUTOMATION-SETUP.md` to name the `release-runner-v2` key and to explain the pinned workspace path. ### Expected savings once merged Measured from run 36424309181. "Removed" is the dependency-compile time that a warm restore removes, minus about 18s to restore the 680 MB entry. The fleet writer's own restore shows this cost. | Lane | Today | Removed | Expected | |---|---|---|---| | Canary Dry Run | 533s | ~150s | ~380s | | Typecheck + Release Registry | 462s | ~155s | ~305s | | Verify Paperclip Runner (vitest 2/2) | 453s | ~245s | ~210s | | Verify Paperclip Runner (rust) | 400s | ~175s | ~225s | | Build | 348s | ~130s | ~220s | | Verify Paperclip Runner (static checks) | 321s | ~170s | ~150s | | Verify Paperclip Runner (vitest 1/2) | 346s | ~70s | ~275s | - Wall clock per PR run: about 533s → about 385s. That is about 2.5 minutes faster to a green check set. `Canary Dry Run` stays the longest check. The rest is the non-cargo work in `release.sh` (standalone package builds ~30s, publish-payload preview ~73s). - Runner time: about 18 runner-minutes saved per PR run across the seven lanes. - The first master push after merge compiles from zero once in the fleet writer (about 4 extra minutes on that one run) and saves the v2 entry. Later PRs hit it. When a PR changes `Cargo.lock`, the prefix restore key still gives a partial hit, as before. ## Verification - Run the guard tests for the three cache lanes: `node --test .github/scripts/tests/pr-runner-rust-cache.test.mjs .github/scripts/tests/release-runner-cache.test.mjs .github/scripts/tests/typecheck-rust-cache.test.mjs` Result: 21 pass, 0 fail. - Run the full guard suite: `node --test '.github/scripts/tests/*.test.mjs'`. Result: 376 pass, 3 fail. The 3 failures are in `docker-canary-promotion.test.mjs`. They hit a sandbox temp-file ENOENT and fail the same way on the unmodified branch. - Run `node --test scripts/__tests__/release-verify-workflow.test.mjs`. Result: 14 pass. - Local archive test: create a tar from the `_work` layout through the symlink (relative `../../../paperclip-runner-rust/target` entries, `tar -P -C $GITHUB_WORKSPACE`, the same way `@actions/cache` does). Extract it on the `work` layout. The files land in the real target directory and the symlink stays intact. - After merge, open any GitHub-hosted PR run and confirm that the seven Rust lanes log `Restored from cache key ...release-runner-v2...` in place of `No cache found`. ## Risks - Low risk. The change touches CI workflows, their tests, and one doc page. No product code changes. - If the pin step fails, `rust-cache` reports a miss and the lane compiles from zero, as it does today. The build does not break. - Both runner layouts sit four levels under `/home/runner`, so the relative `../../../` archive entries line up. The existing `~/.cargo/registry` and `~/.cargo/git` cache paths already rely on this property. A future runner image with a different `$HOME` depth would miss the cache but would not fail the job. - `rm -rf "$pinned"` acts on the symlink itself (no trailing slash), never on the checkout behind it. It only matters on a reused runner. - Squash-merge note: the branch carries commits by `Bender (Fable)`. Add `Co-Authored-By: Bender (Fable) ` to the squash body to keep that authorship. ## Model Used - Anthropic Claude Fable 5.1 (`claude-fable-5-1`), run through Claude Code inside a Paperclip agent heartbeat. Extended thinking was on. Tool use: shell, GitHub CLI, and the GitHub REST API for workflow logs, cache listings, and PR operations. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change and contains no internal ticket id. The agent execution workspace fixed this branch name, so I cannot rename it. Squash-merge drops the branch name. - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 Co-authored-by: Bender (Fable) --- .../tests/pr-runner-rust-cache.test.mjs | 162 ++++++++++++++---- .../tests/release-runner-cache.test.mjs | 9 +- .../tests/typecheck-rust-cache.test.mjs | 27 ++- .github/workflows/pr-trusted.yml | 106 +++++++++++- .github/workflows/release-verify.yml | 50 +++++- doc/RELEASE-AUTOMATION-SETUP.md | 13 +- doc/cloud-build-readiness.md | 9 +- 7 files changed, 317 insertions(+), 59 deletions(-) diff --git a/.github/scripts/tests/pr-runner-rust-cache.test.mjs b/.github/scripts/tests/pr-runner-rust-cache.test.mjs index 1987072442..6089bcec63 100644 --- a/.github/scripts/tests/pr-runner-rust-cache.test.mjs +++ b/.github/scripts/tests/pr-runner-rust-cache.test.mjs @@ -5,8 +5,41 @@ import { readFileSync } from "node:fs"; const read = (name) => readFileSync(new URL(`../../workflows/${name}`, import.meta.url), "utf8"); const prWorkflow = read("pr-trusted.yml"); const releaseWorkflow = read("release-verify.yml"); -const pr = prWorkflow.split(" verify_paperclip_runner:")[1].split(" build:")[0]; -const release = releaseWorkflow.split(" verify_paperclip_runner:")[1].split(" build:")[0]; + +// Slice one job out of a workflow: from its two-space-indented key to the +// next one. Steps sit at six spaces and `with:` at eight, so only job keys +// match the boundary pattern. +const job = (workflow, name) => { + const start = workflow.indexOf(`\n ${name}:\n`); + assert.ok(start >= 0, `workflow is missing the ${name} job`); + const body = workflow.slice(start + 1); + const header = ` ${name}:\n`.length; + const next = body.slice(header).search(/\n [a-z0-9_-]+:\n/); + return next === -1 ? body : body.slice(0, header + next + 1); +}; + +const READ_STEP = " - name: Restore Runner Rust dependencies (read only)"; +const WRITE_STEP = " - name: Cache Runner Rust dependencies"; +const SELECT_STEP = " - name: Select the pinned Runner Rust toolchain"; + +// Every PR job that builds the Runner restores master's `release-runner-v2` +// entry. Each one has to agree with the writer on every key input, or that +// one lane misses and silently recompiles every third-party crate while the +// others hit. +const READER_JOBS = ["typecheck_release_registry", "verify_paperclip_runner", "build", "canary_dry_run"]; +const readers = READER_JOBS.map((name) => [name, job(prWorkflow, name)]); +const release = job(releaseWorkflow, "verify_paperclip_runner"); + +test("every rust-cache restore in the PR workflow belongs to a guarded reader job", () => { + const total = prWorkflow.match(/uses: Swatinem\/rust-cache@/g)?.length ?? 0; + const guarded = readers.filter(([, body]) => /uses: Swatinem\/rust-cache@/.test(body)).length; + assert.equal(guarded, READER_JOBS.length, "each listed reader job must restore the Rust cache"); + assert.equal(total, guarded, "a job restores the Rust cache but is not in READER_JOBS; add it so it is guarded"); + for (const [name, body] of readers) { + assert.equal(body.match(/uses: Swatinem\/rust-cache@/g).length, 1, `${name}: exactly one rust-cache step`); + assert.ok(body.includes(READ_STEP), `${name}: the rust-cache step must be the read-only restore`); + } +}); // The key is computed from these inputs. A pull request that disagrees with // the master writer on any of them misses every time and silently recompiles @@ -14,43 +47,51 @@ const release = releaseWorkflow.split(" verify_paperclip_runner:")[1].split(" // restore exists to remove. const keyInputs = [ /uses: Swatinem\/rust-cache@([0-9a-f]{40}) # v[0-9.]+/, - /workspaces: (packages\/paperclip-runner\/runner -> target)/, - /shared-key: (release-runner-v1)/, + /workspaces: (\$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target)/, + /shared-key: (release-runner-v2)/, /cache-workspace-crates: (false)/, /cache-bin: (false)/, ]; -test("the PR lane restores the Rust cache under the same key the master push writes", () => { - for (const pattern of keyInputs) { - const mine = pr.match(pattern); - const theirs = release.match(pattern); - assert.ok(mine, `PR lane is missing ${pattern}`); - assert.ok(theirs, `master writer is missing ${pattern}`); - assert.equal(mine[1], theirs[1], `key input drifted from the master writer: ${pattern}`); +test("every PR reader restores the Rust cache under the same key the master push writes", () => { + for (const [name, pr] of readers) { + for (const pattern of keyInputs) { + const mine = pr.match(pattern); + const theirs = release.match(pattern); + assert.ok(mine, `${name}: PR lane is missing ${pattern}`); + assert.ok(theirs, `master writer is missing ${pattern}`); + assert.equal(mine[1], theirs[1], `${name}: key input drifted from the master writer: ${pattern}`); + } + assert.doesNotMatch(pr, /prefix-key:|cache-on-failure: true|cache-all-crates: true/, name); } - assert.doesNotMatch(pr, /prefix-key:|cache-on-failure: true|cache-all-crates: true/); }); -test("the PR lane pins the compiler before the key is computed", () => { - const select = pr.indexOf(" - name: Select the pinned Runner Rust toolchain"); - const cache = pr.indexOf(" - name: Restore Runner Rust dependencies (read only)"); - const verify = pr.indexOf(" - name: Verify Paperclip Runner\n"); - assert.ok(select >= 0 && cache > select && verify > cache); - const setup = pr.slice(select, cache); - assert.match(setup, /working-directory: packages\/paperclip-runner/); - assert.match(setup, /rustup show active-toolchain/); - assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/); - // The gate routes to either ubuntu-latest or the public PR fleet, so a - // missing rustup must cost the cache, never the pull request. - assert.match(setup, /command -v rustup/); - assert.doesNotMatch(setup, /set -euo pipefail/); +test("every PR reader pins the compiler before the key is computed", () => { + for (const [name, pr] of readers) { + const select = pr.indexOf(SELECT_STEP); + const cache = pr.indexOf(READ_STEP); + assert.ok(select >= 0 && cache > select, `${name}: the toolchain must be selected before the cache step`); + const setup = pr.slice(select, cache); + // Nothing may sit between the pin and the restore that could change the key. + assert.equal(setup.match(/^ - name: /gm).length, 1, `${name}: no step between the toolchain pin and the restore`); + assert.match(setup, /working-directory: packages\/paperclip-runner/, name); + assert.match(setup, /rustup show active-toolchain/, name); + assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/, name); + // The gate routes to either ubuntu-latest or the public PR fleet, so a + // missing rustup must cost the cache, never the pull request. + assert.match(setup, /command -v rustup/, name); + assert.doesNotMatch(setup, /set -euo pipefail/, name); + } }); test("a pull request never writes to or evicts the master cache entry", () => { - const step = pr.split(" - name: Restore Runner Rust dependencies (read only)")[1] - .split(" - name: Verify Paperclip Runner\n")[0]; - assert.equal(step.match(/^\s*save-if: (.+)$/m)?.[1], "false"); - assert.doesNotMatch(step, /^\s*if:/m, "the restore must not be conditional; a miss is already free"); + for (const [name, pr] of readers) { + const after = pr.split(READ_STEP)[1]; + const nextStep = after.search(/\n - name: /); + const step = nextStep === -1 ? after : after.slice(0, nextStep); + assert.equal(step.match(/^\s*save-if: (.+)$/m)?.[1], "false", `${name}: the restore must set save-if: false`); + assert.doesNotMatch(step, /^\s*if:/m, `${name}: the restore must not be conditional; a miss is already free`); + } assert.doesNotMatch(prWorkflow, /uses: Swatinem\/rust-cache@[0-9a-f]{40}[\s\S]*?save-if: (?!false)/); }); @@ -62,28 +103,75 @@ test("a pull request never writes to or evicts the master cache entry", () => { // the key matches nothing and every run recompiles. const NORMALIZE = /# rust-cache hashes every installed toolchain[\s\S]*?rustup toolchain list\n/; -test("reader and writer strip extra toolchains identically before the key is computed", () => { - const mine = pr.match(NORMALIZE); +test("every reader and the writer strip extra toolchains identically before the key is computed", () => { const theirs = release.match(NORMALIZE); - assert.ok(mine, "pr-trusted.yml must normalize the installed toolchains"); assert.ok(theirs, "release-verify.yml must normalize the installed toolchains"); - assert.equal(mine[0], theirs[0], "the normalization must be identical in both workflows"); - - for (const [name, body] of [["reader", mine[0]], ["writer", theirs[0]]]) { + for (const [name, body] of [["writer", theirs[0]]]) { // Keep the pin, drop the rest, and never fail the job over it. assert.match(body, /grep -vx "\$toolchain"/, name); assert.match(body, /xargs -n1 rustup toolchain uninstall/, name); assert.match(body, /\|\| true/, name); } + for (const [name, pr] of readers) { + const mine = pr.match(NORMALIZE); + assert.ok(mine, `${name}: pr-trusted.yml must normalize the installed toolchains`); + assert.equal(mine[0], theirs[0], `${name}: the normalization must be identical to the writer's`); + } }); test("the toolchain is stripped before the cache step, not after", () => { for (const [name, body, cacheStep] of [ - ["reader", pr, " - name: Restore Runner Rust dependencies (read only)"], - ["writer", release, " - name: Cache Runner Rust dependencies"], + ...readers.map(([name, body]) => [name, body, READ_STEP]), + ["writer", release, WRITE_STEP], ]) { const normalize = body.search(NORMALIZE); const cache = body.indexOf(cacheStep); assert.ok(normalize >= 0 && cache > normalize, `${name}: normalization must precede the cache step`); } }); + +// GitHub matches a cache entry on its key and on a version hash of the +// absolute paths being cached. rust-cache resolves the target directory under +// the checkout, and the checkout root differs by runner (/home/runner/_work on +// the RunsOn fleets that write the cache, /home/runner/work on GitHub-hosted +// runners). With every key input aligned, every GitHub-hosted pull request +// still logged "No cache found" (run 36424309181, 2026-09-28). Both workflows +// therefore hand rust-cache the same checkout-independent path, and they have +// to build it the same way or the version matches nothing. +const PIN = /# rust-cache hashes its absolute cache paths[\s\S]*?echo "path=\$pinned" >> "\$GITHUB_OUTPUT"\n/; + +test("every reader and the writer pin an identical checkout-independent Rust workspace path", () => { + const theirs = release.match(PIN); + assert.ok(theirs, "release-verify.yml must pin the Runner Rust workspace path"); + const pins = [["writer", theirs[0]]]; + for (const [name, pr] of readers) { + const mine = pr.match(PIN); + assert.ok(mine, `${name}: pr-trusted.yml must pin the Runner Rust workspace path`); + assert.equal(mine[0], theirs[0], `${name}: the pinned path must be built identically to the writer's`); + pins.push([name, mine[0]]); + } + + for (const [name, body] of pins) { + assert.match(body, /- name: Pin the Runner Rust workspace path\n\s+id: runner_rust_workspace\n/, name); + // Anchor under $HOME, which both runner images share, never under the checkout. + assert.match(body, /pinned="\$HOME\/[A-Za-z0-9._-]+"/, name); + assert.match(body, /rm -rf "\$pinned"\n\s+ln -s "\$GITHUB_WORKSPACE\/packages\/paperclip-runner\/runner" "\$pinned"/, name); + assert.match(body, /echo "path=\$pinned" >> "\$GITHUB_OUTPUT"/, name); + } +}); + +test("the workspace path is pinned before the cache step and never names the checkout", () => { + for (const [name, body, cacheStep] of [ + ...readers.map(([name, body]) => [name, body, READ_STEP]), + ["writer", release, WRITE_STEP], + ]) { + const pin = body.search(PIN); + const cache = body.indexOf(cacheStep); + assert.ok(pin >= 0 && cache > pin, `${name}: the pin must precede the cache step`); + assert.doesNotMatch(body, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/, `${name}: workspaces must not resolve under the checkout`); + } + // No rust-cache step anywhere in either workflow may point back into the checkout. + for (const [name, workflow] of [["pr-trusted.yml", prWorkflow], ["release-verify.yml", releaseWorkflow]]) { + assert.doesNotMatch(workflow, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/, `${name}: workspaces must not resolve under the checkout`); + } +}); diff --git a/.github/scripts/tests/release-runner-cache.test.mjs b/.github/scripts/tests/release-runner-cache.test.mjs index 7e35ec09c9..96bba1f325 100644 --- a/.github/scripts/tests/release-runner-cache.test.mjs +++ b/.github/scripts/tests/release-runner-cache.test.mjs @@ -15,8 +15,13 @@ test("Runner dependency caching selects the package's pinned compiler before com assert.match(setup, /rustup show active-toolchain/); assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/); assert.match(runner, /uses: Swatinem\/rust-cache@[0-9a-f]{40} # v[0-9.]+/); - assert.match(runner, /workspaces: packages\/paperclip-runner\/runner -> target/); - assert.match(runner, /shared-key: release-runner-v1/); + // The target path feeds the entry's version hash, so it must not resolve + // under the checkout, whose root differs between the fleet and GitHub-hosted + // runners. The pin step publishes a $HOME-anchored path to the same directory. + const pin = runner.indexOf(" - name: Pin the Runner Rust workspace path"); + assert.ok(pin >= 0 && cache > pin, "the workspace path must be pinned before the cache step"); + assert.match(runner, /workspaces: \$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target/); + assert.match(runner, /shared-key: release-runner-v2/); }); test("the shared cache excludes workspace artifacts and only restores or saves the exact master-push source", () => { diff --git a/.github/scripts/tests/typecheck-rust-cache.test.mjs b/.github/scripts/tests/typecheck-rust-cache.test.mjs index c8ff43b880..6f975cc77a 100644 --- a/.github/scripts/tests/typecheck-rust-cache.test.mjs +++ b/.github/scripts/tests/typecheck-rust-cache.test.mjs @@ -28,11 +28,32 @@ for (const [name, overrides, ref, allowed] of [ } test("cache excludes workspace code and executable installs, and preserves full checks", () => { assert.match(cache, /uses: Swatinem\/rust-cache@[a-f0-9]{40}/); - assert.match(cache, /workspaces: packages\/paperclip-runner\/runner -> target/); - assert.match(cache, /shared-key: release-typecheck-v1/); + // The target path feeds the entry's version hash; a checkout-relative path + // hashes differently on the fleet (/home/runner/_work) and on GitHub-hosted + // runners (/home/runner/work), so the pin step publishes a $HOME-anchored one. + assert.match(cache, /workspaces: \$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target/); + assert.match(cache, /shared-key: release-typecheck-v2/); assert.match(cache, /cache-workspace-crates: false/); assert.match(cache, /cache-bin: false/); + const pin = typecheck.indexOf(" - name: Pin the Runner Rust workspace path"); + assert.ok(pin >= 0 && pin < typecheck.indexOf("uses: Swatinem/rust-cache")); assert.ok(typecheck.indexOf('echo "RUSTUP_TOOLCHAIN=$toolchain"') < typecheck.indexOf("uses: Swatinem/rust-cache")); + assert.doesNotMatch(typecheck, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/); +}); +// The typecheck key is written and read only by master pushes on the fleet, so +// its version would still match if this pin drifted. Hold it to the same text +// as the Runner writer anyway: `doc/RELEASE-AUTOMATION-SETUP.md` promises one +// identical pin step before every Rust cache step, and a divergent copy here +// is the first place a later edit would quietly break that promise. +test("the typecheck pin step is identical to the Runner writer's", () => { + const PIN = /# rust-cache hashes its absolute cache paths[\s\S]*?echo "path=\$pinned" >> "\$GITHUB_OUTPUT"\n/; + const writer = workflow.split("\n verify_paperclip_runner:\n")[1]; + assert.ok(writer, "release-verify.yml is missing the verify_paperclip_runner job"); + const theirs = writer.match(PIN); + const mine = typecheck.match(PIN); + assert.ok(theirs, "the Runner writer must pin the Runner Rust workspace path"); + assert.ok(mine, "the typecheck job must pin the Runner Rust workspace path"); + assert.equal(mine[0], theirs[0], "the typecheck pin step drifted from the Runner writer's"); assert.match(typecheck, /run: pnpm -r typecheck/); - assert.match(workflow, /shared-key: release-runner-v1/); + assert.match(workflow, /shared-key: release-runner-v2/); }); diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 7ca7bcac67..b87e5115b9 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -407,6 +407,28 @@ jobs: # rebuilds the Runner release binary; without the shared Rust cache that # is a ~3m40s cold compile of all third-party crates (run 35036001734, # 2026-09-15). Same restore-only contract as Verify Paperclip Runner. + + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -445,8 +467,8 @@ jobs: - name: Restore Runner Rust dependencies (read only) uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-runner-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-runner-v2 # Mirror the master writer: these also feed the cache key. cache-workspace-crates: false cache-bin: false @@ -766,11 +788,33 @@ jobs: # Same restore-only contract as the pnpm store above, for the Rust # dependency tree: master's post-merge verification is the sole writer - # of release-runner-v1, and PR merge refs must not save branch-scoped + # of release-runner-v2, and PR merge refs must not save branch-scoped # copies of a ~680MB target directory. Every key input below has to # match that writer in release-verify.yml exactly or each PR misses and # recompiles all 313 third-party crates in both profiles. A miss is a # slow run, never a wrong one. + + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -809,8 +853,8 @@ jobs: - name: Restore Runner Rust dependencies (read only) uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-runner-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-runner-v2 # Mirror the master writer: these also feed the cache key. cache-workspace-crates: false cache-bin: false @@ -884,6 +928,28 @@ jobs: # shared Rust cache that is a ~3m40s cold compile of all third-party # crates (run 35036001734, 2026-09-15). Same restore-only contract as # Verify Paperclip Runner. + + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -922,8 +988,8 @@ jobs: - name: Restore Runner Rust dependencies (read only) uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-runner-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-runner-v2 # Mirror the master writer: these also feed the cache key. cache-workspace-crates: false cache-bin: false @@ -1094,6 +1160,28 @@ jobs: # build:binary step; without the shared Rust cache that is a ~3m40s cold # compile of all third-party crates (run 35036001734, 2026-09-15). Same # restore-only contract as Verify Paperclip Runner. + + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -1132,8 +1220,8 @@ jobs: - name: Restore Runner Rust dependencies (read only) uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-runner-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-runner-v2 # Mirror the master writer: these also feed the cache key. cache-workspace-crates: false cache-bin: false diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 5144e6b52a..76e5cd28bb 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -42,6 +42,27 @@ jobs: node-version: 24 cache: pnpm + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -56,8 +77,8 @@ jobs: if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-typecheck-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-typecheck-v2 # Rebuild workspace code and rerun every check. Cache only compiled # dependencies; never restore installed executables from cargo/bin. cache-workspace-crates: false @@ -294,6 +315,27 @@ jobs: node-version: 24 cache: pnpm + # rust-cache hashes its absolute cache paths into the entry's version, + # and GitHub only serves an entry whose key and version both match. The + # target directory sits under the checkout, and the checkout root + # differs by runner: /home/runner/_work on the RunsOn fleets that write + # this cache against /home/runner/work on GitHub-hosted runners. Every + # key input agreed, yet all 25 completed pull requests on 2026-09-28 + # logged "No cache found" (run 36424309181) and cold-compiled every + # crate for ~4 minutes in four lanes. Point rust-cache at the same + # directory through a checkout-independent path so both layouts hash + # the same version. Keep this block identical in both workflows: the + # reader and the writer must agree or the version matches nothing. + - name: Pin the Runner Rust workspace path + id: runner_rust_workspace + run: | + set -euo pipefail + pinned="$HOME/paperclip-runner-rust" + # A symlink here is removed as a link, never followed into the checkout. + rm -rf "$pinned" + ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned" + echo "path=$pinned" >> "$GITHUB_OUTPUT" + - name: Select the pinned Runner Rust toolchain working-directory: packages/paperclip-runner run: | @@ -325,8 +367,8 @@ jobs: if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: - workspaces: packages/paperclip-runner/runner -> target - shared-key: release-runner-v1 + workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target + shared-key: release-runner-v2 # Rebuild workspace code and rerun every check. Cache only compiled # dependencies; never restore installed executables from cargo/bin. cache-workspace-crates: false diff --git a/doc/RELEASE-AUTOMATION-SETUP.md b/doc/RELEASE-AUTOMATION-SETUP.md index 5e5afc481d..25890751a4 100644 --- a/doc/RELEASE-AUTOMATION-SETUP.md +++ b/doc/RELEASE-AUTOMATION-SETUP.md @@ -347,11 +347,22 @@ readiness can succeed. A failed lane does not cancel the other lane. Both lanes restore Cargo dependencies with the pinned Rust Cache action. The compiler comes from the Runner package's `rust-toolchain.toml` before the action computes its key. Compiler and Cargo metadata changes select a new cache. The -existing `release-runner-v1` shared key avoids separate copies for these lanes. +`release-runner-v2` shared key avoids separate copies for these lanes. Only the Rust lane saves this cache. After verification it also runs `build:rust` to warm the debug dependencies used by the protocol lane; its own tests already warm release dependencies. The cache writer is shorter than the protocol lane. +GitHub matches a cache entry on its key and on a version hash of the absolute +paths in the entry. The master writer runs on the RunsOn fleet, where the +checkout is `/home/runner/_work/paperclip/paperclip`. The trusted PR workflow +restores the same entry read-only on GitHub-hosted runners, where the checkout +is `/home/runner/work/paperclip/paperclip`. A `Pin the Runner Rust workspace +path` step in both workflows links `$HOME/paperclip-runner-rust` to the Runner +crate and passes that path to the cache action, so both layouts hash the same +paths and the PR lanes can restore master's entry. The guard tests under +`.github/scripts/tests/` require this step, with identical text, before every +Rust cache step in both workflows. + Workspace crates and installed Cargo binaries are excluded. Every run rebuilds workspace code and runs all assigned checks, including on a cache hit. Only an own-repository master-push run verifying that push's exact SHA can restore the diff --git a/doc/cloud-build-readiness.md b/doc/cloud-build-readiness.md index 2a08902fd3..df143f11ec 100644 --- a/doc/cloud-build-readiness.md +++ b/doc/cloud-build-readiness.md @@ -158,10 +158,13 @@ shared infrastructure ownership separately before removing those resources. Source verification's typecheck job builds the native Runner binary through the server's `prepare:runner-vendor` command. It restores and saves compiled Rust dependencies only for canonical master pushes that verify the event's exact SHA. -The `release-typecheck-v1` cache is separate from Runner verification because +The `release-typecheck-v2` cache is separate from Runner verification because those jobs compile different profiles. The pinned toolchain is selected before -cache lookup. Workspace crates and installed cargo binaries are excluded, and -all typechecks still execute. A missing or invalidated cache triggers compilation. +cache lookup, and the cache action receives the Runner crate through the same +checkout-independent `$HOME/paperclip-runner-rust` path as the Runner lanes +(see `RELEASE-AUTOMATION-SETUP.md`). Workspace crates and installed cargo +binaries are excluded, and all typechecks still execute. A missing or +invalidated cache triggers compilation. ### pnpm dependency store cache