mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:24:09 +02:00
ci: pin a checkout-independent Rust cache path so PR lanes hit master's cache (#14394)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip ships a native Runner binary, written in Rust, and seven CI lanes build it on every pull request > - `Canary Dry Run` is the slowest check on every green PR run, and most of its time is `cargo build --release` on third-party crates > - Master saves a Rust dependency cache for these lanes, but every PR lane logs `No cache found` and compiles every crate from zero > - The cache key matches, but GitHub also compares a hash of the absolute cache paths, and the master writer (RunsOn fleet, `/home/runner/_work/...`) and the PR readers (GitHub-hosted, `/home/runner/work/...`) hash different paths > - This pull request gives both sides a checkout-independent workspace path, so the hashes match and the PR lanes restore master's cache > - The benefit is about 2.5 minutes less wall clock per PR run and about 18 fewer runner-minutes per run ## Linked Issues or Issue Description No public issue exists for this problem. The description below follows the enhancement template. Related prior PRs on the same cache: Refs #13194, Refs #13259, Refs #13457, Refs #13459, Refs #13500, Refs #13586. None of them pins the workspace path, so none of them fixes this miss. **What existing behavior does this improve?** The `Swatinem/rust-cache` restore step in the PR workflow lanes that build the Runner: `Canary Dry Run`, `Build`, `Typecheck + Release Registry`, and the four `Verify Paperclip Runner` lanes. **Subsystem affected** CI workflows under `.github/workflows/`, their guard tests under `.github/scripts/tests/`, and `doc/RELEASE-AUTOMATION-SETUP.md`. **Current behavior** Every PR lane logs `No cache found` although master holds an entry with the exact key. Run 36424309181 computed `v0-rust-release-runner-v1-Linux-x64-c3a3ca66-a95b0328`, and master holds a 678 MB entry with that key. GitHub matches a cache entry on the key and on a version hash of the absolute paths in the cache. The master writer runs on the RunsOn fleet, where the checkout is `/home/runner/_work/paperclip/paperclip`. The PR readers run on GitHub-hosted `ubuntu-latest`, where the checkout is `/home/runner/work/paperclip/paperclip`. The stored version `5c40870d…` is the sha256 of the `_work` paths plus `zstd-without-long|1.0`. The `work` paths hash to `1656e9ee…`. The key can never match, so each lane compiles every third-party crate again. **Proposed behavior** The writer and the readers pass the same checkout-independent path to `rust-cache`. Both runner layouts then produce the same version hash, and the PR lanes restore master's cache. **Reason and benefit** `Canary Dry Run` takes 533s on a green run. 251s of that is dependency compilation that a warm cache removes. Seven lanes pay this cost in every PR run. **Breaking changes** None. This change affects CI only. ## What Changed - Add a `Pin the Runner Rust workspace path` step before `rust-cache` in the master writer (`release-verify.yml`, typecheck and runner lanes) and in all four PR readers (`pr-trusted.yml`). The step creates the symlink `$HOME/paperclip-runner-rust` → `$GITHUB_WORKSPACE/packages/paperclip-runner/runner` and passes that path to `rust-cache` as `workspaces: <path> -> target`. `rust-cache` resolves the input with `path.resolve`, which does not follow symlinks, so both runner layouts now produce the same cache paths and the same version hash. `$HOME` is `/home/runner` on both images, which is why the `~/.cargo` paths already agreed. - Bump the shared keys `release-runner-v1` → `release-runner-v2` and `release-typecheck-v1` → `release-typecheck-v2`. The old, unreachable entries are then visibly orphaned instead of sharing a key with the new ones. - Extend the guard tests `pr-runner-rust-cache`, `release-runner-cache`, and `typecheck-rust-cache`. They now require the pin step in both workflows with identical text, placed before the cache step, and they reject a `workspaces:` value that resolves under the checkout. The `pr-runner-rust-cache` test checks all four PR reader jobs and fails if a `rust-cache` step appears in a PR job that is not in its reader list. - Update `doc/RELEASE-AUTOMATION-SETUP.md` to name the `release-runner-v2` key and to explain the pinned workspace path. ### Expected savings once merged Measured from run 36424309181. "Removed" is the dependency-compile time that a warm restore removes, minus about 18s to restore the 680 MB entry. The fleet writer's own restore shows this cost. | Lane | Today | Removed | Expected | |---|---|---|---| | Canary Dry Run | 533s | ~150s | ~380s | | Typecheck + Release Registry | 462s | ~155s | ~305s | | Verify Paperclip Runner (vitest 2/2) | 453s | ~245s | ~210s | | Verify Paperclip Runner (rust) | 400s | ~175s | ~225s | | Build | 348s | ~130s | ~220s | | Verify Paperclip Runner (static checks) | 321s | ~170s | ~150s | | Verify Paperclip Runner (vitest 1/2) | 346s | ~70s | ~275s | - Wall clock per PR run: about 533s → about 385s. That is about 2.5 minutes faster to a green check set. `Canary Dry Run` stays the longest check. The rest is the non-cargo work in `release.sh` (standalone package builds ~30s, publish-payload preview ~73s). - Runner time: about 18 runner-minutes saved per PR run across the seven lanes. - The first master push after merge compiles from zero once in the fleet writer (about 4 extra minutes on that one run) and saves the v2 entry. Later PRs hit it. When a PR changes `Cargo.lock`, the prefix restore key still gives a partial hit, as before. ## Verification - Run the guard tests for the three cache lanes: `node --test .github/scripts/tests/pr-runner-rust-cache.test.mjs .github/scripts/tests/release-runner-cache.test.mjs .github/scripts/tests/typecheck-rust-cache.test.mjs` Result: 21 pass, 0 fail. - Run the full guard suite: `node --test '.github/scripts/tests/*.test.mjs'`. Result: 376 pass, 3 fail. The 3 failures are in `docker-canary-promotion.test.mjs`. They hit a sandbox temp-file ENOENT and fail the same way on the unmodified branch. - Run `node --test scripts/__tests__/release-verify-workflow.test.mjs`. Result: 14 pass. - Local archive test: create a tar from the `_work` layout through the symlink (relative `../../../paperclip-runner-rust/target` entries, `tar -P -C $GITHUB_WORKSPACE`, the same way `@actions/cache` does). Extract it on the `work` layout. The files land in the real target directory and the symlink stays intact. - After merge, open any GitHub-hosted PR run and confirm that the seven Rust lanes log `Restored from cache key ...release-runner-v2...` in place of `No cache found`. ## Risks - Low risk. The change touches CI workflows, their tests, and one doc page. No product code changes. - If the pin step fails, `rust-cache` reports a miss and the lane compiles from zero, as it does today. The build does not break. - Both runner layouts sit four levels under `/home/runner`, so the relative `../../../` archive entries line up. The existing `~/.cargo/registry` and `~/.cargo/git` cache paths already rely on this property. A future runner image with a different `$HOME` depth would miss the cache but would not fail the job. - `rm -rf "$pinned"` acts on the symlink itself (no trailing slash), never on the checkout behind it. It only matters on a reused runner. - Squash-merge note: the branch carries commits by `Bender (Fable)`. Add `Co-Authored-By: Bender (Fable) <bender-fable@paperclip.local>` to the squash body to keep that authorship. ## Model Used - Anthropic Claude Fable 5.1 (`claude-fable-5-1`), run through Claude Code inside a Paperclip agent heartbeat. Extended thinking was on. Tool use: shell, GitHub CLI, and the GitHub REST API for workflow logs, cache listings, and PR operations. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change and contains no internal ticket id. The agent execution workspace fixed this branch name, so I cannot rename it. Squash-merge drops the branch name. - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Bender (Fable) <bender-fable@paperclip.local>
This commit is contained in:
7 files changed
+317
-59
No files matched your search
@@ -5,8 +5,41 @@ import { readFileSync } from "node:fs";
|
||||
const read = (name) => readFileSync(new URL(`../../workflows/${name}`, import.meta.url), "utf8");
|
||||
const prWorkflow = read("pr-trusted.yml");
|
||||
const releaseWorkflow = read("release-verify.yml");
|
||||
const pr = prWorkflow.split(" verify_paperclip_runner:")[1].split(" build:")[0];
|
||||
const release = releaseWorkflow.split(" verify_paperclip_runner:")[1].split(" build:")[0];
|
||||
|
||||
// Slice one job out of a workflow: from its two-space-indented key to the
|
||||
// next one. Steps sit at six spaces and `with:` at eight, so only job keys
|
||||
// match the boundary pattern.
|
||||
const job = (workflow, name) => {
|
||||
const start = workflow.indexOf(`\n ${name}:\n`);
|
||||
assert.ok(start >= 0, `workflow is missing the ${name} job`);
|
||||
const body = workflow.slice(start + 1);
|
||||
const header = ` ${name}:\n`.length;
|
||||
const next = body.slice(header).search(/\n [a-z0-9_-]+:\n/);
|
||||
return next === -1 ? body : body.slice(0, header + next + 1);
|
||||
};
|
||||
|
||||
const READ_STEP = " - name: Restore Runner Rust dependencies (read only)";
|
||||
const WRITE_STEP = " - name: Cache Runner Rust dependencies";
|
||||
const SELECT_STEP = " - name: Select the pinned Runner Rust toolchain";
|
||||
|
||||
// Every PR job that builds the Runner restores master's `release-runner-v2`
|
||||
// entry. Each one has to agree with the writer on every key input, or that
|
||||
// one lane misses and silently recompiles every third-party crate while the
|
||||
// others hit.
|
||||
const READER_JOBS = ["typecheck_release_registry", "verify_paperclip_runner", "build", "canary_dry_run"];
|
||||
const readers = READER_JOBS.map((name) => [name, job(prWorkflow, name)]);
|
||||
const release = job(releaseWorkflow, "verify_paperclip_runner");
|
||||
|
||||
test("every rust-cache restore in the PR workflow belongs to a guarded reader job", () => {
|
||||
const total = prWorkflow.match(/uses: Swatinem\/rust-cache@/g)?.length ?? 0;
|
||||
const guarded = readers.filter(([, body]) => /uses: Swatinem\/rust-cache@/.test(body)).length;
|
||||
assert.equal(guarded, READER_JOBS.length, "each listed reader job must restore the Rust cache");
|
||||
assert.equal(total, guarded, "a job restores the Rust cache but is not in READER_JOBS; add it so it is guarded");
|
||||
for (const [name, body] of readers) {
|
||||
assert.equal(body.match(/uses: Swatinem\/rust-cache@/g).length, 1, `${name}: exactly one rust-cache step`);
|
||||
assert.ok(body.includes(READ_STEP), `${name}: the rust-cache step must be the read-only restore`);
|
||||
}
|
||||
});
|
||||
|
||||
// The key is computed from these inputs. A pull request that disagrees with
|
||||
// the master writer on any of them misses every time and silently recompiles
|
||||
@@ -14,43 +47,51 @@ const release = releaseWorkflow.split(" verify_paperclip_runner:")[1].split("
|
||||
// restore exists to remove.
|
||||
const keyInputs = [
|
||||
/uses: Swatinem\/rust-cache@([0-9a-f]{40}) # v[0-9.]+/,
|
||||
/workspaces: (packages\/paperclip-runner\/runner -> target)/,
|
||||
/shared-key: (release-runner-v1)/,
|
||||
/workspaces: (\$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target)/,
|
||||
/shared-key: (release-runner-v2)/,
|
||||
/cache-workspace-crates: (false)/,
|
||||
/cache-bin: (false)/,
|
||||
];
|
||||
|
||||
test("the PR lane restores the Rust cache under the same key the master push writes", () => {
|
||||
for (const pattern of keyInputs) {
|
||||
const mine = pr.match(pattern);
|
||||
const theirs = release.match(pattern);
|
||||
assert.ok(mine, `PR lane is missing ${pattern}`);
|
||||
assert.ok(theirs, `master writer is missing ${pattern}`);
|
||||
assert.equal(mine[1], theirs[1], `key input drifted from the master writer: ${pattern}`);
|
||||
test("every PR reader restores the Rust cache under the same key the master push writes", () => {
|
||||
for (const [name, pr] of readers) {
|
||||
for (const pattern of keyInputs) {
|
||||
const mine = pr.match(pattern);
|
||||
const theirs = release.match(pattern);
|
||||
assert.ok(mine, `${name}: PR lane is missing ${pattern}`);
|
||||
assert.ok(theirs, `master writer is missing ${pattern}`);
|
||||
assert.equal(mine[1], theirs[1], `${name}: key input drifted from the master writer: ${pattern}`);
|
||||
}
|
||||
assert.doesNotMatch(pr, /prefix-key:|cache-on-failure: true|cache-all-crates: true/, name);
|
||||
}
|
||||
assert.doesNotMatch(pr, /prefix-key:|cache-on-failure: true|cache-all-crates: true/);
|
||||
});
|
||||
|
||||
test("the PR lane pins the compiler before the key is computed", () => {
|
||||
const select = pr.indexOf(" - name: Select the pinned Runner Rust toolchain");
|
||||
const cache = pr.indexOf(" - name: Restore Runner Rust dependencies (read only)");
|
||||
const verify = pr.indexOf(" - name: Verify Paperclip Runner\n");
|
||||
assert.ok(select >= 0 && cache > select && verify > cache);
|
||||
const setup = pr.slice(select, cache);
|
||||
assert.match(setup, /working-directory: packages\/paperclip-runner/);
|
||||
assert.match(setup, /rustup show active-toolchain/);
|
||||
assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/);
|
||||
// The gate routes to either ubuntu-latest or the public PR fleet, so a
|
||||
// missing rustup must cost the cache, never the pull request.
|
||||
assert.match(setup, /command -v rustup/);
|
||||
assert.doesNotMatch(setup, /set -euo pipefail/);
|
||||
test("every PR reader pins the compiler before the key is computed", () => {
|
||||
for (const [name, pr] of readers) {
|
||||
const select = pr.indexOf(SELECT_STEP);
|
||||
const cache = pr.indexOf(READ_STEP);
|
||||
assert.ok(select >= 0 && cache > select, `${name}: the toolchain must be selected before the cache step`);
|
||||
const setup = pr.slice(select, cache);
|
||||
// Nothing may sit between the pin and the restore that could change the key.
|
||||
assert.equal(setup.match(/^ - name: /gm).length, 1, `${name}: no step between the toolchain pin and the restore`);
|
||||
assert.match(setup, /working-directory: packages\/paperclip-runner/, name);
|
||||
assert.match(setup, /rustup show active-toolchain/, name);
|
||||
assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/, name);
|
||||
// The gate routes to either ubuntu-latest or the public PR fleet, so a
|
||||
// missing rustup must cost the cache, never the pull request.
|
||||
assert.match(setup, /command -v rustup/, name);
|
||||
assert.doesNotMatch(setup, /set -euo pipefail/, name);
|
||||
}
|
||||
});
|
||||
|
||||
test("a pull request never writes to or evicts the master cache entry", () => {
|
||||
const step = pr.split(" - name: Restore Runner Rust dependencies (read only)")[1]
|
||||
.split(" - name: Verify Paperclip Runner\n")[0];
|
||||
assert.equal(step.match(/^\s*save-if: (.+)$/m)?.[1], "false");
|
||||
assert.doesNotMatch(step, /^\s*if:/m, "the restore must not be conditional; a miss is already free");
|
||||
for (const [name, pr] of readers) {
|
||||
const after = pr.split(READ_STEP)[1];
|
||||
const nextStep = after.search(/\n - name: /);
|
||||
const step = nextStep === -1 ? after : after.slice(0, nextStep);
|
||||
assert.equal(step.match(/^\s*save-if: (.+)$/m)?.[1], "false", `${name}: the restore must set save-if: false`);
|
||||
assert.doesNotMatch(step, /^\s*if:/m, `${name}: the restore must not be conditional; a miss is already free`);
|
||||
}
|
||||
assert.doesNotMatch(prWorkflow, /uses: Swatinem\/rust-cache@[0-9a-f]{40}[\s\S]*?save-if: (?!false)/);
|
||||
});
|
||||
|
||||
@@ -62,28 +103,75 @@ test("a pull request never writes to or evicts the master cache entry", () => {
|
||||
// the key matches nothing and every run recompiles.
|
||||
const NORMALIZE = /# rust-cache hashes every installed toolchain[\s\S]*?rustup toolchain list\n/;
|
||||
|
||||
test("reader and writer strip extra toolchains identically before the key is computed", () => {
|
||||
const mine = pr.match(NORMALIZE);
|
||||
test("every reader and the writer strip extra toolchains identically before the key is computed", () => {
|
||||
const theirs = release.match(NORMALIZE);
|
||||
assert.ok(mine, "pr-trusted.yml must normalize the installed toolchains");
|
||||
assert.ok(theirs, "release-verify.yml must normalize the installed toolchains");
|
||||
assert.equal(mine[0], theirs[0], "the normalization must be identical in both workflows");
|
||||
|
||||
for (const [name, body] of [["reader", mine[0]], ["writer", theirs[0]]]) {
|
||||
for (const [name, body] of [["writer", theirs[0]]]) {
|
||||
// Keep the pin, drop the rest, and never fail the job over it.
|
||||
assert.match(body, /grep -vx "\$toolchain"/, name);
|
||||
assert.match(body, /xargs -n1 rustup toolchain uninstall/, name);
|
||||
assert.match(body, /\|\| true/, name);
|
||||
}
|
||||
for (const [name, pr] of readers) {
|
||||
const mine = pr.match(NORMALIZE);
|
||||
assert.ok(mine, `${name}: pr-trusted.yml must normalize the installed toolchains`);
|
||||
assert.equal(mine[0], theirs[0], `${name}: the normalization must be identical to the writer's`);
|
||||
}
|
||||
});
|
||||
|
||||
test("the toolchain is stripped before the cache step, not after", () => {
|
||||
for (const [name, body, cacheStep] of [
|
||||
["reader", pr, " - name: Restore Runner Rust dependencies (read only)"],
|
||||
["writer", release, " - name: Cache Runner Rust dependencies"],
|
||||
...readers.map(([name, body]) => [name, body, READ_STEP]),
|
||||
["writer", release, WRITE_STEP],
|
||||
]) {
|
||||
const normalize = body.search(NORMALIZE);
|
||||
const cache = body.indexOf(cacheStep);
|
||||
assert.ok(normalize >= 0 && cache > normalize, `${name}: normalization must precede the cache step`);
|
||||
}
|
||||
});
|
||||
|
||||
// GitHub matches a cache entry on its key and on a version hash of the
|
||||
// absolute paths being cached. rust-cache resolves the target directory under
|
||||
// the checkout, and the checkout root differs by runner (/home/runner/_work on
|
||||
// the RunsOn fleets that write the cache, /home/runner/work on GitHub-hosted
|
||||
// runners). With every key input aligned, every GitHub-hosted pull request
|
||||
// still logged "No cache found" (run 36424309181, 2026-09-28). Both workflows
|
||||
// therefore hand rust-cache the same checkout-independent path, and they have
|
||||
// to build it the same way or the version matches nothing.
|
||||
const PIN = /# rust-cache hashes its absolute cache paths[\s\S]*?echo "path=\$pinned" >> "\$GITHUB_OUTPUT"\n/;
|
||||
|
||||
test("every reader and the writer pin an identical checkout-independent Rust workspace path", () => {
|
||||
const theirs = release.match(PIN);
|
||||
assert.ok(theirs, "release-verify.yml must pin the Runner Rust workspace path");
|
||||
const pins = [["writer", theirs[0]]];
|
||||
for (const [name, pr] of readers) {
|
||||
const mine = pr.match(PIN);
|
||||
assert.ok(mine, `${name}: pr-trusted.yml must pin the Runner Rust workspace path`);
|
||||
assert.equal(mine[0], theirs[0], `${name}: the pinned path must be built identically to the writer's`);
|
||||
pins.push([name, mine[0]]);
|
||||
}
|
||||
|
||||
for (const [name, body] of pins) {
|
||||
assert.match(body, /- name: Pin the Runner Rust workspace path\n\s+id: runner_rust_workspace\n/, name);
|
||||
// Anchor under $HOME, which both runner images share, never under the checkout.
|
||||
assert.match(body, /pinned="\$HOME\/[A-Za-z0-9._-]+"/, name);
|
||||
assert.match(body, /rm -rf "\$pinned"\n\s+ln -s "\$GITHUB_WORKSPACE\/packages\/paperclip-runner\/runner" "\$pinned"/, name);
|
||||
assert.match(body, /echo "path=\$pinned" >> "\$GITHUB_OUTPUT"/, name);
|
||||
}
|
||||
});
|
||||
|
||||
test("the workspace path is pinned before the cache step and never names the checkout", () => {
|
||||
for (const [name, body, cacheStep] of [
|
||||
...readers.map(([name, body]) => [name, body, READ_STEP]),
|
||||
["writer", release, WRITE_STEP],
|
||||
]) {
|
||||
const pin = body.search(PIN);
|
||||
const cache = body.indexOf(cacheStep);
|
||||
assert.ok(pin >= 0 && cache > pin, `${name}: the pin must precede the cache step`);
|
||||
assert.doesNotMatch(body, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/, `${name}: workspaces must not resolve under the checkout`);
|
||||
}
|
||||
// No rust-cache step anywhere in either workflow may point back into the checkout.
|
||||
for (const [name, workflow] of [["pr-trusted.yml", prWorkflow], ["release-verify.yml", releaseWorkflow]]) {
|
||||
assert.doesNotMatch(workflow, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/, `${name}: workspaces must not resolve under the checkout`);
|
||||
}
|
||||
});
|
||||
@@ -15,8 +15,13 @@ test("Runner dependency caching selects the package's pinned compiler before com
|
||||
assert.match(setup, /rustup show active-toolchain/);
|
||||
assert.match(setup, /echo "RUSTUP_TOOLCHAIN=\$toolchain" >> "\$GITHUB_ENV"/);
|
||||
assert.match(runner, /uses: Swatinem\/rust-cache@[0-9a-f]{40} # v[0-9.]+/);
|
||||
assert.match(runner, /workspaces: packages\/paperclip-runner\/runner -> target/);
|
||||
assert.match(runner, /shared-key: release-runner-v1/);
|
||||
// The target path feeds the entry's version hash, so it must not resolve
|
||||
// under the checkout, whose root differs between the fleet and GitHub-hosted
|
||||
// runners. The pin step publishes a $HOME-anchored path to the same directory.
|
||||
const pin = runner.indexOf(" - name: Pin the Runner Rust workspace path");
|
||||
assert.ok(pin >= 0 && cache > pin, "the workspace path must be pinned before the cache step");
|
||||
assert.match(runner, /workspaces: \$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target/);
|
||||
assert.match(runner, /shared-key: release-runner-v2/);
|
||||
});
|
||||
|
||||
test("the shared cache excludes workspace artifacts and only restores or saves the exact master-push source", () => {
|
||||
|
||||
@@ -28,11 +28,32 @@ for (const [name, overrides, ref, allowed] of [
|
||||
}
|
||||
test("cache excludes workspace code and executable installs, and preserves full checks", () => {
|
||||
assert.match(cache, /uses: Swatinem\/rust-cache@[a-f0-9]{40}/);
|
||||
assert.match(cache, /workspaces: packages\/paperclip-runner\/runner -> target/);
|
||||
assert.match(cache, /shared-key: release-typecheck-v1/);
|
||||
// The target path feeds the entry's version hash; a checkout-relative path
|
||||
// hashes differently on the fleet (/home/runner/_work) and on GitHub-hosted
|
||||
// runners (/home/runner/work), so the pin step publishes a $HOME-anchored one.
|
||||
assert.match(cache, /workspaces: \$\{\{ steps\.runner_rust_workspace\.outputs\.path \}\} -> target/);
|
||||
assert.match(cache, /shared-key: release-typecheck-v2/);
|
||||
assert.match(cache, /cache-workspace-crates: false/);
|
||||
assert.match(cache, /cache-bin: false/);
|
||||
const pin = typecheck.indexOf(" - name: Pin the Runner Rust workspace path");
|
||||
assert.ok(pin >= 0 && pin < typecheck.indexOf("uses: Swatinem/rust-cache"));
|
||||
assert.ok(typecheck.indexOf('echo "RUSTUP_TOOLCHAIN=$toolchain"') < typecheck.indexOf("uses: Swatinem/rust-cache"));
|
||||
assert.doesNotMatch(typecheck, /workspaces: (\.|packages\/|\$\{\{ github\.workspace)/);
|
||||
});
|
||||
// The typecheck key is written and read only by master pushes on the fleet, so
|
||||
// its version would still match if this pin drifted. Hold it to the same text
|
||||
// as the Runner writer anyway: `doc/RELEASE-AUTOMATION-SETUP.md` promises one
|
||||
// identical pin step before every Rust cache step, and a divergent copy here
|
||||
// is the first place a later edit would quietly break that promise.
|
||||
test("the typecheck pin step is identical to the Runner writer's", () => {
|
||||
const PIN = /# rust-cache hashes its absolute cache paths[\s\S]*?echo "path=\$pinned" >> "\$GITHUB_OUTPUT"\n/;
|
||||
const writer = workflow.split("\n verify_paperclip_runner:\n")[1];
|
||||
assert.ok(writer, "release-verify.yml is missing the verify_paperclip_runner job");
|
||||
const theirs = writer.match(PIN);
|
||||
const mine = typecheck.match(PIN);
|
||||
assert.ok(theirs, "the Runner writer must pin the Runner Rust workspace path");
|
||||
assert.ok(mine, "the typecheck job must pin the Runner Rust workspace path");
|
||||
assert.equal(mine[0], theirs[0], "the typecheck pin step drifted from the Runner writer's");
|
||||
assert.match(typecheck, /run: pnpm -r typecheck/);
|
||||
assert.match(workflow, /shared-key: release-runner-v1/);
|
||||
assert.match(workflow, /shared-key: release-runner-v2/);
|
||||
});
|
||||
@@ -407,6 +407,28 @@ jobs:
|
||||
# rebuilds the Runner release binary; without the shared Rust cache that
|
||||
# is a ~3m40s cold compile of all third-party crates (run 35036001734,
|
||||
# 2026-09-15). Same restore-only contract as Verify Paperclip Runner.
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -445,8 +467,8 @@ jobs:
|
||||
- name: Restore Runner Rust dependencies (read only)
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-runner-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -766,11 +788,33 @@ jobs:
|
||||
|
||||
# Same restore-only contract as the pnpm store above, for the Rust
|
||||
# dependency tree: master's post-merge verification is the sole writer
|
||||
# of release-runner-v1, and PR merge refs must not save branch-scoped
|
||||
# of release-runner-v2, and PR merge refs must not save branch-scoped
|
||||
# copies of a ~680MB target directory. Every key input below has to
|
||||
# match that writer in release-verify.yml exactly or each PR misses and
|
||||
# recompiles all 313 third-party crates in both profiles. A miss is a
|
||||
# slow run, never a wrong one.
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -809,8 +853,8 @@ jobs:
|
||||
- name: Restore Runner Rust dependencies (read only)
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-runner-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -884,6 +928,28 @@ jobs:
|
||||
# shared Rust cache that is a ~3m40s cold compile of all third-party
|
||||
# crates (run 35036001734, 2026-09-15). Same restore-only contract as
|
||||
# Verify Paperclip Runner.
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -922,8 +988,8 @@ jobs:
|
||||
- name: Restore Runner Rust dependencies (read only)
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-runner-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
@@ -1094,6 +1160,28 @@ jobs:
|
||||
# build:binary step; without the shared Rust cache that is a ~3m40s cold
|
||||
# compile of all third-party crates (run 35036001734, 2026-09-15). Same
|
||||
# restore-only contract as Verify Paperclip Runner.
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -1132,8 +1220,8 @@ jobs:
|
||||
- name: Restore Runner Rust dependencies (read only)
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-runner-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
# Mirror the master writer: these also feed the cache key.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
|
||||
@@ -42,6 +42,27 @@ jobs:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -56,8 +77,8 @@ jobs:
|
||||
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-typecheck-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-typecheck-v2
|
||||
# Rebuild workspace code and rerun every check. Cache only compiled
|
||||
# dependencies; never restore installed executables from cargo/bin.
|
||||
cache-workspace-crates: false
|
||||
@@ -294,6 +315,27 @@ jobs:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
# rust-cache hashes its absolute cache paths into the entry's version,
|
||||
# and GitHub only serves an entry whose key and version both match. The
|
||||
# target directory sits under the checkout, and the checkout root
|
||||
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
||||
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
||||
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
||||
# logged "No cache found" (run 36424309181) and cold-compiled every
|
||||
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
||||
# directory through a checkout-independent path so both layouts hash
|
||||
# the same version. Keep this block identical in both workflows: the
|
||||
# reader and the writer must agree or the version matches nothing.
|
||||
- name: Pin the Runner Rust workspace path
|
||||
id: runner_rust_workspace
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pinned="$HOME/paperclip-runner-rust"
|
||||
# A symlink here is removed as a link, never followed into the checkout.
|
||||
rm -rf "$pinned"
|
||||
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
||||
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select the pinned Runner Rust toolchain
|
||||
working-directory: packages/paperclip-runner
|
||||
run: |
|
||||
@@ -325,8 +367,8 @@ jobs:
|
||||
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
||||
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
workspaces: packages/paperclip-runner/runner -> target
|
||||
shared-key: release-runner-v1
|
||||
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
||||
shared-key: release-runner-v2
|
||||
# Rebuild workspace code and rerun every check. Cache only compiled
|
||||
# dependencies; never restore installed executables from cargo/bin.
|
||||
cache-workspace-crates: false
|
||||
|
||||
@@ -347,11 +347,22 @@ readiness can succeed. A failed lane does not cancel the other lane.
|
||||
Both lanes restore Cargo dependencies with the pinned Rust Cache action. The
|
||||
compiler comes from the Runner package's `rust-toolchain.toml` before the action
|
||||
computes its key. Compiler and Cargo metadata changes select a new cache. The
|
||||
existing `release-runner-v1` shared key avoids separate copies for these lanes.
|
||||
`release-runner-v2` shared key avoids separate copies for these lanes.
|
||||
Only the Rust lane saves this cache. After verification it also runs `build:rust`
|
||||
to warm the debug dependencies used by the protocol lane; its own tests already
|
||||
warm release dependencies. The cache writer is shorter than the protocol lane.
|
||||
|
||||
GitHub matches a cache entry on its key and on a version hash of the absolute
|
||||
paths in the entry. The master writer runs on the RunsOn fleet, where the
|
||||
checkout is `/home/runner/_work/paperclip/paperclip`. The trusted PR workflow
|
||||
restores the same entry read-only on GitHub-hosted runners, where the checkout
|
||||
is `/home/runner/work/paperclip/paperclip`. A `Pin the Runner Rust workspace
|
||||
path` step in both workflows links `$HOME/paperclip-runner-rust` to the Runner
|
||||
crate and passes that path to the cache action, so both layouts hash the same
|
||||
paths and the PR lanes can restore master's entry. The guard tests under
|
||||
`.github/scripts/tests/` require this step, with identical text, before every
|
||||
Rust cache step in both workflows.
|
||||
|
||||
Workspace crates and installed Cargo binaries are excluded. Every run rebuilds
|
||||
workspace code and runs all assigned checks, including on a cache hit. Only an
|
||||
own-repository master-push run verifying that push's exact SHA can restore the
|
||||
|
||||
@@ -158,10 +158,13 @@ shared infrastructure ownership separately before removing those resources.
|
||||
Source verification's typecheck job builds the native Runner binary through the
|
||||
server's `prepare:runner-vendor` command. It restores and saves compiled Rust
|
||||
dependencies only for canonical master pushes that verify the event's exact SHA.
|
||||
The `release-typecheck-v1` cache is separate from Runner verification because
|
||||
The `release-typecheck-v2` cache is separate from Runner verification because
|
||||
those jobs compile different profiles. The pinned toolchain is selected before
|
||||
cache lookup. Workspace crates and installed cargo binaries are excluded, and
|
||||
all typechecks still execute. A missing or invalidated cache triggers compilation.
|
||||
cache lookup, and the cache action receives the Runner crate through the same
|
||||
checkout-independent `$HOME/paperclip-runner-rust` path as the Runner lanes
|
||||
(see `RELEASE-AUTOMATION-SETUP.md`). Workspace crates and installed cargo
|
||||
binaries are excluded, and all typechecks still execute. A missing or
|
||||
invalidated cache triggers compilation.
|
||||
|
||||
### pnpm dependency store cache
|
||||
|
||||
|
||||
Reference in new issue
Block a user