fix: run the cargo-building native-runner CI suite in the Rust-cached vitest lane (#13586)

## Thinking Path

Post-merge of #13557, the slowest check on the freshest fully-green PR
run
([35246999382](https://github.com/paperclipai/paperclip/actions/runs/35246999382))
was `ci / General tests (server (1/12))` at **339s**. The cause is one
suite:
`server/src/services/native-runtime/native-codex-runner.integration.test.ts`
runs 1 test in **277s of a 291s vitest step (95%)** because its
`beforeAll` cargo-builds the Runner release binaries, and the
general-server shards carry no Rust cache — every PR run cold-compiles
the full third-party crate graph. The other 19 suites in that shard
finish in under 70ms each.

The obvious fix (a dedicated Rust-cached matrix lane) requires editing
workflow files, which the available GitHub App credentials cannot push
(`workflows` permission). But the `Verify Paperclip Runner` lanes
**already restore the shared `release-runner-v1` Rust cache read-only**,
and their commands are `pnpm --filter @paperclipai/paperclip-runner
<package script>` — so the suite can move into a Rust-cached lane purely
through script changes.

## What Changed

- `scripts/run-vitest-stable.mjs`: new `general-server-native-runner`
group carrying exactly that suite. Under the PR workflow
(`GITHUB_WORKFLOW == "PR"`, inherited from `pr.yml` by the reusable
`pr-trusted.yml`) the without-chat server shards exclude it and
rebalance to ~211s of tests each. Every other caller — local runs,
`release-verify.yml` under the Release / Cloud readiness workflows —
keeps the suite in the shards, so a renamed or unknown workflow degrades
to today's slower-but-covered behavior instead of dropping coverage.
- `packages/paperclip-runner`: `test:typescript:vitest` now routes
through `scripts/run-pr-vitest-lane.mjs` — the identical
`ensure:eval-build-deps && build:rust && vitest run` chain (shard flags
passed through), plus the native-runner group on the **final PR shard
only** (`--shard=N/M` with `N == M`, i.e. today's `vitest 2/2`, the 122s
lane). With the restored cache the suite's cargo build becomes an
incremental rebuild.
- `scripts/__tests__/run-vitest-stable-shard.test.mjs`: guards pin the
whole contract — PR 12-shard coverage (shards + chat + native-runner =
full server group exactly), Release/local 10-shard runs keep the suite,
`pr.yml` is named `PR`, the vitest lanes partition with exactly one
final shard, the package-script wiring, and the wrapper's shard/workflow
gating via its `--dry-run` plan output.

No workflow files change. `.github/workflows/*` are untouched.

## Verification

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: **36/36 pass**
locally on this branch (includes the new coverage, wiring, and
wrapper-gating guards). Both files run in CI's `Test general-server
shard partition` / `Test release verify workflow wiring` steps.
- Wrapper `--dry-run` plan matrix verified for all six shard/workflow
combinations plus malformed-shard rejection (pinned as a guard test).
- The executing proof is this PR's own CI: `ci / Verify Paperclip Runner
(vitest 2/2)` must go green while running the native-runner suite (its
log will show the `general-server-native-runner` group after the package
vitest shard), and the 12 `ci / General tests (server (x/12))` shards
must go green without it.

## Risks

- The exclusion keys on `GITHUB_WORKFLOW == "PR"`. Failure mode of a
rename is safe (suite falls back into the server shards, slower but
covered) and the guard test on `pr.yml`'s name makes it loud.
- `vitest 2/2` grows from ~122s to an expected ~210–260s — still well
under the ~306s `vitest 1/2` and ~326s e2e shards, and inside the
20-minute lane timeout. If the cache misses (key drift), the lane pays a
cold compile like the server shard does today; a miss is slow, never
wrong.
- Double-run/coverage-loss combinations are enumerated in the wrapper
header and pinned by tests: each caller runs the suite exactly once.

## Model Used

Claude (Bender agent, Paperclip) — Fable 5.

---
Expected savings once merged: the 339s `server (1/12)` check drops to
~265s-equivalent shard levels (~211s of tests), the slowest `ci /` check
becomes the ~326s e2e shard (~13–33s off PR wall time), and every PR run
stops paying ~4.5 min of billed cold Rust compile.

For the merger (squash): please keep the trailer below in the squash
body to preserve authorship.

`Co-Authored-By: Bender (Fable)
<Paperclip-Paperclip@users.noreply.github.com>`

## Related PRs

Searched the GitHub PR list for prior work on this surface — related
groundwork, none duplicate this change:
- #13457 — restored master's Rust dependency cache on the PR runner lane
(the read-only cache this PR relies on)
- #13500 — made that cache key image-toolchain-independent so
GitHub-hosted PR runners actually hit it
- #13521 — rebalanced PR shards and split the Verify Paperclip Runner
lanes this PR extends
- #13557 — previous health-check iteration (split the runnerd transport
suite); this PR targets the next slowest check

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above

Co-authored-by: Bender (Fable) <Paperclip-Paperclip@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-09-18 07:22:12 -07:00
1 parent de9414dd8b
commit 352153b5ed
4 files changed
+278 -33

No files matched your search

@@ -1,5 +1,6 @@
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { readFileSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import test from "node:test";
@@ -21,16 +22,29 @@ const serializedDurationsManifest = path.join(
"serialized-shard-durations.json",
);
function dryRun(args) {
// Membership of general-server-without-chat depends on GITHUB_WORKFLOW (see
// prWorkflowName in run-vitest-stable.mjs), so strip the ambient value and
// make every test pin the caller it mirrors explicitly — these tests
// themselves run inside a workflow on CI.
function workflowEnv(envOverrides = {}) {
const env = { ...process.env, ...envOverrides };
if (!("GITHUB_WORKFLOW" in envOverrides)) {
delete env.GITHUB_WORKFLOW;
}
return env;
}
function dryRun(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [script, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
return result;
}
function dryRunJson(args) {
const result = dryRun(args);
function dryRunJson(args, envOverrides = {}) {
const result = dryRun(args, envOverrides);
assert.equal(result.status, 0, `expected success for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
@@ -222,11 +236,15 @@ test("the real serialized shard partition is duration-balanced", () => {
test("the real shard partition is duration-balanced", () => {
// Mirrors the PR matrix: general-server-without-chat across SHARD_COUNT
// runners, with the chat suite carried by the dedicated general-chat lanes.
// runners, with the chat suite carried by the dedicated general-chat lanes
// and the native-runner suite by the Rust-cached PR vitest lane.
const durations = loadShardDurations(durationsManifest);
const fallback = defaultSuiteWeight(durations);
const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-server-without-chat", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
dryRunJson(
["--mode", "general", "--group", "general-server-without-chat", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)],
{ GITHUB_WORKFLOW: "PR" },
),
);
const totals = shards.map((shard) =>
@@ -235,10 +253,15 @@ test("the real shard partition is duration-balanced", () => {
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the
// bound. The chat suite runs in its own lanes, so exclude it here.
// bound. The chat and native-runner suites run in their own lanes, so
// exclude them here.
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunner =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
const heaviest = Math.max(
...Object.entries(durations).filter(([file]) => file !== chat).map(([, ms]) => ms),
...Object.entries(durations)
.filter(([file]) => file !== chat && file !== nativeRunner)
.map(([, ms]) => ms),
);
assert.ok(
maxTotal - minTotal <= heaviest,
@@ -247,24 +270,131 @@ test("the real shard partition is duration-balanced", () => {
});
// 12 mirrors pr-trusted.yml, 10 mirrors release-verify.yml.
for (const withoutChatShardCount of [10, 12]) {
test(`${withoutChatShardCount} without-chat shards plus the dedicated chat file cover the original server group exactly`, () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
const shards = Array.from({ length: withoutChatShardCount }, (_, index) => dryRunJson([
const chatSuitePath = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunnerSuitePath =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is
// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent
// native-runner suite in the Rust-cached final Verify Paperclip Runner vitest
// shard, so together the three cover the full server group exactly.
test("12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly", () => {
const prEnv = { GITHUB_WORKFLOW: "PR" };
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], prEnv);
const shards = Array.from({ length: 12 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "12",
], prEnv));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
assert.ok(!files.includes(chatSuitePath));
assert.ok(!files.includes(nativeRunnerSuitePath));
assert.deepEqual([...files, chatSuitePath, nativeRunnerSuitePath].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([], prEnv);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
// Mirrors release-verify.yml (10 shards, called by the Release and Cloud
// readiness workflows) and local runs: no Rust-cached vitest lane exists
// there, so the native-runner suite must stay in the server shards.
for (const [caller, envOverrides] of [["Release", { GITHUB_WORKFLOW: "Release" }], ["no ambient workflow", {}]]) {
test(`10 without-chat shards under ${caller} keep the native-runner suite and cover the server group with chat alone`, () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], envOverrides);
const shards = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", String(withoutChatShardCount),
]));
"--shard-index", String(index), "--shard-count", "10",
], envOverrides));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
assert.ok(!files.includes(chat));
assert.deepEqual([...files, chat].sort(), full.selectedGeneralServerSuites.sort());
assert.ok(!files.includes(chatSuitePath));
assert.ok(files.includes(nativeRunnerSuitePath));
assert.deepEqual([...files, chatSuitePath].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([]);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
}
test("the native-runner lane runs exactly the cargo-dependent vertical-slice suite", () => {
const lane = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"]);
assert.deepEqual(lane.selectedGeneralServerSuites, [
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
]);
});
test("shard flags are rejected for the native-runner group", () => {
const result = dryRun(["--mode", "general", "--group", "general-server-native-runner", "--shard-index", "0", "--shard-count", "2"]);
assert.notEqual(result.status, 0, "the native-runner lane is a single suite and must not accept shard flags");
});
// The PR-side exclusion above is safe only while the wiring it assumes holds:
// pr.yml (the caller whose name reusable pr-trusted.yml jobs see as
// GITHUB_WORKFLOW) is named PR, the sharded vitest lanes partition cleanly
// with exactly one final shard, and that lane's package script routes through
// the wrapper that runs the native-runner group.
test("the PR workflow wiring for the native-runner lane holds", () => {
const prWorkflow = readFileSync(path.join(repoRoot, ".github/workflows/pr.yml"), "utf8");
assert.match(prWorkflow, /^name: PR$/m,
"renaming pr.yml silently moves the native-runner suite back into the uncached server shards");
const trustedWorkflow = readFileSync(path.join(repoRoot, ".github/workflows/pr-trusted.yml"), "utf8");
const lanes = [...trustedWorkflow.matchAll(/command: test:typescript:vitest --shard=(\d+)\/(\d+)/g)]
.map((match) => [Number(match[1]), Number(match[2])]);
assert.ok(lanes.length > 0, "expected sharded test:typescript:vitest lanes in pr-trusted.yml");
assert.equal(new Set(lanes.map(([, count]) => count)).size, 1, "vitest lanes must agree on the shard count");
const shardCount = lanes[0][1];
assert.deepEqual(
lanes.map(([index]) => index).sort((left, right) => left - right),
Array.from({ length: shardCount }, (_, index) => index + 1),
"vitest lanes must cover every shard exactly once",
);
assert.equal(lanes.filter(([index, count]) => index === count).length, 1,
"exactly one final vitest shard carries the native-runner group");
const runnerPackage = JSON.parse(
readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8"),
);
assert.equal(runnerPackage.scripts["test:typescript:vitest"], "node ./scripts/run-pr-vitest-lane.mjs");
});
const laneWrapper = path.join(repoRoot, "packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs");
function wrapperPlan(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [laneWrapper, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
assert.equal(result.status, 0, `expected wrapper dry run to succeed for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
test("the PR vitest lane wrapper runs the native-runner group exactly on the final PR shard", () => {
for (const [args, envOverrides, expected] of [
[["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }, false],
[["--shard=2/2"], { GITHUB_WORKFLOW: "PR" }, true],
[[], { GITHUB_WORKFLOW: "PR" }, true],
[["--shard=2/2"], { GITHUB_WORKFLOW: "Release" }, false],
[["--shard=2/2"], {}, false],
[[], {}, false],
]) {
const plan = wrapperPlan(args, envOverrides);
assert.equal(plan.runNativeRunnerGroup, expected,
`args ${JSON.stringify(args)} env ${JSON.stringify(envOverrides)}`);
const commands = plan.plannedCommands.map((planned) => planned.args.join(" "));
assert.ok(commands.some((command) => command.startsWith(`exec vitest run${args.length ? ` ${args.join(" ")}` : ""}`)),
"the wrapper must pass shard flags through to vitest");
assert.equal(
commands.some((command) => command.includes("--group general-server-native-runner")),
expected,
);
}
const malformed = spawnSync(process.execPath, [laneWrapper, "--shard=nonsense", "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv({ GITHUB_WORKFLOW: "PR" }),
});
assert.notEqual(malformed.status, 0, "a malformed shard flag must fail rather than guess a lane");
});
const lineShardFile = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const caseAt = (line, name) => ({ name, file: lineShardFile, projectName: "@paperclipai/server", location: { line, column: 3 } });
+59 -13
View File
@@ -70,13 +70,39 @@ const allModeName = "all";
const generalServerGroupName = "general-server";
const generalServerWithoutChatGroupName = "general-server-without-chat";
const generalChatGroupName = "general-chat";
const generalServerNativeRunnerGroupName = "general-server-native-runner";
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
// This suite rebuilds the Runner release binaries with cargo in beforeAll.
// Inside the PR workflow's plain server shards, which carry no Rust cache,
// that build was a ~4m30s cold compile of every third-party crate on each run
// (277s of a 291s shard vitest step, actions run 35246999382, 2026-09-17).
const nativeRunnerSuite =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
// In the PR workflow (pr.yml, the caller of pr-trusted.yml — reusable
// workflows inherit the caller's GITHUB_WORKFLOW), the last Verify Paperclip
// Runner vitest shard runs the native-runner group instead, because those
// lanes restore the shared release-runner-v1 Rust cache (see
// packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs). Every other
// caller — local runs, release-verify.yml under the Release and Cloud
// readiness workflows — keeps the suite in the server shards, so a renamed or
// unknown workflow degrades to today's slower-but-covered behavior rather
// than dropping the suite.
const prWorkflowName = "PR";
const nativeRunnerSuiteRunsInRustCachedLane = process.env.GITHUB_WORKFLOW === prWorkflowName;
const withoutChatExcludedSuites = nativeRunnerSuiteRunsInRustCachedLane
? [chatSuite, nativeRunnerSuite]
: [chatSuite];
const generalWorkspacesAGroupName = "general-workspaces-a";
const generalWorkspacesBGroupName = "general-workspaces-b";
const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"];
const generalWorkspacesBProjects = nonServerProjects.filter((project) => !generalWorkspacesAProjects.includes(project));
const generalGroupNames = [generalServerGroupName, generalWorkspacesAGroupName, generalWorkspacesBGroupName];
const allowedGeneralGroupNames = [...generalGroupNames, generalServerWithoutChatGroupName, generalChatGroupName];
const allowedGeneralGroupNames = [
...generalGroupNames,
generalServerWithoutChatGroupName,
generalChatGroupName,
generalServerNativeRunnerGroupName,
];
const serializedServerVitestArgs = [
"--no-file-parallelism",
"--maxWorkers=1",
@@ -355,9 +381,21 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =
"chat integration test shard", { index: shardIndex ?? 0, count: shardCount ?? 1 });
return;
}
if (groupName === generalServerNativeRunnerGroupName) {
runVitest(
["--project", "@paperclipai/server", ...serializedServerVitestArgs, nativeRunnerSuite],
"native runner vertical-slice suite",
);
return;
}
if (groupName === generalServerGroupName || groupName === generalServerWithoutChatGroupName) {
// In the PR workflow the without-chat group also leaves the native-runner
// suite to the Rust-cached vitest lane; the full general-server group
// (local runs) keeps both.
const withoutChat = groupName === generalServerWithoutChatGroupName;
const files = withoutChat ? generalServerTestFiles.filter((file) => file !== chatSuite) : generalServerTestFiles;
const files = withoutChat
? generalServerTestFiles.filter((file) => !withoutChatExcludedSuites.includes(file))
: generalServerTestFiles;
if (shardCount !== null && shardCount > 1) {
const shardFiles = selectGeneralServerShard(
files,
@@ -385,7 +423,11 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =
}
const excludeRouteArgs = routeTests.flatMap((file) => ["--exclude", file.serverPath]);
if (withoutChat) excludeRouteArgs.push("--exclude", "src/__tests__/chat-channels.integration.test.ts");
if (withoutChat) {
for (const suite of withoutChatExcludedSuites) {
excludeRouteArgs.push("--exclude", suite.replace(/^server\//, ""));
}
}
runVitest(
[
"--project",
@@ -474,16 +516,20 @@ if (options.dryRun) {
selectedSerializedSuites: serializedSuites.map((routeTest) => routeTest.repoPath),
generalServerSuiteCount: generalServerTestFiles.length,
selectedGeneralServerSuites:
options.mode === generalModeName &&
[generalServerGroupName, generalServerWithoutChatGroupName].includes(options.group) &&
options.shardCount !== null
? selectGeneralServerShard(
options.group === generalServerWithoutChatGroupName ? generalServerTestFiles.filter((file) => file !== chatSuite) : generalServerTestFiles,
options.shardIndex,
options.shardCount,
generalServerShardDurations,
)
: null,
options.mode === generalModeName && options.group === generalServerNativeRunnerGroupName
? [nativeRunnerSuite]
: options.mode === generalModeName &&
[generalServerGroupName, generalServerWithoutChatGroupName].includes(options.group) &&
options.shardCount !== null
? selectGeneralServerShard(
options.group === generalServerWithoutChatGroupName
? generalServerTestFiles.filter((file) => !withoutChatExcludedSuites.includes(file))
: generalServerTestFiles,
options.shardIndex,
options.shardCount,
generalServerShardDurations,
)
: null,
workspaceProjects:
options.group === generalWorkspacesAGroupName
? generalWorkspacesAProjects