From 8dd381f5ddb3501c55c715ab733f0717a12346e5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 10 Oct 2026 20:25:44 -0700 Subject: [PATCH] perf(ci): run the db project as its own workspaces-c lane (#15831) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - The `General tests (workspaces-b)` lane runs every non-server Vitest project except `ui` and `cli` in one job, one project after the other. After #15826 leveled the server shards, this lane became the slowest check: 593s in run 38084171761, 523s in 38083084276 and 499s in 38083020834 (all 2026-10-10). > - The `@paperclipai/db` project alone is 168s to 250s of that lane's 436s to 513s of vitest time. Its `client.test.ts` suite is 117s to 189s and `backup-lib.test.ts` is 70s to 107s. The other seventeen projects together are only 263s to 276s. > - Vitest's native `--shard` does not level this lane. The db project is already file-parallel, and a hash-ordered slice can put both slow suites on one runner. > - This pull request moves the db project into its own `workspaces-c` lane in both `pr-trusted.yml` and `release-verify.yml`. The runner script gains a `general-workspaces-c` group, and `general-workspaces-b` drops the db project. > - The benefit is that both lanes should finish in about 230s to 340s instead of 499s to 593s, and `workspaces-b` leaves the critical path. The cost is one more job per run, about 60s of setup. ## Linked Issues or Issue Description **Describe the enhancement** The `General tests (workspaces-b)` PR check is the slowest check in the run. It runs seventeen Vitest projects in sequence on one runner. The `@paperclipai/db` project is one third to one half of its vitest time. **Why is this needed** The slowest check decides the wall-clock time of every PR run. In the last three fully green runs on the current server-shard manifest, `workspaces-b` took 499s, 523s and 593s (runs 38083020834, 38083084276 and 38084171761, 2026-10-10). The next slowest lanes, `Build` and the server shards, sit at about 480s to 570s. A split of `workspaces-b` removes 170s to 250s from this lane. **Proposed solution** Run the `@paperclipai/db` project as its own lane, `general-workspaces-c`. Keep the other seventeen projects in `general-workspaces-b`. Add the lane to both the PR and the release verification matrix. Extend the guard tests so the three workspace lanes still cover the root Vitest project list exactly once, so the new group rejects shard flags, and so both workflows carry the lane unsharded. **Alternatives considered** - Vitest native `--shard` on `workspaces-b`, as `workspaces-a` does. The db project is already file-parallel and its wall time is set by two long suites, so a hash-ordered slice does not give a predictable split. - Move `adapter-utils` (115s to 117s) into the new lane as well. That makes the db lane 285s to 366s of tests and the rest 147s to 160s, which is less level than db alone (168s to 250s) against the rest (263s to 276s). - Three lanes. Not needed now: after this split both lanes sit under the server shards and `Build`, so a third runner would not shorten the run. ## What Changed - `scripts/run-vitest-stable.mjs`: new `general-workspaces-c` group that runs `@paperclipai/db` alone. `general-workspaces-b` now excludes it. Local `--mode all` runs all four general groups, so every project still runs once. The dry-run output reports the new group's projects. - `.github/workflows/pr-trusted.yml`: a `workspaces-c` entry in the general tests matrix, with a comment that records the 2026-10-10 measurements. - `.github/workflows/release-verify.yml`: the same `workspaces-c` entry, to keep parity with the PR workflow. - `scripts/__tests__/run-vitest-stable-shard.test.mjs`: the coverage test includes the new group; shard flags are rejected for both `workspaces-b` and `workspaces-c`; a new test asserts that `workspaces-c` runs only the db project, that `workspaces-b` no longer carries it, and that both workflows list both lanes without shard arguments. The workflow check parses each `- group:` matrix entry with comment lines removed, so a comment between keys cannot hide a shard setting, and it first confirms the parser sees the shard keys on the `workspaces-a` entries. A small parser test covers the comment-between-keys case (Greptile P2). - `scripts/__tests__/release-verify-workflow.test.mjs`: the release workflow must carry `general-workspaces-c`. ## Verification - `GITHUB_WORKFLOW= node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 32 pass (was 29). - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: the workflow parity test passes. Two unrelated tests in that file (`a publisher lockfile lets a fresh smoke install retain validation after ... drift`) fail on my machine on pristine `master` too, because `pnpm` is not on its PATH. They pass in CI. - `node scripts/run-vitest-stable.mjs --mode general --group general-workspaces-c --dry-run` reports `["@paperclipai/db"]`; `--group general-workspaces-b` reports the other seventeen projects. - Both workflow files parse as YAML; the general tests matrix has 19 entries in `pr-trusted.yml` and 17 in `release-verify.yml`. - This PR's own run exercises the script change but not the workflow change: `pr.yml` calls `pr-trusted.yml@master`, so the matrix comes from `master` and the `workspaces-c` job appears only on PR runs created after the merge. Because the script comes from this branch, the `workspaces-b` job in this PR's run (38090713637) ran the seventeen remaining projects and finished in 340s, inside the expected 325s to 340s band. No job in that run ran `@paperclipai/db`. - To cover the gap, I ran the new lane locally with the same command CI uses, `pnpm test:run:general -- --group general-workspaces-c`: 49 test files, 180 tests, all passed, 111s of vitest time (c324d0080, Node 25, PGlite). Expected result on the first PR run after merge, from the three measured runs: | Check | Before | After (expected) | | --- | --- | --- | | `General tests (workspaces-b)` | 499-593s | ~325-340s (263-276s of vitest time plus ~60s of setup) | | `General tests (workspaces-c)` | (new) | ~230-310s (168-250s of vitest time plus ~60s of setup) | | Slowest check in the run | `workspaces-b` | `Build` (497-570s) or a server shard (~480-520s) | ## Risks - Low risk. The group split only changes which job runs the db project. The guard test asserts that the three workspace lanes cover the root Vitest project list exactly once, so a project cannot be dropped or run twice. - One more job per PR and release verification run, about 60s of setup time on the runner fleet. - The lane names are matrix labels only. The required checks are `ci / verify` and `ci / e2e`, which aggregate the matrix, so no branch ruleset needs an update. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, made the change, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Bender (Fable) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> --- .github/workflows/pr-trusted.yml | 8 ++ .github/workflows/release-verify.yml | 3 + .../release-verify-workflow.test.mjs | 2 +- .../run-vitest-stable-shard.test.mjs | 82 +++++++++++++++++-- scripts/run-vitest-stable.mjs | 31 ++++++- 5 files changed, 117 insertions(+), 9 deletions(-) diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 56b6f594c12..228ac3bd4b7 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -585,8 +585,16 @@ jobs: group_label: workspaces-a (2/2) shard_index: 1 shard_count: 2 + # workspaces-b became the slowest check (499-593s) once the server + # shards were rebalanced: the db project alone was 168-250s of its + # 436-513s of vitest time (PR runs 38084171761, 38083084276 and + # 38083020834, 2026-10-10). The db project now runs as its own + # workspaces-c lane; the rest of workspaces-b is ~265-275s of + # vitest time, so both jobs land near 300-350s with setup. - group: general-workspaces-b group_label: workspaces-b + - group: general-workspaces-c + group_label: workspaces-c steps: - name: Checkout repository diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 492b7e62b47..3fd321edb68 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -174,6 +174,9 @@ jobs: shard_count: 2 - group: general-workspaces-b group_label: workspaces-b + # Keep parity with pr.yml: the db project runs as its own lane. + - group: general-workspaces-c + group_label: workspaces-c steps: - name: Checkout repository diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 043636edb91..9d00a15b79e 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -505,7 +505,7 @@ test("release verify workflow covers the same split test surface as stable PR ve assert.match(buildJob, /persist-credentials: false/); assert.doesNotMatch(buildJob, /cache: pnpm/); - for (const group of ["general-server-without-chat-or-native-runner", "general-chat", "general-workspaces-a", "general-workspaces-b"]) { + for (const group of ["general-server-without-chat-or-native-runner", "general-chat", "general-workspaces-a", "general-workspaces-b", "general-workspaces-c"]) { assert.match(verifyWorkflow, new RegExp(`group: ${group}`)); } for (const [group, count] of [["general-server-without-chat-or-native-runner", 10], ["general-chat", 3]]) { diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjs index a044c67c151..3dd3e9680e0 100644 --- a/scripts/__tests__/run-vitest-stable-shard.test.mjs +++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -107,10 +107,12 @@ test("general shards retain scripts and route-named suites outside the serialize ]) assert.ok(general.selectedGeneralServerSuites.includes(file), `missing configured server suite: ${file}`); }); -test("shard flags are rejected for the workspaces-b group", () => { - const result = dryRun(["--mode", "general", "--group", "general-workspaces-b", "--shard-index", "0", "--shard-count", "3"]); - assert.notEqual(result.status, 0, "workspaces-b must not accept shard flags"); -}); +for (const group of ["general-workspaces-b", "general-workspaces-c"]) { + test(`shard flags are rejected for the ${group} group`, () => { + const result = dryRun(["--mode", "general", "--group", group, "--shard-index", "0", "--shard-count", "3"]); + assert.notEqual(result.status, 0, `${group} must not accept shard flags`); + }); +} test("workspace lanes cover every non-server project in the root Vitest configuration", () => { const config = readFileSync(path.join(repoRoot, "vitest.config.ts"), "utf8"); @@ -118,11 +120,81 @@ test("workspace lanes cover every non-server project in the root Vitest configur assert.ok(roots.includes("server"), "expected the explicit root Vitest project list"); const expected = roots.filter(root => root !== "server") .map(root => JSON.parse(readFileSync(path.join(repoRoot, root, "package.json"), "utf8")).name).sort(); - const actual = ["general-workspaces-a", "general-workspaces-b"] + const actual = ["general-workspaces-a", "general-workspaces-b", "general-workspaces-c"] .flatMap(group => dryRunJson(["--mode", "general", "--group", group]).workspaceProjects).sort(); assert.deepEqual(actual, expected, "no configured project may be silently omitted or run twice"); }); +// Collects every `- group:` matrix entry in a workflow with the keys it sets. +// Comment and blank lines are dropped first, and an entry ends only when the +// indentation returns to the item level, so a comment between two keys cannot +// hide a shard setting from the assertions below. +function matrixEntries(text) { + const entries = []; + let current = null; + for (const line of text.split("\n")) { + if (/^\s*(#|$)/.test(line)) continue; + const indent = line.match(/^ */)[0].length; + const item = line.match(/^ *- group: (\S+)/); + if (item) { + current = { group: item[1], indent, keys: ["group"] }; + entries.push(current); + continue; + } + if (!current) continue; + if (indent <= current.indent) { current = null; continue; } + const key = line.match(/^ *([A-Za-z_]+):/); + if (key) current.keys.push(key[1]); + } + return entries; +} + +test("matrixEntries sees shard keys that sit behind comments or later in the entry", () => { + const entries = matrixEntries([ + " include:", + " - group: general-workspaces-b", + " group_label: workspaces-b", + " # a comment between keys must not hide the shard setting", + " shard_index: 0", + "", + " shard_count: 3", + " - group: general-workspaces-c", + " group_label: workspaces-c", + " steps:", + " - name: unrelated", + " shard_index: 9", + ].join("\n")); + assert.deepEqual(entries.map(entry => entry.group), ["general-workspaces-b", "general-workspaces-c"]); + assert.deepEqual(entries[0].keys, ["group", "group_label", "shard_index", "shard_count"]); + assert.deepEqual(entries[1].keys, ["group", "group_label"], "keys after the matrix must not leak into the last entry"); +}); + +test("workspaces-c runs the db project alone and workspaces-b no longer carries it", () => { + // The db project was 168-250s of the 436-513s workspaces-b vitest time + // (PR runs 38084171761, 38083084276 and 38083020834, 2026-10-10), so it + // runs in its own lane. The PR and release matrices both carry that lane. + const workspacesC = dryRunJson(["--mode", "general", "--group", "general-workspaces-c"]); + assert.deepEqual(workspacesC.workspaceProjects, ["@paperclipai/db"]); + assert.equal(workspacesC.workspacesVitestShard, null); + const workspacesB = dryRunJson(["--mode", "general", "--group", "general-workspaces-b"]); + assert.ok(!workspacesB.workspaceProjects.includes("@paperclipai/db"), "db must not run twice"); + assert.ok(workspacesB.workspaceProjects.length > 0, "workspaces-b must still run its other projects"); + for (const workflow of ["pr-trusted.yml", "release-verify.yml"]) { + const entries = matrixEntries(readFileSync(path.join(repoRoot, ".github", "workflows", workflow), "utf8")); + // The parser must demonstrably see shard keys in this workflow, otherwise + // the unsharded assertions below could pass for the wrong reason. + const sharded = entries.filter(entry => entry.group === "general-workspaces-a"); + assert.ok(sharded.length > 0 && sharded.every(entry => entry.keys.includes("shard_index") && entry.keys.includes("shard_count")), + `${workflow} must carry the sharded workspaces-a entries`); + for (const group of ["general-workspaces-b", "general-workspaces-c"]) { + const matching = entries.filter(entry => entry.group === group); + assert.equal(matching.length, 1, `${workflow} must run ${group} exactly once`); + assert.ok(matching[0].keys.includes("group_label"), `${workflow} must label the ${group} job`); + assert.deepEqual(matching[0].keys.filter(key => key.startsWith("shard_")), [], `${workflow} must run ${group} unsharded`); + } + } +}); + test("workspaces-a shards map to Vitest native --shard slices over a stable project list", () => { const shards = [0, 1].map((index) => dryRunJson([ diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjs index dd00f145cda..a62f90094e1 100644 --- a/scripts/run-vitest-stable.mjs +++ b/scripts/run-vitest-stable.mjs @@ -117,9 +117,27 @@ function excludedServerSuites(groupName) { } const generalWorkspacesAGroupName = "general-workspaces-a"; const generalWorkspacesBGroupName = "general-workspaces-b"; +const generalWorkspacesCGroupName = "general-workspaces-c"; const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"]; -const generalWorkspacesBProjects = nonServerProjects.filter((project) => !generalWorkspacesAProjects.includes(project)); -const generalGroupNames = [generalServerGroupName, generalWorkspacesAGroupName, generalWorkspacesBGroupName]; +// The db project runs in its own lane. Its vitest invocation took 168-250s +// of a 436-513s workspaces-b job (PR runs 38084171761, 38083084276 and +// 38083020834, 2026-10-10), which made workspaces-b the slowest check once +// the server shards were rebalanced. Vitest's native --shard would not help +// here: the project is already file-parallel and client.test.ts alone is +// 117-189s of its wall time, so a hash-ordered slice can land both of its +// slow suites on one runner. Splitting by project is deterministic and +// leaves the remaining projects at ~265-275s of vitest time, roughly level +// with this lane. +const generalWorkspacesCProjects = ["@paperclipai/db"]; +const generalWorkspacesBProjects = nonServerProjects.filter( + (project) => !generalWorkspacesAProjects.includes(project) && !generalWorkspacesCProjects.includes(project), +); +const generalGroupNames = [ + generalServerGroupName, + generalWorkspacesAGroupName, + generalWorkspacesBGroupName, + generalWorkspacesCGroupName, +]; const allowedGeneralGroupNames = [ ...generalGroupNames, generalServerWithoutChatGroupName, @@ -520,6 +538,11 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = return; } + if (groupName === generalWorkspacesCGroupName) { + runProjectGroup(generalWorkspacesCProjects, groupName); + return; + } + fail(`Unknown group "${groupName}".`); } @@ -604,7 +627,9 @@ if (options.dryRun) { ? generalWorkspacesAProjects : options.group === generalWorkspacesBGroupName ? generalWorkspacesBProjects - : null, + : options.group === generalWorkspacesCGroupName + ? generalWorkspacesCProjects + : null, workspacesVitestShard: options.group === generalWorkspacesAGroupName && options.shardCount !== null &&