Files
PaperClipAI/scripts/__tests__/run-vitest-stable-shard.test.mjs
T
DottaandClaude Opus 5.5 0d832a02d3 perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `PR` CI workflow before it can merge
> - The `General tests (server (N/12))` and `General tests (chat (N/3))`
jobs finish last, so they set the wall-clock time of CI
> - Before this change, server shards took 9.5 to 11.7 minutes and chat
shards took 6.4 to 9.6 minutes; the target is 7 minutes or less
> - Measurement shows that the chat lane was unbalanced and that the
server lane is limited by CPU on the 4-vCPU runners
> - This pull request balances the chat lane by recorded test time, runs
three chat partitions per job, and takes general server weights from PR
runs
> - The benefit is a chat lane under 7 minutes and measured evidence for
the shard count that the server lane needs

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The PR CI general server and chat test jobs. They are the slowest jobs
of every pull request.

**Subsystem affected**

CI test scheduling: `scripts/run-vitest-stable.mjs`,
`scripts/test-line-shard.mjs` and
`scripts/ci/refresh-test-durations.mjs`.

**Current behavior**

Baseline: green `PR` runs
[43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012)
and
[43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316),
all on m7i-flex.xlarge (4 vCPU, 16 GB).

- Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step,
effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports
`tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums
1,410 to 1,765 s of file time. The longest file is 303 s
(`deferred-execution-receipt-work.test.ts`), so the floor is total work
divided by workers, not one file.
- Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each
job ran two partitions of `chat-channels.integration.test.ts`. The
partitions were balanced by collected-case count, but one `it.each` line
can run many cases. One of six partitions recorded 532 s of test time
against a 377 s mean.

**Proposed behavior**

- Chat partitions keep each source line whole and balance by recorded
time per collected entry.
- Each chat job runs three partitions in place of two.
- General server weights come from the server jobs of merged PR runs,
which run files with the same concurrency as the lane they place.

**Reason and benefit**

The slowest job sets the CI wall-clock time for every pull request. The
chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes.

## What Changed

- `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional
recorded durations. With durations, it balances source lines by recorded
time; entries without a record take the median. Without durations, the
behavior is unchanged.
- `scripts/chat-shard-durations.json` (new): recorded time per collected
chat entry, from the CI profile records of PR run
[43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171).
926 of 928 collected entries are measured; the other two are the bogus
collection entries that the launcher already drops.
- `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that
manifest from downloaded `ci-profile-*` artifacts. Each executed case
maps to the collected `it`/`it.each` entry that registers it. A case
without an entry fails the refresh.
- `scripts/run-vitest-stable.mjs`: a sharded chat job runs three
partitions in place of two, and the chat launcher passes the recorded
durations. A chat job collects the suite once with `vitest list` in
place of once per partition, and the per-partition filter checks run at
the same time. Before, each partition ran two blocking collections of
the 86,000-line file one after another. The server worker count stays at
three.
- `scripts/ci/refresh-test-durations.mjs` and
`scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general
server weights from the two newest merged-PR runs. Each run must have
the full server shard count from `.github/ci-task-matrix.json` and cover
at least 90% of the lane. Serialized weights still come from master
Cloud readiness runs.
- `scripts/general-server-shard-durations.json`: refreshed with the new
source (PR runs 43439145752 and 43439279537).
- Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`,
`scripts/ci/test-selection-pr-timings.test.mjs` and
`server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted
partition, the entry mapping, the manifest format, the PR-run collection
rules and the three-partition chat launcher.
- `doc/DEVELOPING.md`: describes the three chat partitions, the chat
weights and how to refresh them, and the new source of the server
weights.

## Verification

Experiments ran in this PR's own CI with a temporary resource sampler
(removed before review).

| Setting (run) | Server step vs. same shard's baseline | CPU busy | Max
memory used | Result |
|---|---|---|---|---|
| 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) |
62-73% | 6.8 GB | 1 flaky retry |
| 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean |
| 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard
failure: `decision-models.test.ts` timed out at 15 s twice |
| 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB
| 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at
30 s on m5a |

The server lane is CPU bound: at four workers the CPU is 82-89% busy and
wall time improves by only about 10%. Five workers fail tests. Memory is
not the limit (no swap, no memory pressure, at least 6 GB free). Node's
compile cache and Postgres `fsync=off` showed no attributable gain
(import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to
0.2-0.7% of CPU time), so neither is in this PR.

Final configuration (server: 3 workers with refreshed weights; chat: 3
weighted partitions per job), job minutes / test-step seconds /
effective concurrency:

| Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503
attempt 1 | B: attempt 2 | C: attempt 3 |
|---|---|---|---|---|---|
| server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 /
2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 |
| server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85
| 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 |
| server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81
| 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 |
| server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 /
2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 |
| server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85
| 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 |
| server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69
| 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 |
| server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 /
2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 |
| server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83
| 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 |
| server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68
| 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 |
| server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84
| 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 |
| server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 /
2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 |
| server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 /
2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 |
| chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 |
5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 |
| chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 |
6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 |
| chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 |
5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 |

Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B /
C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server
worker setting is unchanged, so its run-to-run spread is mostly runner
type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min).

Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five
server shards on m5a.xlarge summed 2,545-3,037 s of file time against
about 1,700 s elsewhere; three reached the 20-minute job limit and two
failed tests on 15 s timeouts. The server lane settings in this PR are
the same as master's, so this is a fleet finding: m5a.xlarge cannot run
a server shard inside the job limit (follow-up for the fleet
configuration, outside this PR).

Flakiness of the final setting: the nine chat jobs in A, B and C had no
failure and no retried test. Server shards in A and B had no failure and
no retried test. Before the last commit (single chat collection), three
runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server
retries there (2 in one run, 1 in another) were on the unchanged
three-worker server setting.

Local checks (correctness only):

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs`: 68 passed.
- `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/vitest-chat-shards.test.ts`: passed.
- `pnpm verify` exits 2 (global CI input changed), so the required CI
checks supply the broad coverage.

## Risks

- Three chat partitions per job use more memory: the peak was 11.4 GB of
16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63%
busy.
- Stale chat weights only change balance, never which tests run. Shard
membership is still checked with `vitest list` before each run.
- The weekly refresh now downloads general server logs from merged-PR
runs. Runs that selected a subset of files are skipped before download
because their shard count is smaller.
- The server lane is still above 7 minutes. See the remaining floor
below.

**Remaining floor and shard-count recommendation (for a follow-up under
`.github/`)**

A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs
give at most about 2.8 effective concurrency, so the test step is about
560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test
step of about 360 s, so about 1,000 s of file time per shard. The
current 19,200 s of general server file time therefore needs about 19-20
shards of 4 vCPUs at 3 workers. More workers on the same runners do not
help, because the CPU is already the limit.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
GitHub CLI) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:03 -06:00

899 lines
49 KiB
JavaScript

import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { chmodSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import test from "node:test";
import {
defaultSuiteWeight,
loadShardDurations,
partitionGeneralServerSuites,
} from "../general-server-shard.mjs";
import { assertManifestSource } from "../ci/pr-run-durations.mjs";
import { assertSelectedTests, partitionTestLines } from "../test-line-shard.mjs";
import { chatManifestData, entryDurations } from "../ci/refresh-chat-shard-durations.mjs";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
const script = path.join(repoRoot, "scripts", "run-vitest-stable.mjs");
const durationsManifest = path.join(repoRoot, "scripts", "general-server-shard-durations.json");
const serializedDurationsManifest = path.join(
repoRoot,
"scripts",
"serialized-shard-durations.json",
);
// Membership of general-server-without-chat depends on GITHUB_WORKFLOW (see
// prWorkflowName in run-vitest-stable.mjs), so strip the ambient value and
// make every test pin the caller it mirrors explicitly — these tests
// themselves run inside a workflow on CI.
function workflowEnv(envOverrides = {}) {
const env = { ...process.env, ...envOverrides };
if (!("GITHUB_WORKFLOW" in envOverrides)) {
delete env.GITHUB_WORKFLOW;
}
return env;
}
function dryRun(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [script, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
return result;
}
function dryRunJson(args, envOverrides = {}) {
const result = dryRun(args, envOverrides);
assert.equal(result.status, 0, `expected success for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
const SHARD_COUNT = 12;
const SERIALIZED_SHARD_COUNT = 9;
test("the general-server shards form a complete, non-overlapping partition", () => {
const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
);
const total = shards[0].generalServerSuiteCount;
assert.ok(total > 0, "expected a non-empty general-server suite set");
const seen = new Set();
let selectedTotal = 0;
for (const shard of shards) {
assert.equal(shard.generalServerSuiteCount, total, "suite count must be stable across shards");
for (const file of shard.selectedGeneralServerSuites) {
assert.ok(
!(file.startsWith("server/src/__tests__/") && /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file)),
`route/authz suite must stay in the serialized lane, not general-server: ${file}`,
);
assert.ok(!seen.has(file), `suite assigned to more than one shard: ${file}`);
seen.add(file);
selectedTotal += 1;
}
}
// Every suite runs exactly once: union covers the whole set with no overlap.
assert.equal(selectedTotal, total, "every suite must be selected exactly once");
assert.equal(seen.size, total, "union of shards must cover the whole suite set");
});
test("general shards retain scripts and route-named suites outside the serialized directory", () => {
const general = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
for (const file of [
"server/scripts/verify-runner-vendor-dependencies.test.mjs",
"server/src/services/openrouter-models.test.ts",
"server/src/routes/setup-token-route.test.ts",
]) assert.ok(general.selectedGeneralServerSuites.includes(file), `missing configured server suite: ${file}`);
});
for (const group of ["general-workspaces-b", "general-workspaces-c"]) {
test(`shard flags are rejected for the ${group} group`, () => {
const result = dryRun(["--mode", "general", "--group", group, "--shard-index", "0", "--shard-count", "3"]);
assert.notEqual(result.status, 0, `${group} must not accept shard flags`);
});
}
test("workspace lanes cover every non-server project in the root Vitest configuration", () => {
const config = readFileSync(path.join(repoRoot, "vitest.config.ts"), "utf8");
const roots = [...config.matchAll(/^\s+"([^"]+)",?\s*$/gm)].map(match => match[1]);
assert.ok(roots.includes("server"), "expected the explicit root Vitest project list");
const expected = roots.filter(root => root !== "server")
.map(root => JSON.parse(readFileSync(path.join(repoRoot, root, "package.json"), "utf8")).name).sort();
const actual = ["general-workspaces-a", "general-workspaces-b", "general-workspaces-c"]
.flatMap(group => dryRunJson(["--mode", "general", "--group", group]).workspaceProjects).sort();
assert.deepEqual(actual, expected, "no configured project may be silently omitted or run twice");
});
// Collects every `- group:` matrix entry in a workflow with the keys it sets.
// Comment and blank lines are dropped first, and an entry ends only when the
// indentation returns to the item level, so a comment between two keys cannot
// hide a shard setting from the assertions below.
function matrixEntries(text) {
const entries = [];
let current = null;
for (const line of text.split("\n")) {
if (/^\s*(#|$)/.test(line)) continue;
const indent = line.match(/^ */)[0].length;
const item = line.match(/^ *- group: (\S+)/);
if (item) {
current = { group: item[1], indent, keys: ["group"] };
entries.push(current);
continue;
}
if (!current) continue;
if (indent <= current.indent) { current = null; continue; }
const key = line.match(/^ *([A-Za-z_]+):/);
if (key) current.keys.push(key[1]);
}
return entries;
}
test("matrixEntries sees shard keys that sit behind comments or later in the entry", () => {
const entries = matrixEntries([
" include:",
" - group: general-workspaces-b",
" group_label: workspaces-b",
" # a comment between keys must not hide the shard setting",
" shard_index: 0",
"",
" shard_count: 3",
" - group: general-workspaces-c",
" group_label: workspaces-c",
" steps:",
" - name: unrelated",
" shard_index: 9",
].join("\n"));
assert.deepEqual(entries.map(entry => entry.group), ["general-workspaces-b", "general-workspaces-c"]);
assert.deepEqual(entries[0].keys, ["group", "group_label", "shard_index", "shard_count"]);
assert.deepEqual(entries[1].keys, ["group", "group_label"], "keys after the matrix must not leak into the last entry");
});
test("workspaces-c runs the db project alone and workspaces-b no longer carries it", () => {
// The db project was 168-250s of the 436-513s workspaces-b vitest time
// (PR runs 38084171761, 38083084276 and 38083020834, 2026-10-10), so it
// runs in its own lane. The PR and release matrices both carry that lane.
const workspacesC = dryRunJson(["--mode", "general", "--group", "general-workspaces-c"]);
assert.deepEqual(workspacesC.workspaceProjects, ["@paperclipai/db"]);
assert.equal(workspacesC.workspacesVitestShard, null);
const workspacesB = dryRunJson(["--mode", "general", "--group", "general-workspaces-b"]);
assert.ok(!workspacesB.workspaceProjects.includes("@paperclipai/db"), "db must not run twice");
assert.ok(workspacesB.workspaceProjects.length > 0, "workspaces-b must still run its other projects");
for (const workflow of ["pr-trusted.yml", "release-verify.yml"]) {
const entries = workflow === "pr-trusted.yml"
? JSON.parse(readFileSync(path.join(repoRoot, ".github/ci-task-matrix.json"), "utf8")).general_tests
.map((entry) => ({ ...entry, keys: Object.keys(entry) }))
: matrixEntries(readFileSync(path.join(repoRoot, ".github", "workflows", workflow), "utf8"));
// The parser must demonstrably see shard keys in this workflow, otherwise
// the unsharded assertions below could pass for the wrong reason.
const sharded = entries.filter(entry => entry.group === "general-workspaces-a");
assert.ok(sharded.length > 0 && sharded.every(entry => entry.keys.includes("shard_index") && entry.keys.includes("shard_count")),
`${workflow} must carry the sharded workspaces-a entries`);
for (const group of ["general-workspaces-b", "general-workspaces-c"]) {
const matching = entries.filter(entry => entry.group === group);
assert.equal(matching.length, 1, `${workflow} must run ${group} exactly once`);
assert.ok(matching[0].keys.includes("group_label"), `${workflow} must label the ${group} job`);
assert.deepEqual(matching[0].keys.filter(key => key.startsWith("shard_")), [], `${workflow} must run ${group} unsharded`);
}
}
});
test("workspaces-a shards map to Vitest native --shard slices over a stable project list", () => {
const shards = [0, 1].map((index) =>
dryRunJson([
"--mode", "general", "--group", "general-workspaces-a",
"--shard-index", String(index), "--shard-count", "2",
]),
);
assert.deepEqual(
shards.map((shard) => shard.workspacesVitestShard),
["1/2", "2/2"],
"each matrix job must pass its own --shard slice to vitest",
);
// Vitest's --shard partitions each project's file list deterministically, so
// an identical project list across jobs is what guarantees complete,
// non-overlapping coverage of the lane.
assert.deepEqual(shards[0].workspaceProjects, shards[1].workspaceProjects);
assert.ok(shards[0].workspaceProjects.length > 0, "workspaces-a must run at least one project");
const unsharded = dryRunJson(["--mode", "general", "--group", "general-workspaces-a"]);
assert.deepEqual(
unsharded.workspaceProjects,
shards[0].workspaceProjects,
"sharding must not change which projects the lane covers",
);
assert.equal(unsharded.workspacesVitestShard, null);
});
test("duration-aware partition balances skewed weights better than round-robin", () => {
// Round-robin puts all three heavy suites on shard 0 (indexes 0, 3, 6).
const files = ["a", "b", "c", "d", "e", "f", "g", "h", "i"];
const durations = { a: 30000, d: 30000, g: 30000, b: 100, c: 100, e: 100, f: 100, h: 100, i: 100 };
const shards = partitionGeneralServerSuites(files, 3, durations);
const totals = shards.map((shard) => shard.totalWeight);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
assert.ok(
maxTotal - minTotal <= 200,
`expected near-even shard weights, got ${totals.join(", ")}`,
);
assert.equal(
shards.flatMap((shard) => shard.files).sort().join(","),
files.join(","),
"partition must cover every file exactly once",
);
});
test("the partition is deterministic for identical inputs", () => {
const files = Array.from({ length: 50 }, (_, index) => `suite-${index}.test.ts`);
const durations = Object.fromEntries(files.map((file, index) => [file, (index * 37) % 5000]));
const first = partitionGeneralServerSuites(files, 3, durations);
const second = partitionGeneralServerSuites(files, 3, durations);
assert.deepEqual(first, second, "same inputs must always produce the same partition");
});
test("suites missing from the manifest get the median weight", () => {
assert.equal(defaultSuiteWeight({ a: 100, b: 300, c: 900 }), 300);
assert.equal(defaultSuiteWeight({ a: 100, b: 300, c: 500, d: 900 }), 400);
assert.equal(defaultSuiteWeight({}), 1000, "empty manifest falls back to a fixed weight");
});
test("a missing or malformed manifest degrades to uniform weights", () => {
assert.deepEqual(loadShardDurations(path.join(repoRoot, "scripts", "no-such-manifest.json")), {});
const files = ["a", "b", "c", "d"];
const shards = partitionGeneralServerSuites(files, 2, {});
assert.equal(shards[0].files.length + shards[1].files.length, files.length);
assert.equal(Math.abs(shards[0].files.length - shards[1].files.length), 0);
});
test("the checked-in manifest loads and covers most of the current suite set", () => {
const durations = loadShardDurations(durationsManifest);
assert.ok(Object.keys(durations).length > 0, "manifest must parse to a non-empty duration map");
const shard = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
const currentFiles = shard.selectedGeneralServerSuites;
const known = currentFiles.filter((file) => durations[file] !== undefined).length;
assert.ok(
known / currentFiles.length >= 0.5,
`manifest is stale: only ${known} of ${currentFiles.length} suites have recorded durations — regenerate it from a recent PR run (see the manifest's $comment)`,
);
});
test("the chat integration suite keeps a measured duration for duration-aware fallbacks", () => {
// The PR and release matrices both run the chat suite in dedicated
// line-sharded lanes, but the plain general-server group (local full runs)
// still weighs it into the LPT partition; a median-fallback weight there
// would silently overload whichever shard receives it.
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
const durations = loadShardDurations(durationsManifest);
assert.ok(
Number.isFinite(durations[chatSuite]),
"the full chat cohort must have a measured duration, not the median fallback",
);
});
test("the checked-in serialized manifest loads and covers most of the current suite set", () => {
const durations = loadShardDurations(serializedDurationsManifest);
assert.ok(Object.keys(durations).length > 0, "manifest must parse to a non-empty duration map");
const shard = dryRunJson(["--mode", "serialized", "--shard-index", "0", "--shard-count", "1"]);
const currentFiles = shard.selectedSerializedSuites;
const known = currentFiles.filter((file) => durations[file] !== undefined).length;
assert.ok(
known / currentFiles.length >= 0.5,
`manifest is stale: only ${known} of ${currentFiles.length} suites have recorded durations — regenerate it from a recent PR run (see the manifest's $comment)`,
);
});
test("the real serialized shard partition is complete, non-overlapping and duration-balanced", () => {
const durations = loadShardDurations(serializedDurationsManifest);
const fallback = defaultSuiteWeight(durations);
const shards = Array.from({ length: SERIALIZED_SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "serialized", "--shard-index", String(index), "--shard-count", String(SERIALIZED_SHARD_COUNT)]),
);
const total = shards[0].serializedSuiteCount;
const selected = shards.flatMap((shard) => shard.selectedSerializedSuites);
assert.equal(selected.length, total, "every serialized suite must be selected exactly once");
assert.equal(new Set(selected).size, total, "serialized shards must not overlap");
const totals = shards.map((shard) =>
shard.selectedSerializedSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),
);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the bound.
const heaviest = Math.max(...Object.values(durations));
assert.ok(
maxTotal - minTotal <= heaviest,
`serialized shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
);
});
const chatSuitePath = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunnerSuitePath =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
const ingressRetrySuitePath = "server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts";
const usageDeliverySuitePath = "server/src/services/native-runtime/native-usage-delivery.integration.test.ts";
const warmCheckpointOwnershipSuitePath = "server/src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts";
const pluginWarmOwnerSuitePath = "server/src/services/native-runtime/plugin-runner-task-warm.integration.test.ts";
const pluginReleaseResumeSuitePath = "server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts";
const pluginEvictionSuitePath = "server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts";
const dotRunnerSuitePath = "server/src/__tests__/dot-runner.test.ts";
const restartRecoverySuitePath =
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts";
const cutTurnStartRetrySuitePath = "server/src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts";
const restartWaitSuites = ["lease-renewal", "preparation", "checkpoint-cost", "diskless"]
.map(name => `server/src/services/native-runtime/native-runner-restart-${name}.integration.test.ts`);
// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is
// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent
// native-runner suite in the Rust-cached final Verify Paperclip Runner vitest
// shard, so together the three cover the full server group exactly.
test("12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly", () => {
const durations = loadShardDurations(durationsManifest);
const fallback = defaultSuiteWeight(durations);
const prEnv = { GITHUB_WORKFLOW: "PR" };
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], prEnv);
const shards = Array.from({ length: 12 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "12",
], prEnv));
const totals = shards.map((shard) =>
shard.selectedGeneralServerSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),
);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the
// bound. The chat and native-runner suites run in their own lanes, so
// exclude them here.
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunner =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
const heaviest = Math.max(
...Object.entries(durations)
.filter(([file]) => file !== chat && file !== nativeRunner)
.map(([, ms]) => ms),
);
assert.ok(
maxTotal - minTotal <= heaviest,
`shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
);
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
assert.ok(!files.includes(chatSuitePath));
assert.ok(!files.includes(nativeRunnerSuitePath));
assert.ok(!files.includes(ingressRetrySuitePath));
assert.ok(!files.includes(dotRunnerSuitePath));
assert.ok(!files.includes(restartRecoverySuitePath));
assert.ok(!files.includes(cutTurnStartRetrySuitePath));
assert.ok(!files.includes(usageDeliverySuitePath));
assert.ok(!files.includes(warmCheckpointOwnershipSuitePath));
assert.ok(!files.includes(pluginWarmOwnerSuitePath));
assert.ok(!files.includes(pluginReleaseResumeSuitePath));
assert.ok(!files.includes(pluginEvictionSuitePath));
assert.deepEqual([...files, chatSuitePath, nativeRunnerSuitePath, ingressRetrySuitePath, dotRunnerSuitePath, restartRecoverySuitePath, ...restartWaitSuites, cutTurnStartRetrySuitePath, usageDeliverySuitePath, warmCheckpointOwnershipSuitePath, pluginWarmOwnerSuitePath, pluginReleaseResumeSuitePath, pluginEvictionSuitePath, "server/src/services/native-runtime/plugin-runner-probe.test.ts"].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([], prEnv);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
// The legacy group and local default must remain complete when a caller has
// not selected a separate native lane.
for (const [caller, envOverrides] of [["Release", { GITHUB_WORKFLOW: "Release" }], ["no ambient workflow", {}]]) {
test(`10 without-chat shards under ${caller} keep the native-runner suite and cover the server group with chat alone`, () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], envOverrides);
const shards = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "10",
], envOverrides));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
assert.ok(!files.includes(chatSuitePath));
assert.ok(files.includes(nativeRunnerSuitePath));
assert.ok(files.includes(dotRunnerSuitePath));
assert.ok(files.includes(restartRecoverySuitePath));
assert.ok(files.includes(warmCheckpointOwnershipSuitePath));
assert.ok(files.includes(pluginWarmOwnerSuitePath));
assert.ok(files.includes(pluginReleaseResumeSuitePath));
assert.ok(files.includes(pluginEvictionSuitePath));
assert.deepEqual([...files, chatSuitePath].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
});
}
// Release verification selects its dedicated lane explicitly. Do not infer it
// from the caller name: previews and future workflow callers need the same cover.
for (const caller of ["Release", "Cloud readiness", "another caller"]) {
test(`release server, chat, and native lanes cover every server suite once under ${caller}`, () => {
const env = { GITHUB_WORKFLOW: caller };
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], env);
const files = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat-or-native-runner",
"--shard-index", String(index), "--shard-count", "10",
], env)).flatMap((shard) => shard.selectedGeneralServerSuites);
const native = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"], env);
const combined = [...files, chatSuitePath, ...native.selectedGeneralServerSuites];
assert.equal(new Set(combined).size, combined.length, "lanes must not overlap");
assert.deepEqual(combined.sort(), full.selectedGeneralServerSuites.sort());
});
}
test("the native-runner lane runs exactly the cargo-dependent suites", () => {
const lane = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"]);
assert.deepEqual(lane.selectedGeneralServerSuites, [
"server/src/services/native-runtime/plugin-runner-probe.test.ts",
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
ingressRetrySuitePath,
dotRunnerSuitePath,
restartRecoverySuitePath,
...restartWaitSuites,
cutTurnStartRetrySuitePath,
usageDeliverySuitePath,
warmCheckpointOwnershipSuitePath,
pluginWarmOwnerSuitePath,
pluginReleaseResumeSuitePath,
pluginEvictionSuitePath,
]);
});
test("shard flags are rejected for the native-runner group", () => {
const result = dryRun(["--mode", "general", "--group", "general-server-native-runner", "--shard-index", "0", "--shard-count", "2"]);
assert.notEqual(result.status, 0, "the native-runner lane is a single suite and must not accept shard flags");
});
// The PR-side exclusion above is safe only while the wiring it assumes holds:
// pr.yml (the caller whose name reusable pr-trusted.yml jobs see as
// GITHUB_WORKFLOW) is named PR, the package vitest lanes cover every
// partition exactly once, exactly one lane runs the native-runner group, and
// those lanes' package script routes through the wrapper.
test("the PR workflow wiring for the native-runner lane holds", () => {
const prWorkflow = readFileSync(path.join(repoRoot, ".github/workflows/pr.yml"), "utf8");
assert.match(prWorkflow, /^name: PR$/m,
"renaming pr.yml silently moves the native-runner suite back into the uncached server shards");
const matrix = JSON.parse(readFileSync(path.join(repoRoot, ".github/ci-task-matrix.json"), "utf8"));
const vitestLanes = matrix.verify_paperclip_runner.filter((entry) => entry.command.startsWith("test:typescript:vitest"));
for (const entry of vitestLanes) {
assert.match(entry.command, /^test:typescript:vitest (--package-lane=\d+\/\d+|--native-server)$/, entry.lane_label);
assert.equal(entry.task, "ci:runner-vitest", entry.lane_label);
}
const lanes = vitestLanes
.map((entry) => /--package-lane=(\d+)\/(\d+)$/.exec(entry.command))
.filter(Boolean).map((match) => [Number(match[1]), Number(match[2])]);
assert.ok(lanes.length > 0, "expected package-lane test:typescript:vitest lanes in the task matrix");
assert.equal(new Set(lanes.map(([, count]) => count)).size, 1, "package lanes must agree on the lane count");
const laneCount = lanes[0][1];
assert.deepEqual(
lanes.map(([index]) => index).sort((left, right) => left - right),
Array.from({ length: laneCount }, (_, index) => index + 1),
"package lanes must cover every partition exactly once",
);
const native = vitestLanes.filter((entry) => entry.command.endsWith("--native-server"));
assert.equal(native.length, 1, "exactly one vitest lane carries the native-runner group");
assert.equal(native[0].release_crate, true, "the native-runner lane needs the release crate");
const runnerPackage = JSON.parse(
readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8"),
);
assert.equal(runnerPackage.scripts["test:typescript:vitest"], "node ./scripts/run-pr-vitest-lane.mjs");
});
const laneWrapper = path.join(repoRoot, "packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs");
function wrapperPlan(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [laneWrapper, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
assert.equal(result.status, 0, `expected wrapper dry run to succeed for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
test("the PR lanes separate complete package and native workloads without changing local shards", () => {
for (const [args, envOverrides, packageArgs, native] of [
[["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }, [], false],
[["--shard=2/2"], { GITHUB_WORKFLOW: "PR" }, null, true],
[[], { GITHUB_WORKFLOW: "PR" }, [], true],
[["--shard=2/3"], { GITHUB_WORKFLOW: "PR" }, ["--shard=2/3"], false],
[["--shard=3/3"], { GITHUB_WORKFLOW: "PR" }, ["--shard=3/3"], true],
[["--shard=2/2"], { GITHUB_WORKFLOW: "Release" }, ["--shard=2/2"], false],
[["--shard=2/2"], {}, ["--shard=2/2"], false],
[[], {}, [], false],
]) {
const plan = wrapperPlan(args, envOverrides);
assert.equal(plan.runNativeRunnerGroup, native);
assert.equal(plan.runPackageTests, packageArgs !== null);
const vitest = plan.plannedCommands.find(command => command.args[0] === "exec");
assert.deepEqual(vitest?.args ?? null, packageArgs === null ? null : ["exec", "vitest", "run", ...packageArgs]);
assert.equal(plan.plannedCommands.some(command => command.args.includes("general-server-native-runner")), native);
assert.ok(plan.plannedCommands.some(command => command.args.includes("build:rust")), "both workloads require debug fixtures");
}
const malformed = spawnSync(process.execPath, [laneWrapper, "--shard=nonsense", "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv({ GITHUB_WORKFLOW: "PR" }),
});
assert.notEqual(malformed.status, 0, "a malformed shard flag must fail rather than guess a lane");
});
const runnerRoot = path.join(repoRoot, "packages/paperclip-runner");
const prepCommands = [
{ command: "pnpm", args: ["run", "ensure:eval-build-deps"], cwd: runnerRoot },
{ command: "pnpm", args: ["run", "build:rust"], cwd: runnerRoot },
];
const nativeCommand = {
command: "pnpm",
args: ["test:run:general", "--", "--group", "general-server-native-runner"],
cwd: repoRoot,
};
// Master's trusted workflow keeps sending the --shard commands until the
// package-lane matrix merges, so their exact plans must not move.
test("the master-pinned --shard lane commands plan exactly what they did before package lanes", () => {
assert.deepEqual(wrapperPlan(["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }).plannedCommands, [
...prepCommands,
{ command: "pnpm", args: ["exec", "vitest", "run"], cwd: runnerRoot },
]);
assert.deepEqual(wrapperPlan(["--shard=2/2"], { GITHUB_WORKFLOW: "PR" }).plannedCommands, [...prepCommands, nativeCommand]);
for (const plan of [wrapperPlan(["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }), wrapperPlan(["--shard=2/2"], { GITHUB_WORKFLOW: "PR" })]) {
assert.equal(plan.packageLane, undefined);
}
});
test("--native-server runs only the native group under any workflow", () => {
for (const envOverrides of [{ GITHUB_WORKFLOW: "PR" }, {}]) {
const plan = wrapperPlan(["--native-server"], envOverrides);
assert.deepEqual(plan.plannedCommands, [...prepCommands, nativeCommand]);
assert.equal(plan.runPackageTests, false);
assert.equal(plan.runNativeRunnerGroup, true);
}
});
test("the package lane manifest uses the general server duration schema", () => {
const manifest = JSON.parse(readFileSync(path.join(runnerRoot, "scripts/vitest-lane-durations.json"), "utf8"));
const general = JSON.parse(readFileSync(durationsManifest, "utf8"));
// The refresh script writes exactly these keys; hand-added annotations such
// as the server manifest's splitEstimates are optional.
assert.deepEqual(Object.keys(manifest), ["$comment", "unit", "sources", "coverage", "durations"]);
assert.ok(Object.keys(manifest).every((key) => key in general));
assert.equal(manifest.unit, "ms");
assert.deepEqual(Object.keys(manifest.coverage), Object.keys(general.coverage));
// The weekly refresh samples merged-PR runs for this lane, so its sources
// may lead with a pull number. test-selection-pr-timings.test.mjs passes the
// collector's real output through the same check.
for (const source of manifest.sources) assertManifestSource(source);
const entries = Object.entries(manifest.durations);
assert.ok(entries.length > 100);
for (const [file, ms] of entries) {
assert.match(file, /^packages\/paperclip-runner\/.+\.test\.[cm]?[jt]sx?$/);
assert.ok(Number.isFinite(ms) && ms >= 0, file);
}
});
test("package lane arguments fail rather than guess a lane", () => {
for (const args of [["--package-lane=0/3"], ["--package-lane=4/3"], ["--package-lane=abc"], ["--package-lane", "1/3"],
["--package-lane=1/3", "--package-lane=2/3"], ["--package-lane=1/3", "--shard=1/2"], ["--native-server", "--shard=2/2"],
["--native-server", "--package-lane=1/3"], ["--native-server", "src/foo.test.ts"]]) {
const result = spawnSync(process.execPath, [laneWrapper, ...args, "--dry-run"], {
cwd: repoRoot, encoding: "utf8", env: workflowEnv({ GITHUB_WORKFLOW: "PR" }),
});
assert.notEqual(result.status, 0, args.join(" "));
}
});
const lineShardFile = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const caseAt = (line, name) => ({ name, file: lineShardFile, projectName: "@paperclipai/server", location: { line, column: 3 } });
test("test-line shards cover nested and parameterized cases exactly once without splitting a source line", () => {
const cases = [caseAt(10, "suite > nested > first"), caseAt(10, "suite > nested > second"),
caseAt(20, "same name"), caseAt(30, "same name"), caseAt(40, "last"), caseAt(50, "new case")];
const shards = partitionTestLines(cases, 3, lineShardFile);
assert.deepEqual(shards.map((shard) => shard.tests.length), [2, 2, 2]);
assert.equal(shards.filter((shard) => shard.lines.includes(10)).length, 1);
assert.equal(shards.find((shard) => shard.lines.includes(10)).tests.length, 2);
assert.equal(shards.flatMap((shard) => shard.lines).length, 5);
assert.deepEqual(shards.flatMap((shard) => shard.tests).sort((a, b) => a.location.line - b.location.line), cases);
assert.deepEqual(partitionTestLines([...cases].reverse(), 3, lineShardFile).map((shard) => shard.lines), shards.map((shard) => shard.lines));
});
test("recorded durations balance chat line shards by time without splitting a source line", () => {
const cases = [caseAt(10, "slow"), caseAt(20, "each %s"), caseAt(20, "each %s"), caseAt(30, "a"), caseAt(40, "b"),
caseAt(50, "c"), caseAt(60, "unrecorded")];
const durations = { slow: 9000, "each %s": 1000, a: 3000, b: 3000, c: 2000 };
const shards = partitionTestLines(cases, 2, lineShardFile, durations);
assert.deepEqual(shards.map((shard) => shard.weight), [11000, 11000]);
assert.deepEqual(shards.map((shard) => shard.lines), [[10, 20], [30, 40, 50, 60]]);
assert.deepEqual(shards.flatMap((shard) => shard.tests).sort((a, b) => a.location.line - b.location.line), cases);
assert.deepEqual(partitionTestLines([...cases].reverse(), 2, lineShardFile, durations).map((shard) => shard.lines),
shards.map((shard) => shard.lines));
assert.deepEqual(partitionTestLines(cases, 2, lineShardFile).map((shard) => shard.tests.length), [4, 3]);
});
test("chat durations attribute every executed case to the collected entry that registers it", () => {
const entries = ["sends %s", "suite > sends %s twice", "plain", "keyed $provider ($mode)"];
const { totals, unmatched } = entryDurations(entries, [
{ name: "chat channel control-plane integration > sends a", duration: 5 },
{ name: "chat channel control-plane integration > suite > sends b twice", duration: 7 },
{ name: "chat channel control-plane integration > plain", duration: 11 },
{ name: "chat channel control-plane integration > keyed slack (fast)", duration: 13 },
{ name: "chat channel control-plane integration > keyed github (slow)", duration: 17 },
{ name: "chat channel control-plane integration > unknown", duration: 19 },
]);
assert.deepEqual(Object.fromEntries(totals), { "sends %s": 5, "suite > sends %s twice": 7, plain: 11, "keyed $provider ($mode)": 30 });
assert.deepEqual(unmatched, ["chat channel control-plane integration > unknown"]);
const data = chatManifestData(["plain", "other"], [[{ name: "x > plain", duration: 4 }], [{ name: "x > plain", duration: 9 }],
[{ name: "x > plain", duration: 6 }]], []);
assert.deepEqual(data.durations, { plain: 6 });
assert.deepEqual(data.coverage, { measured: 1, entries: 2 });
assert.throws(() => chatManifestData(["plain"], [[{ name: "x > missing", duration: 1 }]], []));
});
test("the checked-in chat duration manifest is well formed", () => {
const manifest = JSON.parse(readFileSync(path.join(repoRoot, "scripts", "chat-shard-durations.json"), "utf8"));
assert.equal(manifest.unit, "ms");
const entries = Object.entries(manifest.durations);
assert.ok(entries.length > 0);
assert.ok(entries.every(([name, ms]) => name.length > 0 && Number.isSafeInteger(ms) && ms > 0));
assert.deepEqual(loadShardDurations(path.join(repoRoot, "scripts", "chat-shard-durations.json")), manifest.durations);
});
test("line-shard collection rejects empty, foreign, or unlocated tests and invalid shard counts", () => {
const good = caseAt(10, "valid");
for (const input of [[], null, [{ ...good, file: "/another.test.ts" }], [{ ...good, projectName: "wrong" }],
[{ ...good, location: undefined }], [{ ...good, location: { line: 0 } }], [{ ...good, name: "" }]]) {
assert.throws(() => partitionTestLines(input, 1, lineShardFile));
}
for (const count of [0, -1, 1.5, Infinity, 2]) assert.throws(() => partitionTestLines([good], count, lineShardFile));
});
test("filtered collection must match the exact assigned case identities, including duplicates", () => {
const expected = [caseAt(10, "same"), caseAt(10, "same"), caseAt(20, "nested > case")];
assertSelectedTests(expected, [...expected].reverse(), lineShardFile);
for (const actual of [expected.slice(1), [...expected, caseAt(30, "extra")],
[expected[0], expected[1], caseAt(20, "renamed")],
[expected[0], expected[1], caseAt(21, "nested > case")]]) {
assert.throws(() => assertSelectedTests(expected, actual, lineShardFile));
}
});
test("parallel chat jobs form a complete disjoint partition", () => {
for (const count of [2, 3, 4]) {
const shards = Array.from({ length: count }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", String(count)],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards,
);
assert.ok(shards.every((workers) => workers.length === 3));
assert.deepEqual(shards.flat().map(({ index }) => index).sort((a, b) => a - b),
Array.from({ length: count * 3 }, (_, index) => index));
assert.ok(shards.flat().every((shard) => shard.count === count * 3));
}
});
test("shared-database and unsharded chat runs retain one serial partition", () => {
for (let index = 0; index < 3; index++) {
const result = dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", "3"],
{ PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" });
assert.deepEqual(result.chatVitestShards, [{ index, count: 3 }]);
}
assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);
});
test("default chat isolation covers six partitions without dropping native suites", () => {
for (const args of [[], ["--mode", "general"]]) {
for (const workflow of ["PR", "Release", ""]) {
const env = { GITHUB_WORKFLOW: workflow, PAPERCLIP_TEST_DATABASE_URL: "" };
const plan = dryRunJson(args, env);
const full = dryRunJson(["--mode", "general", "--group", "general-server",
"--shard-index", "0", "--shard-count", "1"], env);
assert.deepEqual(plan.chatVitestShards, Array.from({ length: 6 }, (_, index) => ({ index, count: 6 })));
assert.ok(!plan.selectedGeneralServerSuites.includes(chatSuitePath));
assert.equal(new Set(plan.selectedGeneralServerSuites).size, plan.selectedGeneralServerSuites.length);
assert.deepEqual([...plan.selectedGeneralServerSuites, chatSuitePath].sort(), full.selectedGeneralServerSuites.sort());
}
}
});
test("default chat isolation leaves explicit groups and shared databases unchanged", () => {
for (const args of [[], ["--mode", "general"]]) {
const shared = dryRunJson(args, { PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test", GITHUB_WORKFLOW: "PR" });
assert.equal(shared.chatVitestShards, null);
assert.equal(shared.selectedGeneralServerSuites, null);
}
const full = dryRunJson(["--mode", "general", "--group", "general-server",
"--shard-index", "0", "--shard-count", "1"], { PAPERCLIP_TEST_DATABASE_URL: "", GITHUB_WORKFLOW: "PR" });
assert.equal(full.chatVitestShards, null);
assert.ok(full.selectedGeneralServerSuites.includes(chatSuitePath));
assert.ok(full.selectedGeneralServerSuites.includes(nativeRunnerSuitePath));
assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);
assert.equal(dryRunJson(["--mode", "serialized"], { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, null);
});
function defaultLauncher({ shared = false, failureIndex = null } = {}) {
const directory = mkdtempSync(path.join(os.tmpdir(), "default-chat-launch-"));
const file = path.join(repoRoot, chatSuitePath);
const lines = readFileSync(file, "utf8").split("\n")
.flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 6);
assert.equal(lines.length, 6);
const cases = lines.map(line => ({ file, projectName: "@paperclipai/server",
name: `default launcher fixture ${line}`, location: { line } }));
const receipt = path.join(directory, "events.ndjson");
const active = path.join(directory, "active");
const fakePnpm = path.join(directory, "pnpm");
try {
writeFileSync(fakePnpm, `#!${process.execPath}
const fs = require('node:fs');
const args = process.argv.slice(2);
const tests = ${JSON.stringify(cases)};
const selected = args.flatMap(arg => {
const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);
return match ? [Number(match[1])] : [];
});
if (args.includes('list')) {
const output = args.find(arg => arg.startsWith('--json=')).slice(7);
fs.writeFileSync(output, JSON.stringify(selected.length
? tests.filter(test => selected.includes(test.location.line)) : tests));
} else {
const entry = { args, selected, home: process.env.PAPERCLIP_HOME,
config: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID,
tmpdir: process.env.TMPDIR };
const record = edge => fs.appendFileSync(${JSON.stringify(receipt)}, JSON.stringify({ ...entry, edge }) + '\\n');
record('begin');
if (fs.existsSync(${JSON.stringify(active)})) process.exit(91);
fs.writeFileSync(${JSON.stringify(active)}, 'running');
setTimeout(() => {
record('end');
fs.unlinkSync(${JSON.stringify(active)});
process.exit(selected.includes(${JSON.stringify(failureIndex === null ? null : lines[failureIndex])}) ? 7 : 0);
}, selected.length ? 200 : 0);
}
`);
chmodSync(fakePnpm, 0o755);
const result = spawnSync(process.execPath, [script, "--mode", "general"], {
cwd: repoRoot, encoding: "utf8", timeout: 30_000,
env: workflowEnv({ GITHUB_WORKFLOW: "PR", PATH: `${directory}${path.delimiter}${process.env.PATH}`,
PAPERCLIP_TEST_DATABASE_URL: shared ? "postgresql://shared/test" : "", PAPERCLIP_CI_PROFILE: "false" }),
});
const events = existsSync(receipt) ? readFileSync(receipt, "utf8").trim().split("\n").map(line => JSON.parse(line)) : [];
return { result, events, lines };
} finally {
rmSync(directory, { recursive: true, force: true });
}
}
test("default chat isolation executes every partition sequentially before other groups", { skip: process.platform === "win32" }, () => {
const { result, events, lines } = defaultLauncher();
assert.equal(result.status, 0, result.stdout + result.stderr);
const chat = events.filter(event => event.selected.length > 0);
assert.deepEqual(chat.map(event => event.edge), Array.from({ length: 6 }, () => ["begin", "end"]).flat());
const starts = chat.filter(event => event.edge === "begin");
assert.deepEqual(starts.flatMap(event => event.selected).sort((a, b) => a - b), lines);
for (const event of starts) {
assert.ok(event.args.includes("--no-file-parallelism"));
assert.deepEqual(event.args.filter(arg => arg.startsWith("--maxWorkers")), ["--maxWorkers=1"]);
}
for (const field of ["home", "config", "instance", "tmpdir"]) {
assert.equal(new Set(starts.map(event => event[field])).size, 6, `${field} must be isolated per child`);
}
assert.equal(events.findIndex(event => event.selected.length === 0), 12);
const server = events[12];
assert.ok(server.args.includes("@paperclipai/server"));
const excluded = server.args.flatMap((arg, index) => arg === "--exclude" ? [server.args[index + 1]] : []);
assert.equal(excluded.filter(file => file === chatSuitePath.replace(/^server\//, "")).length, 1);
for (const file of [nativeRunnerSuitePath, ingressRetrySuitePath, usageDeliverySuitePath, dotRunnerSuitePath, restartRecoverySuitePath,
"server/src/services/native-runtime/plugin-runner-probe.test.ts"]) {
assert.ok(!excluded.includes(file.replace(/^server\//, "")), `default must retain ${file}`);
}
});
test("default chat isolation propagates failure without starting another partition or group", { skip: process.platform === "win32" }, () => {
const { result, events } = defaultLauncher({ failureIndex: 2 });
assert.equal(result.status, 1, result.stdout + result.stderr);
assert.match(result.stderr, /failed \(7\)/);
assert.equal(events.length, 6);
assert.ok(events.every(event => event.selected.length === 1));
assert.deepEqual(events.map(event => event.edge), ["begin", "end", "begin", "end", "begin", "end"]);
});
test("default chat isolation keeps shared-database execution monolithic and serial", { skip: process.platform === "win32" }, () => {
const { result, events } = defaultLauncher({ shared: true });
assert.equal(result.status, 0, result.stdout + result.stderr);
assert.ok(events.every(event => event.selected.length === 0));
assert.ok(events[0].args.includes("@paperclipai/server"));
assert.ok(events[0].args.includes("--no-file-parallelism"));
assert.deepEqual(events[0].args.filter(arg => arg.startsWith("--maxWorkers")), ["--maxWorkers=1"]);
const excluded = events[0].args.flatMap((arg, index) => arg === "--exclude" ? [events[0].args[index + 1]] : []);
assert.ok(!excluded.includes(chatSuitePath.replace(/^server\//, "")));
assert.ok(!excluded.includes(nativeRunnerSuitePath.replace(/^server\//, "")));
assert.deepEqual(events.map(event => event.edge), Array.from({ length: events.length / 2 }, () => ["begin", "end"]).flat());
});
test("a failed chat child fails the job after its sibling finishes", { skip: process.platform === "win32" }, () => {
const directory = mkdtempSync(path.join(os.tmpdir(), "chat-launch-proof-"));
const file = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const lines = readFileSync(file, "utf8").split("\n")
.flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 9);
assert.equal(lines.length, 9);
const tests = lines.map((line) => ({ file, projectName: "@paperclipai/server",
name: `launcher fixture ${line}`, location: { line } }));
const receipt = path.join(directory, "sibling-finished");
const fakePnpm = path.join(directory, "pnpm");
try {
writeFileSync(fakePnpm, `#!${process.execPath}
const fs = require('node:fs');
const args = process.argv.slice(2);
const tests = ${JSON.stringify(tests)};
const selected = args.flatMap(arg => {
const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);
return match ? [Number(match[1])] : [];
});
if (args.includes('list')) {
const output = args.find(arg => arg.startsWith('--json=')).slice(7);
fs.writeFileSync(output, JSON.stringify(selected.length
? tests.filter(test => selected.includes(test.location.line)) : tests));
} else if (selected.includes(${lines[0]})) process.exit(7);
else setTimeout(() => fs.writeFileSync(${JSON.stringify(receipt)}, 'finished'), 200);
`);
chmodSync(fakePnpm, 0o755);
const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-chat",
"--shard-index", "0", "--shard-count", "3"], {
cwd: repoRoot, encoding: "utf8", timeout: 30_000,
env: { ...process.env, PATH: `${directory}${path.delimiter}${process.env.PATH}`,
PAPERCLIP_TEST_DATABASE_URL: "", PAPERCLIP_CI_PROFILE: "false" },
});
assert.equal(result.status, 1, result.stdout + result.stderr);
assert.match(result.stderr, /failed \(7\)/);
assert.equal(readFileSync(receipt, "utf8"), "finished", "a failed worker must not abandon its sibling");
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
for (const [group, parallelWorkers, shardArgs] of [
["general-server-native-runner", 4, []],
["general-server-without-chat-or-native-runner", 3, ["--shard-index", "0", "--shard-count", "12"]],
]) test(`${group} files overlap only when their database fixtures are isolated`, { skip: process.platform === "win32" }, () => {
const directory = mkdtempSync(path.join(os.tmpdir(), "native-file-workers-"));
const receipt = path.join(directory, "args.json");
const fakePnpm = path.join(directory, "pnpm");
writeFileSync(fakePnpm, `#!/usr/bin/env node\nrequire('node:fs').writeFileSync(${JSON.stringify(receipt)}, JSON.stringify(process.argv.slice(2)));\n`);
chmodSync(fakePnpm, 0o755);
try {
for (const [databaseEnv, workers] of [
[{}, parallelWorkers],
[{ PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" }, 1],
[{ PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL: "postgresql://shared/recovery_test" }, 1],
[{ PAPERCLIP_TEST_POSTGRES_ADMIN_URL: "postgresql://owned/postgres" }, parallelWorkers],
]) {
const env = workflowEnv({ PATH: `${directory}${path.delimiter}${process.env.PATH}`, PAPERCLIP_CI_PROFILE: "false" });
for (const key of Object.keys(env)) if (key.endsWith("_TEST_DATABASE_URL")) delete env[key];
Object.assign(env, databaseEnv);
const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", group, ...shardArgs], {
cwd: repoRoot, encoding: "utf8", env,
});
assert.equal(result.status, 0, result.stdout + result.stderr);
const args = JSON.parse(readFileSync(receipt, "utf8"));
assert.ok(args.includes(`--maxWorkers=${workers}`));
assert.ok(args.includes(workers > 1 ? "--fileParallelism" : "--no-file-parallelism"));
if (workers > 1) assert.ok(args.includes("--pool=forks") && args.includes("--isolate"));
if (group === "general-server-native-runner") {
for (const file of [restartRecoverySuitePath, ...restartWaitSuites]) assert.ok(args.includes(file));
}
}
} finally { rmSync(directory, { recursive: true, force: true }); }
});