Files
PaperClipAI/scripts/ci/test-selection-pr-timings.test.mjs
T
DottaandClaude Opus 5.5 0d832a02d3 perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `PR` CI workflow before it can merge
> - The `General tests (server (N/12))` and `General tests (chat (N/3))`
jobs finish last, so they set the wall-clock time of CI
> - Before this change, server shards took 9.5 to 11.7 minutes and chat
shards took 6.4 to 9.6 minutes; the target is 7 minutes or less
> - Measurement shows that the chat lane was unbalanced and that the
server lane is limited by CPU on the 4-vCPU runners
> - This pull request balances the chat lane by recorded test time, runs
three chat partitions per job, and takes general server weights from PR
runs
> - The benefit is a chat lane under 7 minutes and measured evidence for
the shard count that the server lane needs

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The PR CI general server and chat test jobs. They are the slowest jobs
of every pull request.

**Subsystem affected**

CI test scheduling: `scripts/run-vitest-stable.mjs`,
`scripts/test-line-shard.mjs` and
`scripts/ci/refresh-test-durations.mjs`.

**Current behavior**

Baseline: green `PR` runs
[43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012)
and
[43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316),
all on m7i-flex.xlarge (4 vCPU, 16 GB).

- Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step,
effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports
`tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums
1,410 to 1,765 s of file time. The longest file is 303 s
(`deferred-execution-receipt-work.test.ts`), so the floor is total work
divided by workers, not one file.
- Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each
job ran two partitions of `chat-channels.integration.test.ts`. The
partitions were balanced by collected-case count, but one `it.each` line
can run many cases. One of six partitions recorded 532 s of test time
against a 377 s mean.

**Proposed behavior**

- Chat partitions keep each source line whole and balance by recorded
time per collected entry.
- Each chat job runs three partitions in place of two.
- General server weights come from the server jobs of merged PR runs,
which run files with the same concurrency as the lane they place.

**Reason and benefit**

The slowest job sets the CI wall-clock time for every pull request. The
chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes.

## What Changed

- `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional
recorded durations. With durations, it balances source lines by recorded
time; entries without a record take the median. Without durations, the
behavior is unchanged.
- `scripts/chat-shard-durations.json` (new): recorded time per collected
chat entry, from the CI profile records of PR run
[43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171).
926 of 928 collected entries are measured; the other two are the bogus
collection entries that the launcher already drops.
- `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that
manifest from downloaded `ci-profile-*` artifacts. Each executed case
maps to the collected `it`/`it.each` entry that registers it. A case
without an entry fails the refresh.
- `scripts/run-vitest-stable.mjs`: a sharded chat job runs three
partitions in place of two, and the chat launcher passes the recorded
durations. A chat job collects the suite once with `vitest list` in
place of once per partition, and the per-partition filter checks run at
the same time. Before, each partition ran two blocking collections of
the 86,000-line file one after another. The server worker count stays at
three.
- `scripts/ci/refresh-test-durations.mjs` and
`scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general
server weights from the two newest merged-PR runs. Each run must have
the full server shard count from `.github/ci-task-matrix.json` and cover
at least 90% of the lane. Serialized weights still come from master
Cloud readiness runs.
- `scripts/general-server-shard-durations.json`: refreshed with the new
source (PR runs 43439145752 and 43439279537).
- Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`,
`scripts/ci/test-selection-pr-timings.test.mjs` and
`server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted
partition, the entry mapping, the manifest format, the PR-run collection
rules and the three-partition chat launcher.
- `doc/DEVELOPING.md`: describes the three chat partitions, the chat
weights and how to refresh them, and the new source of the server
weights.

## Verification

Experiments ran in this PR's own CI with a temporary resource sampler
(removed before review).

| Setting (run) | Server step vs. same shard's baseline | CPU busy | Max
memory used | Result |
|---|---|---|---|---|
| 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) |
62-73% | 6.8 GB | 1 flaky retry |
| 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean |
| 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard
failure: `decision-models.test.ts` timed out at 15 s twice |
| 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB
| 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at
30 s on m5a |

The server lane is CPU bound: at four workers the CPU is 82-89% busy and
wall time improves by only about 10%. Five workers fail tests. Memory is
not the limit (no swap, no memory pressure, at least 6 GB free). Node's
compile cache and Postgres `fsync=off` showed no attributable gain
(import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to
0.2-0.7% of CPU time), so neither is in this PR.

Final configuration (server: 3 workers with refreshed weights; chat: 3
weighted partitions per job), job minutes / test-step seconds /
effective concurrency:

| Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503
attempt 1 | B: attempt 2 | C: attempt 3 |
|---|---|---|---|---|---|
| server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 /
2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 |
| server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85
| 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 |
| server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81
| 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 |
| server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 /
2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 |
| server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85
| 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 |
| server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69
| 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 |
| server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 /
2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 |
| server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83
| 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 |
| server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68
| 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 |
| server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84
| 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 |
| server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 /
2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 |
| server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 /
2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 |
| chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 |
5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 |
| chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 |
6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 |
| chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 |
5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 |

Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B /
C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server
worker setting is unchanged, so its run-to-run spread is mostly runner
type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min).

Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five
server shards on m5a.xlarge summed 2,545-3,037 s of file time against
about 1,700 s elsewhere; three reached the 20-minute job limit and two
failed tests on 15 s timeouts. The server lane settings in this PR are
the same as master's, so this is a fleet finding: m5a.xlarge cannot run
a server shard inside the job limit (follow-up for the fleet
configuration, outside this PR).

Flakiness of the final setting: the nine chat jobs in A, B and C had no
failure and no retried test. Server shards in A and B had no failure and
no retried test. Before the last commit (single chat collection), three
runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server
retries there (2 in one run, 1 in another) were on the unchanged
three-worker server setting.

Local checks (correctness only):

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs`: 68 passed.
- `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/vitest-chat-shards.test.ts`: passed.
- `pnpm verify` exits 2 (global CI input changed), so the required CI
checks supply the broad coverage.

## Risks

- Three chat partitions per job use more memory: the peak was 11.4 GB of
16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63%
busy.
- Stale chat weights only change balance, never which tests run. Shard
membership is still checked with `vitest list` before each run.
- The weekly refresh now downloads general server logs from merged-PR
runs. Runs that selected a subset of files are skipped before download
because their shard count is smaller.
- The server lane is still above 7 minutes. See the remaining floor
below.

**Remaining floor and shard-count recommendation (for a follow-up under
`.github/`)**

A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs
give at most about 2.8 effective concurrency, so the test step is about
560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test
step of about 360 s, so about 1,000 s of file time per shard. The
current 19,200 s of general server file time therefore needs about 19-20
shards of 4 vCPUs at 3 workers. More workers on the same runners do not
help, because the CPU is already the limit.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
GitHub CLI) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:03 -06:00

399 lines
26 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import test from "node:test";
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
import { assertManifestSource, attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
import { generalServerJob, generalServerShardCount, oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane,
updatedDurations } from "./refresh-test-durations.mjs";
import { timingPrefix } from "./test-timing-reporter.mjs";
const esc = "\u001b";
const sha = character => character.repeat(40);
const stamp = seconds => new Date(Date.UTC(2026, 10, 6, 21, 0, 0) + seconds * 1000).toISOString().replace("Z", "0000Z");
const logLines = lines => lines.map(([seconds, text]) => `${stamp(seconds)} ${text}`).join("\n");
test("gh api opts into escape sequences, falls back for older gh, and strips colour from logs", async () => {
resetGhApiForTests();
const calls = [];
const modern = async (command, args) => {
calls.push(args);
return { stdout: `${esc}[32m ✓${esc}[39m src/a.test.ts (1 test) 5ms` };
};
assert.equal(await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: modern }), " ✓ src/a.test.ts (1 test) 5ms");
assert.deepEqual(calls[0], ["api", "--allow-escape-sequences", "/repos/x/actions/jobs/1/logs"]);
resetGhApiForTests();
calls.length = 0;
const legacy = async (command, args) => {
calls.push(args);
if (args.includes("--allow-escape-sequences")) {
throw Object.assign(new Error("Command failed"), { stderr: "unknown flag: --allow-escape-sequences\n\nUsage: gh api <endpoint> [flags]" });
}
return { stdout: "{\"ok\":true}" };
};
assert.deepEqual(await ghApi("/a", { run: legacy }), { ok: true });
assert.deepEqual(await ghApi("/b", { run: legacy }), { ok: true });
// The unsupported flag is tried once, then dropped for the process.
assert.deepEqual(calls, [["api", "--allow-escape-sequences", "/a"], ["api", "/a"], ["api", "/b"]]);
resetGhApiForTests();
const failing = async () => { throw Object.assign(new Error("HTTP 404"), { stderr: "gh: Not Found (HTTP 404)" }); };
await assert.rejects(ghApi("/missing", { run: failing }), /HTTP 404/);
resetGhApiForTests();
});
test("gh api allows a large server shard log to finish downloading and never prints a partial log", async () => {
resetGhApiForTests();
const seen = [];
const capture = async (command, args, options) => {
seen.push(options);
return { stdout: "{}" };
};
await ghApi("/repos/x/actions/runs/1/jobs", { run: capture });
await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: capture });
// A 64 MB shard log took 59 s alone (actions run 43133035381); three at once
// overran the old 120 s limit and failed the weekly refresh.
assert.equal(seen[0].timeout, 120_000);
assert.ok(seen[1].timeout >= 600_000, `log timeout ${seen[1].timeout}`);
assert.ok(seen[1].maxBuffer >= 256 * 1024 * 1024, `log buffer ${seen[1].maxBuffer}`);
resetGhApiForTests();
const partial = "x".repeat(10_000);
const timedOut = async () => {
throw Object.assign(new Error(`Command failed: gh api /logs\n${partial}`),
{ killed: true, signal: "SIGTERM", stdout: partial, stderr: "" });
};
await assert.rejects(ghApi("/repos/x/actions/jobs/2/logs", { json: false, run: timedOut }), error => {
assert.match(error.message, /\/repos\/x\/actions\/jobs\/2\/logs failed \(killed by SIGTERM\)/);
assert.ok(!error.message.includes(partial));
assert.equal(error.stdout, undefined);
return true;
});
resetGhApiForTests();
});
test("server timing parsing tolerates colour codes in fetched logs", () => {
const file = "server/src/slow.test.ts";
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
assert.deepEqual(parseTimings(log, new Set([file])).get(file), { ms: 360, complete: true });
});
const shardLine = "shard 2/4 specs: tests/e2e/alpha.spec.ts tests/e2e/smoke-lab-p1-p4.spec.ts tests/e2e/smoke-lab-p5-p7.spec.ts tests/e2e/gone.spec.ts";
const e2eLog = logLines([
[0, `${esc}[36;1mecho "shard 2/4 specs: $specs"${esc}[0m`],
[1, shardLine],
[2, "[WebServer] > vite build"],
[30, "Running 4 tests using 1 worker"],
[31, "[WebServer] ✓ 4 tests/e2e/alpha.spec.ts noise"],
[40, ` ${esc}[32m✓${esc}[39m 1 [chromium] › tests/e2e/alpha.spec.ts:9:1 › first (7.6s)`],
[45, " - 2 [chromium] › tests/e2e/alpha.spec.ts:20:1 › skipped test"],
[70, " ✓ 3 [chromium] › tests/e2e/smoke-lab.shared.ts:316:3 › Smoke Lab scenario catalog mirror (P5-P7) › records the P5-P7 lifecycle @smoke-lab (24.0s)"],
[75, " ✓ 4 [chromium] › tests/e2e/gone.spec.ts:1:1 › removed later (4.0s)"],
[76, " 1 skipped"],
[76, " 3 passed (46.0s)"],
]);
test("e2e weights are gaps between Playwright results, attributed to the running spec", () => {
const inventory = new Set(["tests/e2e/alpha.spec.ts", "tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts"]);
const parsed = parseE2eLog(e2eLog, inventory);
assert.deepEqual(Object.fromEntries(parsed), {
"tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true },
"tests/e2e/smoke-lab-p5-p7.spec.ts": { ms: 25_000, complete: true },
});
assert.throws(() => parseE2eLog(e2eLog.replace("Running 4 tests", "Running 5 tests"), inventory), /Parsed 4 of 5/);
assert.throws(() => parseE2eLog(e2eLog.replace("using 1 worker", "using 2 workers"), inventory), /single Playwright worker/);
assert.throws(() => parseE2eLog(logLines([[0, shardLine]]), inventory), /No Playwright run/);
assert.throws(() => parseE2eLog(e2eLog.replace("tests/e2e/gone.spec.ts:1:1", "tests/e2e/elsewhere.spec.ts:1:1"), inventory),
/not in this shard/);
});
test("shared-file tests map to exactly one shard spec", () => {
const specs = ["tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts", "tests/e2e/agent-chat-projects.spec.ts"];
assert.equal(attributeSpec("tests/e2e/smoke-lab.shared.ts", "Smoke Lab (P1-P4) › records", specs), specs[0]);
assert.equal(attributeSpec("tests/e2e/agent-chat.shared.ts", "anything", specs), specs[2]);
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "no label", specs), /Cannot attribute/);
assert.throws(() => attributeSpec("tests/e2e/helpers/x.ts", "t", specs), /Cannot attribute/);
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "(P1-P4)", null), /No shard spec list/);
});
const runnerLog = logLines([
[0, " RUN v5.0.0 /home/runner/_work/paperclip/paperclip/packages/paperclip-runner"],
[10, ` ${esc}[32m✓${esc}[39m src/live/runnerd-codex-transport.test.ts ${esc}[2m(257 tests)${esc}[22m 126679ms`],
[11, " ✓ src/control-plane/acpx-controller-recovery.test.ts (11 tests | 1 skipped) 78087ms"],
[12, " ✓ scripts/vitest-sequencer.test.mjs (3 tests) 12ms"],
[12, " ✓ a slow test inside a file 5000ms"],
[13, " ✓ src/deleted.test.ts (1 test) 9ms"],
[20, " Test Files 4 passed | 1 skipped (5)"],
[20, " Tests 272 passed | 1 skipped (273)"],
]);
test("Runner weights come from Vitest per-file lines and must account for every passed file", () => {
const inventory = new Set(["packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts",
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts",
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]);
assert.deepEqual(Object.fromEntries(parseRunnerLog(runnerLog, inventory)), {
"packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": { ms: 126_679, complete: false },
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts": { ms: 78_087, complete: false },
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs": { ms: 12, complete: false },
});
assert.throws(() => parseRunnerLog(runnerLog.replace("4 passed", "5 passed"), inventory), /Parsed 4 of 5/);
assert.throws(() => parseRunnerLog(runnerLog.replace(/ Test Files.*\n/, ""), inventory), /No Vitest summary/);
assert.throws(() => parseRunnerLog(runnerLog.replace("src/deleted.test.ts", "src/../../x.test.ts"), inventory));
});
test("the Runner inventory follows the package's Vitest include list", () => {
const inventory = runnerInventory();
assert.ok(inventory.size > 100, `unexpectedly small Runner inventory: ${inventory.size}`);
assert.ok(inventory.has("packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts"));
assert.ok(inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"));
// node --test files in the same package are not Vitest weights.
assert.ok(!inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.mjs"));
assert.ok(![...inventory].some(file => file.startsWith("packages/paperclip-runner/test/")));
});
test("job names select complete, successful shard sets only", () => {
assert.deepEqual(e2eShardJob("ci / e2e shard (3/8)"), { index: 3, count: 8 });
assert.equal(e2eShardJob("ci / e2e"), null);
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package)"), { index: 1, count: 1 });
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (package (2/3))"), { index: 2, count: 3 });
// The lane labels that #19646 shipped keep the "vitest" prefix.
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package (3/3))"), { index: 3, count: 3 });
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
const job = (name, conclusion = "success") => ({ name, conclusion });
const shards = [job("ci / e2e shard (1/2)"), job("ci / e2e shard (2/2)"), job("ci / build")];
assert.equal(completeShardSet(shards, e2eShardJob).length, 2);
assert.equal(completeShardSet(shards.slice(1), e2eShardJob), null);
assert.equal(completeShardSet([shards[0], job("ci / e2e shard (2/2)", "skipped")], e2eShardJob), null);
assert.equal(completeShardSet([shards[0], shards[1], shards[1]], e2eShardJob), null);
assert.equal(completeShardSet([job("ci / build")], e2eShardJob), null);
});
const pull = (number, head, merge, extra = {}) => ({ number, merged_at: `2026-11-0${number % 9 + 1}T00:00:00Z`,
base: { ref: "master", repo: { id: 1170821064 } }, head: { sha: head }, merge_commit_sha: merge, ...extra });
const prRun = (id, head, extra = {}) => ({ id, run_attempt: 1, head_sha: head, event: "pull_request", path: ".github/workflows/pr.yml",
status: "completed", conclusion: "success", html_url: `https://example.test/${id}`,
repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, ...extra });
test("merged-PR runs must be successful pull_request runs of pr.yml at the PR's final head", () => {
const merged = pull(1, sha("a"), sha("b"));
validatePull(merged);
validatePrRun(prRun(5, sha("a")), merged);
for (const override of [{ merged_at: null }, { base: { ref: "release", repo: { id: 1170821064 } } },
{ base: { ref: "master", repo: { id: 7 } } }, { merge_commit_sha: null }]) {
assert.throws(() => validatePull({ ...merged, ...override }));
}
for (const override of [{ head_sha: sha("c") }, { event: "push" }, { path: ".github/workflows/pr-trusted.yml" },
{ conclusion: "failure" }, { status: "in_progress" }, { repository: { id: 7, full_name: "fork/paperclip" } }, { run_attempt: 0 }]) {
assert.throws(() => validatePrRun(prRun(5, sha("a"), override), merged));
}
});
test("collection skips unmerged, absent, stale-head and incomplete runs, and samples each lane per run", async () => {
const api = "/repos/paperclipai/paperclip";
const pulls = [pull(1, sha("1"), sha("a")), pull(2, sha("2"), sha("b")), pull(3, sha("3"), sha("c")),
{ ...pull(4, sha("4"), sha("d")), merged_at: null }, pull(5, sha("5"), sha("e"))];
const e2eJobs = [{ id: 101, name: "ci / e2e shard (1/1)" }, { id: 102, name: "ci / Verify Paperclip Runner (vitest package)" }];
const responses = {
[`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=40`]: pulls,
// PR 1 has only a run of an earlier push; PR 2 is newest-first; PR 5 lacks a Runner job.
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("1")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(11, sha("9"))] },
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("2")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(20, sha("2")), prRun(21, sha("2"))] },
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("5")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(50, sha("5"))] },
[`${api}/actions/runs/21/jobs?filter=latest&per_page=100&page=1`]: { jobs: e2eJobs.map(job => ({ ...job, run_id: 21, conclusion: "success" })) },
[`${api}/actions/runs/50/jobs?filter=latest&per_page=100&page=1`]: { jobs: [{ id: 501, name: "ci / e2e shard (1/1)", run_id: 50, conclusion: "success" }] },
[`${api}/actions/jobs/101/logs`]: e2eLog,
[`${api}/actions/jobs/102/logs`]: runnerLog,
[`${api}/actions/jobs/501/logs`]: e2eLog.replace("Running 4 tests", "Running 9 tests"),
};
const requested = [];
const gh = async endpoint => {
requested.push(endpoint);
assert.ok(endpoint in responses, `unexpected request ${endpoint}`);
return responses[endpoint];
};
const warnings = [];
const lanes = [
{ name: "e2e", classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(["tests/e2e/alpha.spec.ts"]) },
{ name: "runner", classify: runnerPackageJob, parse: parseRunnerLog,
inventory: new Set(["packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]) },
];
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
// The checked-in manifest test uses this same check, so the format the
// collector writes stays accepted even while no manifest carries it yet.
for (const source of [...result.e2e.sources, ...result.runner.sources]) assertManifestSource(source);
assertManifestSource({ run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" });
for (const bad of [{ run: 21, attempt: 1, sha: sha("2") }, { pull: 0, run: 21, attempt: 1, sha: sha("2"), url: "u" },
{ run: 21, pull: 2, attempt: 1, sha: sha("2"), url: "u" }]) assert.throws(() => assertManifestSource(bad));
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.
assert.ok(!requested.some(endpoint => endpoint.includes(sha("3")) || endpoint.includes(sha("4"))));
});
test("general server weights come from complete full-width PR shard sets that cover the lane", async () => {
assert.equal(generalServerShardCount(), 12);
assert.deepEqual(generalServerJob("ci / General tests (server (3/12))", 12), { index: 3, count: 12 });
for (const name of ["ci / General tests (server (1/3))", "ci / General tests (chat (1/3))", "ci / Serialized tests (1/9)"]) {
assert.equal(generalServerJob(name, 12), null);
}
const api = "/repos/paperclipai/paperclip";
const timing = (file, ms) => `2026-11-13T00:00:00.0000000Z ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: ms })}`;
const files = ["server/src/a.test.ts", "server/src/b.test.ts"];
const jobs = (run, ids) => ({ jobs: ids.map((id, index) => ({ id, run_id: run, conclusion: "success",
name: `ci / General tests (server (${index + 1}/${ids.length}))` })) });
const responses = {
[`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=40`]: [pull(1, sha("1"), sha("a")),
pull(2, sha("2"), sha("b")), pull(3, sha("3"), sha("c")), pull(4, sha("4"), sha("d"))],
...Object.fromEntries([1, 2, 3, 4].map(n => [
`${api}/actions/workflows/pr.yml/runs?head_sha=${sha(String(n))}&event=pull_request&status=success&per_page=20`,
{ workflow_runs: [prRun(n * 10, sha(String(n)))] }])),
[`${api}/actions/runs/10/jobs?filter=latest&per_page=100&page=1`]: jobs(10, [101, 102]),
[`${api}/actions/runs/20/jobs?filter=latest&per_page=100&page=1`]: jobs(20, [201]),
[`${api}/actions/runs/30/jobs?filter=latest&per_page=100&page=1`]: jobs(30, [301, 302]),
[`${api}/actions/runs/40/jobs?filter=latest&per_page=100&page=1`]: jobs(40, [401, 402]),
[`${api}/actions/jobs/101/logs`]: timing(files[0], 1000),
[`${api}/actions/jobs/102/logs`]: timing(files[1], 3000),
[`${api}/actions/jobs/301/logs`]: timing(files[0], 2000),
[`${api}/actions/jobs/302/logs`]: "",
[`${api}/actions/jobs/401/logs`]: timing(files[0], 4000),
[`${api}/actions/jobs/402/logs`]: timing(files[1], 5000),
};
const requested = [];
const gh = async endpoint => {
requested.push(endpoint);
assert.ok(endpoint in responses, `unexpected request ${endpoint}`);
return responses[endpoint];
};
const lane = { name: "general", classify: name => generalServerJob(name, 2), parse: parseTimings,
inventory: new Set(files), maxRuns: 2, minFiles: 2 };
const result = await collectMergedPrSamples({ gh, lanes: [lane], isMerged: () => true, log: () => {} });
assert.deepEqual(result.general.sources.map(source => source.run), [40, 10]);
assert.deepEqual(result.general.samples.map(sample => Object.fromEntries([...sample].map(([file, value]) => [file, value.ms]))),
[{ [files[0]]: 4000, [files[1]]: 5000 }, { [files[0]]: 1000, [files[1]]: 3000 }]);
assert.ok(!requested.some(endpoint => endpoint.includes("/jobs/201/")), "a narrower selected run is never downloaded");
});
test("merged-PR manifests use the shared schema and only reorder placement", () => {
const inventory = new Set(["tests/e2e/a.spec.ts", "tests/e2e/b.spec.ts"]);
const samples = [new Map([["tests/e2e/a.spec.ts", { ms: 1000, complete: true }], ["tests/e2e/b.spec.ts", { ms: 50, complete: true }]]),
new Map([["tests/e2e/a.spec.ts", { ms: 3000, complete: true }]])];
const result = updatedDurations({ durations: {} }, inventory, samples, [{ pull: 1, run: 2 }], { comment: "c" });
assert.deepEqual(result, { $comment: "c", unit: "ms", sources: [{ pull: 1, run: 2 }],
coverage: { measured: 2, withSetup: 2, inventory: 2 }, durations: { "tests/e2e/a.spec.ts": 2000, "tests/e2e/b.spec.ts": 100 } });
const manifest = JSON.parse(readFileSync("scripts/e2e-shard-durations.json", "utf8"));
assert.match(manifest.$comment, /never change which specs run/);
assert.equal(manifest.unit, "ms");
assert.ok(manifest.sources.length >= 2 && manifest.sources.every(source => Number.isSafeInteger(source.pull)));
});
test("oversized files are listed in the refresh PR body", () => {
const template = readFileSync(".github/ci-duration-pr.md", "utf8");
assert.ok(template.includes(oversizedPlaceholder));
const manifests = [{ data: { durations: { "server/src/a.test.ts": 200_000, "server/src/b.test.ts": 1000 } } },
{ data: { durations: { "tests/e2e/c.spec.ts": 400_000 } } }];
const oversized = oversizedFiles(manifests, 180_000);
assert.deepEqual(oversized, [["tests/e2e/c.spec.ts", 400_000], ["server/src/a.test.ts", 200_000]]);
const body = renderPrBody(template, oversized, 180_000);
assert.ok(!body.includes(oversizedPlaceholder));
assert.match(body, /2 test file\(s\) exceed the 180s per-file budget/);
assert.match(body, / - `tests\/e2e\/c\.spec\.ts`: 400s\n - `server\/src\/a\.test\.ts`: 200s/);
assert.match(renderPrBody(template, [], 180_000), /No measured test file exceeds the 180s/);
assert.throws(() => renderPrBody("no placeholder", [], 180_000), /placeholder/);
});
test("master sampling recognizes only server timing jobs", () => {
assert.equal(serverLane("ci / General tests (server (3/12))"), "general");
assert.equal(serverLane("verify / Serialized tests (2/4)"), "serialized");
// Admission-only master runs have no server jobs and are passed over.
for (const name of ["Cloud candidate admission v1", "verify", "ci / General tests (ui (1/2))"]) assert.equal(serverLane(name), null);
});
test("refresh arguments keep explicit master run IDs and add dry-run, PR body and lane selection", () => {
const options = parseArgs(["123", "--dry-run", "--pr-body=/tmp/body.md", "--lanes=e2e,runner"]);
assert.deepEqual(options.runIds, ["123"]);
assert.equal(options.dryRun, true);
assert.equal(options.prBody, "/tmp/body.md");
assert.deepEqual([...options.lanes], ["e2e", "runner"]);
assert.throws(() => parseArgs(["--lanes=ui"]), /Unknown lane/);
assert.throws(() => parseArgs(["--force"]), /Unexpected argument/);
});
test("the refresh workflow may only commit the four timing manifests and uses the rendered PR body", () => {
const workflow = readFileSync(".github/workflows/ci-test-durations.yml", "utf8");
for (const file of ["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json",
"scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]) {
assert.ok(workflow.split(file).length >= 3, `${file} must be both allowed and staged`);
}
assert.match(workflow, /refresh-test-durations\.mjs "--pr-body=\$RUNNER_TEMP\/ci-duration-pr\.md"/);
assert.match(workflow, /--body-file "\$RUNNER_TEMP\/ci-duration-pr\.md"/);
});
test("oversized-file reporters warn above the configurable budget and never throw", () => {
assert.equal(oversizedThresholdMs({}), 180_000);
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "90" }), 90_000);
for (const invalid of ["", "0", "-5", "soon"]) {
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: invalid }), 180_000);
}
assert.match(oversizedAnnotation("a,b:c.test.ts", 200_000, 180_000),
/^::warning file=a%2Cb%3Ac\.test\.ts,title=Oversized test file::a,b:c\.test\.ts took 200s, over the 180s per-file budget/);
const lines = [];
const reporter = new OversizedTestFileReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
const module = (file, total, state = "passed") => ({ moduleId: `/repo/${file}`, state: () => state,
diagnostic: () => ({ duration: total - 3, collectDuration: 1, setupDuration: 1, prepareDuration: 1, environmentSetupDuration: 0 }) });
reporter.onTestModuleEnd(module("server/src/slow.test.ts", 180_001));
reporter.onTestModuleEnd(module("server/src/failed-slow.test.ts", 200_000, "failed"));
reporter.onTestModuleEnd(module("server/src/fast.test.ts", 180_000));
reporter.onTestModuleEnd(module("server/src/skipped.test.ts", 900_000, "skipped"));
reporter.onTestModuleEnd({ moduleId: "/repo/x", state: () => { throw new Error("broken"); } });
assert.deepEqual(lines.map(line => /file=([^,]+)/.exec(line)[1]), ["server/src/slow.test.ts", "server/src/failed-slow.test.ts"]);
lines.length = 0;
const custom = new OversizedTestFileReporter({ environment: { PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "60" },
write: line => lines.push(line), root: "/repo" });
custom.onTestModuleEnd(module("server/src/medium.test.ts", 61_000));
assert.equal(lines.length, 1);
lines.length = 0;
const playwright = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
assert.equal(playwright.printsToStdio(), false);
const spec = file => ({ location: { file: `/repo/${file}` } });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 100_000 });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 90_000 });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "skipped", duration: 500_000 });
playwright.onTestEnd(spec("tests/e2e/short.spec.ts"), { status: "passed", duration: 170_000 });
playwright.onEnd();
assert.equal(lines.length, 1);
assert.match(lines[0], /^::warning file=tests\/e2e\/long\.spec\.ts,.*took 190s/);
// Two specs that declare their tests through one shared helper are timed
// per spec, not added together under the helper.
lines.length = 0;
const shared = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
const viaHelper = specPath => ({
location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
parent: { type: "describe", location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
parent: { type: "file", location: { file: `/repo/${specPath}` }, parent: { type: "project" } } },
});
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p1-p4.spec.ts"), { status: "passed", duration: 120_000 });
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 120_000 });
shared.onEnd();
assert.deepEqual(lines, [], "neither spec is over the limit; the helper must not sum them");
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 70_000 });
shared.onEnd();
assert.equal(lines.length, 1);
assert.match(lines[0], /^::warning file=tests\/e2e\/smoke-lab-p5-p7\.spec\.ts,.*took 190s/);
});
test("CI test configs wire the oversized-file reporters for GitHub Actions", () => {
assert.match(readFileSync("vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files\.mjs/);
assert.match(readFileSync("packages/paperclip-runner/vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[^\n]*oversized-test-files\.mjs/);
assert.match(readFileSync("tests/e2e/playwright.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files-playwright\.mjs/);
});