Files
PaperClipAI/scripts/ci/test-selection-pr-timings.test.mjs
T
Nicky LeachandClaude Opus 5.5 97953e9b59 fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - CI test shards are balanced with per-file durations that a weekly
workflow refreshes
> - The weekly refresh has failed since 2026-11-02, and the e2e weights
have not been measured since 2026-09-15. Today e2e shards 3 and 6 take
7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes
> - A single file can also grow until it sets the time of its shard, and
nothing tells us about it
> - This pull request repairs the refresh, extends it to the e2e and
Runner package lanes, and warns about test files over 180 seconds
> - The benefit is balanced shards that stay balanced, and early notice
of oversized test files

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The weekly refresh of the test-duration weights that balance CI shards.

**Subsystem affected**

CI: `scripts/ci/refresh-test-durations.mjs`,
`.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright
CI reporters.

**Current behavior**

- The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now
refuses output that contains terminal escape sequences.
- Most successful `cloud-readiness.yml` master runs are admission-only
and have no server jobs. The script took the newest 2 runs, so it would
fail with "No server timing jobs found" on most weeks.
- The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh
and are from 2026-09-15.
- No signal tells us when one test file becomes too slow.

**Proposed behavior**

The refresh runs again, selects master runs that ran the server jobs,
refreshes the e2e and Runner package-lane weights from merged PRs, and
lists oversized files. CI writes a warning for each test file over 180
seconds.

**Reason and benefit**

Unbalanced shards add minutes to every PR. The current e2e weights put
472 seconds on the slowest shard. The new weights put about 300 seconds
on each shard.

**Breaking changes**

None. Weights change only shard placement, never which tests run.

## What Changed

- Add `scripts/ci/gh-api.mjs`. It always passes
`--allow-escape-sequences`, retries without the flag when an older `gh`
does not know it, and strips ANSI codes from logs.
- `scripts/ci/refresh-test-durations.mjs`:
- scans up to 50 master runs and uses the 2 newest that ran the server
jobs;
- adds the `--dry-run`, `--pr-body=<path>` and
`--lanes=server,e2e,runner` options;
- validates every lane before it writes a file. If a merged-PR lane has
fewer than 2 valid runs, it keeps the current weights and writes a
warning.
- Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing
`pr.yml` run of each merged PR, and it validates:
  - the PR is merged into `master`;
  - the merge commit is in the checkout;
  - the run's head SHA is the PR's final head SHA;
  - every shard of the run succeeded.

  It parses Playwright per-spec times and Runner per-file times.
- Add `scripts/ci/oversized-test-files.mjs` (Vitest) and
`scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub
Actions they write a `::warning` for each test file over
`PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never
fail a job. The root and Runner Vitest configs and the e2e Playwright
config load them.
- `scripts/ci/test-timing-reporter.mjs`: move the per-file duration
calculation into an exported `moduleDurationMs()` helper. The behavior
does not change.
- `.github/workflows/ci-test-durations.yml` and
`.github/ci-duration-pr.md`:
  - the bot PR can also update the e2e and Runner manifests;
  - the PR body lists oversized files;
  - the verify step also runs the new tests and the e2e shard tests.
- Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured
(before: 30), from merged PRs #19613, #19191 and #19176. The file now
uses the shared schema with `sources` and `coverage`.

## Verification

- `node --test scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e
shard balance check with the new weights.
- `node scripts/ci/refresh-test-durations.mjs --dry-run` against real
GitHub data, all lanes:

| Manifest | Measured | Sources |
| --- | --- | --- |
| `general-server-shard-durations.json` | 1488/1534 files | master runs
42296943853, 42294279748 |
| `serialized-shard-durations.json` | 167/167 files | the same two runs
|
| `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365,
42297700476, 42296994179 |
| Runner `vitest-lane-durations.json` | 310/311 files | the same three
PR runs |

- e2e placement over 8 shards, with the new medians as the real costs:
  - with the old weights, the slowest shard has 472 seconds;
  - with the new weights, every shard has about 300 seconds.
- The dry run lists 12 files over 180 seconds. The largest is
`server/src/services/workspace-recovery.test.ts`, at 636 seconds.
- In this PR, only the e2e weight file changes. The weekly run refreshes
the server weights. The Runner manifest is added by #19646.

## Risks

- The `--allow-escape-sequences` path ran only in unit tests. The local
`gh` did not support the flag, so the real dry run used the fallback.
The next weekly run is the first real test of the flag.
- The changes to the bot-PR shell steps did not run in Actions yet.
- The e2e weights come from 3 PR runs. Real shard times vary more than
the estimate.
- This PR and #19637 both edit the `reporters` lists in
`vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The
second PR to merge needs a small merge resolution.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:50:02 -08:00

319 lines
21 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import test from "node:test";
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs";
import { timingPrefix } from "./test-timing-reporter.mjs";
const esc = "\u001b";
const sha = character => character.repeat(40);
const stamp = seconds => new Date(Date.UTC(2026, 10, 6, 21, 0, 0) + seconds * 1000).toISOString().replace("Z", "0000Z");
const logLines = lines => lines.map(([seconds, text]) => `${stamp(seconds)} ${text}`).join("\n");
test("gh api opts into escape sequences, falls back for older gh, and strips colour from logs", async () => {
resetGhApiForTests();
const calls = [];
const modern = async (command, args) => {
calls.push(args);
return { stdout: `${esc}[32m ✓${esc}[39m src/a.test.ts (1 test) 5ms` };
};
assert.equal(await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: modern }), " ✓ src/a.test.ts (1 test) 5ms");
assert.deepEqual(calls[0], ["api", "--allow-escape-sequences", "/repos/x/actions/jobs/1/logs"]);
resetGhApiForTests();
calls.length = 0;
const legacy = async (command, args) => {
calls.push(args);
if (args.includes("--allow-escape-sequences")) {
throw Object.assign(new Error("Command failed"), { stderr: "unknown flag: --allow-escape-sequences\n\nUsage: gh api <endpoint> [flags]" });
}
return { stdout: "{\"ok\":true}" };
};
assert.deepEqual(await ghApi("/a", { run: legacy }), { ok: true });
assert.deepEqual(await ghApi("/b", { run: legacy }), { ok: true });
// The unsupported flag is tried once, then dropped for the process.
assert.deepEqual(calls, [["api", "--allow-escape-sequences", "/a"], ["api", "/a"], ["api", "/b"]]);
resetGhApiForTests();
const failing = async () => { throw Object.assign(new Error("HTTP 404"), { stderr: "gh: Not Found (HTTP 404)" }); };
await assert.rejects(ghApi("/missing", { run: failing }), /HTTP 404/);
resetGhApiForTests();
});
test("server timing parsing tolerates colour codes in fetched logs", () => {
const file = "server/src/slow.test.ts";
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
assert.deepEqual(parseTimings(log, new Set([file])).get(file), { ms: 360, complete: true });
});
const shardLine = "shard 2/4 specs: tests/e2e/alpha.spec.ts tests/e2e/smoke-lab-p1-p4.spec.ts tests/e2e/smoke-lab-p5-p7.spec.ts tests/e2e/gone.spec.ts";
const e2eLog = logLines([
[0, `${esc}[36;1mecho "shard 2/4 specs: $specs"${esc}[0m`],
[1, shardLine],
[2, "[WebServer] > vite build"],
[30, "Running 4 tests using 1 worker"],
[31, "[WebServer] ✓ 4 tests/e2e/alpha.spec.ts noise"],
[40, ` ${esc}[32m✓${esc}[39m 1 [chromium] › tests/e2e/alpha.spec.ts:9:1 › first (7.6s)`],
[45, " - 2 [chromium] › tests/e2e/alpha.spec.ts:20:1 › skipped test"],
[70, " ✓ 3 [chromium] › tests/e2e/smoke-lab.shared.ts:316:3 › Smoke Lab scenario catalog mirror (P5-P7) › records the P5-P7 lifecycle @smoke-lab (24.0s)"],
[75, " ✓ 4 [chromium] › tests/e2e/gone.spec.ts:1:1 › removed later (4.0s)"],
[76, " 1 skipped"],
[76, " 3 passed (46.0s)"],
]);
test("e2e weights are gaps between Playwright results, attributed to the running spec", () => {
const inventory = new Set(["tests/e2e/alpha.spec.ts", "tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts"]);
const parsed = parseE2eLog(e2eLog, inventory);
assert.deepEqual(Object.fromEntries(parsed), {
"tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true },
"tests/e2e/smoke-lab-p5-p7.spec.ts": { ms: 25_000, complete: true },
});
assert.throws(() => parseE2eLog(e2eLog.replace("Running 4 tests", "Running 5 tests"), inventory), /Parsed 4 of 5/);
assert.throws(() => parseE2eLog(e2eLog.replace("using 1 worker", "using 2 workers"), inventory), /single Playwright worker/);
assert.throws(() => parseE2eLog(logLines([[0, shardLine]]), inventory), /No Playwright run/);
assert.throws(() => parseE2eLog(e2eLog.replace("tests/e2e/gone.spec.ts:1:1", "tests/e2e/elsewhere.spec.ts:1:1"), inventory),
/not in this shard/);
});
test("shared-file tests map to exactly one shard spec", () => {
const specs = ["tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts", "tests/e2e/agent-chat-projects.spec.ts"];
assert.equal(attributeSpec("tests/e2e/smoke-lab.shared.ts", "Smoke Lab (P1-P4) › records", specs), specs[0]);
assert.equal(attributeSpec("tests/e2e/agent-chat.shared.ts", "anything", specs), specs[2]);
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "no label", specs), /Cannot attribute/);
assert.throws(() => attributeSpec("tests/e2e/helpers/x.ts", "t", specs), /Cannot attribute/);
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "(P1-P4)", null), /No shard spec list/);
});
const runnerLog = logLines([
[0, " RUN v5.0.0 /home/runner/_work/paperclip/paperclip/packages/paperclip-runner"],
[10, ` ${esc}[32m✓${esc}[39m src/live/runnerd-codex-transport.test.ts ${esc}[2m(257 tests)${esc}[22m 126679ms`],
[11, " ✓ src/control-plane/acpx-controller-recovery.test.ts (11 tests | 1 skipped) 78087ms"],
[12, " ✓ scripts/vitest-sequencer.test.mjs (3 tests) 12ms"],
[12, " ✓ a slow test inside a file 5000ms"],
[13, " ✓ src/deleted.test.ts (1 test) 9ms"],
[20, " Test Files 4 passed | 1 skipped (5)"],
[20, " Tests 272 passed | 1 skipped (273)"],
]);
test("Runner weights come from Vitest per-file lines and must account for every passed file", () => {
const inventory = new Set(["packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts",
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts",
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]);
assert.deepEqual(Object.fromEntries(parseRunnerLog(runnerLog, inventory)), {
"packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": { ms: 126_679, complete: false },
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts": { ms: 78_087, complete: false },
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs": { ms: 12, complete: false },
});
assert.throws(() => parseRunnerLog(runnerLog.replace("4 passed", "5 passed"), inventory), /Parsed 4 of 5/);
assert.throws(() => parseRunnerLog(runnerLog.replace(/ Test Files.*\n/, ""), inventory), /No Vitest summary/);
assert.throws(() => parseRunnerLog(runnerLog.replace("src/deleted.test.ts", "src/../../x.test.ts"), inventory));
});
test("the Runner inventory follows the package's Vitest include list", () => {
const inventory = runnerInventory();
assert.ok(inventory.size > 100, `unexpectedly small Runner inventory: ${inventory.size}`);
assert.ok(inventory.has("packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts"));
assert.ok(inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"));
// node --test files in the same package are not Vitest weights.
assert.ok(!inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.mjs"));
assert.ok(![...inventory].some(file => file.startsWith("packages/paperclip-runner/test/")));
});
test("job names select complete, successful shard sets only", () => {
assert.deepEqual(e2eShardJob("ci / e2e shard (3/8)"), { index: 3, count: 8 });
assert.equal(e2eShardJob("ci / e2e"), null);
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package)"), { index: 1, count: 1 });
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (package (2/3))"), { index: 2, count: 3 });
// The lane labels that #19646 shipped keep the "vitest" prefix.
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package (3/3))"), { index: 3, count: 3 });
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
const job = (name, conclusion = "success") => ({ name, conclusion });
const shards = [job("ci / e2e shard (1/2)"), job("ci / e2e shard (2/2)"), job("ci / build")];
assert.equal(completeShardSet(shards, e2eShardJob).length, 2);
assert.equal(completeShardSet(shards.slice(1), e2eShardJob), null);
assert.equal(completeShardSet([shards[0], job("ci / e2e shard (2/2)", "skipped")], e2eShardJob), null);
assert.equal(completeShardSet([shards[0], shards[1], shards[1]], e2eShardJob), null);
assert.equal(completeShardSet([job("ci / build")], e2eShardJob), null);
});
const pull = (number, head, merge, extra = {}) => ({ number, merged_at: `2026-11-0${number % 9 + 1}T00:00:00Z`,
base: { ref: "master", repo: { id: 1170821064 } }, head: { sha: head }, merge_commit_sha: merge, ...extra });
const prRun = (id, head, extra = {}) => ({ id, run_attempt: 1, head_sha: head, event: "pull_request", path: ".github/workflows/pr.yml",
status: "completed", conclusion: "success", html_url: `https://example.test/${id}`,
repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, ...extra });
test("merged-PR runs must be successful pull_request runs of pr.yml at the PR's final head", () => {
const merged = pull(1, sha("a"), sha("b"));
validatePull(merged);
validatePrRun(prRun(5, sha("a")), merged);
for (const override of [{ merged_at: null }, { base: { ref: "release", repo: { id: 1170821064 } } },
{ base: { ref: "master", repo: { id: 7 } } }, { merge_commit_sha: null }]) {
assert.throws(() => validatePull({ ...merged, ...override }));
}
for (const override of [{ head_sha: sha("c") }, { event: "push" }, { path: ".github/workflows/pr-trusted.yml" },
{ conclusion: "failure" }, { status: "in_progress" }, { repository: { id: 7, full_name: "fork/paperclip" } }, { run_attempt: 0 }]) {
assert.throws(() => validatePrRun(prRun(5, sha("a"), override), merged));
}
});
test("collection skips unmerged, absent, stale-head and incomplete runs, and samples each lane per run", async () => {
const api = "/repos/paperclipai/paperclip";
const pulls = [pull(1, sha("1"), sha("a")), pull(2, sha("2"), sha("b")), pull(3, sha("3"), sha("c")),
{ ...pull(4, sha("4"), sha("d")), merged_at: null }, pull(5, sha("5"), sha("e"))];
const e2eJobs = [{ id: 101, name: "ci / e2e shard (1/1)" }, { id: 102, name: "ci / Verify Paperclip Runner (vitest package)" }];
const responses = {
[`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=40`]: pulls,
// PR 1 has only a run of an earlier push; PR 2 is newest-first; PR 5 lacks a Runner job.
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("1")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(11, sha("9"))] },
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("2")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(20, sha("2")), prRun(21, sha("2"))] },
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("5")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(50, sha("5"))] },
[`${api}/actions/runs/21/jobs?filter=latest&per_page=100&page=1`]: { jobs: e2eJobs.map(job => ({ ...job, run_id: 21, conclusion: "success" })) },
[`${api}/actions/runs/50/jobs?filter=latest&per_page=100&page=1`]: { jobs: [{ id: 501, name: "ci / e2e shard (1/1)", run_id: 50, conclusion: "success" }] },
[`${api}/actions/jobs/101/logs`]: e2eLog,
[`${api}/actions/jobs/102/logs`]: runnerLog,
[`${api}/actions/jobs/501/logs`]: e2eLog.replace("Running 4 tests", "Running 9 tests"),
};
const requested = [];
const gh = async endpoint => {
requested.push(endpoint);
assert.ok(endpoint in responses, `unexpected request ${endpoint}`);
return responses[endpoint];
};
const warnings = [];
const lanes = [
{ name: "e2e", classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(["tests/e2e/alpha.spec.ts"]) },
{ name: "runner", classify: runnerPackageJob, parse: parseRunnerLog,
inventory: new Set(["packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]) },
];
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.
assert.ok(!requested.some(endpoint => endpoint.includes(sha("3")) || endpoint.includes(sha("4"))));
});
test("merged-PR manifests use the shared schema and only reorder placement", () => {
const inventory = new Set(["tests/e2e/a.spec.ts", "tests/e2e/b.spec.ts"]);
const samples = [new Map([["tests/e2e/a.spec.ts", { ms: 1000, complete: true }], ["tests/e2e/b.spec.ts", { ms: 50, complete: true }]]),
new Map([["tests/e2e/a.spec.ts", { ms: 3000, complete: true }]])];
const result = updatedDurations({ durations: {} }, inventory, samples, [{ pull: 1, run: 2 }], { comment: "c" });
assert.deepEqual(result, { $comment: "c", unit: "ms", sources: [{ pull: 1, run: 2 }],
coverage: { measured: 2, withSetup: 2, inventory: 2 }, durations: { "tests/e2e/a.spec.ts": 2000, "tests/e2e/b.spec.ts": 100 } });
const manifest = JSON.parse(readFileSync("scripts/e2e-shard-durations.json", "utf8"));
assert.match(manifest.$comment, /never change which specs run/);
assert.equal(manifest.unit, "ms");
assert.ok(manifest.sources.length >= 2 && manifest.sources.every(source => Number.isSafeInteger(source.pull)));
});
test("oversized files are listed in the refresh PR body", () => {
const template = readFileSync(".github/ci-duration-pr.md", "utf8");
assert.ok(template.includes(oversizedPlaceholder));
const manifests = [{ data: { durations: { "server/src/a.test.ts": 200_000, "server/src/b.test.ts": 1000 } } },
{ data: { durations: { "tests/e2e/c.spec.ts": 400_000 } } }];
const oversized = oversizedFiles(manifests, 180_000);
assert.deepEqual(oversized, [["tests/e2e/c.spec.ts", 400_000], ["server/src/a.test.ts", 200_000]]);
const body = renderPrBody(template, oversized, 180_000);
assert.ok(!body.includes(oversizedPlaceholder));
assert.match(body, /2 test file\(s\) exceed the 180s per-file budget/);
assert.match(body, / - `tests\/e2e\/c\.spec\.ts`: 400s\n - `server\/src\/a\.test\.ts`: 200s/);
assert.match(renderPrBody(template, [], 180_000), /No measured test file exceeds the 180s/);
assert.throws(() => renderPrBody("no placeholder", [], 180_000), /placeholder/);
});
test("master sampling recognizes only server timing jobs", () => {
assert.equal(serverLane("ci / General tests (server (3/12))"), "general");
assert.equal(serverLane("verify / Serialized tests (2/4)"), "serialized");
// Admission-only master runs have no server jobs and are passed over.
for (const name of ["Cloud candidate admission v1", "verify", "ci / General tests (ui (1/2))"]) assert.equal(serverLane(name), null);
});
test("refresh arguments keep explicit master run IDs and add dry-run, PR body and lane selection", () => {
const options = parseArgs(["123", "--dry-run", "--pr-body=/tmp/body.md", "--lanes=e2e,runner"]);
assert.deepEqual(options.runIds, ["123"]);
assert.equal(options.dryRun, true);
assert.equal(options.prBody, "/tmp/body.md");
assert.deepEqual([...options.lanes], ["e2e", "runner"]);
assert.throws(() => parseArgs(["--lanes=ui"]), /Unknown lane/);
assert.throws(() => parseArgs(["--force"]), /Unexpected argument/);
});
test("the refresh workflow may only commit the four timing manifests and uses the rendered PR body", () => {
const workflow = readFileSync(".github/workflows/ci-test-durations.yml", "utf8");
for (const file of ["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json",
"scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]) {
assert.ok(workflow.split(file).length >= 3, `${file} must be both allowed and staged`);
}
assert.match(workflow, /refresh-test-durations\.mjs "--pr-body=\$RUNNER_TEMP\/ci-duration-pr\.md"/);
assert.match(workflow, /--body-file "\$RUNNER_TEMP\/ci-duration-pr\.md"/);
});
test("oversized-file reporters warn above the configurable budget and never throw", () => {
assert.equal(oversizedThresholdMs({}), 180_000);
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "90" }), 90_000);
for (const invalid of ["", "0", "-5", "soon"]) {
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: invalid }), 180_000);
}
assert.match(oversizedAnnotation("a,b:c.test.ts", 200_000, 180_000),
/^::warning file=a%2Cb%3Ac\.test\.ts,title=Oversized test file::a,b:c\.test\.ts took 200s, over the 180s per-file budget/);
const lines = [];
const reporter = new OversizedTestFileReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
const module = (file, total, state = "passed") => ({ moduleId: `/repo/${file}`, state: () => state,
diagnostic: () => ({ duration: total - 3, collectDuration: 1, setupDuration: 1, prepareDuration: 1, environmentSetupDuration: 0 }) });
reporter.onTestModuleEnd(module("server/src/slow.test.ts", 180_001));
reporter.onTestModuleEnd(module("server/src/failed-slow.test.ts", 200_000, "failed"));
reporter.onTestModuleEnd(module("server/src/fast.test.ts", 180_000));
reporter.onTestModuleEnd(module("server/src/skipped.test.ts", 900_000, "skipped"));
reporter.onTestModuleEnd({ moduleId: "/repo/x", state: () => { throw new Error("broken"); } });
assert.deepEqual(lines.map(line => /file=([^,]+)/.exec(line)[1]), ["server/src/slow.test.ts", "server/src/failed-slow.test.ts"]);
lines.length = 0;
const custom = new OversizedTestFileReporter({ environment: { PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "60" },
write: line => lines.push(line), root: "/repo" });
custom.onTestModuleEnd(module("server/src/medium.test.ts", 61_000));
assert.equal(lines.length, 1);
lines.length = 0;
const playwright = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
assert.equal(playwright.printsToStdio(), false);
const spec = file => ({ location: { file: `/repo/${file}` } });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 100_000 });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 90_000 });
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "skipped", duration: 500_000 });
playwright.onTestEnd(spec("tests/e2e/short.spec.ts"), { status: "passed", duration: 170_000 });
playwright.onEnd();
assert.equal(lines.length, 1);
assert.match(lines[0], /^::warning file=tests\/e2e\/long\.spec\.ts,.*took 190s/);
// Two specs that declare their tests through one shared helper are timed
// per spec, not added together under the helper.
lines.length = 0;
const shared = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
const viaHelper = specPath => ({
location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
parent: { type: "describe", location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
parent: { type: "file", location: { file: `/repo/${specPath}` }, parent: { type: "project" } } },
});
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p1-p4.spec.ts"), { status: "passed", duration: 120_000 });
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 120_000 });
shared.onEnd();
assert.deepEqual(lines, [], "neither spec is over the limit; the helper must not sum them");
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 70_000 });
shared.onEnd();
assert.equal(lines.length, 1);
assert.match(lines[0], /^::warning file=tests\/e2e\/smoke-lab-p5-p7\.spec\.ts,.*took 190s/);
});
test("CI test configs wire the oversized-file reporters for GitHub Actions", () => {
assert.match(readFileSync("vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files\.mjs/);
assert.match(readFileSync("packages/paperclip-runner/vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[^\n]*oversized-test-files\.mjs/);
assert.match(readFileSync("tests/e2e/playwright.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files-playwright\.mjs/);
});