mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - CI test shards are balanced with per-file durations that a weekly workflow refreshes > - The weekly refresh has failed since 2026-11-02, and the e2e weights have not been measured since 2026-09-15. Today e2e shards 3 and 6 take 7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes > - A single file can also grow until it sets the time of its shard, and nothing tells us about it > - This pull request repairs the refresh, extends it to the e2e and Runner package lanes, and warns about test files over 180 seconds > - The benefit is balanced shards that stay balanced, and early notice of oversized test files ## Linked Issues or Issue Description **What existing behavior does this improve?** The weekly refresh of the test-duration weights that balance CI shards. **Subsystem affected** CI: `scripts/ci/refresh-test-durations.mjs`, `.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright CI reporters. **Current behavior** - The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now refuses output that contains terminal escape sequences. - Most successful `cloud-readiness.yml` master runs are admission-only and have no server jobs. The script took the newest 2 runs, so it would fail with "No server timing jobs found" on most weeks. - The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh and are from 2026-09-15. - No signal tells us when one test file becomes too slow. **Proposed behavior** The refresh runs again, selects master runs that ran the server jobs, refreshes the e2e and Runner package-lane weights from merged PRs, and lists oversized files. CI writes a warning for each test file over 180 seconds. **Reason and benefit** Unbalanced shards add minutes to every PR. The current e2e weights put 472 seconds on the slowest shard. The new weights put about 300 seconds on each shard. **Breaking changes** None. Weights change only shard placement, never which tests run. ## What Changed - Add `scripts/ci/gh-api.mjs`. It always passes `--allow-escape-sequences`, retries without the flag when an older `gh` does not know it, and strips ANSI codes from logs. - `scripts/ci/refresh-test-durations.mjs`: - scans up to 50 master runs and uses the 2 newest that ran the server jobs; - adds the `--dry-run`, `--pr-body=<path>` and `--lanes=server,e2e,runner` options; - validates every lane before it writes a file. If a merged-PR lane has fewer than 2 valid runs, it keeps the current weights and writes a warning. - Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing `pr.yml` run of each merged PR, and it validates: - the PR is merged into `master`; - the merge commit is in the checkout; - the run's head SHA is the PR's final head SHA; - every shard of the run succeeded. It parses Playwright per-spec times and Runner per-file times. - Add `scripts/ci/oversized-test-files.mjs` (Vitest) and `scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub Actions they write a `::warning` for each test file over `PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never fail a job. The root and Runner Vitest configs and the e2e Playwright config load them. - `scripts/ci/test-timing-reporter.mjs`: move the per-file duration calculation into an exported `moduleDurationMs()` helper. The behavior does not change. - `.github/workflows/ci-test-durations.yml` and `.github/ci-duration-pr.md`: - the bot PR can also update the e2e and Runner manifests; - the PR body lists oversized files; - the verify step also runs the new tests and the e2e shard tests. - Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured (before: 30), from merged PRs #19613, #19191 and #19176. The file now uses the shared schema with `sources` and `coverage`. ## Verification - `node --test scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e shard balance check with the new weights. - `node scripts/ci/refresh-test-durations.mjs --dry-run` against real GitHub data, all lanes: | Manifest | Measured | Sources | | --- | --- | --- | | `general-server-shard-durations.json` | 1488/1534 files | master runs 42296943853, 42294279748 | | `serialized-shard-durations.json` | 167/167 files | the same two runs | | `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365, 42297700476, 42296994179 | | Runner `vitest-lane-durations.json` | 310/311 files | the same three PR runs | - e2e placement over 8 shards, with the new medians as the real costs: - with the old weights, the slowest shard has 472 seconds; - with the new weights, every shard has about 300 seconds. - The dry run lists 12 files over 180 seconds. The largest is `server/src/services/workspace-recovery.test.ts`, at 636 seconds. - In this PR, only the e2e weight file changes. The weekly run refreshes the server weights. The Runner manifest is added by #19646. ## Risks - The `--allow-escape-sequences` path ran only in unit tests. The local `gh` did not support the flag, so the real dry run used the fallback. The next weekly run is the first real test of the flag. - The changes to the bot-PR shell steps did not run in Actions yet. - The e2e weights come from 3 PR runs. Real shard times vary more than the estimate. - This PR and #19637 both edit the `reporters` lists in `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The second PR to merge needs a small merge resolution. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed
1 parent
fb1b458409
commit
97953e9b59
13 files changed
+957
-82
No files matched your search
@@ -4,7 +4,7 @@
|
||||
> - Contributors need predictable CI feedback.
|
||||
> - Test shards use historical per-file costs to distribute their work.
|
||||
> - New and changed tests make those costs stale.
|
||||
> - This automated PR refreshes the costs from successful protected-master runs.
|
||||
> - This automated PR refreshes the costs from successful protected-master runs and, for the e2e and Runner lanes that only run on PRs, from merged PRs' final-head runs.
|
||||
> - The refresh changes placement, while preserving test inventory and runner authorization.
|
||||
|
||||
## Linked Issues or Issue Description
|
||||
@@ -19,15 +19,16 @@ Parallel test jobs should have similar workloads. Timing metadata must never aut
|
||||
|
||||
**Steps to reproduce**
|
||||
|
||||
Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access and a full master checkout.
|
||||
Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access and a full master checkout. Add `--dry-run` to print the changes without writing manifests.
|
||||
|
||||
## What Changed
|
||||
|
||||
- Refresh general-server and serialized timing weights from recent successful master verification runs.
|
||||
- Refresh e2e spec and Runner package-lane weights from successful PR-workflow runs of recently merged PRs, each pinned to the PR's final head SHA. A lane with fewer than two valid runs keeps its current weights.
|
||||
|
||||
## Verification
|
||||
|
||||
- The collector validates repository, workflow, branch, event, attempt, successful jobs, commit ancestry and measurement coverage.
|
||||
- The collector validates repository, workflow, branch, event, attempt, successful jobs, commit ancestry and measurement coverage. Merged-PR sources must be `pull_request` runs of `.github/workflows/pr.yml` whose head SHA equals the merged PR's final head and whose merge commit is in this checkout.
|
||||
- Timing validation and full shard coverage tests pass before this branch is pushed.
|
||||
- Review the source run links and coverage recorded in both manifests.
|
||||
- Required PR checks and review must pass before merge. GitHub may require approval to start checks for a PR created with `GITHUB_TOKEN`.
|
||||
@@ -36,10 +37,11 @@ Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access
|
||||
|
||||
- Runner load can distort measurements. Medians reduce this noise. Missing measurements retain previous weights.
|
||||
- The changes affect balance, not which tests execute. There is no automatic merge or AWS authorization change.
|
||||
<!-- oversized-test-files -->
|
||||
|
||||
## Model Used
|
||||
|
||||
- No model ran this refresh. The generator was authored with OpenAI Codex (GPT-6); exact deployment ID and context window were not exposed. It uses code execution and repository inspection.
|
||||
- No model ran this refresh. The generator was authored with OpenAI Codex (GPT-6); exact deployment ID and context window were not exposed. The merged-PR lanes and oversized-file report were added with Anthropic Claude Opus 5.5 (`claude-opus-5-5`). Both use code execution and repository inspection.
|
||||
|
||||
## Checklist
|
||||
|
||||
|
||||
@@ -36,34 +36,40 @@ jobs:
|
||||
run: |
|
||||
count="$(gh pr list --repo paperclipai/paperclip --state open --search 'in:title "refresh measured test durations"' --limit 100 --json headRefName --jq '[.[] | select(.headRefName | startswith("codex/refresh-ci-durations-"))] | length')"
|
||||
echo "count=$count" >> "$GITHUB_OUTPUT"
|
||||
- name: Refresh from successful protected-master runs
|
||||
- name: Refresh from successful protected-master and merged-PR runs
|
||||
if: steps.pending.outputs.count == '0'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: node scripts/ci/refresh-test-durations.mjs
|
||||
run: node scripts/ci/refresh-test-durations.mjs "--pr-body=$RUNNER_TEMP/ci-duration-pr.md"
|
||||
- name: Verify timing data and complete shard coverage
|
||||
if: steps.pending.outputs.count == '0'
|
||||
run: node --test scripts/ci/test-selection-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs
|
||||
run: >-
|
||||
node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs
|
||||
scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs
|
||||
- name: Open a timing-only pull request
|
||||
if: steps.pending.outputs.count == '0'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
manifests="scripts/general-server-shard-durations.json scripts/serialized-shard-durations.json scripts/e2e-shard-durations.json packages/paperclip-runner/scripts/vitest-lane-durations.json"
|
||||
# The Runner manifest can be new; record intent so the diff sees it.
|
||||
for file in $manifests; do [ ! -e "$file" ] || git add -N -- "$file"; done
|
||||
if git diff --quiet; then exit 0; fi
|
||||
# The collector must not cause unrelated code to enter the bot PR.
|
||||
git diff --name-only -z | node --input-type=module -e '
|
||||
import {readFileSync} from "node:fs";
|
||||
import assert from "node:assert/strict";
|
||||
const allowed = new Set(["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json"]);
|
||||
const allowed = new Set(["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json",
|
||||
"scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]);
|
||||
for (const file of readFileSync(0,"utf8").split("\0").filter(Boolean)) assert.ok(allowed.has(file), file);
|
||||
'
|
||||
branch="codex/refresh-ci-durations-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
|
||||
git switch -c "$branch"
|
||||
git config user.name 'github-actions[bot]'
|
||||
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
||||
git add scripts/general-server-shard-durations.json scripts/serialized-shard-durations.json
|
||||
for file in $manifests; do [ ! -e "$file" ] || git add -- "$file"; done
|
||||
git commit -m 'chore(ci): refresh measured test durations' -m 'Co-Authored-By: Paperclip <noreply@paperclip.ing>'
|
||||
gh auth setup-git
|
||||
git push origin "$branch"
|
||||
gh pr create --repo paperclipai/paperclip --base master --head "$branch" \
|
||||
--title 'chore(ci): refresh measured test durations' --body-file .github/ci-duration-pr.md
|
||||
--title 'chore(ci): refresh measured test durations' --body-file "$RUNNER_TEMP/ci-duration-pr.md"
|
||||
@@ -6,6 +6,10 @@ import { configDefaults, defineConfig } from "vitest/config";
|
||||
import { RunnerSequencer } from "./scripts/vitest-sequencer.mjs";
|
||||
|
||||
const packageRoot = fileURLToPath(new URL(".", import.meta.url));
|
||||
const ciReporters = [
|
||||
...(process.env.GITHUB_ACTIONS === "true" ? [resolve(packageRoot, "../../scripts/ci/oversized-test-files.mjs")] : []),
|
||||
...(process.env.PAPERCLIP_CI_PROFILE === "true" ? [resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] : []),
|
||||
];
|
||||
|
||||
export default defineConfig({
|
||||
resolve: {
|
||||
@@ -15,9 +19,7 @@ export default defineConfig({
|
||||
],
|
||||
},
|
||||
test: {
|
||||
...(process.env.PAPERCLIP_CI_PROFILE === "true"
|
||||
? { reporters: [...configDefaults.reporters, resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] }
|
||||
: {}),
|
||||
...(ciReporters.length ? { reporters: [...configDefaults.reporters, ...ciReporters] } : {}),
|
||||
environment: "node",
|
||||
// Hermetic files also launch Runner and sidecar processes. Reserve CPU
|
||||
// capacity for their startup within the original readiness deadlines.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { promisify, stripVTControlCharacters } from "node:util";
|
||||
|
||||
const execute = promisify(execFile);
|
||||
const escapeFlag = "--allow-escape-sequences";
|
||||
// Recent gh releases refuse to print a response containing terminal escape
|
||||
// sequences unless the caller opts in. Job logs always contain colour codes,
|
||||
// so the refresh opts in and strips them itself. Older gh releases reject
|
||||
// the flag as unknown; remember that once and retry without it.
|
||||
let escapeFlagSupported = true;
|
||||
|
||||
export function isUnknownEscapeFlagError(error) {
|
||||
return /unknown flag:?\s+--allow-escape-sequences/.test(String(error?.stderr ?? error?.message ?? ""));
|
||||
}
|
||||
|
||||
// Calls `gh api` with argv only; no shell sees the endpoint. `run` exists for tests.
|
||||
export async function ghApi(endpoint, { json = true, run = execute } = {}) {
|
||||
const options = { encoding: "utf8", maxBuffer: json ? 16 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000 };
|
||||
const call = flags => run("gh", ["api", ...flags, endpoint], options);
|
||||
let result;
|
||||
if (escapeFlagSupported) {
|
||||
try {
|
||||
result = await call([escapeFlag]);
|
||||
} catch (error) {
|
||||
if (!isUnknownEscapeFlagError(error)) throw error;
|
||||
escapeFlagSupported = false;
|
||||
}
|
||||
}
|
||||
result ??= await call([]);
|
||||
return json ? JSON.parse(result.stdout) : stripVTControlCharacters(result.stdout);
|
||||
}
|
||||
|
||||
export function resetGhApiForTests() {
|
||||
escapeFlagSupported = true;
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
import { oversizedAnnotation, oversizedThresholdMs, repoRelative } from "./oversized-test-files.mjs";
|
||||
|
||||
// The spec file the scheduler places on a shard. A test declared in a shared
|
||||
// helper (smoke-lab.shared.ts) has that helper as its location, so walk up to
|
||||
// the enclosing file suite instead.
|
||||
export function specFile(test) {
|
||||
for (let suite = test.parent; suite; suite = suite.parent) {
|
||||
if (suite.type === "file" && suite.location?.file) return suite.location.file;
|
||||
}
|
||||
return test.location.file;
|
||||
}
|
||||
|
||||
// Playwright reporter. A spec's cost is the sum of its test results, which
|
||||
// include hooks and fixtures. Wired for GitHub Actions only; it never fails a run.
|
||||
export default class OversizedSpecReporter {
|
||||
constructor({ environment = process.env, write = line => console.log(line), root } = {}) {
|
||||
this.thresholdMs = oversizedThresholdMs(environment);
|
||||
this.write = write;
|
||||
this.root = root;
|
||||
this.totals = new Map();
|
||||
}
|
||||
|
||||
printsToStdio() {
|
||||
return false;
|
||||
}
|
||||
|
||||
onTestEnd(test, result) {
|
||||
if (result.status === "skipped" || !Number.isFinite(result.duration)) return;
|
||||
const file = specFile(test);
|
||||
this.totals.set(file, (this.totals.get(file) ?? 0) + result.duration);
|
||||
}
|
||||
|
||||
onEnd() {
|
||||
try {
|
||||
for (const [file, durationMs] of [...this.totals].sort((a, b) => b[1] - a[1])) {
|
||||
if (durationMs <= this.thresholdMs) continue;
|
||||
this.write(oversizedAnnotation(repoRelative(file, this.root), durationMs, this.thresholdMs));
|
||||
}
|
||||
} catch {
|
||||
// A diagnostic reporter must not change a test result.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { moduleDurationMs } from "./test-timing-reporter.mjs";
|
||||
|
||||
// One file over this budget caps how evenly a lane can shard: the scheduler
|
||||
// cannot split a file. Warn so authors split it; never fail the run.
|
||||
export const oversizedThresholdEnv = "PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS";
|
||||
export const defaultOversizedThresholdSeconds = 180;
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
|
||||
export function oversizedThresholdMs(environment = process.env) {
|
||||
const raw = environment[oversizedThresholdEnv];
|
||||
const seconds = raw === undefined || raw === "" ? NaN : Number(raw);
|
||||
return (Number.isFinite(seconds) && seconds > 0 ? seconds : defaultOversizedThresholdSeconds) * 1000;
|
||||
}
|
||||
|
||||
export function repoRelative(file, root = repoRoot) {
|
||||
return path.relative(root, file).split(path.sep).join("/");
|
||||
}
|
||||
|
||||
// https://docs.github.com/actions/reference/workflow-commands-for-github-actions
|
||||
const escapeData = value => String(value).replaceAll("%", "%25").replaceAll("\r", "%0D").replaceAll("\n", "%0A");
|
||||
const escapeProperty = value => escapeData(value).replaceAll(":", "%3A").replaceAll(",", "%2C");
|
||||
|
||||
const seconds = ms => `${Number((ms / 1000).toFixed(ms < 10_000 ? 2 : 0))}s`;
|
||||
|
||||
export function oversizedAnnotation(file, durationMs, thresholdMs) {
|
||||
return `::warning file=${escapeProperty(file)},title=${escapeProperty("Oversized test file")}::`
|
||||
+ escapeData(`${file} took ${seconds(durationMs)}, over the ${seconds(thresholdMs)} per-file budget (${oversizedThresholdEnv}). `
|
||||
+ "Shards cannot split a file, so this bounds lane wall-clock time. Consider splitting it.");
|
||||
}
|
||||
|
||||
// Vitest reporter. Wired for GitHub Actions only; it prints and never throws.
|
||||
export default class OversizedTestFileReporter {
|
||||
constructor({ environment = process.env, write = line => console.log(line), root = repoRoot } = {}) {
|
||||
this.thresholdMs = oversizedThresholdMs(environment);
|
||||
this.write = write;
|
||||
this.root = root;
|
||||
}
|
||||
|
||||
onTestModuleEnd(module) {
|
||||
try {
|
||||
if (module.state() === "skipped") return;
|
||||
const durationMs = moduleDurationMs(module.diagnostic());
|
||||
if (durationMs === null || durationMs <= this.thresholdMs) return;
|
||||
this.write(oversizedAnnotation(repoRelative(module.moduleId, this.root), durationMs, this.thresholdMs));
|
||||
} catch {
|
||||
// A diagnostic reporter must not change a test result.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
|
||||
// e2e and the Runner package lane never run on master. Their weights come from
|
||||
// PR-workflow runs of PRs that merged into master, pinned to the final head SHA.
|
||||
// Logs are parsed as data; nothing from a run is executed.
|
||||
export const repository = "paperclipai/paperclip";
|
||||
export const repositoryId = 1170821064;
|
||||
const api = `/repos/${repository}`;
|
||||
|
||||
export const runnerPackageRoot = "packages/paperclip-runner";
|
||||
export const e2eManifest = "scripts/e2e-shard-durations.json";
|
||||
export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`;
|
||||
|
||||
export function validatePull(pull) {
|
||||
assert.equal(pull.base?.ref, "master");
|
||||
assert.equal(pull.base?.repo?.id, repositoryId);
|
||||
assert.ok(pull.merged_at, `PR #${pull.number} is not merged`);
|
||||
assert.match(pull.head?.sha ?? "", /^[a-f0-9]{40}$/);
|
||||
assert.match(pull.merge_commit_sha ?? "", /^[a-f0-9]{40}$/);
|
||||
}
|
||||
|
||||
export function validatePrRun(run, pull) {
|
||||
assert.equal(run.repository?.full_name, repository);
|
||||
assert.equal(run.repository?.id, repositoryId);
|
||||
assert.equal(run.event, "pull_request");
|
||||
assert.equal(run.path, ".github/workflows/pr.yml");
|
||||
assert.equal(run.status, "completed");
|
||||
assert.equal(run.conclusion, "success");
|
||||
// A run of an earlier push measures code that never merged.
|
||||
assert.equal(run.head_sha, pull.head.sha);
|
||||
assert.ok(Number.isSafeInteger(run.id) && run.id > 0);
|
||||
assert.ok(Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0);
|
||||
}
|
||||
|
||||
export function e2eShardJob(name) {
|
||||
const match = /^ci \/ e2e shard \((\d+)\/(\d+)\)$/.exec(name);
|
||||
return match ? { index: Number(match[1]), count: Number(match[2]) } : null;
|
||||
}
|
||||
|
||||
// Before #19646: "ci / Verify Paperclip Runner (vitest package)", one lane.
|
||||
// Since #19646: "ci / Verify Paperclip Runner (vitest package (K/N))".
|
||||
export function runnerPackageJob(name) {
|
||||
if (name === "ci / Verify Paperclip Runner (vitest package)") return { index: 1, count: 1 };
|
||||
const match = /^ci \/ Verify Paperclip Runner \((?:vitest )?package \((\d+)\/(\d+)\)\)$/.exec(name);
|
||||
return match ? { index: Number(match[1]), count: Number(match[2]) } : null;
|
||||
}
|
||||
|
||||
// Returns the jobs only when they form one complete, successful set of shards.
|
||||
export function completeShardSet(jobs, classify) {
|
||||
const selected = jobs.map(job => ({ job, shard: classify(job.name) })).filter(item => item.shard);
|
||||
if (!selected.length) return null;
|
||||
const count = selected[0].shard.count;
|
||||
const indexes = new Set(selected.map(item => item.shard.index));
|
||||
if (selected.some(item => item.shard.count !== count || item.job.conclusion !== "success")) return null;
|
||||
if (indexes.size !== selected.length || indexes.size !== count) return null;
|
||||
for (let index = 1; index <= count; index++) if (!indexes.has(index)) return null;
|
||||
return selected.map(item => item.job);
|
||||
}
|
||||
|
||||
function splitLogLine(raw) {
|
||||
const line = stripVTControlCharacters(raw).replace(/\r$/, "");
|
||||
const match = /^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d+))?Z ?(.*)$/.exec(line);
|
||||
if (!match) return { time: null, text: line };
|
||||
const fraction = Number(`0.${match[2] ?? "0"}`);
|
||||
return { time: Date.parse(`${match[1]}Z`) + fraction * 1000, text: match[3] };
|
||||
}
|
||||
|
||||
// A location outside a spec (e.g. smoke-lab.shared.ts) belongs to the shard's
|
||||
// spec named after it. Several candidates are told apart by the suffix that the
|
||||
// describe title repeats, e.g. "(P1-P4)" for smoke-lab-p1-p4.spec.ts.
|
||||
export function attributeSpec(location, title, shardSpecs) {
|
||||
if (location.endsWith(".spec.ts")) {
|
||||
assert.ok(!shardSpecs || shardSpecs.includes(location), `${location} is not in this shard`);
|
||||
return location;
|
||||
}
|
||||
assert.ok(shardSpecs, `No shard spec list to attribute ${location}`);
|
||||
const directory = path.posix.dirname(location);
|
||||
const stem = path.posix.basename(location).replace(/(\.shared)?\.[cm]?[jt]sx?$/, "");
|
||||
const candidates = shardSpecs.filter(spec => path.posix.dirname(spec) === directory
|
||||
&& path.posix.basename(spec).startsWith(`${stem}-`));
|
||||
if (candidates.length === 1) return candidates[0];
|
||||
const lower = title.toLowerCase();
|
||||
const named = candidates.filter(spec => lower.includes(path.posix.basename(spec).slice(stem.length + 1, -".spec.ts".length)));
|
||||
assert.equal(named.length, 1, `Cannot attribute ${location} (${title}) to one spec`);
|
||||
return named[0];
|
||||
}
|
||||
|
||||
// The e2e lane runs one Playwright worker, so each gap between consecutive
|
||||
// result lines is the finishing test's cost including fixtures and navigation.
|
||||
// The first gap starts when Playwright reports "Running N tests".
|
||||
export function parseE2eLog(log, inventory) {
|
||||
let shardSpecs = null, start = null, previous = null, expected = null, seen = 0;
|
||||
const totals = new Map();
|
||||
for (const raw of log.split("\n")) {
|
||||
const { time, text } = splitLogLine(raw);
|
||||
if (time === null || text.startsWith("[WebServer]")) continue;
|
||||
const specs = /^shard \d+\/\d+ specs: (.+)$/.exec(text);
|
||||
if (specs) {
|
||||
shardSpecs = specs[1].trim().split(/\s+/);
|
||||
continue;
|
||||
}
|
||||
const running = /^Running (\d+) tests? using (\d+) workers?\b/.exec(text);
|
||||
if (running) {
|
||||
assert.equal(start, null, "Expected one Playwright run per job");
|
||||
assert.equal(Number(running[2]), 1, "Gap attribution requires a single Playwright worker");
|
||||
expected = Number(running[1]);
|
||||
start = previous = time;
|
||||
continue;
|
||||
}
|
||||
if (start === null) continue;
|
||||
const result = /^\s*\S+\s+\d+\s+\[[^\]]+\]\s+›\s+(tests\/e2e\/[^\s:]+):\d+:\d+\s+›\s+(.*)$/.exec(text);
|
||||
if (!result) continue;
|
||||
assert.ok(!result[1].split("/").some(part => part === ".." || part === "."), result[1]);
|
||||
seen++;
|
||||
const spec = attributeSpec(result[1], result[2], shardSpecs);
|
||||
const gap = Math.max(0, time - previous);
|
||||
previous = time;
|
||||
totals.set(spec, (totals.get(spec) ?? 0) + gap);
|
||||
}
|
||||
assert.notEqual(start, null, "No Playwright run found");
|
||||
assert.equal(seen, expected, `Parsed ${seen} of ${expected} Playwright results`);
|
||||
const parsed = new Map();
|
||||
for (const [spec, ms] of totals) {
|
||||
assert.ok(ms <= 3_600_000, `${spec}: implausible ${ms}ms`);
|
||||
// Deleted or renamed specs still appear in older trusted runs.
|
||||
if (inventory.has(spec)) parsed.set(spec, { ms: Math.round(ms), complete: true });
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
// Vitest prints one line per file, relative to the package:
|
||||
// ✓ src/live/runnerd-codex-transport.test.ts (257 tests) 126679ms
|
||||
// The value covers tests and hooks, not collection or setup.
|
||||
export function parseRunnerLog(log, inventory) {
|
||||
const parsed = new Map();
|
||||
let expected = 0, seen = 0, summaries = 0;
|
||||
for (const raw of log.split("\n")) {
|
||||
const { text } = splitLogLine(raw);
|
||||
const summary = /^\s*Test Files\s+(?:\d+ failed \| )?(\d+) passed\b/.exec(text);
|
||||
if (summary) {
|
||||
summaries++;
|
||||
expected += Number(summary[1]);
|
||||
continue;
|
||||
}
|
||||
const match = /^\s*✓\s+(\S+\.test\.[cm]?[jt]sx?)\s+\(\d+ tests?[^)]*\)\s+(\d+)ms\s*$/.exec(text);
|
||||
if (!match) continue;
|
||||
assert.ok(!match[1].split("/").some(part => part === ".." || part === "." || part === ""), match[1]);
|
||||
assert.ok(!match[1].includes("\\"));
|
||||
seen++;
|
||||
const file = `${runnerPackageRoot}/${match[1]}`, ms = Number(match[2]);
|
||||
assert.ok(ms <= 3_600_000, `${file}: implausible ${ms}ms`);
|
||||
if (inventory.has(file)) parsed.set(file, { ms: Math.max(parsed.get(file)?.ms ?? 0, ms), complete: false });
|
||||
}
|
||||
assert.ok(summaries > 0, "No Vitest summary found");
|
||||
assert.equal(seen, expected, `Parsed ${seen} of ${expected} passed Vitest files`);
|
||||
return parsed;
|
||||
}
|
||||
|
||||
// The Runner include globs, read as text so the refresh needs no installed
|
||||
// dependencies. Tests assert the list stays non-empty and matches real files.
|
||||
export function runnerIncludeGlobs(repoRoot = process.cwd()) {
|
||||
const config = readFileSync(path.join(repoRoot, runnerPackageRoot, "vitest.config.ts"), "utf8");
|
||||
const block = /\binclude:\s*\[([\s\S]*?)\]/.exec(config);
|
||||
assert.ok(block, "Runner vitest.config.ts has no include list");
|
||||
const globs = [...block[1].matchAll(/"([^"]+)"/g)].map(match => match[1]);
|
||||
assert.ok(globs.length > 0, "Runner include list is empty");
|
||||
return globs;
|
||||
}
|
||||
|
||||
export function runnerInventory(repoRoot = process.cwd()) {
|
||||
const globs = runnerIncludeGlobs(repoRoot);
|
||||
const tracked = execFileSync("git", ["ls-files", "-z", "--", runnerPackageRoot], { cwd: repoRoot, encoding: "utf8" })
|
||||
.split("\0").filter(Boolean);
|
||||
return new Set(tracked.filter(file => {
|
||||
const relative = file.slice(runnerPackageRoot.length + 1);
|
||||
return globs.some(glob => path.matchesGlob(relative, glob));
|
||||
}));
|
||||
}
|
||||
|
||||
export function isMergedIntoHead(sha, repoRoot = process.cwd()) {
|
||||
try {
|
||||
execFileSync("git", ["merge-base", "--is-ancestor", sha, "HEAD"], { cwd: repoRoot, stdio: "ignore" });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function keepLargest(target, parsed) {
|
||||
for (const [file, value] of parsed) {
|
||||
const previous = target.get(file);
|
||||
if (!previous || value.ms > previous.ms) target.set(file, value);
|
||||
}
|
||||
}
|
||||
|
||||
// lanes: [{ name, classify(jobName) -> {index,count}|null, parse(log, inventory), inventory }]
|
||||
// Returns { [lane]: { samples: Map[], sources: [] } } with at most maxRuns samples each.
|
||||
export async function collectMergedPrSamples({ gh, lanes, maxRuns = 3, pullLimit = 40,
|
||||
isMerged = isMergedIntoHead, log = console.log } = {}) {
|
||||
const result = Object.fromEntries(lanes.map(lane => [lane.name, { samples: [], sources: [] }]));
|
||||
const pulls = await gh(`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=${pullLimit}`);
|
||||
const merged = pulls.filter(pull => pull.merged_at).sort((a, b) => b.merged_at.localeCompare(a.merged_at));
|
||||
for (const pull of merged) {
|
||||
const pending = lanes.filter(lane => result[lane.name].samples.length < maxRuns);
|
||||
if (!pending.length) break;
|
||||
validatePull(pull);
|
||||
// Only measure code present in this checkout.
|
||||
if (!isMerged(pull.merge_commit_sha)) continue;
|
||||
const listed = await gh(`${api}/actions/workflows/pr.yml/runs?head_sha=${pull.head.sha}&event=pull_request&status=success&per_page=20`);
|
||||
const runs = listed.workflow_runs.filter(run => {
|
||||
try { validatePrRun(run, pull); return true; } catch { return false; }
|
||||
}).sort((a, b) => b.id - a.id);
|
||||
if (!runs.length) continue;
|
||||
const run = runs[0];
|
||||
const jobs = [];
|
||||
for (let page = 1; ; page++) {
|
||||
const response = await gh(`${api}/actions/runs/${run.id}/jobs?filter=latest&per_page=100&page=${page}`);
|
||||
jobs.push(...response.jobs);
|
||||
if (response.jobs.length < 100) break;
|
||||
}
|
||||
for (const lane of pending) {
|
||||
const selected = completeShardSet(jobs, lane.classify);
|
||||
if (!selected) continue;
|
||||
const sample = new Map();
|
||||
try {
|
||||
for (const job of selected) {
|
||||
assert.equal(job.run_id, run.id);
|
||||
keepLargest(sample, lane.parse(await gh(`${api}/actions/jobs/${job.id}/logs`, false), lane.inventory));
|
||||
}
|
||||
} catch (error) {
|
||||
log(`::warning::${lane.name}: skipped run ${run.id} for PR #${pull.number}: ${error.message.split("\n")[0]}`);
|
||||
continue;
|
||||
}
|
||||
result[lane.name].samples.push(sample);
|
||||
result[lane.name].sources.push({ pull: pull.number, run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url });
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -1,20 +1,22 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile, execFileSync } from "node:child_process";
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify, stripVTControlCharacters } from "node:util";
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
import { listE2eSpecs } from "../e2e-shard.mjs";
|
||||
import { ghApi } from "./gh-api.mjs";
|
||||
import { oversizedThresholdEnv, oversizedThresholdMs } from "./oversized-test-files.mjs";
|
||||
import { collectMergedPrSamples, e2eManifest, e2eShardJob, isMergedIntoHead, parseE2eLog, parseRunnerLog,
|
||||
runnerInventory, runnerManifest, runnerPackageJob } from "./pr-run-durations.mjs";
|
||||
import { timingPrefix } from "./test-timing-reporter.mjs";
|
||||
|
||||
const repository = "paperclipai/paperclip";
|
||||
const api = `/repos/${repository}`;
|
||||
const execute = promisify(execFile);
|
||||
const github = async (endpoint, json = true) => {
|
||||
const { stdout } = await execute("gh", ["api", endpoint], {
|
||||
encoding: "utf8", maxBuffer: json ? 4 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000,
|
||||
});
|
||||
return json ? JSON.parse(stdout) : stdout;
|
||||
};
|
||||
const github = (endpoint, json = true) => ghApi(endpoint, { json });
|
||||
const masterComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.";
|
||||
const e2eComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback.";
|
||||
const runnerComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per file: the Vitest file duration printed by the Verify Paperclip Runner package job (tests and hooks; collection and setup are not included). Median across runs. Weights only place files on lanes; they never change which tests run. Missing files retain their prior weight or the median fallback.";
|
||||
|
||||
export function validateRun(run) {
|
||||
assert.equal(run.repository?.full_name, repository);
|
||||
@@ -59,7 +61,7 @@ export function parseTimings(log, inventory) {
|
||||
.concat([...measured].map(([file, ms]) => [file, { ms, complete: true }])));
|
||||
}
|
||||
|
||||
export function updatedDurations(previous, inventory, samples, sources) {
|
||||
export function updatedDurations(previous, inventory, samples, sources, { comment = masterComment } = {}) {
|
||||
const measured = [...inventory].filter(file => samples.some(sample => sample.has(file)));
|
||||
assert.ok(measured.length >= inventory.size * 0.9, `Incomplete timing coverage: ${measured.length}/${inventory.size}`);
|
||||
const durations = {};
|
||||
@@ -78,12 +80,62 @@ export function updatedDurations(previous, inventory, samples, sources) {
|
||||
durations[file] = Math.max(100, Math.ceil(ms.length % 2 ? ms[middle] : (ms[middle - 1] + ms[middle]) / 2));
|
||||
}
|
||||
return {
|
||||
$comment: "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.",
|
||||
$comment: comment,
|
||||
unit: "ms", sources, coverage: { measured: measured.length, withSetup: complete, inventory: inventory.size }, durations,
|
||||
};
|
||||
}
|
||||
|
||||
export async function refreshDurations(runIds = []) {
|
||||
// Files over the per-file budget, slowest first, across every refreshed manifest.
|
||||
export function oversizedFiles(manifests, thresholdMs) {
|
||||
return manifests.flatMap(({ data }) => Object.entries(data.durations))
|
||||
.filter(([, ms]) => ms > thresholdMs).sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
|
||||
}
|
||||
|
||||
export const oversizedPlaceholder = "<!-- oversized-test-files -->";
|
||||
|
||||
export function renderPrBody(template, oversized, thresholdMs) {
|
||||
assert.ok(template.includes(oversizedPlaceholder), "PR body template lacks the oversized-file placeholder");
|
||||
const seconds = Math.round(thresholdMs / 1000);
|
||||
const lines = oversized.length
|
||||
? [`- ${oversized.length} test file(s) exceed the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`). A shard cannot split a file, so each one bounds its lane's wall-clock time. Consider splitting:`,
|
||||
...oversized.map(([file, ms]) => ` - \`${file}\`: ${Math.round(ms / 1000)}s`)]
|
||||
: [`- No measured test file exceeds the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`).`];
|
||||
return template.replace(oversizedPlaceholder, lines.join("\n"));
|
||||
}
|
||||
|
||||
function summarizeChange(file, previous, next) {
|
||||
const before = previous?.durations ?? {}, after = next.durations;
|
||||
const changes = Object.keys(after).map(key => [key, before[key], after[key]])
|
||||
.filter(([, old, value]) => old !== value)
|
||||
.sort((a, b) => Math.abs(b[2] - (b[1] ?? 0)) - Math.abs(a[2] - (a[1] ?? 0)));
|
||||
const removed = Object.keys(before).filter(key => !(key in after));
|
||||
console.log(`${file}: ${next.coverage.measured}/${next.coverage.inventory} measured; ${next.coverage.withSetup} include setup; `
|
||||
+ `${changes.length} changed, ${removed.length} removed; sources ${next.sources.map(source => source.pull ? `#${source.pull}/${source.run}` : source.run).join(", ")}`);
|
||||
for (const [key, old, value] of changes.slice(0, 12)) console.log(` ${key}: ${old ?? "(new)"} -> ${value}`);
|
||||
}
|
||||
|
||||
export function parseArgs(argv) {
|
||||
const options = { runIds: [], dryRun: false, prBody: null, lanes: new Set(["server", "e2e", "runner"]) };
|
||||
for (const arg of argv) {
|
||||
if (arg === "--dry-run") options.dryRun = true;
|
||||
else if (arg.startsWith("--pr-body=")) options.prBody = arg.slice("--pr-body=".length);
|
||||
else if (arg.startsWith("--lanes=")) {
|
||||
options.lanes = new Set(arg.slice("--lanes=".length).split(","));
|
||||
for (const lane of options.lanes) assert.ok(["server", "e2e", "runner"].includes(lane), `Unknown lane ${lane}`);
|
||||
} else {
|
||||
assert.match(arg, /^\d+$/, `Unexpected argument ${arg}`);
|
||||
options.runIds.push(arg);
|
||||
}
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
export const serverLane = name => /General tests \(server \(\d+\/\d+\)\)$/.test(name) ? "general"
|
||||
: /Serialized tests \(\d+\/\d+\)$/.test(name) ? "serialized" : null;
|
||||
|
||||
const readManifest = file => existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : { durations: {} };
|
||||
|
||||
async function refreshMasterLanes(runIds) {
|
||||
const inventory = JSON.parse(execFileSync(process.execPath, ["scripts/run-vitest-stable.mjs", "--mode", "general",
|
||||
"--group", "general-server", "--shard-index", "0", "--shard-count", "1", "--dry-run"], {
|
||||
encoding: "utf8", env: { ...process.env, GITHUB_WORKFLOW: "PR", PAPERCLIP_CI_TEST_SELECTION: "" },
|
||||
@@ -92,26 +144,43 @@ export async function refreshDurations(runIds = []) {
|
||||
general: { files: new Set(inventory.selectedGeneralServerSuites), path: "scripts/general-server-shard-durations.json", samples: [] },
|
||||
serialized: { files: new Set(inventory.selectedSerializedSuites), path: "scripts/serialized-shard-durations.json", samples: [] },
|
||||
};
|
||||
if (!runIds.length) {
|
||||
const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=3`);
|
||||
runIds = listed.workflow_runs.map(run => run.id).slice(0, 2);
|
||||
}
|
||||
assert.ok(runIds.length > 0 && runIds.length <= 3, "Expected one to three successful master runs");
|
||||
const sources = [];
|
||||
for (const id of runIds) {
|
||||
assert.match(String(id), /^\d+$/);
|
||||
const run = await github(`${api}/actions/runs/${id}`);
|
||||
validateRun(run);
|
||||
execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]);
|
||||
const runJobs = async run => {
|
||||
const jobs = [];
|
||||
for (let page = 1; ; page++) {
|
||||
const result = await github(`${api}/actions/runs/${id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
||||
const result = await github(`${api}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
||||
jobs.push(...result.jobs);
|
||||
if (result.jobs.length < 100) break;
|
||||
}
|
||||
return jobs;
|
||||
};
|
||||
const candidates = [];
|
||||
if (runIds.length) {
|
||||
assert.ok(runIds.length <= 3, "Expected one to three successful master runs");
|
||||
for (const id of runIds) {
|
||||
assert.match(String(id), /^\d+$/);
|
||||
const run = await github(`${api}/actions/runs/${id}`);
|
||||
validateRun(run);
|
||||
execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]);
|
||||
candidates.push({ run, jobs: await runJobs(run) });
|
||||
}
|
||||
} else {
|
||||
// Most master pushes only run Cloud admission and skip verification, and a
|
||||
// newer push can land after this checkout. Use the two newest runs this
|
||||
// checkout contains that actually ran the server test jobs.
|
||||
const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=50`);
|
||||
for (const run of listed.workflow_runs) {
|
||||
if (candidates.length === 2) break;
|
||||
if (!isMergedIntoHead(run.head_sha)) continue;
|
||||
validateRun(run);
|
||||
const jobs = await runJobs(run);
|
||||
if (jobs.some(job => serverLane(job.name))) candidates.push({ run, jobs });
|
||||
}
|
||||
}
|
||||
assert.ok(candidates.length > 0, "Expected one to three successful master runs");
|
||||
const sources = [];
|
||||
for (const { run, jobs } of candidates) {
|
||||
const samples = { general: new Map(), serialized: new Map() };
|
||||
const selected = jobs.map(job => ({ job, lane: /General tests \(server \(\d+\/\d+\)\)$/.test(job.name) ? "general"
|
||||
: /Serialized tests \(\d+\/\d+\)$/.test(job.name) ? "serialized" : null })).filter(item => item.lane);
|
||||
const selected = jobs.map(job => ({ job, lane: serverLane(job.name) })).filter(item => item.lane);
|
||||
assert.ok(selected.length, "No server timing jobs found");
|
||||
// Keep API traffic and log buffers bounded. Logs are parsed data, never code.
|
||||
for (let start = 0; start < selected.length; start += 3) {
|
||||
@@ -129,13 +198,56 @@ export async function refreshDurations(runIds = []) {
|
||||
for (const lane of Object.keys(lanes)) lanes[lane].samples.push(samples[lane]);
|
||||
sources.push({ run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url });
|
||||
}
|
||||
// Validate both lanes before replacing either checked-in manifest.
|
||||
const updates = Object.values(lanes).map(lane => [lane.path,
|
||||
updatedDurations(JSON.parse(readFileSync(lane.path, "utf8")), lane.files, lane.samples, sources)]);
|
||||
for (const [file, data] of updates) {
|
||||
writeFileSync(file, JSON.stringify(data, null, 2) + "\n");
|
||||
console.log(`${file}: ${data.coverage.measured}/${data.coverage.inventory} measured; ${data.coverage.withSetup} include setup`);
|
||||
return Object.values(lanes).map(lane => {
|
||||
const previous = readManifest(lane.path);
|
||||
return { path: lane.path, previous, data: updatedDurations(previous, lane.files, lane.samples, sources) };
|
||||
});
|
||||
}
|
||||
|
||||
// Fewer than two valid runs leaves a manifest unchanged with a visible warning;
|
||||
// one run is too noisy to replace measured weights.
|
||||
async function refreshPrLanes(names) {
|
||||
if (!names.length) return [];
|
||||
const definitions = {
|
||||
e2e: () => ({ classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(listE2eSpecs()), path: e2eManifest, comment: e2eComment }),
|
||||
runner: () => ({ classify: runnerPackageJob, parse: parseRunnerLog, inventory: runnerInventory(), path: runnerManifest, comment: runnerComment }),
|
||||
};
|
||||
const lanes = names.map(name => ({ name, ...definitions[name]() }));
|
||||
const collected = await collectMergedPrSamples({ gh: github, lanes });
|
||||
const updates = [];
|
||||
for (const lane of lanes) {
|
||||
const { samples, sources } = collected[lane.name];
|
||||
if (samples.length < 2) {
|
||||
console.log(`::warning::${lane.path}: only ${samples.length} valid merged-PR run(s); keeping the current weights`);
|
||||
continue;
|
||||
}
|
||||
const previous = readManifest(lane.path);
|
||||
updates.push({ path: lane.path, previous,
|
||||
data: updatedDurations(previous, lane.inventory, samples, sources, { comment: lane.comment }) });
|
||||
}
|
||||
return updates;
|
||||
}
|
||||
|
||||
export async function refreshDurations(argv = []) {
|
||||
const options = parseArgs(argv.map(String));
|
||||
const updates = [
|
||||
...(options.lanes.has("server") ? await refreshMasterLanes(options.runIds) : []),
|
||||
...await refreshPrLanes(["e2e", "runner"].filter(lane => options.lanes.has(lane))),
|
||||
];
|
||||
// Validate every lane before replacing any checked-in manifest.
|
||||
for (const { path: file, previous, data } of updates) {
|
||||
summarizeChange(file, previous, data);
|
||||
if (!options.dryRun) writeFileSync(file, JSON.stringify(data, null, 2) + "\n");
|
||||
}
|
||||
const thresholdMs = oversizedThresholdMs();
|
||||
const oversized = oversizedFiles(updates, thresholdMs);
|
||||
console.log(`${oversized.length} measured test file(s) exceed ${Math.round(thresholdMs / 1000)}s`);
|
||||
for (const [file, ms] of oversized) console.log(` ${file}: ${Math.round(ms / 1000)}s`);
|
||||
if (options.prBody) {
|
||||
writeFileSync(options.prBody, renderPrBody(readFileSync(".github/ci-duration-pr.md", "utf8"), oversized, thresholdMs));
|
||||
}
|
||||
if (options.dryRun) console.log("Dry run: no manifest was written");
|
||||
return updates;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
|
||||
@@ -0,0 +1,318 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
|
||||
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
|
||||
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
|
||||
import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
|
||||
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
|
||||
import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs";
|
||||
import { timingPrefix } from "./test-timing-reporter.mjs";
|
||||
|
||||
const esc = "\u001b";
|
||||
const sha = character => character.repeat(40);
|
||||
const stamp = seconds => new Date(Date.UTC(2026, 10, 6, 21, 0, 0) + seconds * 1000).toISOString().replace("Z", "0000Z");
|
||||
const logLines = lines => lines.map(([seconds, text]) => `${stamp(seconds)} ${text}`).join("\n");
|
||||
|
||||
test("gh api opts into escape sequences, falls back for older gh, and strips colour from logs", async () => {
|
||||
resetGhApiForTests();
|
||||
const calls = [];
|
||||
const modern = async (command, args) => {
|
||||
calls.push(args);
|
||||
return { stdout: `${esc}[32m ✓${esc}[39m src/a.test.ts (1 test) 5ms` };
|
||||
};
|
||||
assert.equal(await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: modern }), " ✓ src/a.test.ts (1 test) 5ms");
|
||||
assert.deepEqual(calls[0], ["api", "--allow-escape-sequences", "/repos/x/actions/jobs/1/logs"]);
|
||||
|
||||
resetGhApiForTests();
|
||||
calls.length = 0;
|
||||
const legacy = async (command, args) => {
|
||||
calls.push(args);
|
||||
if (args.includes("--allow-escape-sequences")) {
|
||||
throw Object.assign(new Error("Command failed"), { stderr: "unknown flag: --allow-escape-sequences\n\nUsage: gh api <endpoint> [flags]" });
|
||||
}
|
||||
return { stdout: "{\"ok\":true}" };
|
||||
};
|
||||
assert.deepEqual(await ghApi("/a", { run: legacy }), { ok: true });
|
||||
assert.deepEqual(await ghApi("/b", { run: legacy }), { ok: true });
|
||||
// The unsupported flag is tried once, then dropped for the process.
|
||||
assert.deepEqual(calls, [["api", "--allow-escape-sequences", "/a"], ["api", "/a"], ["api", "/b"]]);
|
||||
|
||||
resetGhApiForTests();
|
||||
const failing = async () => { throw Object.assign(new Error("HTTP 404"), { stderr: "gh: Not Found (HTTP 404)" }); };
|
||||
await assert.rejects(ghApi("/missing", { run: failing }), /HTTP 404/);
|
||||
resetGhApiForTests();
|
||||
});
|
||||
|
||||
test("server timing parsing tolerates colour codes in fetched logs", () => {
|
||||
const file = "server/src/slow.test.ts";
|
||||
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
|
||||
assert.deepEqual(parseTimings(log, new Set([file])).get(file), { ms: 360, complete: true });
|
||||
});
|
||||
|
||||
const shardLine = "shard 2/4 specs: tests/e2e/alpha.spec.ts tests/e2e/smoke-lab-p1-p4.spec.ts tests/e2e/smoke-lab-p5-p7.spec.ts tests/e2e/gone.spec.ts";
|
||||
const e2eLog = logLines([
|
||||
[0, `${esc}[36;1mecho "shard 2/4 specs: $specs"${esc}[0m`],
|
||||
[1, shardLine],
|
||||
[2, "[WebServer] > vite build"],
|
||||
[30, "Running 4 tests using 1 worker"],
|
||||
[31, "[WebServer] ✓ 4 tests/e2e/alpha.spec.ts noise"],
|
||||
[40, ` ${esc}[32m✓${esc}[39m 1 [chromium] › tests/e2e/alpha.spec.ts:9:1 › first (7.6s)`],
|
||||
[45, " - 2 [chromium] › tests/e2e/alpha.spec.ts:20:1 › skipped test"],
|
||||
[70, " ✓ 3 [chromium] › tests/e2e/smoke-lab.shared.ts:316:3 › Smoke Lab scenario catalog mirror (P5-P7) › records the P5-P7 lifecycle @smoke-lab (24.0s)"],
|
||||
[75, " ✓ 4 [chromium] › tests/e2e/gone.spec.ts:1:1 › removed later (4.0s)"],
|
||||
[76, " 1 skipped"],
|
||||
[76, " 3 passed (46.0s)"],
|
||||
]);
|
||||
|
||||
test("e2e weights are gaps between Playwright results, attributed to the running spec", () => {
|
||||
const inventory = new Set(["tests/e2e/alpha.spec.ts", "tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts"]);
|
||||
const parsed = parseE2eLog(e2eLog, inventory);
|
||||
assert.deepEqual(Object.fromEntries(parsed), {
|
||||
"tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true },
|
||||
"tests/e2e/smoke-lab-p5-p7.spec.ts": { ms: 25_000, complete: true },
|
||||
});
|
||||
assert.throws(() => parseE2eLog(e2eLog.replace("Running 4 tests", "Running 5 tests"), inventory), /Parsed 4 of 5/);
|
||||
assert.throws(() => parseE2eLog(e2eLog.replace("using 1 worker", "using 2 workers"), inventory), /single Playwright worker/);
|
||||
assert.throws(() => parseE2eLog(logLines([[0, shardLine]]), inventory), /No Playwright run/);
|
||||
assert.throws(() => parseE2eLog(e2eLog.replace("tests/e2e/gone.spec.ts:1:1", "tests/e2e/elsewhere.spec.ts:1:1"), inventory),
|
||||
/not in this shard/);
|
||||
});
|
||||
|
||||
test("shared-file tests map to exactly one shard spec", () => {
|
||||
const specs = ["tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts", "tests/e2e/agent-chat-projects.spec.ts"];
|
||||
assert.equal(attributeSpec("tests/e2e/smoke-lab.shared.ts", "Smoke Lab (P1-P4) › records", specs), specs[0]);
|
||||
assert.equal(attributeSpec("tests/e2e/agent-chat.shared.ts", "anything", specs), specs[2]);
|
||||
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "no label", specs), /Cannot attribute/);
|
||||
assert.throws(() => attributeSpec("tests/e2e/helpers/x.ts", "t", specs), /Cannot attribute/);
|
||||
assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "(P1-P4)", null), /No shard spec list/);
|
||||
});
|
||||
|
||||
const runnerLog = logLines([
|
||||
[0, " RUN v5.0.0 /home/runner/_work/paperclip/paperclip/packages/paperclip-runner"],
|
||||
[10, ` ${esc}[32m✓${esc}[39m src/live/runnerd-codex-transport.test.ts ${esc}[2m(257 tests)${esc}[22m 126679ms`],
|
||||
[11, " ✓ src/control-plane/acpx-controller-recovery.test.ts (11 tests | 1 skipped) 78087ms"],
|
||||
[12, " ✓ scripts/vitest-sequencer.test.mjs (3 tests) 12ms"],
|
||||
[12, " ✓ a slow test inside a file 5000ms"],
|
||||
[13, " ✓ src/deleted.test.ts (1 test) 9ms"],
|
||||
[20, " Test Files 4 passed | 1 skipped (5)"],
|
||||
[20, " Tests 272 passed | 1 skipped (273)"],
|
||||
]);
|
||||
|
||||
test("Runner weights come from Vitest per-file lines and must account for every passed file", () => {
|
||||
const inventory = new Set(["packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts",
|
||||
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts",
|
||||
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]);
|
||||
assert.deepEqual(Object.fromEntries(parseRunnerLog(runnerLog, inventory)), {
|
||||
"packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": { ms: 126_679, complete: false },
|
||||
"packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts": { ms: 78_087, complete: false },
|
||||
"packages/paperclip-runner/scripts/vitest-sequencer.test.mjs": { ms: 12, complete: false },
|
||||
});
|
||||
assert.throws(() => parseRunnerLog(runnerLog.replace("4 passed", "5 passed"), inventory), /Parsed 4 of 5/);
|
||||
assert.throws(() => parseRunnerLog(runnerLog.replace(/ Test Files.*\n/, ""), inventory), /No Vitest summary/);
|
||||
assert.throws(() => parseRunnerLog(runnerLog.replace("src/deleted.test.ts", "src/../../x.test.ts"), inventory));
|
||||
});
|
||||
|
||||
test("the Runner inventory follows the package's Vitest include list", () => {
|
||||
const inventory = runnerInventory();
|
||||
assert.ok(inventory.size > 100, `unexpectedly small Runner inventory: ${inventory.size}`);
|
||||
assert.ok(inventory.has("packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts"));
|
||||
assert.ok(inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"));
|
||||
// node --test files in the same package are not Vitest weights.
|
||||
assert.ok(!inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.mjs"));
|
||||
assert.ok(![...inventory].some(file => file.startsWith("packages/paperclip-runner/test/")));
|
||||
});
|
||||
|
||||
test("job names select complete, successful shard sets only", () => {
|
||||
assert.deepEqual(e2eShardJob("ci / e2e shard (3/8)"), { index: 3, count: 8 });
|
||||
assert.equal(e2eShardJob("ci / e2e"), null);
|
||||
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package)"), { index: 1, count: 1 });
|
||||
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (package (2/3))"), { index: 2, count: 3 });
|
||||
// The lane labels that #19646 shipped keep the "vitest" prefix.
|
||||
assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package (3/3))"), { index: 3, count: 3 });
|
||||
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
|
||||
assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null);
|
||||
const job = (name, conclusion = "success") => ({ name, conclusion });
|
||||
const shards = [job("ci / e2e shard (1/2)"), job("ci / e2e shard (2/2)"), job("ci / build")];
|
||||
assert.equal(completeShardSet(shards, e2eShardJob).length, 2);
|
||||
assert.equal(completeShardSet(shards.slice(1), e2eShardJob), null);
|
||||
assert.equal(completeShardSet([shards[0], job("ci / e2e shard (2/2)", "skipped")], e2eShardJob), null);
|
||||
assert.equal(completeShardSet([shards[0], shards[1], shards[1]], e2eShardJob), null);
|
||||
assert.equal(completeShardSet([job("ci / build")], e2eShardJob), null);
|
||||
});
|
||||
|
||||
const pull = (number, head, merge, extra = {}) => ({ number, merged_at: `2026-11-0${number % 9 + 1}T00:00:00Z`,
|
||||
base: { ref: "master", repo: { id: 1170821064 } }, head: { sha: head }, merge_commit_sha: merge, ...extra });
|
||||
const prRun = (id, head, extra = {}) => ({ id, run_attempt: 1, head_sha: head, event: "pull_request", path: ".github/workflows/pr.yml",
|
||||
status: "completed", conclusion: "success", html_url: `https://example.test/${id}`,
|
||||
repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, ...extra });
|
||||
|
||||
test("merged-PR runs must be successful pull_request runs of pr.yml at the PR's final head", () => {
|
||||
const merged = pull(1, sha("a"), sha("b"));
|
||||
validatePull(merged);
|
||||
validatePrRun(prRun(5, sha("a")), merged);
|
||||
for (const override of [{ merged_at: null }, { base: { ref: "release", repo: { id: 1170821064 } } },
|
||||
{ base: { ref: "master", repo: { id: 7 } } }, { merge_commit_sha: null }]) {
|
||||
assert.throws(() => validatePull({ ...merged, ...override }));
|
||||
}
|
||||
for (const override of [{ head_sha: sha("c") }, { event: "push" }, { path: ".github/workflows/pr-trusted.yml" },
|
||||
{ conclusion: "failure" }, { status: "in_progress" }, { repository: { id: 7, full_name: "fork/paperclip" } }, { run_attempt: 0 }]) {
|
||||
assert.throws(() => validatePrRun(prRun(5, sha("a"), override), merged));
|
||||
}
|
||||
});
|
||||
|
||||
test("collection skips unmerged, absent, stale-head and incomplete runs, and samples each lane per run", async () => {
|
||||
const api = "/repos/paperclipai/paperclip";
|
||||
const pulls = [pull(1, sha("1"), sha("a")), pull(2, sha("2"), sha("b")), pull(3, sha("3"), sha("c")),
|
||||
{ ...pull(4, sha("4"), sha("d")), merged_at: null }, pull(5, sha("5"), sha("e"))];
|
||||
const e2eJobs = [{ id: 101, name: "ci / e2e shard (1/1)" }, { id: 102, name: "ci / Verify Paperclip Runner (vitest package)" }];
|
||||
const responses = {
|
||||
[`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=40`]: pulls,
|
||||
// PR 1 has only a run of an earlier push; PR 2 is newest-first; PR 5 lacks a Runner job.
|
||||
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("1")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(11, sha("9"))] },
|
||||
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("2")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(20, sha("2")), prRun(21, sha("2"))] },
|
||||
[`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("5")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(50, sha("5"))] },
|
||||
[`${api}/actions/runs/21/jobs?filter=latest&per_page=100&page=1`]: { jobs: e2eJobs.map(job => ({ ...job, run_id: 21, conclusion: "success" })) },
|
||||
[`${api}/actions/runs/50/jobs?filter=latest&per_page=100&page=1`]: { jobs: [{ id: 501, name: "ci / e2e shard (1/1)", run_id: 50, conclusion: "success" }] },
|
||||
[`${api}/actions/jobs/101/logs`]: e2eLog,
|
||||
[`${api}/actions/jobs/102/logs`]: runnerLog,
|
||||
[`${api}/actions/jobs/501/logs`]: e2eLog.replace("Running 4 tests", "Running 9 tests"),
|
||||
};
|
||||
const requested = [];
|
||||
const gh = async endpoint => {
|
||||
requested.push(endpoint);
|
||||
assert.ok(endpoint in responses, `unexpected request ${endpoint}`);
|
||||
return responses[endpoint];
|
||||
};
|
||||
const warnings = [];
|
||||
const lanes = [
|
||||
{ name: "e2e", classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(["tests/e2e/alpha.spec.ts"]) },
|
||||
{ name: "runner", classify: runnerPackageJob, parse: parseRunnerLog,
|
||||
inventory: new Set(["packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]) },
|
||||
];
|
||||
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
|
||||
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
|
||||
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
|
||||
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
|
||||
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
|
||||
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.
|
||||
assert.ok(!requested.some(endpoint => endpoint.includes(sha("3")) || endpoint.includes(sha("4"))));
|
||||
});
|
||||
|
||||
test("merged-PR manifests use the shared schema and only reorder placement", () => {
|
||||
const inventory = new Set(["tests/e2e/a.spec.ts", "tests/e2e/b.spec.ts"]);
|
||||
const samples = [new Map([["tests/e2e/a.spec.ts", { ms: 1000, complete: true }], ["tests/e2e/b.spec.ts", { ms: 50, complete: true }]]),
|
||||
new Map([["tests/e2e/a.spec.ts", { ms: 3000, complete: true }]])];
|
||||
const result = updatedDurations({ durations: {} }, inventory, samples, [{ pull: 1, run: 2 }], { comment: "c" });
|
||||
assert.deepEqual(result, { $comment: "c", unit: "ms", sources: [{ pull: 1, run: 2 }],
|
||||
coverage: { measured: 2, withSetup: 2, inventory: 2 }, durations: { "tests/e2e/a.spec.ts": 2000, "tests/e2e/b.spec.ts": 100 } });
|
||||
const manifest = JSON.parse(readFileSync("scripts/e2e-shard-durations.json", "utf8"));
|
||||
assert.match(manifest.$comment, /never change which specs run/);
|
||||
assert.equal(manifest.unit, "ms");
|
||||
assert.ok(manifest.sources.length >= 2 && manifest.sources.every(source => Number.isSafeInteger(source.pull)));
|
||||
});
|
||||
|
||||
test("oversized files are listed in the refresh PR body", () => {
|
||||
const template = readFileSync(".github/ci-duration-pr.md", "utf8");
|
||||
assert.ok(template.includes(oversizedPlaceholder));
|
||||
const manifests = [{ data: { durations: { "server/src/a.test.ts": 200_000, "server/src/b.test.ts": 1000 } } },
|
||||
{ data: { durations: { "tests/e2e/c.spec.ts": 400_000 } } }];
|
||||
const oversized = oversizedFiles(manifests, 180_000);
|
||||
assert.deepEqual(oversized, [["tests/e2e/c.spec.ts", 400_000], ["server/src/a.test.ts", 200_000]]);
|
||||
const body = renderPrBody(template, oversized, 180_000);
|
||||
assert.ok(!body.includes(oversizedPlaceholder));
|
||||
assert.match(body, /2 test file\(s\) exceed the 180s per-file budget/);
|
||||
assert.match(body, / - `tests\/e2e\/c\.spec\.ts`: 400s\n - `server\/src\/a\.test\.ts`: 200s/);
|
||||
assert.match(renderPrBody(template, [], 180_000), /No measured test file exceeds the 180s/);
|
||||
assert.throws(() => renderPrBody("no placeholder", [], 180_000), /placeholder/);
|
||||
});
|
||||
|
||||
test("master sampling recognizes only server timing jobs", () => {
|
||||
assert.equal(serverLane("ci / General tests (server (3/12))"), "general");
|
||||
assert.equal(serverLane("verify / Serialized tests (2/4)"), "serialized");
|
||||
// Admission-only master runs have no server jobs and are passed over.
|
||||
for (const name of ["Cloud candidate admission v1", "verify", "ci / General tests (ui (1/2))"]) assert.equal(serverLane(name), null);
|
||||
});
|
||||
|
||||
test("refresh arguments keep explicit master run IDs and add dry-run, PR body and lane selection", () => {
|
||||
const options = parseArgs(["123", "--dry-run", "--pr-body=/tmp/body.md", "--lanes=e2e,runner"]);
|
||||
assert.deepEqual(options.runIds, ["123"]);
|
||||
assert.equal(options.dryRun, true);
|
||||
assert.equal(options.prBody, "/tmp/body.md");
|
||||
assert.deepEqual([...options.lanes], ["e2e", "runner"]);
|
||||
assert.throws(() => parseArgs(["--lanes=ui"]), /Unknown lane/);
|
||||
assert.throws(() => parseArgs(["--force"]), /Unexpected argument/);
|
||||
});
|
||||
|
||||
test("the refresh workflow may only commit the four timing manifests and uses the rendered PR body", () => {
|
||||
const workflow = readFileSync(".github/workflows/ci-test-durations.yml", "utf8");
|
||||
for (const file of ["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json",
|
||||
"scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]) {
|
||||
assert.ok(workflow.split(file).length >= 3, `${file} must be both allowed and staged`);
|
||||
}
|
||||
assert.match(workflow, /refresh-test-durations\.mjs "--pr-body=\$RUNNER_TEMP\/ci-duration-pr\.md"/);
|
||||
assert.match(workflow, /--body-file "\$RUNNER_TEMP\/ci-duration-pr\.md"/);
|
||||
});
|
||||
|
||||
test("oversized-file reporters warn above the configurable budget and never throw", () => {
|
||||
assert.equal(oversizedThresholdMs({}), 180_000);
|
||||
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "90" }), 90_000);
|
||||
for (const invalid of ["", "0", "-5", "soon"]) {
|
||||
assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: invalid }), 180_000);
|
||||
}
|
||||
assert.match(oversizedAnnotation("a,b:c.test.ts", 200_000, 180_000),
|
||||
/^::warning file=a%2Cb%3Ac\.test\.ts,title=Oversized test file::a,b:c\.test\.ts took 200s, over the 180s per-file budget/);
|
||||
|
||||
const lines = [];
|
||||
const reporter = new OversizedTestFileReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
|
||||
const module = (file, total, state = "passed") => ({ moduleId: `/repo/${file}`, state: () => state,
|
||||
diagnostic: () => ({ duration: total - 3, collectDuration: 1, setupDuration: 1, prepareDuration: 1, environmentSetupDuration: 0 }) });
|
||||
reporter.onTestModuleEnd(module("server/src/slow.test.ts", 180_001));
|
||||
reporter.onTestModuleEnd(module("server/src/failed-slow.test.ts", 200_000, "failed"));
|
||||
reporter.onTestModuleEnd(module("server/src/fast.test.ts", 180_000));
|
||||
reporter.onTestModuleEnd(module("server/src/skipped.test.ts", 900_000, "skipped"));
|
||||
reporter.onTestModuleEnd({ moduleId: "/repo/x", state: () => { throw new Error("broken"); } });
|
||||
assert.deepEqual(lines.map(line => /file=([^,]+)/.exec(line)[1]), ["server/src/slow.test.ts", "server/src/failed-slow.test.ts"]);
|
||||
|
||||
lines.length = 0;
|
||||
const custom = new OversizedTestFileReporter({ environment: { PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "60" },
|
||||
write: line => lines.push(line), root: "/repo" });
|
||||
custom.onTestModuleEnd(module("server/src/medium.test.ts", 61_000));
|
||||
assert.equal(lines.length, 1);
|
||||
|
||||
lines.length = 0;
|
||||
const playwright = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
|
||||
assert.equal(playwright.printsToStdio(), false);
|
||||
const spec = file => ({ location: { file: `/repo/${file}` } });
|
||||
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 100_000 });
|
||||
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 90_000 });
|
||||
playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "skipped", duration: 500_000 });
|
||||
playwright.onTestEnd(spec("tests/e2e/short.spec.ts"), { status: "passed", duration: 170_000 });
|
||||
playwright.onEnd();
|
||||
assert.equal(lines.length, 1);
|
||||
assert.match(lines[0], /^::warning file=tests\/e2e\/long\.spec\.ts,.*took 190s/);
|
||||
|
||||
// Two specs that declare their tests through one shared helper are timed
|
||||
// per spec, not added together under the helper.
|
||||
lines.length = 0;
|
||||
const shared = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" });
|
||||
const viaHelper = specPath => ({
|
||||
location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
|
||||
parent: { type: "describe", location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" },
|
||||
parent: { type: "file", location: { file: `/repo/${specPath}` }, parent: { type: "project" } } },
|
||||
});
|
||||
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p1-p4.spec.ts"), { status: "passed", duration: 120_000 });
|
||||
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 120_000 });
|
||||
shared.onEnd();
|
||||
assert.deepEqual(lines, [], "neither spec is over the limit; the helper must not sum them");
|
||||
shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 70_000 });
|
||||
shared.onEnd();
|
||||
assert.equal(lines.length, 1);
|
||||
assert.match(lines[0], /^::warning file=tests\/e2e\/smoke-lab-p5-p7\.spec\.ts,.*took 190s/);
|
||||
});
|
||||
|
||||
test("CI test configs wire the oversized-file reporters for GitHub Actions", () => {
|
||||
assert.match(readFileSync("vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files\.mjs/);
|
||||
assert.match(readFileSync("packages/paperclip-runner/vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[^\n]*oversized-test-files\.mjs/);
|
||||
assert.match(readFileSync("tests/e2e/playwright.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files-playwright\.mjs/);
|
||||
});
|
||||
@@ -8,11 +8,16 @@ export function fileTiming(module, root = process.cwd()) {
|
||||
if (module.project.name !== "@paperclipai/server" || module.state() !== "passed") return null;
|
||||
const file = path.relative(root, module.moduleId).split(path.sep).join("/");
|
||||
if (!/^server\/(?:src|scripts)\/.+\.test\.[cm]?[jt]s$/.test(file)) return null;
|
||||
const timing = module.diagnostic();
|
||||
const phases = [timing.duration, timing.collectDuration, timing.setupDuration,
|
||||
timing.prepareDuration, timing.environmentSetupDuration];
|
||||
const durationMs = moduleDurationMs(module.diagnostic());
|
||||
return durationMs === null ? null : { version: 1, file, durationMs };
|
||||
}
|
||||
|
||||
// A file's whole cost: tests and hooks plus collection, setup and environment.
|
||||
export function moduleDurationMs(timing) {
|
||||
const phases = [timing?.duration, timing?.collectDuration, timing?.setupDuration,
|
||||
timing?.prepareDuration, timing?.environmentSetupDuration];
|
||||
if (phases.some(value => !Number.isFinite(value) || value < 0)) return null;
|
||||
return { version: 1, file, durationMs: Math.ceil(phases.reduce((sum, value) => sum + value, 0)) };
|
||||
return Math.ceil(phases.reduce((sum, value) => sum + value, 0));
|
||||
}
|
||||
|
||||
export default class TestTimingReporter {
|
||||
|
||||
@@ -1,36 +1,90 @@
|
||||
{
|
||||
"$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 35036001734 and 35024948947, 2026-09-15) by diffing consecutive test-completion timestamps in the 'Run e2e tests' logs, attributing each gap to the finishing test's spec — that captures fixture and navigation time the list-reporter durations alone miss. Smoke Lab tests report under tests/e2e/smoke-lab.shared.ts; their time is mapped back to the p1-p4/p5-p7 spec by scenario tag. Specs missing here get the median weight, so the manifest only needs occasional refreshes: regenerate the same way when e2e shard steps drift apart. chat-adapters-ui and agent-chat were split 2026-09-15 into providers/messaging specs; their weights are the per-describe sums from the same two runs. agent-chat halves are weighted by their per-test sums scaled to the measured spec total.",
|
||||
"$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback.",
|
||||
"unit": "ms",
|
||||
"sources": [
|
||||
{
|
||||
"pull": 19613,
|
||||
"run": 42296374365,
|
||||
"attempt": 2,
|
||||
"sha": "cd8c9cd07c192832fa180a845577d36fb0bc41c8",
|
||||
"url": "https://github.com/paperclipai/paperclip/actions/runs/42296374365"
|
||||
},
|
||||
{
|
||||
"pull": 19191,
|
||||
"run": 42297700476,
|
||||
"attempt": 1,
|
||||
"sha": "36c0400008a12ec74ed65a0e17951f1b47221a02",
|
||||
"url": "https://github.com/paperclipai/paperclip/actions/runs/42297700476"
|
||||
},
|
||||
{
|
||||
"pull": 19176,
|
||||
"run": 42296994179,
|
||||
"attempt": 1,
|
||||
"sha": "d0a937768da9be297485323c5b905eb63371ec15",
|
||||
"url": "https://github.com/paperclipai/paperclip/actions/runs/42296994179"
|
||||
}
|
||||
],
|
||||
"coverage": {
|
||||
"measured": 56,
|
||||
"withSetup": 56,
|
||||
"inventory": 56
|
||||
},
|
||||
"durations": {
|
||||
"tests/e2e/acp-stop-continuation.spec.ts": 44152,
|
||||
"tests/e2e/action-test-agent-picker.spec.ts": 10050,
|
||||
"tests/e2e/agent-chat-projects.spec.ts": 159100,
|
||||
"tests/e2e/agent-chat-sessions.spec.ts": 151200,
|
||||
"tests/e2e/agentmail.spec.ts": 8944,
|
||||
"tests/e2e/app-not-connected.spec.ts": 15684,
|
||||
"tests/e2e/application-delete-screenshot.spec.ts": 3585,
|
||||
"tests/e2e/applications-crud.spec.ts": 10924,
|
||||
"tests/e2e/apps-dark-mode-shots.spec.ts": 27798,
|
||||
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 19557,
|
||||
"tests/e2e/archived-company-url.spec.ts": 11149,
|
||||
"tests/e2e/board-attachment-receipts.spec.ts": 102785,
|
||||
"tests/e2e/chat-adapters-ui-messaging.spec.ts": 241600,
|
||||
"tests/e2e/chat-adapters-ui-providers.spec.ts": 200200,
|
||||
"tests/e2e/composer-stop.spec.ts": 57673,
|
||||
"tests/e2e/conference-room-typing-intro.spec.ts": 15196,
|
||||
"tests/e2e/connection-intents.spec.ts": 14451,
|
||||
"tests/e2e/connection-reviews.spec.ts": 141760,
|
||||
"tests/e2e/legacy-failure-continuation.spec.ts": 74050,
|
||||
"tests/e2e/mcp-user-stories.spec.ts": 38899,
|
||||
"tests/e2e/nux-phase4-screenshots.spec.ts": 18776,
|
||||
"tests/e2e/onboarding.spec.ts": 21295,
|
||||
"tests/e2e/paused-composer.spec.ts": 33211,
|
||||
"tests/e2e/planning-mode-visual-verification.spec.ts": 26193,
|
||||
"tests/e2e/project-repositories.spec.ts": 21634,
|
||||
"tests/e2e/runner-e2e-dashboard.spec.ts": 2028,
|
||||
"tests/e2e/sidebar-takeover.spec.ts": 19034,
|
||||
"tests/e2e/signoff-policy.spec.ts": 12121,
|
||||
"tests/e2e/smoke-lab-p1-p4.spec.ts": 95529,
|
||||
"tests/e2e/smoke-lab-p5-p7.spec.ts": 64328
|
||||
"tests/e2e/acp-stop-continuation.spec.ts": 67299,
|
||||
"tests/e2e/action-test-agent-picker.spec.ts": 13115,
|
||||
"tests/e2e/agent-chat-messages.spec.ts": 23569,
|
||||
"tests/e2e/agent-chat-projects.spec.ts": 206156,
|
||||
"tests/e2e/agent-chat-sessions.spec.ts": 117218,
|
||||
"tests/e2e/agent-configuration.spec.ts": 105794,
|
||||
"tests/e2e/agentmail.spec.ts": 27227,
|
||||
"tests/e2e/ai-personal-connection-recovery.spec.ts": 12373,
|
||||
"tests/e2e/announcement-first-session.spec.ts": 9986,
|
||||
"tests/e2e/announcement-placement.spec.ts": 18485,
|
||||
"tests/e2e/app-not-connected.spec.ts": 10832,
|
||||
"tests/e2e/application-delete-screenshot.spec.ts": 2970,
|
||||
"tests/e2e/applications-crud.spec.ts": 23859,
|
||||
"tests/e2e/apps-dark-mode-shots.spec.ts": 18910,
|
||||
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 13276,
|
||||
"tests/e2e/archived-company-url.spec.ts": 6145,
|
||||
"tests/e2e/artifact-tab-arrival.spec.ts": 20087,
|
||||
"tests/e2e/artifact-uploads.spec.ts": 28200,
|
||||
"tests/e2e/board-attachment-receipts.spec.ts": 112020,
|
||||
"tests/e2e/cases-read-only.spec.ts": 9657,
|
||||
"tests/e2e/chat-adapters-ui-messaging.spec.ts": 181303,
|
||||
"tests/e2e/chat-adapters-ui-providers.spec.ts": 181541,
|
||||
"tests/e2e/chunk-load-recovery.spec.ts": 16330,
|
||||
"tests/e2e/classic-workspace.spec.ts": 6483,
|
||||
"tests/e2e/cloud-auth.spec.ts": 33711,
|
||||
"tests/e2e/company-context-refresh.spec.ts": 4128,
|
||||
"tests/e2e/completed-task-paused-assignee.spec.ts": 25016,
|
||||
"tests/e2e/composer-stop.spec.ts": 74978,
|
||||
"tests/e2e/conference-room-typing-intro.spec.ts": 30042,
|
||||
"tests/e2e/connection-intents.spec.ts": 23775,
|
||||
"tests/e2e/connection-reviews.spec.ts": 134583,
|
||||
"tests/e2e/coordinator-inbox-views.spec.ts": 18241,
|
||||
"tests/e2e/daytona-connector.spec.ts": 2603,
|
||||
"tests/e2e/disconnect-resilience.spec.ts": 200278,
|
||||
"tests/e2e/disconnect-write-recovery.spec.ts": 51072,
|
||||
"tests/e2e/document-image-gallery.spec.ts": 46344,
|
||||
"tests/e2e/legacy-failure-continuation.spec.ts": 89961,
|
||||
"tests/e2e/mcp-user-stories.spec.ts": 37417,
|
||||
"tests/e2e/new-task-picker-touch.spec.ts": 22514,
|
||||
"tests/e2e/nux-phase4-screenshots.spec.ts": 11330,
|
||||
"tests/e2e/onboarding.spec.ts": 62933,
|
||||
"tests/e2e/paused-composer.spec.ts": 38240,
|
||||
"tests/e2e/plan-task-fixture.spec.ts": 7828,
|
||||
"tests/e2e/planning-mode-visual-verification.spec.ts": 26325,
|
||||
"tests/e2e/project-repositories.spec.ts": 13403,
|
||||
"tests/e2e/railway-catalog.spec.ts": 2813,
|
||||
"tests/e2e/runner-e2e-dashboard.spec.ts": 4985,
|
||||
"tests/e2e/runner-task-creation.spec.ts": 32437,
|
||||
"tests/e2e/sidebar-takeover.spec.ts": 9119,
|
||||
"tests/e2e/signoff-policy.spec.ts": 52229,
|
||||
"tests/e2e/smoke-lab-p1-p4.spec.ts": 22390,
|
||||
"tests/e2e/smoke-lab-p5-p7.spec.ts": 16416,
|
||||
"tests/e2e/task-loading-frame.spec.ts": 31309,
|
||||
"tests/e2e/task-reload.spec.ts": 20988,
|
||||
"tests/e2e/text-attachment-tabs.spec.ts": 12758,
|
||||
"tests/e2e/unikraft-connector.spec.ts": 2613
|
||||
}
|
||||
}
|
||||
@@ -97,6 +97,9 @@ export default defineConfig({
|
||||
outputDir: "./test-results",
|
||||
reporter: [
|
||||
["list"], ["html", { open: "never", outputFolder: "./playwright-report" }],
|
||||
...(process.env.GITHUB_ACTIONS === "true"
|
||||
? [[path.resolve(import.meta.dirname, "../../scripts/ci/oversized-test-files-playwright.mjs")] as [string]]
|
||||
: []),
|
||||
...(process.env.PAPERCLIP_CI_PROFILE === "true"
|
||||
? [[path.resolve(import.meta.dirname, "../../scripts/ci-profile/playwright-reporter.mjs")] as [string]]
|
||||
: []),
|
||||
|
||||
+2
-1
@@ -3,7 +3,8 @@ import { configDefaults, defineConfig } from "vitest/config";
|
||||
export default defineConfig({
|
||||
test: {
|
||||
reporters: [...configDefaults.reporters,
|
||||
...(process.env.GITHUB_ACTIONS === "true" ? ["./scripts/ci/test-timing-reporter.mjs"] : []),
|
||||
...(process.env.GITHUB_ACTIONS === "true"
|
||||
? ["./scripts/ci/test-timing-reporter.mjs", "./scripts/ci/oversized-test-files.mjs"] : []),
|
||||
...(process.env.PAPERCLIP_CI_PROFILE === "true" ? ["./scripts/ci-profile/vitest-reporter.mjs"] : [])],
|
||||
projects: [
|
||||
"packages/shared",
|
||||
|
||||
Reference in new issue
Block a user