From 97953e9b59bd5f1fd494a96b18ba5ebc0d580b9d Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 15:50:02 -0800 Subject: [PATCH] fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - CI test shards are balanced with per-file durations that a weekly workflow refreshes > - The weekly refresh has failed since 2026-11-02, and the e2e weights have not been measured since 2026-09-15. Today e2e shards 3 and 6 take 7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes > - A single file can also grow until it sets the time of its shard, and nothing tells us about it > - This pull request repairs the refresh, extends it to the e2e and Runner package lanes, and warns about test files over 180 seconds > - The benefit is balanced shards that stay balanced, and early notice of oversized test files ## Linked Issues or Issue Description **What existing behavior does this improve?** The weekly refresh of the test-duration weights that balance CI shards. **Subsystem affected** CI: `scripts/ci/refresh-test-durations.mjs`, `.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright CI reporters. **Current behavior** - The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now refuses output that contains terminal escape sequences. - Most successful `cloud-readiness.yml` master runs are admission-only and have no server jobs. The script took the newest 2 runs, so it would fail with "No server timing jobs found" on most weeks. - The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh and are from 2026-09-15. - No signal tells us when one test file becomes too slow. **Proposed behavior** The refresh runs again, selects master runs that ran the server jobs, refreshes the e2e and Runner package-lane weights from merged PRs, and lists oversized files. CI writes a warning for each test file over 180 seconds. **Reason and benefit** Unbalanced shards add minutes to every PR. The current e2e weights put 472 seconds on the slowest shard. The new weights put about 300 seconds on each shard. **Breaking changes** None. Weights change only shard placement, never which tests run. ## What Changed - Add `scripts/ci/gh-api.mjs`. It always passes `--allow-escape-sequences`, retries without the flag when an older `gh` does not know it, and strips ANSI codes from logs. - `scripts/ci/refresh-test-durations.mjs`: - scans up to 50 master runs and uses the 2 newest that ran the server jobs; - adds the `--dry-run`, `--pr-body=` and `--lanes=server,e2e,runner` options; - validates every lane before it writes a file. If a merged-PR lane has fewer than 2 valid runs, it keeps the current weights and writes a warning. - Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing `pr.yml` run of each merged PR, and it validates: - the PR is merged into `master`; - the merge commit is in the checkout; - the run's head SHA is the PR's final head SHA; - every shard of the run succeeded. It parses Playwright per-spec times and Runner per-file times. - Add `scripts/ci/oversized-test-files.mjs` (Vitest) and `scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub Actions they write a `::warning` for each test file over `PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never fail a job. The root and Runner Vitest configs and the e2e Playwright config load them. - `scripts/ci/test-timing-reporter.mjs`: move the per-file duration calculation into an exported `moduleDurationMs()` helper. The behavior does not change. - `.github/workflows/ci-test-durations.yml` and `.github/ci-duration-pr.md`: - the bot PR can also update the e2e and Runner manifests; - the PR body lists oversized files; - the verify step also runs the new tests and the e2e shard tests. - Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured (before: 30), from merged PRs #19613, #19191 and #19176. The file now uses the shared schema with `sources` and `coverage`. ## Verification - `node --test scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e shard balance check with the new weights. - `node scripts/ci/refresh-test-durations.mjs --dry-run` against real GitHub data, all lanes: | Manifest | Measured | Sources | | --- | --- | --- | | `general-server-shard-durations.json` | 1488/1534 files | master runs 42296943853, 42294279748 | | `serialized-shard-durations.json` | 167/167 files | the same two runs | | `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365, 42297700476, 42296994179 | | Runner `vitest-lane-durations.json` | 310/311 files | the same three PR runs | - e2e placement over 8 shards, with the new medians as the real costs: - with the old weights, the slowest shard has 472 seconds; - with the new weights, every shard has about 300 seconds. - The dry run lists 12 files over 180 seconds. The largest is `server/src/services/workspace-recovery.test.ts`, at 636 seconds. - In this PR, only the e2e weight file changes. The weekly run refreshes the server weights. The Runner manifest is added by #19646. ## Risks - The `--allow-escape-sequences` path ran only in unit tests. The local `gh` did not support the flag, so the real dry run used the fallback. The next weekly run is the first real test of the flag. - The changes to the bot-PR shell steps did not run in Actions yet. - The e2e weights come from 3 PR runs. Real shard times vary more than the estimate. - This PR and #19637 both edit the `reporters` lists in `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The second PR to merge needs a small merge resolution. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .github/ci-duration-pr.md | 10 +- .github/workflows/ci-test-durations.yml | 18 +- packages/paperclip-runner/vitest.config.ts | 8 +- scripts/ci/gh-api.mjs | 35 ++ .../ci/oversized-test-files-playwright.mjs | 43 +++ scripts/ci/oversized-test-files.mjs | 51 +++ scripts/ci/pr-run-durations.mjs | 243 +++++++++++++ scripts/ci/refresh-test-durations.mjs | 178 ++++++++-- scripts/ci/test-selection-pr-timings.test.mjs | 318 ++++++++++++++++++ scripts/ci/test-timing-reporter.mjs | 13 +- scripts/e2e-shard-durations.json | 116 +++++-- tests/e2e/playwright.config.ts | 3 + vitest.config.ts | 3 +- 13 files changed, 957 insertions(+), 82 deletions(-) create mode 100644 scripts/ci/gh-api.mjs create mode 100644 scripts/ci/oversized-test-files-playwright.mjs create mode 100644 scripts/ci/oversized-test-files.mjs create mode 100644 scripts/ci/pr-run-durations.mjs create mode 100644 scripts/ci/test-selection-pr-timings.test.mjs diff --git a/.github/ci-duration-pr.md b/.github/ci-duration-pr.md index 92ca142ac68..53ac2ce3214 100644 --- a/.github/ci-duration-pr.md +++ b/.github/ci-duration-pr.md @@ -4,7 +4,7 @@ > - Contributors need predictable CI feedback. > - Test shards use historical per-file costs to distribute their work. > - New and changed tests make those costs stale. -> - This automated PR refreshes the costs from successful protected-master runs. +> - This automated PR refreshes the costs from successful protected-master runs and, for the e2e and Runner lanes that only run on PRs, from merged PRs' final-head runs. > - The refresh changes placement, while preserving test inventory and runner authorization. ## Linked Issues or Issue Description @@ -19,15 +19,16 @@ Parallel test jobs should have similar workloads. Timing metadata must never aut **Steps to reproduce** -Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access and a full master checkout. +Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access and a full master checkout. Add `--dry-run` to print the changes without writing manifests. ## What Changed - Refresh general-server and serialized timing weights from recent successful master verification runs. +- Refresh e2e spec and Runner package-lane weights from successful PR-workflow runs of recently merged PRs, each pinned to the PR's final head SHA. A lane with fewer than two valid runs keeps its current weights. ## Verification -- The collector validates repository, workflow, branch, event, attempt, successful jobs, commit ancestry and measurement coverage. +- The collector validates repository, workflow, branch, event, attempt, successful jobs, commit ancestry and measurement coverage. Merged-PR sources must be `pull_request` runs of `.github/workflows/pr.yml` whose head SHA equals the merged PR's final head and whose merge commit is in this checkout. - Timing validation and full shard coverage tests pass before this branch is pushed. - Review the source run links and coverage recorded in both manifests. - Required PR checks and review must pass before merge. GitHub may require approval to start checks for a PR created with `GITHUB_TOKEN`. @@ -36,10 +37,11 @@ Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access - Runner load can distort measurements. Medians reduce this noise. Missing measurements retain previous weights. - The changes affect balance, not which tests execute. There is no automatic merge or AWS authorization change. + ## Model Used -- No model ran this refresh. The generator was authored with OpenAI Codex (GPT-6); exact deployment ID and context window were not exposed. It uses code execution and repository inspection. +- No model ran this refresh. The generator was authored with OpenAI Codex (GPT-6); exact deployment ID and context window were not exposed. The merged-PR lanes and oversized-file report were added with Anthropic Claude Opus 5.5 (`claude-opus-5-5`). Both use code execution and repository inspection. ## Checklist diff --git a/.github/workflows/ci-test-durations.yml b/.github/workflows/ci-test-durations.yml index a17f6f45e1f..366f449816c 100644 --- a/.github/workflows/ci-test-durations.yml +++ b/.github/workflows/ci-test-durations.yml @@ -36,34 +36,40 @@ jobs: run: | count="$(gh pr list --repo paperclipai/paperclip --state open --search 'in:title "refresh measured test durations"' --limit 100 --json headRefName --jq '[.[] | select(.headRefName | startswith("codex/refresh-ci-durations-"))] | length')" echo "count=$count" >> "$GITHUB_OUTPUT" - - name: Refresh from successful protected-master runs + - name: Refresh from successful protected-master and merged-PR runs if: steps.pending.outputs.count == '0' env: GH_TOKEN: ${{ github.token }} - run: node scripts/ci/refresh-test-durations.mjs + run: node scripts/ci/refresh-test-durations.mjs "--pr-body=$RUNNER_TEMP/ci-duration-pr.md" - name: Verify timing data and complete shard coverage if: steps.pending.outputs.count == '0' - run: node --test scripts/ci/test-selection-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs + run: >- + node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs + scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs - name: Open a timing-only pull request if: steps.pending.outputs.count == '0' env: GH_TOKEN: ${{ github.token }} run: | + manifests="scripts/general-server-shard-durations.json scripts/serialized-shard-durations.json scripts/e2e-shard-durations.json packages/paperclip-runner/scripts/vitest-lane-durations.json" + # The Runner manifest can be new; record intent so the diff sees it. + for file in $manifests; do [ ! -e "$file" ] || git add -N -- "$file"; done if git diff --quiet; then exit 0; fi # The collector must not cause unrelated code to enter the bot PR. git diff --name-only -z | node --input-type=module -e ' import {readFileSync} from "node:fs"; import assert from "node:assert/strict"; - const allowed = new Set(["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json"]); + const allowed = new Set(["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json", + "scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]); for (const file of readFileSync(0,"utf8").split("\0").filter(Boolean)) assert.ok(allowed.has(file), file); ' branch="codex/refresh-ci-durations-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" git switch -c "$branch" git config user.name 'github-actions[bot]' git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add scripts/general-server-shard-durations.json scripts/serialized-shard-durations.json + for file in $manifests; do [ ! -e "$file" ] || git add -- "$file"; done git commit -m 'chore(ci): refresh measured test durations' -m 'Co-Authored-By: Paperclip ' gh auth setup-git git push origin "$branch" gh pr create --repo paperclipai/paperclip --base master --head "$branch" \ - --title 'chore(ci): refresh measured test durations' --body-file .github/ci-duration-pr.md + --title 'chore(ci): refresh measured test durations' --body-file "$RUNNER_TEMP/ci-duration-pr.md" diff --git a/packages/paperclip-runner/vitest.config.ts b/packages/paperclip-runner/vitest.config.ts index 488531c4144..5c96b64dfb3 100644 --- a/packages/paperclip-runner/vitest.config.ts +++ b/packages/paperclip-runner/vitest.config.ts @@ -6,6 +6,10 @@ import { configDefaults, defineConfig } from "vitest/config"; import { RunnerSequencer } from "./scripts/vitest-sequencer.mjs"; const packageRoot = fileURLToPath(new URL(".", import.meta.url)); +const ciReporters = [ + ...(process.env.GITHUB_ACTIONS === "true" ? [resolve(packageRoot, "../../scripts/ci/oversized-test-files.mjs")] : []), + ...(process.env.PAPERCLIP_CI_PROFILE === "true" ? [resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] : []), +]; export default defineConfig({ resolve: { @@ -15,9 +19,7 @@ export default defineConfig({ ], }, test: { - ...(process.env.PAPERCLIP_CI_PROFILE === "true" - ? { reporters: [...configDefaults.reporters, resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] } - : {}), + ...(ciReporters.length ? { reporters: [...configDefaults.reporters, ...ciReporters] } : {}), environment: "node", // Hermetic files also launch Runner and sidecar processes. Reserve CPU // capacity for their startup within the original readiness deadlines. diff --git a/scripts/ci/gh-api.mjs b/scripts/ci/gh-api.mjs new file mode 100644 index 00000000000..6845a6f403b --- /dev/null +++ b/scripts/ci/gh-api.mjs @@ -0,0 +1,35 @@ +import { execFile } from "node:child_process"; +import { promisify, stripVTControlCharacters } from "node:util"; + +const execute = promisify(execFile); +const escapeFlag = "--allow-escape-sequences"; +// Recent gh releases refuse to print a response containing terminal escape +// sequences unless the caller opts in. Job logs always contain colour codes, +// so the refresh opts in and strips them itself. Older gh releases reject +// the flag as unknown; remember that once and retry without it. +let escapeFlagSupported = true; + +export function isUnknownEscapeFlagError(error) { + return /unknown flag:?\s+--allow-escape-sequences/.test(String(error?.stderr ?? error?.message ?? "")); +} + +// Calls `gh api` with argv only; no shell sees the endpoint. `run` exists for tests. +export async function ghApi(endpoint, { json = true, run = execute } = {}) { + const options = { encoding: "utf8", maxBuffer: json ? 16 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000 }; + const call = flags => run("gh", ["api", ...flags, endpoint], options); + let result; + if (escapeFlagSupported) { + try { + result = await call([escapeFlag]); + } catch (error) { + if (!isUnknownEscapeFlagError(error)) throw error; + escapeFlagSupported = false; + } + } + result ??= await call([]); + return json ? JSON.parse(result.stdout) : stripVTControlCharacters(result.stdout); +} + +export function resetGhApiForTests() { + escapeFlagSupported = true; +} diff --git a/scripts/ci/oversized-test-files-playwright.mjs b/scripts/ci/oversized-test-files-playwright.mjs new file mode 100644 index 00000000000..a4058533112 --- /dev/null +++ b/scripts/ci/oversized-test-files-playwright.mjs @@ -0,0 +1,43 @@ +import { oversizedAnnotation, oversizedThresholdMs, repoRelative } from "./oversized-test-files.mjs"; + +// The spec file the scheduler places on a shard. A test declared in a shared +// helper (smoke-lab.shared.ts) has that helper as its location, so walk up to +// the enclosing file suite instead. +export function specFile(test) { + for (let suite = test.parent; suite; suite = suite.parent) { + if (suite.type === "file" && suite.location?.file) return suite.location.file; + } + return test.location.file; +} + +// Playwright reporter. A spec's cost is the sum of its test results, which +// include hooks and fixtures. Wired for GitHub Actions only; it never fails a run. +export default class OversizedSpecReporter { + constructor({ environment = process.env, write = line => console.log(line), root } = {}) { + this.thresholdMs = oversizedThresholdMs(environment); + this.write = write; + this.root = root; + this.totals = new Map(); + } + + printsToStdio() { + return false; + } + + onTestEnd(test, result) { + if (result.status === "skipped" || !Number.isFinite(result.duration)) return; + const file = specFile(test); + this.totals.set(file, (this.totals.get(file) ?? 0) + result.duration); + } + + onEnd() { + try { + for (const [file, durationMs] of [...this.totals].sort((a, b) => b[1] - a[1])) { + if (durationMs <= this.thresholdMs) continue; + this.write(oversizedAnnotation(repoRelative(file, this.root), durationMs, this.thresholdMs)); + } + } catch { + // A diagnostic reporter must not change a test result. + } + } +} diff --git a/scripts/ci/oversized-test-files.mjs b/scripts/ci/oversized-test-files.mjs new file mode 100644 index 00000000000..fdd7dac4a74 --- /dev/null +++ b/scripts/ci/oversized-test-files.mjs @@ -0,0 +1,51 @@ +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { moduleDurationMs } from "./test-timing-reporter.mjs"; + +// One file over this budget caps how evenly a lane can shard: the scheduler +// cannot split a file. Warn so authors split it; never fail the run. +export const oversizedThresholdEnv = "PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS"; +export const defaultOversizedThresholdSeconds = 180; +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", ".."); + +export function oversizedThresholdMs(environment = process.env) { + const raw = environment[oversizedThresholdEnv]; + const seconds = raw === undefined || raw === "" ? NaN : Number(raw); + return (Number.isFinite(seconds) && seconds > 0 ? seconds : defaultOversizedThresholdSeconds) * 1000; +} + +export function repoRelative(file, root = repoRoot) { + return path.relative(root, file).split(path.sep).join("/"); +} + +// https://docs.github.com/actions/reference/workflow-commands-for-github-actions +const escapeData = value => String(value).replaceAll("%", "%25").replaceAll("\r", "%0D").replaceAll("\n", "%0A"); +const escapeProperty = value => escapeData(value).replaceAll(":", "%3A").replaceAll(",", "%2C"); + +const seconds = ms => `${Number((ms / 1000).toFixed(ms < 10_000 ? 2 : 0))}s`; + +export function oversizedAnnotation(file, durationMs, thresholdMs) { + return `::warning file=${escapeProperty(file)},title=${escapeProperty("Oversized test file")}::` + + escapeData(`${file} took ${seconds(durationMs)}, over the ${seconds(thresholdMs)} per-file budget (${oversizedThresholdEnv}). ` + + "Shards cannot split a file, so this bounds lane wall-clock time. Consider splitting it."); +} + +// Vitest reporter. Wired for GitHub Actions only; it prints and never throws. +export default class OversizedTestFileReporter { + constructor({ environment = process.env, write = line => console.log(line), root = repoRoot } = {}) { + this.thresholdMs = oversizedThresholdMs(environment); + this.write = write; + this.root = root; + } + + onTestModuleEnd(module) { + try { + if (module.state() === "skipped") return; + const durationMs = moduleDurationMs(module.diagnostic()); + if (durationMs === null || durationMs <= this.thresholdMs) return; + this.write(oversizedAnnotation(repoRelative(module.moduleId, this.root), durationMs, this.thresholdMs)); + } catch { + // A diagnostic reporter must not change a test result. + } + } +} diff --git a/scripts/ci/pr-run-durations.mjs b/scripts/ci/pr-run-durations.mjs new file mode 100644 index 00000000000..93b1b11e0bc --- /dev/null +++ b/scripts/ci/pr-run-durations.mjs @@ -0,0 +1,243 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { stripVTControlCharacters } from "node:util"; + +// e2e and the Runner package lane never run on master. Their weights come from +// PR-workflow runs of PRs that merged into master, pinned to the final head SHA. +// Logs are parsed as data; nothing from a run is executed. +export const repository = "paperclipai/paperclip"; +export const repositoryId = 1170821064; +const api = `/repos/${repository}`; + +export const runnerPackageRoot = "packages/paperclip-runner"; +export const e2eManifest = "scripts/e2e-shard-durations.json"; +export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`; + +export function validatePull(pull) { + assert.equal(pull.base?.ref, "master"); + assert.equal(pull.base?.repo?.id, repositoryId); + assert.ok(pull.merged_at, `PR #${pull.number} is not merged`); + assert.match(pull.head?.sha ?? "", /^[a-f0-9]{40}$/); + assert.match(pull.merge_commit_sha ?? "", /^[a-f0-9]{40}$/); +} + +export function validatePrRun(run, pull) { + assert.equal(run.repository?.full_name, repository); + assert.equal(run.repository?.id, repositoryId); + assert.equal(run.event, "pull_request"); + assert.equal(run.path, ".github/workflows/pr.yml"); + assert.equal(run.status, "completed"); + assert.equal(run.conclusion, "success"); + // A run of an earlier push measures code that never merged. + assert.equal(run.head_sha, pull.head.sha); + assert.ok(Number.isSafeInteger(run.id) && run.id > 0); + assert.ok(Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0); +} + +export function e2eShardJob(name) { + const match = /^ci \/ e2e shard \((\d+)\/(\d+)\)$/.exec(name); + return match ? { index: Number(match[1]), count: Number(match[2]) } : null; +} + +// Before #19646: "ci / Verify Paperclip Runner (vitest package)", one lane. +// Since #19646: "ci / Verify Paperclip Runner (vitest package (K/N))". +export function runnerPackageJob(name) { + if (name === "ci / Verify Paperclip Runner (vitest package)") return { index: 1, count: 1 }; + const match = /^ci \/ Verify Paperclip Runner \((?:vitest )?package \((\d+)\/(\d+)\)\)$/.exec(name); + return match ? { index: Number(match[1]), count: Number(match[2]) } : null; +} + +// Returns the jobs only when they form one complete, successful set of shards. +export function completeShardSet(jobs, classify) { + const selected = jobs.map(job => ({ job, shard: classify(job.name) })).filter(item => item.shard); + if (!selected.length) return null; + const count = selected[0].shard.count; + const indexes = new Set(selected.map(item => item.shard.index)); + if (selected.some(item => item.shard.count !== count || item.job.conclusion !== "success")) return null; + if (indexes.size !== selected.length || indexes.size !== count) return null; + for (let index = 1; index <= count; index++) if (!indexes.has(index)) return null; + return selected.map(item => item.job); +} + +function splitLogLine(raw) { + const line = stripVTControlCharacters(raw).replace(/\r$/, ""); + const match = /^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d+))?Z ?(.*)$/.exec(line); + if (!match) return { time: null, text: line }; + const fraction = Number(`0.${match[2] ?? "0"}`); + return { time: Date.parse(`${match[1]}Z`) + fraction * 1000, text: match[3] }; +} + +// A location outside a spec (e.g. smoke-lab.shared.ts) belongs to the shard's +// spec named after it. Several candidates are told apart by the suffix that the +// describe title repeats, e.g. "(P1-P4)" for smoke-lab-p1-p4.spec.ts. +export function attributeSpec(location, title, shardSpecs) { + if (location.endsWith(".spec.ts")) { + assert.ok(!shardSpecs || shardSpecs.includes(location), `${location} is not in this shard`); + return location; + } + assert.ok(shardSpecs, `No shard spec list to attribute ${location}`); + const directory = path.posix.dirname(location); + const stem = path.posix.basename(location).replace(/(\.shared)?\.[cm]?[jt]sx?$/, ""); + const candidates = shardSpecs.filter(spec => path.posix.dirname(spec) === directory + && path.posix.basename(spec).startsWith(`${stem}-`)); + if (candidates.length === 1) return candidates[0]; + const lower = title.toLowerCase(); + const named = candidates.filter(spec => lower.includes(path.posix.basename(spec).slice(stem.length + 1, -".spec.ts".length))); + assert.equal(named.length, 1, `Cannot attribute ${location} (${title}) to one spec`); + return named[0]; +} + +// The e2e lane runs one Playwright worker, so each gap between consecutive +// result lines is the finishing test's cost including fixtures and navigation. +// The first gap starts when Playwright reports "Running N tests". +export function parseE2eLog(log, inventory) { + let shardSpecs = null, start = null, previous = null, expected = null, seen = 0; + const totals = new Map(); + for (const raw of log.split("\n")) { + const { time, text } = splitLogLine(raw); + if (time === null || text.startsWith("[WebServer]")) continue; + const specs = /^shard \d+\/\d+ specs: (.+)$/.exec(text); + if (specs) { + shardSpecs = specs[1].trim().split(/\s+/); + continue; + } + const running = /^Running (\d+) tests? using (\d+) workers?\b/.exec(text); + if (running) { + assert.equal(start, null, "Expected one Playwright run per job"); + assert.equal(Number(running[2]), 1, "Gap attribution requires a single Playwright worker"); + expected = Number(running[1]); + start = previous = time; + continue; + } + if (start === null) continue; + const result = /^\s*\S+\s+\d+\s+\[[^\]]+\]\s+›\s+(tests\/e2e\/[^\s:]+):\d+:\d+\s+›\s+(.*)$/.exec(text); + if (!result) continue; + assert.ok(!result[1].split("/").some(part => part === ".." || part === "."), result[1]); + seen++; + const spec = attributeSpec(result[1], result[2], shardSpecs); + const gap = Math.max(0, time - previous); + previous = time; + totals.set(spec, (totals.get(spec) ?? 0) + gap); + } + assert.notEqual(start, null, "No Playwright run found"); + assert.equal(seen, expected, `Parsed ${seen} of ${expected} Playwright results`); + const parsed = new Map(); + for (const [spec, ms] of totals) { + assert.ok(ms <= 3_600_000, `${spec}: implausible ${ms}ms`); + // Deleted or renamed specs still appear in older trusted runs. + if (inventory.has(spec)) parsed.set(spec, { ms: Math.round(ms), complete: true }); + } + return parsed; +} + +// Vitest prints one line per file, relative to the package: +// ✓ src/live/runnerd-codex-transport.test.ts (257 tests) 126679ms +// The value covers tests and hooks, not collection or setup. +export function parseRunnerLog(log, inventory) { + const parsed = new Map(); + let expected = 0, seen = 0, summaries = 0; + for (const raw of log.split("\n")) { + const { text } = splitLogLine(raw); + const summary = /^\s*Test Files\s+(?:\d+ failed \| )?(\d+) passed\b/.exec(text); + if (summary) { + summaries++; + expected += Number(summary[1]); + continue; + } + const match = /^\s*✓\s+(\S+\.test\.[cm]?[jt]sx?)\s+\(\d+ tests?[^)]*\)\s+(\d+)ms\s*$/.exec(text); + if (!match) continue; + assert.ok(!match[1].split("/").some(part => part === ".." || part === "." || part === ""), match[1]); + assert.ok(!match[1].includes("\\")); + seen++; + const file = `${runnerPackageRoot}/${match[1]}`, ms = Number(match[2]); + assert.ok(ms <= 3_600_000, `${file}: implausible ${ms}ms`); + if (inventory.has(file)) parsed.set(file, { ms: Math.max(parsed.get(file)?.ms ?? 0, ms), complete: false }); + } + assert.ok(summaries > 0, "No Vitest summary found"); + assert.equal(seen, expected, `Parsed ${seen} of ${expected} passed Vitest files`); + return parsed; +} + +// The Runner include globs, read as text so the refresh needs no installed +// dependencies. Tests assert the list stays non-empty and matches real files. +export function runnerIncludeGlobs(repoRoot = process.cwd()) { + const config = readFileSync(path.join(repoRoot, runnerPackageRoot, "vitest.config.ts"), "utf8"); + const block = /\binclude:\s*\[([\s\S]*?)\]/.exec(config); + assert.ok(block, "Runner vitest.config.ts has no include list"); + const globs = [...block[1].matchAll(/"([^"]+)"/g)].map(match => match[1]); + assert.ok(globs.length > 0, "Runner include list is empty"); + return globs; +} + +export function runnerInventory(repoRoot = process.cwd()) { + const globs = runnerIncludeGlobs(repoRoot); + const tracked = execFileSync("git", ["ls-files", "-z", "--", runnerPackageRoot], { cwd: repoRoot, encoding: "utf8" }) + .split("\0").filter(Boolean); + return new Set(tracked.filter(file => { + const relative = file.slice(runnerPackageRoot.length + 1); + return globs.some(glob => path.matchesGlob(relative, glob)); + })); +} + +export function isMergedIntoHead(sha, repoRoot = process.cwd()) { + try { + execFileSync("git", ["merge-base", "--is-ancestor", sha, "HEAD"], { cwd: repoRoot, stdio: "ignore" }); + return true; + } catch { + return false; + } +} + +function keepLargest(target, parsed) { + for (const [file, value] of parsed) { + const previous = target.get(file); + if (!previous || value.ms > previous.ms) target.set(file, value); + } +} + +// lanes: [{ name, classify(jobName) -> {index,count}|null, parse(log, inventory), inventory }] +// Returns { [lane]: { samples: Map[], sources: [] } } with at most maxRuns samples each. +export async function collectMergedPrSamples({ gh, lanes, maxRuns = 3, pullLimit = 40, + isMerged = isMergedIntoHead, log = console.log } = {}) { + const result = Object.fromEntries(lanes.map(lane => [lane.name, { samples: [], sources: [] }])); + const pulls = await gh(`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=${pullLimit}`); + const merged = pulls.filter(pull => pull.merged_at).sort((a, b) => b.merged_at.localeCompare(a.merged_at)); + for (const pull of merged) { + const pending = lanes.filter(lane => result[lane.name].samples.length < maxRuns); + if (!pending.length) break; + validatePull(pull); + // Only measure code present in this checkout. + if (!isMerged(pull.merge_commit_sha)) continue; + const listed = await gh(`${api}/actions/workflows/pr.yml/runs?head_sha=${pull.head.sha}&event=pull_request&status=success&per_page=20`); + const runs = listed.workflow_runs.filter(run => { + try { validatePrRun(run, pull); return true; } catch { return false; } + }).sort((a, b) => b.id - a.id); + if (!runs.length) continue; + const run = runs[0]; + const jobs = []; + for (let page = 1; ; page++) { + const response = await gh(`${api}/actions/runs/${run.id}/jobs?filter=latest&per_page=100&page=${page}`); + jobs.push(...response.jobs); + if (response.jobs.length < 100) break; + } + for (const lane of pending) { + const selected = completeShardSet(jobs, lane.classify); + if (!selected) continue; + const sample = new Map(); + try { + for (const job of selected) { + assert.equal(job.run_id, run.id); + keepLargest(sample, lane.parse(await gh(`${api}/actions/jobs/${job.id}/logs`, false), lane.inventory)); + } + } catch (error) { + log(`::warning::${lane.name}: skipped run ${run.id} for PR #${pull.number}: ${error.message.split("\n")[0]}`); + continue; + } + result[lane.name].samples.push(sample); + result[lane.name].sources.push({ pull: pull.number, run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url }); + } + } + return result; +} diff --git a/scripts/ci/refresh-test-durations.mjs b/scripts/ci/refresh-test-durations.mjs index 8e6753f1542..f98e6215eac 100644 --- a/scripts/ci/refresh-test-durations.mjs +++ b/scripts/ci/refresh-test-durations.mjs @@ -1,20 +1,22 @@ import assert from "node:assert/strict"; -import { execFile, execFileSync } from "node:child_process"; -import { readFileSync, writeFileSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; -import { promisify, stripVTControlCharacters } from "node:util"; +import { stripVTControlCharacters } from "node:util"; +import { listE2eSpecs } from "../e2e-shard.mjs"; +import { ghApi } from "./gh-api.mjs"; +import { oversizedThresholdEnv, oversizedThresholdMs } from "./oversized-test-files.mjs"; +import { collectMergedPrSamples, e2eManifest, e2eShardJob, isMergedIntoHead, parseE2eLog, parseRunnerLog, + runnerInventory, runnerManifest, runnerPackageJob } from "./pr-run-durations.mjs"; import { timingPrefix } from "./test-timing-reporter.mjs"; const repository = "paperclipai/paperclip"; const api = `/repos/${repository}`; -const execute = promisify(execFile); -const github = async (endpoint, json = true) => { - const { stdout } = await execute("gh", ["api", endpoint], { - encoding: "utf8", maxBuffer: json ? 4 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000, - }); - return json ? JSON.parse(stdout) : stdout; -}; +const github = (endpoint, json = true) => ghApi(endpoint, { json }); +const masterComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test."; +const e2eComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback."; +const runnerComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per file: the Vitest file duration printed by the Verify Paperclip Runner package job (tests and hooks; collection and setup are not included). Median across runs. Weights only place files on lanes; they never change which tests run. Missing files retain their prior weight or the median fallback."; export function validateRun(run) { assert.equal(run.repository?.full_name, repository); @@ -59,7 +61,7 @@ export function parseTimings(log, inventory) { .concat([...measured].map(([file, ms]) => [file, { ms, complete: true }]))); } -export function updatedDurations(previous, inventory, samples, sources) { +export function updatedDurations(previous, inventory, samples, sources, { comment = masterComment } = {}) { const measured = [...inventory].filter(file => samples.some(sample => sample.has(file))); assert.ok(measured.length >= inventory.size * 0.9, `Incomplete timing coverage: ${measured.length}/${inventory.size}`); const durations = {}; @@ -78,12 +80,62 @@ export function updatedDurations(previous, inventory, samples, sources) { durations[file] = Math.max(100, Math.ceil(ms.length % 2 ? ms[middle] : (ms[middle - 1] + ms[middle]) / 2)); } return { - $comment: "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.", + $comment: comment, unit: "ms", sources, coverage: { measured: measured.length, withSetup: complete, inventory: inventory.size }, durations, }; } -export async function refreshDurations(runIds = []) { +// Files over the per-file budget, slowest first, across every refreshed manifest. +export function oversizedFiles(manifests, thresholdMs) { + return manifests.flatMap(({ data }) => Object.entries(data.durations)) + .filter(([, ms]) => ms > thresholdMs).sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])); +} + +export const oversizedPlaceholder = ""; + +export function renderPrBody(template, oversized, thresholdMs) { + assert.ok(template.includes(oversizedPlaceholder), "PR body template lacks the oversized-file placeholder"); + const seconds = Math.round(thresholdMs / 1000); + const lines = oversized.length + ? [`- ${oversized.length} test file(s) exceed the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`). A shard cannot split a file, so each one bounds its lane's wall-clock time. Consider splitting:`, + ...oversized.map(([file, ms]) => ` - \`${file}\`: ${Math.round(ms / 1000)}s`)] + : [`- No measured test file exceeds the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`).`]; + return template.replace(oversizedPlaceholder, lines.join("\n")); +} + +function summarizeChange(file, previous, next) { + const before = previous?.durations ?? {}, after = next.durations; + const changes = Object.keys(after).map(key => [key, before[key], after[key]]) + .filter(([, old, value]) => old !== value) + .sort((a, b) => Math.abs(b[2] - (b[1] ?? 0)) - Math.abs(a[2] - (a[1] ?? 0))); + const removed = Object.keys(before).filter(key => !(key in after)); + console.log(`${file}: ${next.coverage.measured}/${next.coverage.inventory} measured; ${next.coverage.withSetup} include setup; ` + + `${changes.length} changed, ${removed.length} removed; sources ${next.sources.map(source => source.pull ? `#${source.pull}/${source.run}` : source.run).join(", ")}`); + for (const [key, old, value] of changes.slice(0, 12)) console.log(` ${key}: ${old ?? "(new)"} -> ${value}`); +} + +export function parseArgs(argv) { + const options = { runIds: [], dryRun: false, prBody: null, lanes: new Set(["server", "e2e", "runner"]) }; + for (const arg of argv) { + if (arg === "--dry-run") options.dryRun = true; + else if (arg.startsWith("--pr-body=")) options.prBody = arg.slice("--pr-body=".length); + else if (arg.startsWith("--lanes=")) { + options.lanes = new Set(arg.slice("--lanes=".length).split(",")); + for (const lane of options.lanes) assert.ok(["server", "e2e", "runner"].includes(lane), `Unknown lane ${lane}`); + } else { + assert.match(arg, /^\d+$/, `Unexpected argument ${arg}`); + options.runIds.push(arg); + } + } + return options; +} + +export const serverLane = name => /General tests \(server \(\d+\/\d+\)\)$/.test(name) ? "general" + : /Serialized tests \(\d+\/\d+\)$/.test(name) ? "serialized" : null; + +const readManifest = file => existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : { durations: {} }; + +async function refreshMasterLanes(runIds) { const inventory = JSON.parse(execFileSync(process.execPath, ["scripts/run-vitest-stable.mjs", "--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1", "--dry-run"], { encoding: "utf8", env: { ...process.env, GITHUB_WORKFLOW: "PR", PAPERCLIP_CI_TEST_SELECTION: "" }, @@ -92,26 +144,43 @@ export async function refreshDurations(runIds = []) { general: { files: new Set(inventory.selectedGeneralServerSuites), path: "scripts/general-server-shard-durations.json", samples: [] }, serialized: { files: new Set(inventory.selectedSerializedSuites), path: "scripts/serialized-shard-durations.json", samples: [] }, }; - if (!runIds.length) { - const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=3`); - runIds = listed.workflow_runs.map(run => run.id).slice(0, 2); - } - assert.ok(runIds.length > 0 && runIds.length <= 3, "Expected one to three successful master runs"); - const sources = []; - for (const id of runIds) { - assert.match(String(id), /^\d+$/); - const run = await github(`${api}/actions/runs/${id}`); - validateRun(run); - execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]); + const runJobs = async run => { const jobs = []; for (let page = 1; ; page++) { - const result = await github(`${api}/actions/runs/${id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`); + const result = await github(`${api}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`); jobs.push(...result.jobs); if (result.jobs.length < 100) break; } + return jobs; + }; + const candidates = []; + if (runIds.length) { + assert.ok(runIds.length <= 3, "Expected one to three successful master runs"); + for (const id of runIds) { + assert.match(String(id), /^\d+$/); + const run = await github(`${api}/actions/runs/${id}`); + validateRun(run); + execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]); + candidates.push({ run, jobs: await runJobs(run) }); + } + } else { + // Most master pushes only run Cloud admission and skip verification, and a + // newer push can land after this checkout. Use the two newest runs this + // checkout contains that actually ran the server test jobs. + const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=50`); + for (const run of listed.workflow_runs) { + if (candidates.length === 2) break; + if (!isMergedIntoHead(run.head_sha)) continue; + validateRun(run); + const jobs = await runJobs(run); + if (jobs.some(job => serverLane(job.name))) candidates.push({ run, jobs }); + } + } + assert.ok(candidates.length > 0, "Expected one to three successful master runs"); + const sources = []; + for (const { run, jobs } of candidates) { const samples = { general: new Map(), serialized: new Map() }; - const selected = jobs.map(job => ({ job, lane: /General tests \(server \(\d+\/\d+\)\)$/.test(job.name) ? "general" - : /Serialized tests \(\d+\/\d+\)$/.test(job.name) ? "serialized" : null })).filter(item => item.lane); + const selected = jobs.map(job => ({ job, lane: serverLane(job.name) })).filter(item => item.lane); assert.ok(selected.length, "No server timing jobs found"); // Keep API traffic and log buffers bounded. Logs are parsed data, never code. for (let start = 0; start < selected.length; start += 3) { @@ -129,13 +198,56 @@ export async function refreshDurations(runIds = []) { for (const lane of Object.keys(lanes)) lanes[lane].samples.push(samples[lane]); sources.push({ run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url }); } - // Validate both lanes before replacing either checked-in manifest. - const updates = Object.values(lanes).map(lane => [lane.path, - updatedDurations(JSON.parse(readFileSync(lane.path, "utf8")), lane.files, lane.samples, sources)]); - for (const [file, data] of updates) { - writeFileSync(file, JSON.stringify(data, null, 2) + "\n"); - console.log(`${file}: ${data.coverage.measured}/${data.coverage.inventory} measured; ${data.coverage.withSetup} include setup`); + return Object.values(lanes).map(lane => { + const previous = readManifest(lane.path); + return { path: lane.path, previous, data: updatedDurations(previous, lane.files, lane.samples, sources) }; + }); +} + +// Fewer than two valid runs leaves a manifest unchanged with a visible warning; +// one run is too noisy to replace measured weights. +async function refreshPrLanes(names) { + if (!names.length) return []; + const definitions = { + e2e: () => ({ classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(listE2eSpecs()), path: e2eManifest, comment: e2eComment }), + runner: () => ({ classify: runnerPackageJob, parse: parseRunnerLog, inventory: runnerInventory(), path: runnerManifest, comment: runnerComment }), + }; + const lanes = names.map(name => ({ name, ...definitions[name]() })); + const collected = await collectMergedPrSamples({ gh: github, lanes }); + const updates = []; + for (const lane of lanes) { + const { samples, sources } = collected[lane.name]; + if (samples.length < 2) { + console.log(`::warning::${lane.path}: only ${samples.length} valid merged-PR run(s); keeping the current weights`); + continue; + } + const previous = readManifest(lane.path); + updates.push({ path: lane.path, previous, + data: updatedDurations(previous, lane.inventory, samples, sources, { comment: lane.comment }) }); } + return updates; +} + +export async function refreshDurations(argv = []) { + const options = parseArgs(argv.map(String)); + const updates = [ + ...(options.lanes.has("server") ? await refreshMasterLanes(options.runIds) : []), + ...await refreshPrLanes(["e2e", "runner"].filter(lane => options.lanes.has(lane))), + ]; + // Validate every lane before replacing any checked-in manifest. + for (const { path: file, previous, data } of updates) { + summarizeChange(file, previous, data); + if (!options.dryRun) writeFileSync(file, JSON.stringify(data, null, 2) + "\n"); + } + const thresholdMs = oversizedThresholdMs(); + const oversized = oversizedFiles(updates, thresholdMs); + console.log(`${oversized.length} measured test file(s) exceed ${Math.round(thresholdMs / 1000)}s`); + for (const [file, ms] of oversized) console.log(` ${file}: ${Math.round(ms / 1000)}s`); + if (options.prBody) { + writeFileSync(options.prBody, renderPrBody(readFileSync(".github/ci-duration-pr.md", "utf8"), oversized, thresholdMs)); + } + if (options.dryRun) console.log("Dry run: no manifest was written"); + return updates; } if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { diff --git a/scripts/ci/test-selection-pr-timings.test.mjs b/scripts/ci/test-selection-pr-timings.test.mjs new file mode 100644 index 00000000000..6993110fafa --- /dev/null +++ b/scripts/ci/test-selection-pr-timings.test.mjs @@ -0,0 +1,318 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import test from "node:test"; +import { ghApi, resetGhApiForTests } from "./gh-api.mjs"; +import OversizedSpecReporter from "./oversized-test-files-playwright.mjs"; +import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs"; +import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog, + runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs"; +import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs"; +import { timingPrefix } from "./test-timing-reporter.mjs"; + +const esc = "\u001b"; +const sha = character => character.repeat(40); +const stamp = seconds => new Date(Date.UTC(2026, 10, 6, 21, 0, 0) + seconds * 1000).toISOString().replace("Z", "0000Z"); +const logLines = lines => lines.map(([seconds, text]) => `${stamp(seconds)} ${text}`).join("\n"); + +test("gh api opts into escape sequences, falls back for older gh, and strips colour from logs", async () => { + resetGhApiForTests(); + const calls = []; + const modern = async (command, args) => { + calls.push(args); + return { stdout: `${esc}[32m ✓${esc}[39m src/a.test.ts (1 test) 5ms` }; + }; + assert.equal(await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: modern }), " ✓ src/a.test.ts (1 test) 5ms"); + assert.deepEqual(calls[0], ["api", "--allow-escape-sequences", "/repos/x/actions/jobs/1/logs"]); + + resetGhApiForTests(); + calls.length = 0; + const legacy = async (command, args) => { + calls.push(args); + if (args.includes("--allow-escape-sequences")) { + throw Object.assign(new Error("Command failed"), { stderr: "unknown flag: --allow-escape-sequences\n\nUsage: gh api [flags]" }); + } + return { stdout: "{\"ok\":true}" }; + }; + assert.deepEqual(await ghApi("/a", { run: legacy }), { ok: true }); + assert.deepEqual(await ghApi("/b", { run: legacy }), { ok: true }); + // The unsupported flag is tried once, then dropped for the process. + assert.deepEqual(calls, [["api", "--allow-escape-sequences", "/a"], ["api", "/a"], ["api", "/b"]]); + + resetGhApiForTests(); + const failing = async () => { throw Object.assign(new Error("HTTP 404"), { stderr: "gh: Not Found (HTTP 404)" }); }; + await assert.rejects(ghApi("/missing", { run: failing }), /HTTP 404/); + resetGhApiForTests(); +}); + +test("server timing parsing tolerates colour codes in fetched logs", () => { + const file = "server/src/slow.test.ts"; + const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`; + assert.deepEqual(parseTimings(log, new Set([file])).get(file), { ms: 360, complete: true }); +}); + +const shardLine = "shard 2/4 specs: tests/e2e/alpha.spec.ts tests/e2e/smoke-lab-p1-p4.spec.ts tests/e2e/smoke-lab-p5-p7.spec.ts tests/e2e/gone.spec.ts"; +const e2eLog = logLines([ + [0, `${esc}[36;1mecho "shard 2/4 specs: $specs"${esc}[0m`], + [1, shardLine], + [2, "[WebServer] > vite build"], + [30, "Running 4 tests using 1 worker"], + [31, "[WebServer] ✓ 4 tests/e2e/alpha.spec.ts noise"], + [40, ` ${esc}[32m✓${esc}[39m 1 [chromium] › tests/e2e/alpha.spec.ts:9:1 › first (7.6s)`], + [45, " - 2 [chromium] › tests/e2e/alpha.spec.ts:20:1 › skipped test"], + [70, " ✓ 3 [chromium] › tests/e2e/smoke-lab.shared.ts:316:3 › Smoke Lab scenario catalog mirror (P5-P7) › records the P5-P7 lifecycle @smoke-lab (24.0s)"], + [75, " ✓ 4 [chromium] › tests/e2e/gone.spec.ts:1:1 › removed later (4.0s)"], + [76, " 1 skipped"], + [76, " 3 passed (46.0s)"], +]); + +test("e2e weights are gaps between Playwright results, attributed to the running spec", () => { + const inventory = new Set(["tests/e2e/alpha.spec.ts", "tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts"]); + const parsed = parseE2eLog(e2eLog, inventory); + assert.deepEqual(Object.fromEntries(parsed), { + "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true }, + "tests/e2e/smoke-lab-p5-p7.spec.ts": { ms: 25_000, complete: true }, + }); + assert.throws(() => parseE2eLog(e2eLog.replace("Running 4 tests", "Running 5 tests"), inventory), /Parsed 4 of 5/); + assert.throws(() => parseE2eLog(e2eLog.replace("using 1 worker", "using 2 workers"), inventory), /single Playwright worker/); + assert.throws(() => parseE2eLog(logLines([[0, shardLine]]), inventory), /No Playwright run/); + assert.throws(() => parseE2eLog(e2eLog.replace("tests/e2e/gone.spec.ts:1:1", "tests/e2e/elsewhere.spec.ts:1:1"), inventory), + /not in this shard/); +}); + +test("shared-file tests map to exactly one shard spec", () => { + const specs = ["tests/e2e/smoke-lab-p1-p4.spec.ts", "tests/e2e/smoke-lab-p5-p7.spec.ts", "tests/e2e/agent-chat-projects.spec.ts"]; + assert.equal(attributeSpec("tests/e2e/smoke-lab.shared.ts", "Smoke Lab (P1-P4) › records", specs), specs[0]); + assert.equal(attributeSpec("tests/e2e/agent-chat.shared.ts", "anything", specs), specs[2]); + assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "no label", specs), /Cannot attribute/); + assert.throws(() => attributeSpec("tests/e2e/helpers/x.ts", "t", specs), /Cannot attribute/); + assert.throws(() => attributeSpec("tests/e2e/smoke-lab.shared.ts", "(P1-P4)", null), /No shard spec list/); +}); + +const runnerLog = logLines([ + [0, " RUN v5.0.0 /home/runner/_work/paperclip/paperclip/packages/paperclip-runner"], + [10, ` ${esc}[32m✓${esc}[39m src/live/runnerd-codex-transport.test.ts ${esc}[2m(257 tests)${esc}[22m 126679ms`], + [11, " ✓ src/control-plane/acpx-controller-recovery.test.ts (11 tests | 1 skipped) 78087ms"], + [12, " ✓ scripts/vitest-sequencer.test.mjs (3 tests) 12ms"], + [12, " ✓ a slow test inside a file 5000ms"], + [13, " ✓ src/deleted.test.ts (1 test) 9ms"], + [20, " Test Files 4 passed | 1 skipped (5)"], + [20, " Tests 272 passed | 1 skipped (273)"], +]); + +test("Runner weights come from Vitest per-file lines and must account for every passed file", () => { + const inventory = new Set(["packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts", + "packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts", + "packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]); + assert.deepEqual(Object.fromEntries(parseRunnerLog(runnerLog, inventory)), { + "packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": { ms: 126_679, complete: false }, + "packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts": { ms: 78_087, complete: false }, + "packages/paperclip-runner/scripts/vitest-sequencer.test.mjs": { ms: 12, complete: false }, + }); + assert.throws(() => parseRunnerLog(runnerLog.replace("4 passed", "5 passed"), inventory), /Parsed 4 of 5/); + assert.throws(() => parseRunnerLog(runnerLog.replace(/ Test Files.*\n/, ""), inventory), /No Vitest summary/); + assert.throws(() => parseRunnerLog(runnerLog.replace("src/deleted.test.ts", "src/../../x.test.ts"), inventory)); +}); + +test("the Runner inventory follows the package's Vitest include list", () => { + const inventory = runnerInventory(); + assert.ok(inventory.size > 100, `unexpectedly small Runner inventory: ${inventory.size}`); + assert.ok(inventory.has("packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts")); + assert.ok(inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.test.mjs")); + // node --test files in the same package are not Vitest weights. + assert.ok(!inventory.has("packages/paperclip-runner/scripts/vitest-sequencer.mjs")); + assert.ok(![...inventory].some(file => file.startsWith("packages/paperclip-runner/test/"))); +}); + +test("job names select complete, successful shard sets only", () => { + assert.deepEqual(e2eShardJob("ci / e2e shard (3/8)"), { index: 3, count: 8 }); + assert.equal(e2eShardJob("ci / e2e"), null); + assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package)"), { index: 1, count: 1 }); + assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (package (2/3))"), { index: 2, count: 3 }); + // The lane labels that #19646 shipped keep the "vitest" prefix. + assert.deepEqual(runnerPackageJob("ci / Verify Paperclip Runner (vitest package (3/3))"), { index: 3, count: 3 }); + assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null); + assert.equal(runnerPackageJob("ci / Verify Paperclip Runner (vitest native server)"), null); + const job = (name, conclusion = "success") => ({ name, conclusion }); + const shards = [job("ci / e2e shard (1/2)"), job("ci / e2e shard (2/2)"), job("ci / build")]; + assert.equal(completeShardSet(shards, e2eShardJob).length, 2); + assert.equal(completeShardSet(shards.slice(1), e2eShardJob), null); + assert.equal(completeShardSet([shards[0], job("ci / e2e shard (2/2)", "skipped")], e2eShardJob), null); + assert.equal(completeShardSet([shards[0], shards[1], shards[1]], e2eShardJob), null); + assert.equal(completeShardSet([job("ci / build")], e2eShardJob), null); +}); + +const pull = (number, head, merge, extra = {}) => ({ number, merged_at: `2026-11-0${number % 9 + 1}T00:00:00Z`, + base: { ref: "master", repo: { id: 1170821064 } }, head: { sha: head }, merge_commit_sha: merge, ...extra }); +const prRun = (id, head, extra = {}) => ({ id, run_attempt: 1, head_sha: head, event: "pull_request", path: ".github/workflows/pr.yml", + status: "completed", conclusion: "success", html_url: `https://example.test/${id}`, + repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, ...extra }); + +test("merged-PR runs must be successful pull_request runs of pr.yml at the PR's final head", () => { + const merged = pull(1, sha("a"), sha("b")); + validatePull(merged); + validatePrRun(prRun(5, sha("a")), merged); + for (const override of [{ merged_at: null }, { base: { ref: "release", repo: { id: 1170821064 } } }, + { base: { ref: "master", repo: { id: 7 } } }, { merge_commit_sha: null }]) { + assert.throws(() => validatePull({ ...merged, ...override })); + } + for (const override of [{ head_sha: sha("c") }, { event: "push" }, { path: ".github/workflows/pr-trusted.yml" }, + { conclusion: "failure" }, { status: "in_progress" }, { repository: { id: 7, full_name: "fork/paperclip" } }, { run_attempt: 0 }]) { + assert.throws(() => validatePrRun(prRun(5, sha("a"), override), merged)); + } +}); + +test("collection skips unmerged, absent, stale-head and incomplete runs, and samples each lane per run", async () => { + const api = "/repos/paperclipai/paperclip"; + const pulls = [pull(1, sha("1"), sha("a")), pull(2, sha("2"), sha("b")), pull(3, sha("3"), sha("c")), + { ...pull(4, sha("4"), sha("d")), merged_at: null }, pull(5, sha("5"), sha("e"))]; + const e2eJobs = [{ id: 101, name: "ci / e2e shard (1/1)" }, { id: 102, name: "ci / Verify Paperclip Runner (vitest package)" }]; + const responses = { + [`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=40`]: pulls, + // PR 1 has only a run of an earlier push; PR 2 is newest-first; PR 5 lacks a Runner job. + [`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("1")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(11, sha("9"))] }, + [`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("2")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(20, sha("2")), prRun(21, sha("2"))] }, + [`${api}/actions/workflows/pr.yml/runs?head_sha=${sha("5")}&event=pull_request&status=success&per_page=20`]: { workflow_runs: [prRun(50, sha("5"))] }, + [`${api}/actions/runs/21/jobs?filter=latest&per_page=100&page=1`]: { jobs: e2eJobs.map(job => ({ ...job, run_id: 21, conclusion: "success" })) }, + [`${api}/actions/runs/50/jobs?filter=latest&per_page=100&page=1`]: { jobs: [{ id: 501, name: "ci / e2e shard (1/1)", run_id: 50, conclusion: "success" }] }, + [`${api}/actions/jobs/101/logs`]: e2eLog, + [`${api}/actions/jobs/102/logs`]: runnerLog, + [`${api}/actions/jobs/501/logs`]: e2eLog.replace("Running 4 tests", "Running 9 tests"), + }; + const requested = []; + const gh = async endpoint => { + requested.push(endpoint); + assert.ok(endpoint in responses, `unexpected request ${endpoint}`); + return responses[endpoint]; + }; + const warnings = []; + const lanes = [ + { name: "e2e", classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(["tests/e2e/alpha.spec.ts"]) }, + { name: "runner", classify: runnerPackageJob, parse: parseRunnerLog, + inventory: new Set(["packages/paperclip-runner/scripts/vitest-sequencer.test.mjs"]) }, + ]; + const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) }); + assert.deepEqual(result.e2e.sources.map(source => source.run), [21]); + assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]); + assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } }); + assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/); + // PR 3 is not in this checkout and PR 4 never merged: neither is queried. + assert.ok(!requested.some(endpoint => endpoint.includes(sha("3")) || endpoint.includes(sha("4")))); +}); + +test("merged-PR manifests use the shared schema and only reorder placement", () => { + const inventory = new Set(["tests/e2e/a.spec.ts", "tests/e2e/b.spec.ts"]); + const samples = [new Map([["tests/e2e/a.spec.ts", { ms: 1000, complete: true }], ["tests/e2e/b.spec.ts", { ms: 50, complete: true }]]), + new Map([["tests/e2e/a.spec.ts", { ms: 3000, complete: true }]])]; + const result = updatedDurations({ durations: {} }, inventory, samples, [{ pull: 1, run: 2 }], { comment: "c" }); + assert.deepEqual(result, { $comment: "c", unit: "ms", sources: [{ pull: 1, run: 2 }], + coverage: { measured: 2, withSetup: 2, inventory: 2 }, durations: { "tests/e2e/a.spec.ts": 2000, "tests/e2e/b.spec.ts": 100 } }); + const manifest = JSON.parse(readFileSync("scripts/e2e-shard-durations.json", "utf8")); + assert.match(manifest.$comment, /never change which specs run/); + assert.equal(manifest.unit, "ms"); + assert.ok(manifest.sources.length >= 2 && manifest.sources.every(source => Number.isSafeInteger(source.pull))); +}); + +test("oversized files are listed in the refresh PR body", () => { + const template = readFileSync(".github/ci-duration-pr.md", "utf8"); + assert.ok(template.includes(oversizedPlaceholder)); + const manifests = [{ data: { durations: { "server/src/a.test.ts": 200_000, "server/src/b.test.ts": 1000 } } }, + { data: { durations: { "tests/e2e/c.spec.ts": 400_000 } } }]; + const oversized = oversizedFiles(manifests, 180_000); + assert.deepEqual(oversized, [["tests/e2e/c.spec.ts", 400_000], ["server/src/a.test.ts", 200_000]]); + const body = renderPrBody(template, oversized, 180_000); + assert.ok(!body.includes(oversizedPlaceholder)); + assert.match(body, /2 test file\(s\) exceed the 180s per-file budget/); + assert.match(body, / - `tests\/e2e\/c\.spec\.ts`: 400s\n - `server\/src\/a\.test\.ts`: 200s/); + assert.match(renderPrBody(template, [], 180_000), /No measured test file exceeds the 180s/); + assert.throws(() => renderPrBody("no placeholder", [], 180_000), /placeholder/); +}); + +test("master sampling recognizes only server timing jobs", () => { + assert.equal(serverLane("ci / General tests (server (3/12))"), "general"); + assert.equal(serverLane("verify / Serialized tests (2/4)"), "serialized"); + // Admission-only master runs have no server jobs and are passed over. + for (const name of ["Cloud candidate admission v1", "verify", "ci / General tests (ui (1/2))"]) assert.equal(serverLane(name), null); +}); + +test("refresh arguments keep explicit master run IDs and add dry-run, PR body and lane selection", () => { + const options = parseArgs(["123", "--dry-run", "--pr-body=/tmp/body.md", "--lanes=e2e,runner"]); + assert.deepEqual(options.runIds, ["123"]); + assert.equal(options.dryRun, true); + assert.equal(options.prBody, "/tmp/body.md"); + assert.deepEqual([...options.lanes], ["e2e", "runner"]); + assert.throws(() => parseArgs(["--lanes=ui"]), /Unknown lane/); + assert.throws(() => parseArgs(["--force"]), /Unexpected argument/); +}); + +test("the refresh workflow may only commit the four timing manifests and uses the rendered PR body", () => { + const workflow = readFileSync(".github/workflows/ci-test-durations.yml", "utf8"); + for (const file of ["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json", + "scripts/e2e-shard-durations.json", "packages/paperclip-runner/scripts/vitest-lane-durations.json"]) { + assert.ok(workflow.split(file).length >= 3, `${file} must be both allowed and staged`); + } + assert.match(workflow, /refresh-test-durations\.mjs "--pr-body=\$RUNNER_TEMP\/ci-duration-pr\.md"/); + assert.match(workflow, /--body-file "\$RUNNER_TEMP\/ci-duration-pr\.md"/); +}); + +test("oversized-file reporters warn above the configurable budget and never throw", () => { + assert.equal(oversizedThresholdMs({}), 180_000); + assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "90" }), 90_000); + for (const invalid of ["", "0", "-5", "soon"]) { + assert.equal(oversizedThresholdMs({ PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: invalid }), 180_000); + } + assert.match(oversizedAnnotation("a,b:c.test.ts", 200_000, 180_000), + /^::warning file=a%2Cb%3Ac\.test\.ts,title=Oversized test file::a,b:c\.test\.ts took 200s, over the 180s per-file budget/); + + const lines = []; + const reporter = new OversizedTestFileReporter({ environment: {}, write: line => lines.push(line), root: "/repo" }); + const module = (file, total, state = "passed") => ({ moduleId: `/repo/${file}`, state: () => state, + diagnostic: () => ({ duration: total - 3, collectDuration: 1, setupDuration: 1, prepareDuration: 1, environmentSetupDuration: 0 }) }); + reporter.onTestModuleEnd(module("server/src/slow.test.ts", 180_001)); + reporter.onTestModuleEnd(module("server/src/failed-slow.test.ts", 200_000, "failed")); + reporter.onTestModuleEnd(module("server/src/fast.test.ts", 180_000)); + reporter.onTestModuleEnd(module("server/src/skipped.test.ts", 900_000, "skipped")); + reporter.onTestModuleEnd({ moduleId: "/repo/x", state: () => { throw new Error("broken"); } }); + assert.deepEqual(lines.map(line => /file=([^,]+)/.exec(line)[1]), ["server/src/slow.test.ts", "server/src/failed-slow.test.ts"]); + + lines.length = 0; + const custom = new OversizedTestFileReporter({ environment: { PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS: "60" }, + write: line => lines.push(line), root: "/repo" }); + custom.onTestModuleEnd(module("server/src/medium.test.ts", 61_000)); + assert.equal(lines.length, 1); + + lines.length = 0; + const playwright = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" }); + assert.equal(playwright.printsToStdio(), false); + const spec = file => ({ location: { file: `/repo/${file}` } }); + playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 100_000 }); + playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "passed", duration: 90_000 }); + playwright.onTestEnd(spec("tests/e2e/long.spec.ts"), { status: "skipped", duration: 500_000 }); + playwright.onTestEnd(spec("tests/e2e/short.spec.ts"), { status: "passed", duration: 170_000 }); + playwright.onEnd(); + assert.equal(lines.length, 1); + assert.match(lines[0], /^::warning file=tests\/e2e\/long\.spec\.ts,.*took 190s/); + + // Two specs that declare their tests through one shared helper are timed + // per spec, not added together under the helper. + lines.length = 0; + const shared = new OversizedSpecReporter({ environment: {}, write: line => lines.push(line), root: "/repo" }); + const viaHelper = specPath => ({ + location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" }, + parent: { type: "describe", location: { file: "/repo/tests/e2e/smoke-lab.shared.ts" }, + parent: { type: "file", location: { file: `/repo/${specPath}` }, parent: { type: "project" } } }, + }); + shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p1-p4.spec.ts"), { status: "passed", duration: 120_000 }); + shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 120_000 }); + shared.onEnd(); + assert.deepEqual(lines, [], "neither spec is over the limit; the helper must not sum them"); + shared.onTestEnd(viaHelper("tests/e2e/smoke-lab-p5-p7.spec.ts"), { status: "passed", duration: 70_000 }); + shared.onEnd(); + assert.equal(lines.length, 1); + assert.match(lines[0], /^::warning file=tests\/e2e\/smoke-lab-p5-p7\.spec\.ts,.*took 190s/); +}); + +test("CI test configs wire the oversized-file reporters for GitHub Actions", () => { + assert.match(readFileSync("vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files\.mjs/); + assert.match(readFileSync("packages/paperclip-runner/vitest.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[^\n]*oversized-test-files\.mjs/); + assert.match(readFileSync("tests/e2e/playwright.config.ts", "utf8"), /GITHUB_ACTIONS === "true"[\s\S]{0,120}oversized-test-files-playwright\.mjs/); +}); diff --git a/scripts/ci/test-timing-reporter.mjs b/scripts/ci/test-timing-reporter.mjs index cbc46660e2b..ab3f376d516 100644 --- a/scripts/ci/test-timing-reporter.mjs +++ b/scripts/ci/test-timing-reporter.mjs @@ -8,11 +8,16 @@ export function fileTiming(module, root = process.cwd()) { if (module.project.name !== "@paperclipai/server" || module.state() !== "passed") return null; const file = path.relative(root, module.moduleId).split(path.sep).join("/"); if (!/^server\/(?:src|scripts)\/.+\.test\.[cm]?[jt]s$/.test(file)) return null; - const timing = module.diagnostic(); - const phases = [timing.duration, timing.collectDuration, timing.setupDuration, - timing.prepareDuration, timing.environmentSetupDuration]; + const durationMs = moduleDurationMs(module.diagnostic()); + return durationMs === null ? null : { version: 1, file, durationMs }; +} + +// A file's whole cost: tests and hooks plus collection, setup and environment. +export function moduleDurationMs(timing) { + const phases = [timing?.duration, timing?.collectDuration, timing?.setupDuration, + timing?.prepareDuration, timing?.environmentSetupDuration]; if (phases.some(value => !Number.isFinite(value) || value < 0)) return null; - return { version: 1, file, durationMs: Math.ceil(phases.reduce((sum, value) => sum + value, 0)) }; + return Math.ceil(phases.reduce((sum, value) => sum + value, 0)); } export default class TestTimingReporter { diff --git a/scripts/e2e-shard-durations.json b/scripts/e2e-shard-durations.json index 1e053bff8f7..7b91c42c0a7 100644 --- a/scripts/e2e-shard-durations.json +++ b/scripts/e2e-shard-durations.json @@ -1,36 +1,90 @@ { - "$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 35036001734 and 35024948947, 2026-09-15) by diffing consecutive test-completion timestamps in the 'Run e2e tests' logs, attributing each gap to the finishing test's spec — that captures fixture and navigation time the list-reporter durations alone miss. Smoke Lab tests report under tests/e2e/smoke-lab.shared.ts; their time is mapped back to the p1-p4/p5-p7 spec by scenario tag. Specs missing here get the median weight, so the manifest only needs occasional refreshes: regenerate the same way when e2e shard steps drift apart. chat-adapters-ui and agent-chat were split 2026-09-15 into providers/messaging specs; their weights are the per-describe sums from the same two runs. agent-chat halves are weighted by their per-test sums scaled to the measured spec total.", + "$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback.", "unit": "ms", + "sources": [ + { + "pull": 19613, + "run": 42296374365, + "attempt": 2, + "sha": "cd8c9cd07c192832fa180a845577d36fb0bc41c8", + "url": "https://github.com/paperclipai/paperclip/actions/runs/42296374365" + }, + { + "pull": 19191, + "run": 42297700476, + "attempt": 1, + "sha": "36c0400008a12ec74ed65a0e17951f1b47221a02", + "url": "https://github.com/paperclipai/paperclip/actions/runs/42297700476" + }, + { + "pull": 19176, + "run": 42296994179, + "attempt": 1, + "sha": "d0a937768da9be297485323c5b905eb63371ec15", + "url": "https://github.com/paperclipai/paperclip/actions/runs/42296994179" + } + ], + "coverage": { + "measured": 56, + "withSetup": 56, + "inventory": 56 + }, "durations": { - "tests/e2e/acp-stop-continuation.spec.ts": 44152, - "tests/e2e/action-test-agent-picker.spec.ts": 10050, - "tests/e2e/agent-chat-projects.spec.ts": 159100, - "tests/e2e/agent-chat-sessions.spec.ts": 151200, - "tests/e2e/agentmail.spec.ts": 8944, - "tests/e2e/app-not-connected.spec.ts": 15684, - "tests/e2e/application-delete-screenshot.spec.ts": 3585, - "tests/e2e/applications-crud.spec.ts": 10924, - "tests/e2e/apps-dark-mode-shots.spec.ts": 27798, - "tests/e2e/apps-prosumer-mcp-flow.spec.ts": 19557, - "tests/e2e/archived-company-url.spec.ts": 11149, - "tests/e2e/board-attachment-receipts.spec.ts": 102785, - "tests/e2e/chat-adapters-ui-messaging.spec.ts": 241600, - "tests/e2e/chat-adapters-ui-providers.spec.ts": 200200, - "tests/e2e/composer-stop.spec.ts": 57673, - "tests/e2e/conference-room-typing-intro.spec.ts": 15196, - "tests/e2e/connection-intents.spec.ts": 14451, - "tests/e2e/connection-reviews.spec.ts": 141760, - "tests/e2e/legacy-failure-continuation.spec.ts": 74050, - "tests/e2e/mcp-user-stories.spec.ts": 38899, - "tests/e2e/nux-phase4-screenshots.spec.ts": 18776, - "tests/e2e/onboarding.spec.ts": 21295, - "tests/e2e/paused-composer.spec.ts": 33211, - "tests/e2e/planning-mode-visual-verification.spec.ts": 26193, - "tests/e2e/project-repositories.spec.ts": 21634, - "tests/e2e/runner-e2e-dashboard.spec.ts": 2028, - "tests/e2e/sidebar-takeover.spec.ts": 19034, - "tests/e2e/signoff-policy.spec.ts": 12121, - "tests/e2e/smoke-lab-p1-p4.spec.ts": 95529, - "tests/e2e/smoke-lab-p5-p7.spec.ts": 64328 + "tests/e2e/acp-stop-continuation.spec.ts": 67299, + "tests/e2e/action-test-agent-picker.spec.ts": 13115, + "tests/e2e/agent-chat-messages.spec.ts": 23569, + "tests/e2e/agent-chat-projects.spec.ts": 206156, + "tests/e2e/agent-chat-sessions.spec.ts": 117218, + "tests/e2e/agent-configuration.spec.ts": 105794, + "tests/e2e/agentmail.spec.ts": 27227, + "tests/e2e/ai-personal-connection-recovery.spec.ts": 12373, + "tests/e2e/announcement-first-session.spec.ts": 9986, + "tests/e2e/announcement-placement.spec.ts": 18485, + "tests/e2e/app-not-connected.spec.ts": 10832, + "tests/e2e/application-delete-screenshot.spec.ts": 2970, + "tests/e2e/applications-crud.spec.ts": 23859, + "tests/e2e/apps-dark-mode-shots.spec.ts": 18910, + "tests/e2e/apps-prosumer-mcp-flow.spec.ts": 13276, + "tests/e2e/archived-company-url.spec.ts": 6145, + "tests/e2e/artifact-tab-arrival.spec.ts": 20087, + "tests/e2e/artifact-uploads.spec.ts": 28200, + "tests/e2e/board-attachment-receipts.spec.ts": 112020, + "tests/e2e/cases-read-only.spec.ts": 9657, + "tests/e2e/chat-adapters-ui-messaging.spec.ts": 181303, + "tests/e2e/chat-adapters-ui-providers.spec.ts": 181541, + "tests/e2e/chunk-load-recovery.spec.ts": 16330, + "tests/e2e/classic-workspace.spec.ts": 6483, + "tests/e2e/cloud-auth.spec.ts": 33711, + "tests/e2e/company-context-refresh.spec.ts": 4128, + "tests/e2e/completed-task-paused-assignee.spec.ts": 25016, + "tests/e2e/composer-stop.spec.ts": 74978, + "tests/e2e/conference-room-typing-intro.spec.ts": 30042, + "tests/e2e/connection-intents.spec.ts": 23775, + "tests/e2e/connection-reviews.spec.ts": 134583, + "tests/e2e/coordinator-inbox-views.spec.ts": 18241, + "tests/e2e/daytona-connector.spec.ts": 2603, + "tests/e2e/disconnect-resilience.spec.ts": 200278, + "tests/e2e/disconnect-write-recovery.spec.ts": 51072, + "tests/e2e/document-image-gallery.spec.ts": 46344, + "tests/e2e/legacy-failure-continuation.spec.ts": 89961, + "tests/e2e/mcp-user-stories.spec.ts": 37417, + "tests/e2e/new-task-picker-touch.spec.ts": 22514, + "tests/e2e/nux-phase4-screenshots.spec.ts": 11330, + "tests/e2e/onboarding.spec.ts": 62933, + "tests/e2e/paused-composer.spec.ts": 38240, + "tests/e2e/plan-task-fixture.spec.ts": 7828, + "tests/e2e/planning-mode-visual-verification.spec.ts": 26325, + "tests/e2e/project-repositories.spec.ts": 13403, + "tests/e2e/railway-catalog.spec.ts": 2813, + "tests/e2e/runner-e2e-dashboard.spec.ts": 4985, + "tests/e2e/runner-task-creation.spec.ts": 32437, + "tests/e2e/sidebar-takeover.spec.ts": 9119, + "tests/e2e/signoff-policy.spec.ts": 52229, + "tests/e2e/smoke-lab-p1-p4.spec.ts": 22390, + "tests/e2e/smoke-lab-p5-p7.spec.ts": 16416, + "tests/e2e/task-loading-frame.spec.ts": 31309, + "tests/e2e/task-reload.spec.ts": 20988, + "tests/e2e/text-attachment-tabs.spec.ts": 12758, + "tests/e2e/unikraft-connector.spec.ts": 2613 } } diff --git a/tests/e2e/playwright.config.ts b/tests/e2e/playwright.config.ts index ff52aaa342d..3b99ee9dd79 100644 --- a/tests/e2e/playwright.config.ts +++ b/tests/e2e/playwright.config.ts @@ -97,6 +97,9 @@ export default defineConfig({ outputDir: "./test-results", reporter: [ ["list"], ["html", { open: "never", outputFolder: "./playwright-report" }], + ...(process.env.GITHUB_ACTIONS === "true" + ? [[path.resolve(import.meta.dirname, "../../scripts/ci/oversized-test-files-playwright.mjs")] as [string]] + : []), ...(process.env.PAPERCLIP_CI_PROFILE === "true" ? [[path.resolve(import.meta.dirname, "../../scripts/ci-profile/playwright-reporter.mjs")] as [string]] : []), diff --git a/vitest.config.ts b/vitest.config.ts index 8b1cf431d44..f8408d06df2 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -3,7 +3,8 @@ import { configDefaults, defineConfig } from "vitest/config"; export default defineConfig({ test: { reporters: [...configDefaults.reporters, - ...(process.env.GITHUB_ACTIONS === "true" ? ["./scripts/ci/test-timing-reporter.mjs"] : []), + ...(process.env.GITHUB_ACTIONS === "true" + ? ["./scripts/ci/test-timing-reporter.mjs", "./scripts/ci/oversized-test-files.mjs"] : []), ...(process.env.PAPERCLIP_CI_PROFILE === "true" ? ["./scripts/ci-profile/vitest-reporter.mjs"] : [])], projects: [ "packages/shared",