mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `PR` CI workflow before it can merge > - The `General tests (server (N/12))` and `General tests (chat (N/3))` jobs finish last, so they set the wall-clock time of CI > - Before this change, server shards took 9.5 to 11.7 minutes and chat shards took 6.4 to 9.6 minutes; the target is 7 minutes or less > - Measurement shows that the chat lane was unbalanced and that the server lane is limited by CPU on the 4-vCPU runners > - This pull request balances the chat lane by recorded test time, runs three chat partitions per job, and takes general server weights from PR runs > - The benefit is a chat lane under 7 minutes and measured evidence for the shard count that the server lane needs ## Linked Issues or Issue Description **What existing behavior does this improve?** The PR CI general server and chat test jobs. They are the slowest jobs of every pull request. **Subsystem affected** CI test scheduling: `scripts/run-vitest-stable.mjs`, `scripts/test-line-shard.mjs` and `scripts/ci/refresh-test-durations.mjs`. **Current behavior** Baseline: green `PR` runs [43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012) and [43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316), all on m7i-flex.xlarge (4 vCPU, 16 GB). - Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step, effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports `tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums 1,410 to 1,765 s of file time. The longest file is 303 s (`deferred-execution-receipt-work.test.ts`), so the floor is total work divided by workers, not one file. - Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each job ran two partitions of `chat-channels.integration.test.ts`. The partitions were balanced by collected-case count, but one `it.each` line can run many cases. One of six partitions recorded 532 s of test time against a 377 s mean. **Proposed behavior** - Chat partitions keep each source line whole and balance by recorded time per collected entry. - Each chat job runs three partitions in place of two. - General server weights come from the server jobs of merged PR runs, which run files with the same concurrency as the lane they place. **Reason and benefit** The slowest job sets the CI wall-clock time for every pull request. The chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes. ## What Changed - `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional recorded durations. With durations, it balances source lines by recorded time; entries without a record take the median. Without durations, the behavior is unchanged. - `scripts/chat-shard-durations.json` (new): recorded time per collected chat entry, from the CI profile records of PR run [43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171). 926 of 928 collected entries are measured; the other two are the bogus collection entries that the launcher already drops. - `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that manifest from downloaded `ci-profile-*` artifacts. Each executed case maps to the collected `it`/`it.each` entry that registers it. A case without an entry fails the refresh. - `scripts/run-vitest-stable.mjs`: a sharded chat job runs three partitions in place of two, and the chat launcher passes the recorded durations. A chat job collects the suite once with `vitest list` in place of once per partition, and the per-partition filter checks run at the same time. Before, each partition ran two blocking collections of the 86,000-line file one after another. The server worker count stays at three. - `scripts/ci/refresh-test-durations.mjs` and `scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general server weights from the two newest merged-PR runs. Each run must have the full server shard count from `.github/ci-task-matrix.json` and cover at least 90% of the lane. Serialized weights still come from master Cloud readiness runs. - `scripts/general-server-shard-durations.json`: refreshed with the new source (PR runs 43439145752 and 43439279537). - Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`, `scripts/ci/test-selection-pr-timings.test.mjs` and `server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted partition, the entry mapping, the manifest format, the PR-run collection rules and the three-partition chat launcher. - `doc/DEVELOPING.md`: describes the three chat partitions, the chat weights and how to refresh them, and the new source of the server weights. ## Verification Experiments ran in this PR's own CI with a temporary resource sampler (removed before review). | Setting (run) | Server step vs. same shard's baseline | CPU busy | Max memory used | Result | |---|---|---|---|---| | 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) | 62-73% | 6.8 GB | 1 flaky retry | | 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean | | 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard failure: `decision-models.test.ts` timed out at 15 s twice | | 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB | 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at 30 s on m5a | The server lane is CPU bound: at four workers the CPU is 82-89% busy and wall time improves by only about 10%. Five workers fail tests. Memory is not the limit (no swap, no memory pressure, at least 6 GB free). Node's compile cache and Postgres `fsync=off` showed no attributable gain (import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to 0.2-0.7% of CPU time), so neither is in this PR. Final configuration (server: 3 workers with refreshed weights; chat: 3 weighted partitions per job), job minutes / test-step seconds / effective concurrency: | Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503 attempt 1 | B: attempt 2 | C: attempt 3 | |---|---|---|---|---|---| | server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 / 2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 | | server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85 | 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 | | server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81 | 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 | | server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 / 2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 | | server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85 | 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 | | server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69 | 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 | | server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 / 2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 | | server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83 | 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 | | server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68 | 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 | | server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84 | 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 | | server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 / 2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 | | server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 / 2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 | | chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 | 5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 | | chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 | 6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 | | chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 | 5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 | Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B / C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server worker setting is unchanged, so its run-to-run spread is mostly runner type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min). Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five server shards on m5a.xlarge summed 2,545-3,037 s of file time against about 1,700 s elsewhere; three reached the 20-minute job limit and two failed tests on 15 s timeouts. The server lane settings in this PR are the same as master's, so this is a fleet finding: m5a.xlarge cannot run a server shard inside the job limit (follow-up for the fleet configuration, outside this PR). Flakiness of the final setting: the nine chat jobs in A, B and C had no failure and no retried test. Server shards in A and B had no failure and no retried test. Before the last commit (single chat collection), three runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server retries there (2 in one run, 1 in another) were on the unchanged three-worker server setting. Local checks (correctness only): - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs`: 68 passed. - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/vitest-chat-shards.test.ts`: passed. - `pnpm verify` exits 2 (global CI input changed), so the required CI checks supply the broad coverage. ## Risks - Three chat partitions per job use more memory: the peak was 11.4 GB of 16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63% busy. - Stale chat weights only change balance, never which tests run. Shard membership is still checked with `vitest list` before each run. - The weekly refresh now downloads general server logs from merged-PR runs. Runs that selected a subset of files are skipped before download because their shard count is smaller. - The server lane is still above 7 minutes. See the remaining floor below. **Remaining floor and shard-count recommendation (for a follow-up under `.github/`)** A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs give at most about 2.8 effective concurrency, so the test step is about 560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test step of about 360 s, so about 1,000 s of file time per shard. The current 19,200 s of general server file time therefore needs about 19-20 shards of 4 vCPUs at 3 workers. More workers on the same runners do not help, because the CPU is already the limit. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, GitHub CLI) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
282 lines
16 KiB
JavaScript
282 lines
16 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import { execFileSync } from "node:child_process";
|
|
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { stripVTControlCharacters } from "node:util";
|
|
import { listE2eSpecs } from "../e2e-shard.mjs";
|
|
import { ghApi } from "./gh-api.mjs";
|
|
import { oversizedThresholdEnv, oversizedThresholdMs } from "./oversized-test-files.mjs";
|
|
import { collectMergedPrSamples, e2eManifest, e2eShardJob, isMergedIntoHead, parseE2eLog, parseRunnerLog,
|
|
runnerInventory, runnerManifest, runnerPackageJob } from "./pr-run-durations.mjs";
|
|
import { timingPrefix } from "./test-timing-reporter.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const api = `/repos/${repository}`;
|
|
const github = (endpoint, json = true) => ghApi(endpoint, { json });
|
|
const masterComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.";
|
|
const e2eComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback.";
|
|
const generalComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA and each covering at least 90% of the general server lane. Per file: the cost the timing reporter prints in the General tests (server) jobs, which includes tests, hooks, collection, setup and environment preparation under the lane's own file concurrency. Median across runs. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.";
|
|
const runnerComment = "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per file: the Vitest file duration printed by the Verify Paperclip Runner package job (tests and hooks; collection and setup are not included). Median across runs. Weights only place files on lanes; they never change which tests run. Missing files retain their prior weight or the median fallback.";
|
|
|
|
export function validateRun(run) {
|
|
assert.equal(run.repository?.full_name, repository);
|
|
assert.equal(run.repository?.id, 1170821064);
|
|
assert.equal(run.head_repository?.id, 1170821064);
|
|
assert.equal(run.head_branch, "master");
|
|
assert.equal(run.event, "push");
|
|
assert.equal(run.path, ".github/workflows/cloud-readiness.yml");
|
|
assert.equal(run.status, "completed");
|
|
assert.equal(run.conclusion, "success");
|
|
assert.match(run.head_sha, /^[a-f0-9]{40}$/);
|
|
assert.ok(Number.isSafeInteger(run.id) && run.id > 0);
|
|
assert.ok(Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0);
|
|
}
|
|
|
|
export function parseTimings(log, inventory) {
|
|
const fallback = new Map(), measured = new Map();
|
|
for (const raw of log.split("\n")) {
|
|
const line = stripVTControlCharacters(raw);
|
|
const offset = line.indexOf(timingPrefix);
|
|
if (offset >= 0) {
|
|
const record = JSON.parse(line.slice(offset + timingPrefix.length));
|
|
assert.equal(record.version, 1);
|
|
assert.equal(typeof record.file, "string");
|
|
assert.match(record.file, /^server\/(src|scripts)\/.+\.test\.[cm]?[jt]s$/);
|
|
assert.ok(!record.file.split("/").some(part => part === ".." || part === "." || part === ""));
|
|
assert.ok(!record.file.includes("\\"));
|
|
assert.ok(Number.isFinite(record.durationMs) && record.durationMs >= 0 && record.durationMs <= 3_600_000);
|
|
// A trusted source run can predate a deletion or a lane reassignment.
|
|
if (!inventory.has(record.file)) continue;
|
|
measured.set(record.file, Math.max(measured.get(record.file) ?? 0, record.durationMs));
|
|
continue;
|
|
}
|
|
// Bootstrap from pre-reporter runs. These values omit import/setup cost;
|
|
// structured measurements take precedence as soon as master emits them.
|
|
const match = line.match(/@paperclipai\/server\s+([^\s]+\.test\.[cm]?[jt]s)\s+\(\d+ tests?[^)]*\)\s+(\d+)ms\s*$/);
|
|
if (!match) continue;
|
|
const file = `server/${match[1]}`, ms = Number(match[2]);
|
|
if (inventory.has(file) && ms <= 3_600_000) fallback.set(file, Math.max(fallback.get(file) ?? 0, ms));
|
|
}
|
|
return new Map([...fallback].map(([file, ms]) => [file, { ms, complete: false }])
|
|
.concat([...measured].map(([file, ms]) => [file, { ms, complete: true }])));
|
|
}
|
|
|
|
export function updatedDurations(previous, inventory, samples, sources, { comment = masterComment } = {}) {
|
|
const measured = [...inventory].filter(file => samples.some(sample => sample.has(file)));
|
|
assert.ok(measured.length >= inventory.size * 0.9, `Incomplete timing coverage: ${measured.length}/${inventory.size}`);
|
|
const durations = {};
|
|
let complete = 0;
|
|
for (const file of [...inventory].sort()) {
|
|
const values = samples.flatMap(sample => sample.has(file) ? [sample.get(file)] : []);
|
|
const full = values.filter(value => value.complete);
|
|
if (full.length) complete++;
|
|
const ms = (full.length ? full : values).map(value => value.ms).sort((a, b) => a - b);
|
|
if (!ms.length) {
|
|
if (Number.isFinite(previous.durations[file]) && previous.durations[file] >= 0) durations[file] = previous.durations[file];
|
|
continue;
|
|
}
|
|
const middle = Math.floor(ms.length / 2);
|
|
// A small floor prevents clock resolution from giving a suite zero cost.
|
|
durations[file] = Math.max(100, Math.ceil(ms.length % 2 ? ms[middle] : (ms[middle - 1] + ms[middle]) / 2));
|
|
}
|
|
return {
|
|
$comment: comment,
|
|
unit: "ms", sources, coverage: { measured: measured.length, withSetup: complete, inventory: inventory.size }, durations,
|
|
};
|
|
}
|
|
|
|
// Files over the per-file budget, slowest first, across every refreshed manifest.
|
|
export function oversizedFiles(manifests, thresholdMs) {
|
|
return manifests.flatMap(({ data }) => Object.entries(data.durations))
|
|
.filter(([, ms]) => ms > thresholdMs).sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
|
|
}
|
|
|
|
export const oversizedPlaceholder = "<!-- oversized-test-files -->";
|
|
|
|
export function renderPrBody(template, oversized, thresholdMs) {
|
|
assert.ok(template.includes(oversizedPlaceholder), "PR body template lacks the oversized-file placeholder");
|
|
const seconds = Math.round(thresholdMs / 1000);
|
|
const lines = oversized.length
|
|
? [`- ${oversized.length} test file(s) exceed the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`). A shard cannot split a file, so each one bounds its lane's wall-clock time. Consider splitting:`,
|
|
...oversized.map(([file, ms]) => ` - \`${file}\`: ${Math.round(ms / 1000)}s`)]
|
|
: [`- No measured test file exceeds the ${seconds}s per-file budget (\`${oversizedThresholdEnv}\`).`];
|
|
return template.replace(oversizedPlaceholder, lines.join("\n"));
|
|
}
|
|
|
|
function summarizeChange(file, previous, next) {
|
|
const before = previous?.durations ?? {}, after = next.durations;
|
|
const changes = Object.keys(after).map(key => [key, before[key], after[key]])
|
|
.filter(([, old, value]) => old !== value)
|
|
.sort((a, b) => Math.abs(b[2] - (b[1] ?? 0)) - Math.abs(a[2] - (a[1] ?? 0)));
|
|
const removed = Object.keys(before).filter(key => !(key in after));
|
|
console.log(`${file}: ${next.coverage.measured}/${next.coverage.inventory} measured; ${next.coverage.withSetup} include setup; `
|
|
+ `${changes.length} changed, ${removed.length} removed; sources ${next.sources.map(source => source.pull ? `#${source.pull}/${source.run}` : source.run).join(", ")}`);
|
|
for (const [key, old, value] of changes.slice(0, 12)) console.log(` ${key}: ${old ?? "(new)"} -> ${value}`);
|
|
}
|
|
|
|
export function parseArgs(argv) {
|
|
const options = { runIds: [], dryRun: false, prBody: null, lanes: new Set(["server", "e2e", "runner"]) };
|
|
for (const arg of argv) {
|
|
if (arg === "--dry-run") options.dryRun = true;
|
|
else if (arg.startsWith("--pr-body=")) options.prBody = arg.slice("--pr-body=".length);
|
|
else if (arg.startsWith("--lanes=")) {
|
|
options.lanes = new Set(arg.slice("--lanes=".length).split(","));
|
|
for (const lane of options.lanes) assert.ok(["server", "e2e", "runner"].includes(lane), `Unknown lane ${lane}`);
|
|
} else {
|
|
assert.match(arg, /^\d+$/, `Unexpected argument ${arg}`);
|
|
options.runIds.push(arg);
|
|
}
|
|
}
|
|
return options;
|
|
}
|
|
|
|
export function generalServerShardCount(matrixFile = ".github/ci-task-matrix.json") {
|
|
const entries = JSON.parse(readFileSync(matrixFile, "utf8")).general_tests
|
|
.filter(entry => entry.group === "general-server-without-chat");
|
|
assert.ok(entries.length > 0 && entries.every(entry => entry.shard_count === entries.length));
|
|
return entries.length;
|
|
}
|
|
|
|
export function generalServerJob(name, fullCount = generalServerShardCount()) {
|
|
const match = /^ci \/ General tests \(server \((\d+)\/(\d+)\)\)$/.exec(name);
|
|
return match && Number(match[2]) === fullCount ? { index: Number(match[1]), count: Number(match[2]) } : null;
|
|
}
|
|
|
|
export const serverLane = name => /General tests \(server \(\d+\/\d+\)\)$/.test(name) ? "general"
|
|
: /Serialized tests \(\d+\/\d+\)$/.test(name) ? "serialized" : null;
|
|
|
|
const readManifest = file => existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : { durations: {} };
|
|
|
|
async function refreshMasterLanes(runIds) {
|
|
const inventory = JSON.parse(execFileSync(process.execPath, ["scripts/run-vitest-stable.mjs", "--mode", "general",
|
|
"--group", "general-server", "--shard-index", "0", "--shard-count", "1", "--dry-run"], {
|
|
encoding: "utf8", env: { ...process.env, GITHUB_WORKFLOW: "PR", PAPERCLIP_CI_TEST_SELECTION: "" },
|
|
}));
|
|
const lanes = {
|
|
serialized: { files: new Set(inventory.selectedSerializedSuites), path: "scripts/serialized-shard-durations.json", samples: [] },
|
|
};
|
|
const runJobs = async run => {
|
|
const jobs = [];
|
|
for (let page = 1; ; page++) {
|
|
const result = await github(`${api}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
|
jobs.push(...result.jobs);
|
|
if (result.jobs.length < 100) break;
|
|
}
|
|
return jobs;
|
|
};
|
|
const candidates = [];
|
|
if (runIds.length) {
|
|
assert.ok(runIds.length <= 3, "Expected one to three successful master runs");
|
|
for (const id of runIds) {
|
|
assert.match(String(id), /^\d+$/);
|
|
const run = await github(`${api}/actions/runs/${id}`);
|
|
validateRun(run);
|
|
execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]);
|
|
candidates.push({ run, jobs: await runJobs(run) });
|
|
}
|
|
} else {
|
|
// Most master pushes only run Cloud admission and skip verification, and a
|
|
// newer push can land after this checkout. Use the two newest runs this
|
|
// checkout contains that actually ran the server test jobs.
|
|
const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=50`);
|
|
for (const run of listed.workflow_runs) {
|
|
if (candidates.length === 2) break;
|
|
if (!isMergedIntoHead(run.head_sha)) continue;
|
|
validateRun(run);
|
|
const jobs = await runJobs(run);
|
|
if (jobs.some(job => serverLane(job.name) === "serialized")) candidates.push({ run, jobs });
|
|
}
|
|
}
|
|
assert.ok(candidates.length > 0, "Expected one to three successful master runs");
|
|
const sources = [];
|
|
for (const { run, jobs } of candidates) {
|
|
const samples = { serialized: new Map() };
|
|
const selected = jobs.map(job => ({ job, lane: serverLane(job.name) })).filter(item => item.lane === "serialized");
|
|
assert.ok(selected.length, "No server timing jobs found");
|
|
// Keep API traffic and log buffers bounded. Logs are parsed data, never code.
|
|
for (let start = 0; start < selected.length; start += 3) {
|
|
await Promise.all(selected.slice(start, start + 3).map(async ({ job, lane }) => {
|
|
assert.equal(job.run_id, run.id);
|
|
assert.equal(job.conclusion, "success");
|
|
const parsed = parseTimings(await github(`${api}/actions/jobs/${job.id}/logs`, false), lanes[lane].files);
|
|
for (const [file, value] of parsed) {
|
|
const previous = samples[lane].get(file);
|
|
if (!previous || (value.complete && !previous.complete)
|
|
|| (value.complete === previous.complete && value.ms > previous.ms)) samples[lane].set(file, value);
|
|
}
|
|
}));
|
|
}
|
|
for (const lane of Object.keys(lanes)) lanes[lane].samples.push(samples[lane]);
|
|
sources.push({ run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url });
|
|
}
|
|
return Object.values(lanes).map(lane => {
|
|
const previous = readManifest(lane.path);
|
|
return { path: lane.path, previous, data: updatedDurations(previous, lane.files, lane.samples, sources) };
|
|
});
|
|
}
|
|
|
|
// Fewer than two valid runs leaves a manifest unchanged with a visible warning;
|
|
// one run is too noisy to replace measured weights.
|
|
export function generalServerInventory() {
|
|
return new Set(JSON.parse(execFileSync(process.execPath, ["scripts/run-vitest-stable.mjs", "--mode", "general",
|
|
"--group", "general-server", "--shard-index", "0", "--shard-count", "1", "--dry-run"], {
|
|
encoding: "utf8", env: { ...process.env, GITHUB_WORKFLOW: "PR", PAPERCLIP_CI_TEST_SELECTION: "" },
|
|
})).selectedGeneralServerSuites);
|
|
}
|
|
|
|
async function refreshPrLanes(names) {
|
|
if (!names.length) return [];
|
|
const definitions = {
|
|
general: () => {
|
|
const inventory = generalServerInventory();
|
|
const fullCount = generalServerShardCount();
|
|
return { classify: name => generalServerJob(name, fullCount), parse: parseTimings, inventory, path: "scripts/general-server-shard-durations.json",
|
|
comment: generalComment, maxRuns: 2, minFiles: Math.ceil(inventory.size * 0.9) };
|
|
},
|
|
e2e: () => ({ classify: e2eShardJob, parse: parseE2eLog, inventory: new Set(listE2eSpecs()), path: e2eManifest, comment: e2eComment }),
|
|
runner: () => ({ classify: runnerPackageJob, parse: parseRunnerLog, inventory: runnerInventory(), path: runnerManifest, comment: runnerComment }),
|
|
};
|
|
const lanes = names.map(name => ({ name, ...definitions[name]() }));
|
|
const collected = await collectMergedPrSamples({ gh: github, lanes });
|
|
const updates = [];
|
|
for (const lane of lanes) {
|
|
const { samples, sources } = collected[lane.name];
|
|
if (samples.length < 2) {
|
|
console.log(`::warning::${lane.path}: only ${samples.length} valid merged-PR run(s); keeping the current weights`);
|
|
continue;
|
|
}
|
|
const previous = readManifest(lane.path);
|
|
updates.push({ path: lane.path, previous,
|
|
data: updatedDurations(previous, lane.inventory, samples, sources, { comment: lane.comment }) });
|
|
}
|
|
return updates;
|
|
}
|
|
|
|
export async function refreshDurations(argv = []) {
|
|
const options = parseArgs(argv.map(String));
|
|
const updates = [
|
|
...(options.lanes.has("server") ? await refreshMasterLanes(options.runIds) : []),
|
|
...await refreshPrLanes([...(options.lanes.has("server") ? ["general"] : []),
|
|
...["e2e", "runner"].filter(lane => options.lanes.has(lane))]),
|
|
];
|
|
// Validate every lane before replacing any checked-in manifest.
|
|
for (const { path: file, previous, data } of updates) {
|
|
summarizeChange(file, previous, data);
|
|
if (!options.dryRun) writeFileSync(file, JSON.stringify(data, null, 2) + "\n");
|
|
}
|
|
const thresholdMs = oversizedThresholdMs();
|
|
const oversized = oversizedFiles(updates, thresholdMs);
|
|
console.log(`${oversized.length} measured test file(s) exceed ${Math.round(thresholdMs / 1000)}s`);
|
|
for (const [file, ms] of oversized) console.log(` ${file}: ${Math.round(ms / 1000)}s`);
|
|
if (options.prBody) {
|
|
writeFileSync(options.prBody, renderPrBody(readFileSync(".github/ci-duration-pr.md", "utf8"), oversized, thresholdMs));
|
|
}
|
|
if (options.dryRun) console.log("Dry run: no manifest was written");
|
|
return updates;
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
refreshDurations(process.argv.slice(2)).catch(error => { console.error(error); process.exitCode = 1; });
|
|
}
|