mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `PR` CI workflow before it can merge > - The `General tests (server (N/12))` and `General tests (chat (N/3))` jobs finish last, so they set the wall-clock time of CI > - Before this change, server shards took 9.5 to 11.7 minutes and chat shards took 6.4 to 9.6 minutes; the target is 7 minutes or less > - Measurement shows that the chat lane was unbalanced and that the server lane is limited by CPU on the 4-vCPU runners > - This pull request balances the chat lane by recorded test time, runs three chat partitions per job, and takes general server weights from PR runs > - The benefit is a chat lane under 7 minutes and measured evidence for the shard count that the server lane needs ## Linked Issues or Issue Description **What existing behavior does this improve?** The PR CI general server and chat test jobs. They are the slowest jobs of every pull request. **Subsystem affected** CI test scheduling: `scripts/run-vitest-stable.mjs`, `scripts/test-line-shard.mjs` and `scripts/ci/refresh-test-durations.mjs`. **Current behavior** Baseline: green `PR` runs [43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012) and [43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316), all on m7i-flex.xlarge (4 vCPU, 16 GB). - Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step, effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports `tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums 1,410 to 1,765 s of file time. The longest file is 303 s (`deferred-execution-receipt-work.test.ts`), so the floor is total work divided by workers, not one file. - Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each job ran two partitions of `chat-channels.integration.test.ts`. The partitions were balanced by collected-case count, but one `it.each` line can run many cases. One of six partitions recorded 532 s of test time against a 377 s mean. **Proposed behavior** - Chat partitions keep each source line whole and balance by recorded time per collected entry. - Each chat job runs three partitions in place of two. - General server weights come from the server jobs of merged PR runs, which run files with the same concurrency as the lane they place. **Reason and benefit** The slowest job sets the CI wall-clock time for every pull request. The chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes. ## What Changed - `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional recorded durations. With durations, it balances source lines by recorded time; entries without a record take the median. Without durations, the behavior is unchanged. - `scripts/chat-shard-durations.json` (new): recorded time per collected chat entry, from the CI profile records of PR run [43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171). 926 of 928 collected entries are measured; the other two are the bogus collection entries that the launcher already drops. - `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that manifest from downloaded `ci-profile-*` artifacts. Each executed case maps to the collected `it`/`it.each` entry that registers it. A case without an entry fails the refresh. - `scripts/run-vitest-stable.mjs`: a sharded chat job runs three partitions in place of two, and the chat launcher passes the recorded durations. A chat job collects the suite once with `vitest list` in place of once per partition, and the per-partition filter checks run at the same time. Before, each partition ran two blocking collections of the 86,000-line file one after another. The server worker count stays at three. - `scripts/ci/refresh-test-durations.mjs` and `scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general server weights from the two newest merged-PR runs. Each run must have the full server shard count from `.github/ci-task-matrix.json` and cover at least 90% of the lane. Serialized weights still come from master Cloud readiness runs. - `scripts/general-server-shard-durations.json`: refreshed with the new source (PR runs 43439145752 and 43439279537). - Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`, `scripts/ci/test-selection-pr-timings.test.mjs` and `server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted partition, the entry mapping, the manifest format, the PR-run collection rules and the three-partition chat launcher. - `doc/DEVELOPING.md`: describes the three chat partitions, the chat weights and how to refresh them, and the new source of the server weights. ## Verification Experiments ran in this PR's own CI with a temporary resource sampler (removed before review). | Setting (run) | Server step vs. same shard's baseline | CPU busy | Max memory used | Result | |---|---|---|---|---| | 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) | 62-73% | 6.8 GB | 1 flaky retry | | 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean | | 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard failure: `decision-models.test.ts` timed out at 15 s twice | | 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB | 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at 30 s on m5a | The server lane is CPU bound: at four workers the CPU is 82-89% busy and wall time improves by only about 10%. Five workers fail tests. Memory is not the limit (no swap, no memory pressure, at least 6 GB free). Node's compile cache and Postgres `fsync=off` showed no attributable gain (import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to 0.2-0.7% of CPU time), so neither is in this PR. Final configuration (server: 3 workers with refreshed weights; chat: 3 weighted partitions per job), job minutes / test-step seconds / effective concurrency: | Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503 attempt 1 | B: attempt 2 | C: attempt 3 | |---|---|---|---|---|---| | server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 / 2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 | | server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85 | 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 | | server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81 | 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 | | server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 / 2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 | | server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85 | 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 | | server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69 | 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 | | server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 / 2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 | | server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83 | 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 | | server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68 | 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 | | server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84 | 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 | | server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 / 2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 | | server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 / 2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 | | chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 | 5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 | | chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 | 6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 | | chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 | 5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 | Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B / C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server worker setting is unchanged, so its run-to-run spread is mostly runner type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min). Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five server shards on m5a.xlarge summed 2,545-3,037 s of file time against about 1,700 s elsewhere; three reached the 20-minute job limit and two failed tests on 15 s timeouts. The server lane settings in this PR are the same as master's, so this is a fleet finding: m5a.xlarge cannot run a server shard inside the job limit (follow-up for the fleet configuration, outside this PR). Flakiness of the final setting: the nine chat jobs in A, B and C had no failure and no retried test. Server shards in A and B had no failure and no retried test. Before the last commit (single chat collection), three runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server retries there (2 in one run, 1 in another) were on the unchanged three-worker server setting. Local checks (correctness only): - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs`: 68 passed. - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/vitest-chat-shards.test.ts`: passed. - `pnpm verify` exits 2 (global CI input changed), so the required CI checks supply the broad coverage. ## Risks - Three chat partitions per job use more memory: the peak was 11.4 GB of 16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63% busy. - Stale chat weights only change balance, never which tests run. Shard membership is still checked with `vitest list` before each run. - The weekly refresh now downloads general server logs from merged-PR runs. Runs that selected a subset of files are skipped before download because their shard count is smaller. - The server lane is still above 7 minutes. See the remaining floor below. **Remaining floor and shard-count recommendation (for a follow-up under `.github/`)** A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs give at most about 2.8 effective concurrency, so the test step is about 560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test step of about 360 s, so about 1,000 s of file time per shard. The current 19,200 s of general server file time therefore needs about 19-20 shards of 4 vCPUs at 3 workers. More workers on the same runners do not help, because the CPU is already the limit. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, GitHub CLI) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
259 lines
12 KiB
JavaScript
259 lines
12 KiB
JavaScript
import assert from "node:assert/strict";
|
||
import { execFileSync } from "node:child_process";
|
||
import { readFileSync } from "node:fs";
|
||
import path from "node:path";
|
||
import { stripVTControlCharacters } from "node:util";
|
||
|
||
// e2e and the Runner package lane never run on master. Their weights come from
|
||
// PR-workflow runs of PRs that merged into master, pinned to the final head SHA.
|
||
// Logs are parsed as data; nothing from a run is executed.
|
||
export const repository = "paperclipai/paperclip";
|
||
export const repositoryId = 1170821064;
|
||
const api = `/repos/${repository}`;
|
||
|
||
export const runnerPackageRoot = "packages/paperclip-runner";
|
||
export const e2eManifest = "scripts/e2e-shard-durations.json";
|
||
export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`;
|
||
|
||
// The source record shape every timing manifest may carry. Protected-master
|
||
// sources have run, attempt, sha and url; merged-PR sources (written by
|
||
// collectMergedPrSamples below) add a leading pull number. Rejecting that key
|
||
// failed the 2026-11-09 weekly refresh and froze the shard weights.
|
||
export function assertManifestSource(source) {
|
||
const keys = Object.keys(source);
|
||
assert.deepEqual(keys[0] === "pull" ? keys.slice(1) : keys, ["run", "attempt", "sha", "url"]);
|
||
if (keys[0] === "pull") assert.ok(Number.isSafeInteger(source.pull) && source.pull > 0);
|
||
assert.ok(Number.isSafeInteger(source.run) && source.run > 0);
|
||
assert.ok(Number.isSafeInteger(source.attempt) && source.attempt > 0);
|
||
assert.match(source.sha, /^[a-f0-9]{40}$/);
|
||
assert.equal(typeof source.url, "string");
|
||
}
|
||
|
||
export function validatePull(pull) {
|
||
assert.equal(pull.base?.ref, "master");
|
||
assert.equal(pull.base?.repo?.id, repositoryId);
|
||
assert.ok(pull.merged_at, `PR #${pull.number} is not merged`);
|
||
assert.match(pull.head?.sha ?? "", /^[a-f0-9]{40}$/);
|
||
assert.match(pull.merge_commit_sha ?? "", /^[a-f0-9]{40}$/);
|
||
}
|
||
|
||
export function validatePrRun(run, pull) {
|
||
assert.equal(run.repository?.full_name, repository);
|
||
assert.equal(run.repository?.id, repositoryId);
|
||
assert.equal(run.event, "pull_request");
|
||
assert.equal(run.path, ".github/workflows/pr.yml");
|
||
assert.equal(run.status, "completed");
|
||
assert.equal(run.conclusion, "success");
|
||
// A run of an earlier push measures code that never merged.
|
||
assert.equal(run.head_sha, pull.head.sha);
|
||
assert.ok(Number.isSafeInteger(run.id) && run.id > 0);
|
||
assert.ok(Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0);
|
||
}
|
||
|
||
export function e2eShardJob(name) {
|
||
const match = /^ci \/ e2e shard \((\d+)\/(\d+)\)$/.exec(name);
|
||
return match ? { index: Number(match[1]), count: Number(match[2]) } : null;
|
||
}
|
||
|
||
// Before #19646: "ci / Verify Paperclip Runner (vitest package)", one lane.
|
||
// Since #19646: "ci / Verify Paperclip Runner (vitest package (K/N))".
|
||
export function runnerPackageJob(name) {
|
||
if (name === "ci / Verify Paperclip Runner (vitest package)") return { index: 1, count: 1 };
|
||
const match = /^ci \/ Verify Paperclip Runner \((?:vitest )?package \((\d+)\/(\d+)\)\)$/.exec(name);
|
||
return match ? { index: Number(match[1]), count: Number(match[2]) } : null;
|
||
}
|
||
|
||
// Returns the jobs only when they form one complete, successful set of shards.
|
||
export function completeShardSet(jobs, classify) {
|
||
const selected = jobs.map(job => ({ job, shard: classify(job.name) })).filter(item => item.shard);
|
||
if (!selected.length) return null;
|
||
const count = selected[0].shard.count;
|
||
const indexes = new Set(selected.map(item => item.shard.index));
|
||
if (selected.some(item => item.shard.count !== count || item.job.conclusion !== "success")) return null;
|
||
if (indexes.size !== selected.length || indexes.size !== count) return null;
|
||
for (let index = 1; index <= count; index++) if (!indexes.has(index)) return null;
|
||
return selected.map(item => item.job);
|
||
}
|
||
|
||
function splitLogLine(raw) {
|
||
const line = stripVTControlCharacters(raw).replace(/\r$/, "");
|
||
const match = /^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d+))?Z ?(.*)$/.exec(line);
|
||
if (!match) return { time: null, text: line };
|
||
const fraction = Number(`0.${match[2] ?? "0"}`);
|
||
return { time: Date.parse(`${match[1]}Z`) + fraction * 1000, text: match[3] };
|
||
}
|
||
|
||
// A location outside a spec (e.g. smoke-lab.shared.ts) belongs to the shard's
|
||
// spec named after it. Several candidates are told apart by the suffix that the
|
||
// describe title repeats, e.g. "(P1-P4)" for smoke-lab-p1-p4.spec.ts.
|
||
export function attributeSpec(location, title, shardSpecs) {
|
||
if (location.endsWith(".spec.ts")) {
|
||
assert.ok(!shardSpecs || shardSpecs.includes(location), `${location} is not in this shard`);
|
||
return location;
|
||
}
|
||
assert.ok(shardSpecs, `No shard spec list to attribute ${location}`);
|
||
const directory = path.posix.dirname(location);
|
||
const stem = path.posix.basename(location).replace(/(\.shared)?\.[cm]?[jt]sx?$/, "");
|
||
const candidates = shardSpecs.filter(spec => path.posix.dirname(spec) === directory
|
||
&& path.posix.basename(spec).startsWith(`${stem}-`));
|
||
if (candidates.length === 1) return candidates[0];
|
||
const lower = title.toLowerCase();
|
||
const named = candidates.filter(spec => lower.includes(path.posix.basename(spec).slice(stem.length + 1, -".spec.ts".length)));
|
||
assert.equal(named.length, 1, `Cannot attribute ${location} (${title}) to one spec`);
|
||
return named[0];
|
||
}
|
||
|
||
// The e2e lane runs one Playwright worker, so each gap between consecutive
|
||
// result lines is the finishing test's cost including fixtures and navigation.
|
||
// The first gap starts when Playwright reports "Running N tests".
|
||
export function parseE2eLog(log, inventory) {
|
||
let shardSpecs = null, start = null, previous = null, expected = null, seen = 0;
|
||
const totals = new Map();
|
||
for (const raw of log.split("\n")) {
|
||
const { time, text } = splitLogLine(raw);
|
||
if (time === null || text.startsWith("[WebServer]")) continue;
|
||
const specs = /^shard \d+\/\d+ specs: (.+)$/.exec(text);
|
||
if (specs) {
|
||
shardSpecs = specs[1].trim().split(/\s+/);
|
||
continue;
|
||
}
|
||
const running = /^Running (\d+) tests? using (\d+) workers?\b/.exec(text);
|
||
if (running) {
|
||
assert.equal(start, null, "Expected one Playwright run per job");
|
||
assert.equal(Number(running[2]), 1, "Gap attribution requires a single Playwright worker");
|
||
expected = Number(running[1]);
|
||
start = previous = time;
|
||
continue;
|
||
}
|
||
if (start === null) continue;
|
||
const result = /^\s*\S+\s+\d+\s+\[[^\]]+\]\s+›\s+(tests\/e2e\/[^\s:]+):\d+:\d+\s+›\s+(.*)$/.exec(text);
|
||
if (!result) continue;
|
||
assert.ok(!result[1].split("/").some(part => part === ".." || part === "."), result[1]);
|
||
seen++;
|
||
const spec = attributeSpec(result[1], result[2], shardSpecs);
|
||
const gap = Math.max(0, time - previous);
|
||
previous = time;
|
||
totals.set(spec, (totals.get(spec) ?? 0) + gap);
|
||
}
|
||
assert.notEqual(start, null, "No Playwright run found");
|
||
assert.equal(seen, expected, `Parsed ${seen} of ${expected} Playwright results`);
|
||
const parsed = new Map();
|
||
for (const [spec, ms] of totals) {
|
||
assert.ok(ms <= 3_600_000, `${spec}: implausible ${ms}ms`);
|
||
// Deleted or renamed specs still appear in older trusted runs.
|
||
if (inventory.has(spec)) parsed.set(spec, { ms: Math.round(ms), complete: true });
|
||
}
|
||
return parsed;
|
||
}
|
||
|
||
// Vitest prints one line per file, relative to the package:
|
||
// ✓ src/live/runnerd-codex-transport.test.ts (257 tests) 126679ms
|
||
// The value covers tests and hooks, not collection or setup.
|
||
export function parseRunnerLog(log, inventory) {
|
||
const parsed = new Map();
|
||
let expected = 0, seen = 0, summaries = 0;
|
||
for (const raw of log.split("\n")) {
|
||
const { text } = splitLogLine(raw);
|
||
const summary = /^\s*Test Files\s+(?:\d+ failed \| )?(\d+) passed\b/.exec(text);
|
||
if (summary) {
|
||
summaries++;
|
||
expected += Number(summary[1]);
|
||
continue;
|
||
}
|
||
const match = /^\s*✓\s+(\S+\.test\.[cm]?[jt]sx?)\s+\(\d+ tests?[^)]*\)\s+(\d+)ms\s*$/.exec(text);
|
||
if (!match) continue;
|
||
assert.ok(!match[1].split("/").some(part => part === ".." || part === "." || part === ""), match[1]);
|
||
assert.ok(!match[1].includes("\\"));
|
||
seen++;
|
||
const file = `${runnerPackageRoot}/${match[1]}`, ms = Number(match[2]);
|
||
assert.ok(ms <= 3_600_000, `${file}: implausible ${ms}ms`);
|
||
if (inventory.has(file)) parsed.set(file, { ms: Math.max(parsed.get(file)?.ms ?? 0, ms), complete: false });
|
||
}
|
||
assert.ok(summaries > 0, "No Vitest summary found");
|
||
assert.equal(seen, expected, `Parsed ${seen} of ${expected} passed Vitest files`);
|
||
return parsed;
|
||
}
|
||
|
||
// The Runner include globs, read as text so the refresh needs no installed
|
||
// dependencies. Tests assert the list stays non-empty and matches real files.
|
||
export function runnerIncludeGlobs(repoRoot = process.cwd()) {
|
||
const config = readFileSync(path.join(repoRoot, runnerPackageRoot, "vitest.config.ts"), "utf8");
|
||
const block = /\binclude:\s*\[([\s\S]*?)\]/.exec(config);
|
||
assert.ok(block, "Runner vitest.config.ts has no include list");
|
||
const globs = [...block[1].matchAll(/"([^"]+)"/g)].map(match => match[1]);
|
||
assert.ok(globs.length > 0, "Runner include list is empty");
|
||
return globs;
|
||
}
|
||
|
||
export function runnerInventory(repoRoot = process.cwd()) {
|
||
const globs = runnerIncludeGlobs(repoRoot);
|
||
const tracked = execFileSync("git", ["ls-files", "-z", "--", runnerPackageRoot], { cwd: repoRoot, encoding: "utf8" })
|
||
.split("\0").filter(Boolean);
|
||
return new Set(tracked.filter(file => {
|
||
const relative = file.slice(runnerPackageRoot.length + 1);
|
||
return globs.some(glob => path.matchesGlob(relative, glob));
|
||
}));
|
||
}
|
||
|
||
export function isMergedIntoHead(sha, repoRoot = process.cwd()) {
|
||
try {
|
||
execFileSync("git", ["merge-base", "--is-ancestor", sha, "HEAD"], { cwd: repoRoot, stdio: "ignore" });
|
||
return true;
|
||
} catch {
|
||
return false;
|
||
}
|
||
}
|
||
|
||
function keepLargest(target, parsed) {
|
||
for (const [file, value] of parsed) {
|
||
const previous = target.get(file);
|
||
if (!previous || value.ms > previous.ms) target.set(file, value);
|
||
}
|
||
}
|
||
|
||
// lanes: [{ name, classify(jobName) -> {index,count}|null, parse(log, inventory), inventory }]
|
||
// Returns { [lane]: { samples: Map[], sources: [] } } with at most maxRuns samples each.
|
||
export async function collectMergedPrSamples({ gh, lanes, maxRuns = 3, pullLimit = 40,
|
||
isMerged = isMergedIntoHead, log = console.log } = {}) {
|
||
const result = Object.fromEntries(lanes.map(lane => [lane.name, { samples: [], sources: [] }]));
|
||
const pulls = await gh(`${api}/pulls?state=closed&base=master&sort=updated&direction=desc&per_page=${pullLimit}`);
|
||
const merged = pulls.filter(pull => pull.merged_at).sort((a, b) => b.merged_at.localeCompare(a.merged_at));
|
||
for (const pull of merged) {
|
||
const pending = lanes.filter(lane => result[lane.name].samples.length < (lane.maxRuns ?? maxRuns));
|
||
if (!pending.length) break;
|
||
validatePull(pull);
|
||
// Only measure code present in this checkout.
|
||
if (!isMerged(pull.merge_commit_sha)) continue;
|
||
const listed = await gh(`${api}/actions/workflows/pr.yml/runs?head_sha=${pull.head.sha}&event=pull_request&status=success&per_page=20`);
|
||
const runs = listed.workflow_runs.filter(run => {
|
||
try { validatePrRun(run, pull); return true; } catch { return false; }
|
||
}).sort((a, b) => b.id - a.id);
|
||
if (!runs.length) continue;
|
||
const run = runs[0];
|
||
const jobs = [];
|
||
for (let page = 1; ; page++) {
|
||
const response = await gh(`${api}/actions/runs/${run.id}/jobs?filter=latest&per_page=100&page=${page}`);
|
||
jobs.push(...response.jobs);
|
||
if (response.jobs.length < 100) break;
|
||
}
|
||
for (const lane of pending) {
|
||
const selected = completeShardSet(jobs, lane.classify);
|
||
if (!selected) continue;
|
||
const sample = new Map();
|
||
try {
|
||
for (const job of selected) {
|
||
assert.equal(job.run_id, run.id);
|
||
keepLargest(sample, lane.parse(await gh(`${api}/actions/jobs/${job.id}/logs`, false), lane.inventory));
|
||
}
|
||
} catch (error) {
|
||
log(`::warning::${lane.name}: skipped run ${run.id} for PR #${pull.number}: ${error.message.split("\n")[0]}`);
|
||
continue;
|
||
}
|
||
if (lane.minFiles && sample.size < lane.minFiles) continue;
|
||
result[lane.name].samples.push(sample);
|
||
result[lane.name].sources.push({ pull: pull.number, run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url });
|
||
}
|
||
}
|
||
return result;
|
||
}
|