mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Core image is published only after the release verification of the same commit passes (`cloud-readiness.yml` calls `release-verify.yml`) > - The image is ready about 3 minutes before that gate opens, so the slowest verification job sets the staging release time > - The slowest job is a general-server test shard: 11.8-12.7 min, while other server shards take 8-9 min > - The shards are balanced by a duration manifest, but the manifest has not changed since 2026-10-31. The weekly refresh that updates it failed on 2026-11-02 and 2026-11-09 > - About 40 newer suites per shard use the 1 s median fallback weight, but several of them run for 90-200 s, so they pile up on one shard > - This pull request repairs the weekly refresh and refreshes the server weights from three current master verifications > - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8 min of test time) on every master verification and every PR run ## Linked Issues or Issue Description **What happened?** The Core image publication waits about 13.5 min for source verification. The critical path is one general-server shard. In three master verifications (runs 43133035381, 43134251278, 43145252393) the slowest server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and 8.4 min. **Root cause** `scripts/general-server-shard-durations.json` came from one run on 2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice: - 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A server shard log is 30-65 MB. One download took 59 s locally. Three parallel downloads overran the 120 s `execFile` timeout. - 2026-11-09 (run 42580964686): the refresh's own check failed. Merged-PR sources for the Runner lane have a leading `pull` key, and the test accepted only `run, attempt, sha, url`. **Expected behavior** The weekly refresh finishes and keeps the shards balanced as suites are added. ## What Changed - `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a 512 MB buffer. JSON calls keep 120 s. A failed call now gives a short error and does not print the partial 60 MB log. - `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`. It accepts the optional leading `pull` key that `collectMergedPrSamples` writes. The checked-in Runner manifest test and the collector test both use it, so the collector format stays covered. - `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 -> 45 min (the server lanes alone fetch about 30 large logs). - `scripts/general-server-shard-durations.json` and `scripts/serialized-shard-durations.json`: refreshed with `node scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381 43134251278 43145252393` (median of three successful protected-master Cloud readiness runs; 1757/1815 general files and 173/173 serialized files measured). - No test is added, removed, skipped or reassigned to a different lane. No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check are not touched. ## Verification Measured job timings (minutes from the first job start; three master verifications): | Job | 43133035381 | 43134251278 | 43145252393 | | --- | --- | --- | --- | | Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7 | | Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 | | Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 | | Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 | | Chat (slowest of 3) | 8.8 | 10.5 | 9.4 | | Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 | | Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 | | Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 | - Queue to runner start is 0.5-1.1 min for every job. In a server shard, setup (checkout, pnpm, Node, install) is about 50 s and the preflight plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8 min). Setup is not worth caching. - Shard prediction, using the refreshed medians as the cost of each file and the 3 fork workers per shard: - general-server, 10 shards: old weights max 10.3 / min 8.1 min; new weights 8.8 / 8.8 min. - serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights 6.6 / 6.6 min. - The old-weight prediction matches what CI saw: predicted 10.3 and 9.9 min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in run 43133035381. - Red/green: - `node --test --test-name-pattern="gh api" scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix (`log timeout 120000`) and passes after it. - With a `pull` key in the Runner manifest sources, the old "package lane manifest" test failed with the same assertion as run 42580964686. The new test passes. - The refresh script, which died before the fix, completed locally in 8 min after it. - `node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the complete, non-overlapping shard cover checks. - `actionlint .github/workflows/ci-test-durations.yml`: clean. - Not run: the release-verify workflow itself, the scheduled refresh workflow, and the Runner/e2e lane refresh. The shard times above are predictions from logged per-file timings, not a measured CI run on this branch. ## Risks - Low risk. Weights only move files between shards of the same lane. The shard tests prove every file still runs exactly once. - The predicted gain is about 1.5 min on the slowest server shard (12-12.7 -> about 10.5 min job time). The gate then waits for "Pre-release Runner chaos evals / Restart, replay, trace, and recovery faults" (10.2-11.9 min). So the expected gate saving is about 1 min (13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a shard is about ±1 min. - A larger log timeout can make a stuck download wait up to 15 min. The 45 min job timeout still bounds the run. Candidates for the owner (not changed here): - The chaos-eval job is now the bound. Its "server finalization" step takes 7-9 min. Decide if it must gate image publication, or split it. - Server shard logs are up to 811k lines, mostly repeated stack traces from `server/src/services/execution-preparation-timing.ts`. That slows log upload and this refresh. - 9 test files run over 180 s (largest: `chat-channels.integration.test.ts` 429 s, `heartbeat-process-recovery.test.ts` 344 s). One file cannot be split across shards. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, GitHub CLI read-only). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Related: #19650 (earlier repair of the weekly refresh), #21013 (Core image publish time; separate files). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f8e1fb0278
commit
bed5f70278
7 files changed
+1972
-1485
No files matched your search
+22
-2
@@ -13,10 +13,30 @@ export function isUnknownEscapeFlagError(error) {
|
||||
return /unknown flag:?\s+--allow-escape-sequences/.test(String(error?.stderr ?? error?.message ?? ""));
|
||||
}
|
||||
|
||||
// A server test shard's job log is 30-65 MB (actions run 43133035381) and one
|
||||
// download took 59 s on its own. Three in parallel overran the old 120 s limit,
|
||||
// so the weekly refresh died mid-download and the shard weights stopped moving.
|
||||
const jsonFetchOptions = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
|
||||
const logFetchOptions = { maxBuffer: 512 * 1024 * 1024, timeout: 900_000 };
|
||||
|
||||
// A failed log download carries the partial log on stdout; never print it.
|
||||
function conciseError(endpoint, error) {
|
||||
const reason = error?.killed ? `killed by ${error.signal ?? "timeout"}` : `exit ${error?.code ?? "unknown"}`;
|
||||
const stderr = String(error?.stderr ?? "").trim().split("\n").slice(-3).join(" | ");
|
||||
return Object.assign(new Error(`gh api ${endpoint} failed (${reason})${stderr ? `: ${stderr}` : ""}`),
|
||||
{ stderr: error?.stderr, code: error?.code, killed: error?.killed, signal: error?.signal });
|
||||
}
|
||||
|
||||
// Calls `gh api` with argv only; no shell sees the endpoint. `run` exists for tests.
|
||||
export async function ghApi(endpoint, { json = true, run = execute } = {}) {
|
||||
const options = { encoding: "utf8", maxBuffer: json ? 16 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000 };
|
||||
const call = flags => run("gh", ["api", ...flags, endpoint], options);
|
||||
const options = { encoding: "utf8", ...(json ? jsonFetchOptions : logFetchOptions) };
|
||||
const call = async flags => {
|
||||
try {
|
||||
return await run("gh", ["api", ...flags, endpoint], options);
|
||||
} catch (error) {
|
||||
throw conciseError(endpoint, error);
|
||||
}
|
||||
};
|
||||
let result;
|
||||
if (escapeFlagSupported) {
|
||||
try {
|
||||
|
||||
@@ -15,6 +15,20 @@ export const runnerPackageRoot = "packages/paperclip-runner";
|
||||
export const e2eManifest = "scripts/e2e-shard-durations.json";
|
||||
export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`;
|
||||
|
||||
// The source record shape every timing manifest may carry. Protected-master
|
||||
// sources have run, attempt, sha and url; merged-PR sources (written by
|
||||
// collectMergedPrSamples below) add a leading pull number. Rejecting that key
|
||||
// failed the 2026-11-09 weekly refresh and froze the shard weights.
|
||||
export function assertManifestSource(source) {
|
||||
const keys = Object.keys(source);
|
||||
assert.deepEqual(keys[0] === "pull" ? keys.slice(1) : keys, ["run", "attempt", "sha", "url"]);
|
||||
if (keys[0] === "pull") assert.ok(Number.isSafeInteger(source.pull) && source.pull > 0);
|
||||
assert.ok(Number.isSafeInteger(source.run) && source.run > 0);
|
||||
assert.ok(Number.isSafeInteger(source.attempt) && source.attempt > 0);
|
||||
assert.match(source.sha, /^[a-f0-9]{40}$/);
|
||||
assert.equal(typeof source.url, "string");
|
||||
}
|
||||
|
||||
export function validatePull(pull) {
|
||||
assert.equal(pull.base?.ref, "master");
|
||||
assert.equal(pull.base?.repo?.id, repositoryId);
|
||||
|
||||
@@ -4,7 +4,7 @@ import test from "node:test";
|
||||
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
|
||||
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
|
||||
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
|
||||
import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
|
||||
import { assertManifestSource, attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
|
||||
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
|
||||
import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs";
|
||||
import { timingPrefix } from "./test-timing-reporter.mjs";
|
||||
@@ -44,6 +44,36 @@ test("gh api opts into escape sequences, falls back for older gh, and strips col
|
||||
resetGhApiForTests();
|
||||
});
|
||||
|
||||
test("gh api allows a large server shard log to finish downloading and never prints a partial log", async () => {
|
||||
resetGhApiForTests();
|
||||
const seen = [];
|
||||
const capture = async (command, args, options) => {
|
||||
seen.push(options);
|
||||
return { stdout: "{}" };
|
||||
};
|
||||
await ghApi("/repos/x/actions/runs/1/jobs", { run: capture });
|
||||
await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: capture });
|
||||
// A 64 MB shard log took 59 s alone (actions run 43133035381); three at once
|
||||
// overran the old 120 s limit and failed the weekly refresh.
|
||||
assert.equal(seen[0].timeout, 120_000);
|
||||
assert.ok(seen[1].timeout >= 600_000, `log timeout ${seen[1].timeout}`);
|
||||
assert.ok(seen[1].maxBuffer >= 256 * 1024 * 1024, `log buffer ${seen[1].maxBuffer}`);
|
||||
|
||||
resetGhApiForTests();
|
||||
const partial = "x".repeat(10_000);
|
||||
const timedOut = async () => {
|
||||
throw Object.assign(new Error(`Command failed: gh api /logs\n${partial}`),
|
||||
{ killed: true, signal: "SIGTERM", stdout: partial, stderr: "" });
|
||||
};
|
||||
await assert.rejects(ghApi("/repos/x/actions/jobs/2/logs", { json: false, run: timedOut }), error => {
|
||||
assert.match(error.message, /\/repos\/x\/actions\/jobs\/2\/logs failed \(killed by SIGTERM\)/);
|
||||
assert.ok(!error.message.includes(partial));
|
||||
assert.equal(error.stdout, undefined);
|
||||
return true;
|
||||
});
|
||||
resetGhApiForTests();
|
||||
});
|
||||
|
||||
test("server timing parsing tolerates colour codes in fetched logs", () => {
|
||||
const file = "server/src/slow.test.ts";
|
||||
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
|
||||
@@ -193,6 +223,12 @@ test("collection skips unmerged, absent, stale-head and incomplete runs, and sam
|
||||
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
|
||||
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
|
||||
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
|
||||
// The checked-in manifest test uses this same check, so the format the
|
||||
// collector writes stays accepted even while no manifest carries it yet.
|
||||
for (const source of [...result.e2e.sources, ...result.runner.sources]) assertManifestSource(source);
|
||||
assertManifestSource({ run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" });
|
||||
for (const bad of [{ run: 21, attempt: 1, sha: sha("2") }, { pull: 0, run: 21, attempt: 1, sha: sha("2"), url: "u" },
|
||||
{ run: 21, pull: 2, attempt: 1, sha: sha("2"), url: "u" }]) assert.throws(() => assertManifestSource(bad));
|
||||
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
|
||||
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
|
||||
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.
|
||||
|
||||
Reference in new issue
Block a user