perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after the release verification of the
same commit passes (`cloud-readiness.yml` calls `release-verify.yml`)
> - The image is ready about 3 minutes before that gate opens, so the
slowest verification job sets the staging release time
> - The slowest job is a general-server test shard: 11.8-12.7 min, while
other server shards take 8-9 min
> - The shards are balanced by a duration manifest, but the manifest has
not changed since 2026-10-31. The weekly refresh that updates it failed
on 2026-11-02 and 2026-11-09
> - About 40 newer suites per shard use the 1 s median fallback weight,
but several of them run for 90-200 s, so they pile up on one shard
> - This pull request repairs the weekly refresh and refreshes the
server weights from three current master verifications
> - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8
min of test time) on every master verification and every PR run

## Linked Issues or Issue Description

**What happened?**
The Core image publication waits about 13.5 min for source verification.
The critical path is one general-server shard. In three master
verifications (runs 43133035381, 43134251278, 43145252393) the slowest
server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and
8.4 min.

**Root cause**
`scripts/general-server-shard-durations.json` came from one run on
2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice:
- 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A
server shard log is 30-65 MB. One download took 59 s locally. Three
parallel downloads overran the 120 s `execFile` timeout.
- 2026-11-09 (run 42580964686): the refresh's own check failed.
Merged-PR sources for the Runner lane have a leading `pull` key, and the
test accepted only `run, attempt, sha, url`.

**Expected behavior**
The weekly refresh finishes and keeps the shards balanced as suites are
added.

## What Changed

- `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a
512 MB buffer. JSON calls keep 120 s. A failed call now gives a short
error and does not print the partial 60 MB log.
- `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`.
It accepts the optional leading `pull` key that `collectMergedPrSamples`
writes. The checked-in Runner manifest test and the collector test both
use it, so the collector format stays covered.
- `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 ->
45 min (the server lanes alone fetch about 30 large logs).
- `scripts/general-server-shard-durations.json` and
`scripts/serialized-shard-durations.json`: refreshed with `node
scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381
43134251278 43145252393` (median of three successful protected-master
Cloud readiness runs; 1757/1815 general files and 173/173 serialized
files measured).
- No test is added, removed, skipped or reassigned to a different lane.
No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check
are not touched.

## Verification

Measured job timings (minutes from the first job start; three master
verifications):

| Job | 43133035381 | 43134251278 | 43145252393 |
| --- | --- | --- | --- |
| Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7
|
| Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 |
| Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 |
| Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 |
| Chat (slowest of 3) | 8.8 | 10.5 | 9.4 |
| Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 |
| Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 |
| Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 |

- Queue to runner start is 0.5-1.1 min for every job. In a server shard,
setup (checkout, pnpm, Node, install) is about 50 s and the preflight
plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8
min). Setup is not worth caching.
- Shard prediction, using the refreshed medians as the cost of each file
and the 3 fork workers per shard:
- general-server, 10 shards: old weights max 10.3 / min 8.1 min; new
weights 8.8 / 8.8 min.
- serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights
6.6 / 6.6 min.
- The old-weight prediction matches what CI saw: predicted 10.3 and 9.9
min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in
run 43133035381.
- Red/green:
- `node --test --test-name-pattern="gh api"
scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix
(`log timeout 120000`) and passes after it.
- With a `pull` key in the Runner manifest sources, the old "package
lane manifest" test failed with the same assertion as run 42580964686.
The new test passes.
- The refresh script, which died before the fix, completed locally in 8
min after it.
- `node --test scripts/ci/test-selection-timings.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the
complete, non-overlapping shard cover checks.
- `actionlint .github/workflows/ci-test-durations.yml`: clean.
- Not run: the release-verify workflow itself, the scheduled refresh
workflow, and the Runner/e2e lane refresh. The shard times above are
predictions from logged per-file timings, not a measured CI run on this
branch.

## Risks

- Low risk. Weights only move files between shards of the same lane. The
shard tests prove every file still runs exactly once.
- The predicted gain is about 1.5 min on the slowest server shard
(12-12.7 -> about 10.5 min job time). The gate then waits for
"Pre-release Runner chaos evals / Restart, replay, trace, and recovery
faults" (10.2-11.9 min). So the expected gate saving is about 1 min
(13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a
shard is about ±1 min.
- A larger log timeout can make a stuck download wait up to 15 min. The
45 min job timeout still bounds the run.

Candidates for the owner (not changed here):
- The chaos-eval job is now the bound. Its "server finalization" step
takes 7-9 min. Decide if it must gate image publication, or split it.
- Server shard logs are up to 811k lines, mostly repeated stack traces
from `server/src/services/execution-preparation-timing.ts`. That slows
log upload and this refresh.
- 9 test files run over 180 s (largest:
`chat-channels.integration.test.ts` 429 s,
`heartbeat-process-recovery.test.ts` 344 s). One file cannot be split
across shards.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI read-only). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #19650 (earlier repair of the weekly refresh), #21013 (Core
image publish time; separate files).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 05:29:27 -06:00
1 parent f8e1fb0278
commit bed5f70278
7 files changed
+1972 -1485

No files matched your search

+22 -2
View File
@@ -13,10 +13,30 @@ export function isUnknownEscapeFlagError(error) {
return /unknown flag:?\s+--allow-escape-sequences/.test(String(error?.stderr ?? error?.message ?? ""));
}
// A server test shard's job log is 30-65 MB (actions run 43133035381) and one
// download took 59 s on its own. Three in parallel overran the old 120 s limit,
// so the weekly refresh died mid-download and the shard weights stopped moving.
const jsonFetchOptions = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
const logFetchOptions = { maxBuffer: 512 * 1024 * 1024, timeout: 900_000 };
// A failed log download carries the partial log on stdout; never print it.
function conciseError(endpoint, error) {
const reason = error?.killed ? `killed by ${error.signal ?? "timeout"}` : `exit ${error?.code ?? "unknown"}`;
const stderr = String(error?.stderr ?? "").trim().split("\n").slice(-3).join(" | ");
return Object.assign(new Error(`gh api ${endpoint} failed (${reason})${stderr ? `: ${stderr}` : ""}`),
{ stderr: error?.stderr, code: error?.code, killed: error?.killed, signal: error?.signal });
}
// Calls `gh api` with argv only; no shell sees the endpoint. `run` exists for tests.
export async function ghApi(endpoint, { json = true, run = execute } = {}) {
const options = { encoding: "utf8", maxBuffer: json ? 16 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000 };
const call = flags => run("gh", ["api", ...flags, endpoint], options);
const options = { encoding: "utf8", ...(json ? jsonFetchOptions : logFetchOptions) };
const call = async flags => {
try {
return await run("gh", ["api", ...flags, endpoint], options);
} catch (error) {
throw conciseError(endpoint, error);
}
};
let result;
if (escapeFlagSupported) {
try {
+14
View File
@@ -15,6 +15,20 @@ export const runnerPackageRoot = "packages/paperclip-runner";
export const e2eManifest = "scripts/e2e-shard-durations.json";
export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`;
// The source record shape every timing manifest may carry. Protected-master
// sources have run, attempt, sha and url; merged-PR sources (written by
// collectMergedPrSamples below) add a leading pull number. Rejecting that key
// failed the 2026-11-09 weekly refresh and froze the shard weights.
export function assertManifestSource(source) {
const keys = Object.keys(source);
assert.deepEqual(keys[0] === "pull" ? keys.slice(1) : keys, ["run", "attempt", "sha", "url"]);
if (keys[0] === "pull") assert.ok(Number.isSafeInteger(source.pull) && source.pull > 0);
assert.ok(Number.isSafeInteger(source.run) && source.run > 0);
assert.ok(Number.isSafeInteger(source.attempt) && source.attempt > 0);
assert.match(source.sha, /^[a-f0-9]{40}$/);
assert.equal(typeof source.url, "string");
}
export function validatePull(pull) {
assert.equal(pull.base?.ref, "master");
assert.equal(pull.base?.repo?.id, repositoryId);
+37 -1
View File
@@ -4,7 +4,7 @@ import test from "node:test";
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
import { assertManifestSource, attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs";
import { timingPrefix } from "./test-timing-reporter.mjs";
@@ -44,6 +44,36 @@ test("gh api opts into escape sequences, falls back for older gh, and strips col
resetGhApiForTests();
});
test("gh api allows a large server shard log to finish downloading and never prints a partial log", async () => {
resetGhApiForTests();
const seen = [];
const capture = async (command, args, options) => {
seen.push(options);
return { stdout: "{}" };
};
await ghApi("/repos/x/actions/runs/1/jobs", { run: capture });
await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: capture });
// A 64 MB shard log took 59 s alone (actions run 43133035381); three at once
// overran the old 120 s limit and failed the weekly refresh.
assert.equal(seen[0].timeout, 120_000);
assert.ok(seen[1].timeout >= 600_000, `log timeout ${seen[1].timeout}`);
assert.ok(seen[1].maxBuffer >= 256 * 1024 * 1024, `log buffer ${seen[1].maxBuffer}`);
resetGhApiForTests();
const partial = "x".repeat(10_000);
const timedOut = async () => {
throw Object.assign(new Error(`Command failed: gh api /logs\n${partial}`),
{ killed: true, signal: "SIGTERM", stdout: partial, stderr: "" });
};
await assert.rejects(ghApi("/repos/x/actions/jobs/2/logs", { json: false, run: timedOut }), error => {
assert.match(error.message, /\/repos\/x\/actions\/jobs\/2\/logs failed \(killed by SIGTERM\)/);
assert.ok(!error.message.includes(partial));
assert.equal(error.stdout, undefined);
return true;
});
resetGhApiForTests();
});
test("server timing parsing tolerates colour codes in fetched logs", () => {
const file = "server/src/slow.test.ts";
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
@@ -193,6 +223,12 @@ test("collection skips unmerged, absent, stale-head and incomplete runs, and sam
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
// The checked-in manifest test uses this same check, so the format the
// collector writes stays accepted even while no manifest carries it yet.
for (const source of [...result.e2e.sources, ...result.runner.sources]) assertManifestSource(source);
assertManifestSource({ run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" });
for (const bad of [{ run: 21, attempt: 1, sha: sha("2") }, { pull: 0, run: 21, attempt: 1, sha: sha("2"), url: "u" },
{ run: 21, pull: 2, attempt: 1, sha: sha("2"), url: "u" }]) assert.throws(() => assertManifestSource(bad));
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.