mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
ci: cut PR wall clock from ~16 to ~6 minutes (#13521)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the Trusted PR CI workflow before merge > - The test suites roughly tripled in six weeks, and shard balance did not keep up, so PR runs crept from ~4 to ~17 minutes > - Slow CI delays every merge and every contributor > - This pull request rebalances the shards from fresh measurements, splits the largest test files, reuses the Rust build cache in three more jobs, and takes the policy job off the critical path > - The benefit is a PR wall clock near 6 minutes with the same coverage ## Linked Issues or Issue Description **What existing behavior does this improve?** PR CI wall clock. A typical green run took 16-17 minutes. Two months ago it took about 4 minutes. **Subsystem affected** The Trusted PR CI workflow (`.github/workflows/pr-trusted.yml`), the shard-duration manifests, the vitest shard runner scripts, the `paperclip-runner` package scripts, and the dry-run branch of `release.sh`. **Current behavior** The shard-duration manifests were stale. The general-server manifest had durations for ~400 of 649 suites. The e2e manifest was missing 14 of 29 specs. Stale median weights made shard steps range 417s-806s (server) and 277s-745s (e2e). Three jobs each paid a ~3m40s cold cargo release build. Every test lane waited ~60s for the policy job before it could start. **Proposed behavior** All lanes finish in a narrow ~200-290s band. The manifests carry fresh measured durations for every suite. The three largest test files are split so no single file caps a shard. The Rust cache restore runs in every job that builds the Runner binary. Test lanes start as soon as the gate resolves. **Reason and benefit** Merges stop waiting on CI. The projected wall clock is ~6 minutes for the same test coverage. ## What Changed - Rebuild `scripts/general-server-shard-durations.json` (646 suites) and `scripts/e2e-shard-durations.json` (all specs) from per-suite completion timestamps in runs 35036001734 and 35024948947. - Move the PR server lane to the release-verify shape: `general-server-without-chat` across twelve duration-balanced shards, plus the chat integration suite split by collected test location across three dedicated lanes. - Split `tests/e2e/chat-adapters-ui.spec.ts` into `-providers` and `-messaging` specs, and `tests/e2e/agent-chat.spec.ts` into `-sessions` and `-projects` specs. Each pair shares fixtures through a `.shared.ts` module. Playwright collects the same test sets (39 and 20 tests). - Raise e2e shards to eight and serialized shards to nine. - Run the runner package's `check:all` as four matrix lanes: `check:static`, `check:runner`, and two native vitest `--shard` halves. The union is exactly `check:all`. - Add the read-only Rust cache restore (toolchain pin, `save-if: false`) to the Canary Dry Run, Build, and Typecheck jobs. - Make release.sh preview publish payloads concurrently in batches of eight during `--dry-run`. The real publish path stays strictly serial. - Drop the policy-job lockfile artifact chain. Each lane installs with `--frozen-lockfile` and falls back to an inline `--resolution-only` regeneration. The policy job stays a required check through the `verify` and `e2e` aggregates. - Update the shard-count mirrors and workflow assertions in the partition and gate tests. ## Verification - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs` — 30 pass. - `node --test '.github/scripts/tests/'*.test.mjs` — 410 pass. - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/cloud-source-verification.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs` — 42 pass. - `playwright test --list` collects 39 tests across the chat-adapters split and 20 across the agent-chat split, equal to the original files. - A local vitest collection of the chat suite partitions 995 tests into 498/497 line shards. - Projected shard weights: server 230s x12, chat ~143s x3, e2e 207-242s x8, serialized ~216s x9. ## Risks - The split spec files reorder tests relative to the original files. Every describe seeds its own company, so the specs stay independent; a hidden cross-describe dependency would surface as a deterministic failure in one shard. - The inline lockfile fallback changes install behavior for manifest-changing and stacked PRs. The policy job still validates resolution as a required check. - `release.sh` changes are confined to the `--dry-run` preview branch. The publish loop is untouched. `bash -n` passes and the release dry-run tests pass. - One PR now schedules ~44 fleet runners. If the RunsOn fleet caps concurrency, queueing may absorb part of the gain; watch the first runs. ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley authored and GitHub committed
1 parent
c49336acdf
commit
d08abcba15
17 files changed
+5330
-4607
No files matched your search
@@ -17,7 +17,7 @@ const prCallerWorkflow = path.join(repoRoot, ".github", "workflows", "pr.yml");
|
||||
const trustedPrWorkflowPath = ".github/workflows/pr-trusted.yml";
|
||||
const trustedPrWorkflow = path.join(repoRoot, trustedPrWorkflowPath);
|
||||
|
||||
const SHARD_COUNT = 3;
|
||||
const SHARD_COUNT = 8;
|
||||
|
||||
function runShard(args) {
|
||||
const result = spawnSync(process.execPath, [script, ...args], { cwd: repoRoot, encoding: "utf8" });
|
||||
@@ -317,23 +317,25 @@ test("the trusted PR workflow regenerates stale stacked lockfiles", () => {
|
||||
"resolution-only policy must not restore or save a dependency store",
|
||||
);
|
||||
assert.match(
|
||||
workflow,
|
||||
policy,
|
||||
/pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile/,
|
||||
"the policy job must resolve the complete merge tree without rewriting platform metadata",
|
||||
);
|
||||
assert.match(
|
||||
workflow,
|
||||
/cmp -s "\$RUNNER_TEMP\/pnpm-lock\.before\.yaml" pnpm-lock\.yaml/,
|
||||
"the policy job must upload a lockfile only when regeneration changed it",
|
||||
);
|
||||
|
||||
const restoreSteps = workflow.match(
|
||||
/- name: Restore regenerated PR lockfile \(if policy uploaded one\)\n if: needs\.policy\.outputs\.lockfile_regenerated == '1'/g,
|
||||
// Test lanes no longer wait on a policy-job artifact: each install step
|
||||
// resolves a stale lockfile inline, so a manifest-changing or stacked PR
|
||||
// still installs while the policy job validates resolution in parallel.
|
||||
const fallbackInstalls = workflow.match(
|
||||
/if ! pnpm install --frozen-lockfile; then\n[\s\S]{0,240}?pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile\n\s+pnpm install --frozen-lockfile\n\s+fi/g,
|
||||
) ?? [];
|
||||
assert.equal(restoreSteps.length, 7, "every downstream install job must restore a required regenerated artifact");
|
||||
assert.equal(
|
||||
fallbackInstalls.length,
|
||||
7,
|
||||
"every downstream install job must resolve a stale lockfile inline",
|
||||
);
|
||||
assert.doesNotMatch(
|
||||
workflow,
|
||||
/- name: Restore regenerated PR lockfile \(if policy uploaded one\)[\s\S]{0,220}continue-on-error:/,
|
||||
"a missing artifact must fail after the policy job says it uploaded one",
|
||||
/Restore regenerated PR lockfile|lockfile_regenerated|name: pr-lockfile/,
|
||||
"the policy lockfile artifact chain must stay removed; it put the policy job on every lane's critical path",
|
||||
);
|
||||
});
|
||||
@@ -36,7 +36,14 @@ git_current_branch() { printf 'master\\n'; }
|
||||
get_last_stable_tag() { printf 'v2026.709.0\\n'; }
|
||||
get_current_stable_version() { printf '2026.709.0\\n'; }
|
||||
utc_date_iso() { printf '2026-07-10\\n'; }
|
||||
list_public_package_info() { printf 'cli\\tpaperclipai\\t0.0.0\\n'; }
|
||||
list_public_package_info() {
|
||||
if [ -n "\${FAKE_PACKAGE_INFO:-}" ]; then
|
||||
printf '%b' "$FAKE_PACKAGE_INFO"
|
||||
else
|
||||
printf 'cli\\tpaperclipai\\t0.0.0\\n'
|
||||
fi
|
||||
}
|
||||
package_publish_tool() { printf 'pnpm\\n'; }
|
||||
next_stable_version() { printf '2026.710.0\\n'; }
|
||||
next_prerelease_version() { printf '2026.710.0-%s.0\\n' "$1"; }
|
||||
release_notes_file() { printf '%s/releases/v%s.md\\n' "$REPO_ROOT" "$1"; }
|
||||
@@ -56,6 +63,7 @@ require_channel_tag_absent_at_head() {
|
||||
fi
|
||||
echo "[fixture] require_channel_tag_absent_at_head $1"
|
||||
}
|
||||
require_on_master_branch() { :; }
|
||||
require_clean_worktree() { :; }
|
||||
require_npm_publish_auth() { :; }
|
||||
git_local_tag_exists() { return 1; }
|
||||
@@ -112,9 +120,21 @@ exit 0
|
||||
set -euo pipefail
|
||||
printf 'pnpm %s\\n' "$*" >> "$FAKE_CALL_LOG"
|
||||
if [ "$*" = "build" ]; then
|
||||
if [ "\${FAKE_BUILD_OK:-}" = "1" ]; then
|
||||
exit 0
|
||||
fi
|
||||
echo "fixture stopped at workspace build"
|
||||
exit 42
|
||||
fi
|
||||
if [ "$1" = "publish" ]; then
|
||||
echo "fixture publish preview in $PWD"
|
||||
case "$PWD" in
|
||||
*"\${FAKE_FAIL_PUBLISH_DIR:-__none__}")
|
||||
echo "fixture preview failure" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
exit 0
|
||||
`,
|
||||
);
|
||||
@@ -122,8 +142,11 @@ exit 0
|
||||
return { binDir, callLog, fixtureDir, script: join(scriptsDir, "release.sh") };
|
||||
}
|
||||
|
||||
function runRelease(args, extraEnv = {}) {
|
||||
function runRelease(args, extraEnv = {}, prepare = null) {
|
||||
const fixture = createReleaseFixture();
|
||||
if (prepare) {
|
||||
prepare(fixture);
|
||||
}
|
||||
const result = spawnSync(fixture.script, args, {
|
||||
cwd: fixture.fixtureDir,
|
||||
encoding: "utf8",
|
||||
@@ -247,3 +270,92 @@ test("beta refuses commits that never shipped a nightly", () => {
|
||||
assert.match(result.output, /HEAD has no nightly\/v\* tag/);
|
||||
assert.doesNotMatch(result.calls, /^pnpm /m);
|
||||
});
|
||||
|
||||
// --- Step 5 concurrent publish-payload previews (dry-run only) ---
|
||||
|
||||
const PREVIEW_PACKAGE_COUNT = 10; // more than one batch of 8
|
||||
|
||||
function preparePreviewFixture(fixture) {
|
||||
const { fixtureDir } = fixture;
|
||||
// Step 2 copies the skills tree into these package dirs.
|
||||
mkdirSync(join(fixtureDir, "skills"), { recursive: true });
|
||||
for (const dir of ["server", "packages/adapters/claude-local", "packages/adapters/codex-local"]) {
|
||||
mkdirSync(join(fixtureDir, dir), { recursive: true });
|
||||
}
|
||||
// Step 2 build helpers that live outside the pnpm stub.
|
||||
writeExecutable(
|
||||
join(fixtureDir, "scripts", "build-standalone-public-packages.mjs"),
|
||||
"#!/usr/bin/env node\nprocess.exit(0);\n",
|
||||
);
|
||||
writeExecutable(join(fixtureDir, "scripts", "prepare-server-ui-dist.sh"), "#!/usr/bin/env bash\nexit 0\n");
|
||||
writeExecutable(join(fixtureDir, "scripts", "build-npm.sh"), "#!/usr/bin/env bash\nexit 0\n");
|
||||
// Step 4 verifies the CLI package version against the target canary version.
|
||||
mkdirSync(join(fixtureDir, "cli"), { recursive: true });
|
||||
writeFileSync(
|
||||
join(fixtureDir, "cli", "package.json"),
|
||||
JSON.stringify({ name: "paperclipai", version: "2026.710.0-canary.0" }),
|
||||
);
|
||||
// Step 5 cds into each previewed package directory.
|
||||
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
|
||||
mkdirSync(join(fixtureDir, previewPackageDir(index)), { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
function previewPackageDir(index) {
|
||||
return `pkg-${String(index).padStart(2, "0")}`;
|
||||
}
|
||||
|
||||
function previewPackageInfo() {
|
||||
return Array.from({ length: PREVIEW_PACKAGE_COUNT }, (_, i) => {
|
||||
const dir = previewPackageDir(i + 1);
|
||||
return `${dir}\\t@fixture/${dir}\\t0.0.0\\n`;
|
||||
}).join("");
|
||||
}
|
||||
|
||||
test("canary dry-run previews every publish payload and replays logs in package order", () => {
|
||||
const result = runRelease(
|
||||
["canary", "--skip-verify", "--dry-run"],
|
||||
{ FAKE_BUILD_OK: "1", FAKE_PACKAGE_INFO: previewPackageInfo() },
|
||||
preparePreviewFixture,
|
||||
);
|
||||
|
||||
assert.equal(result.status, 0, result.output);
|
||||
const headerOffsets = [];
|
||||
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
|
||||
const offset = result.output.indexOf(`--- ${previewPackageDir(index)} ---`);
|
||||
assert.notEqual(offset, -1, `missing preview header for ${previewPackageDir(index)}`);
|
||||
headerOffsets.push(offset);
|
||||
}
|
||||
assert.deepEqual(
|
||||
headerOffsets,
|
||||
[...headerOffsets].sort((a, b) => a - b),
|
||||
"preview logs must replay in package order even though previews run concurrently",
|
||||
);
|
||||
const previews = result.output.match(/fixture publish preview in /g) ?? [];
|
||||
assert.equal(previews.length, PREVIEW_PACKAGE_COUNT, "every package must be previewed exactly once");
|
||||
assert.match(result.output, /Would create git tag/);
|
||||
});
|
||||
|
||||
test("canary dry-run fails when any concurrent payload preview fails, after replaying every log", () => {
|
||||
const result = runRelease(
|
||||
["canary", "--skip-verify", "--dry-run"],
|
||||
{
|
||||
FAKE_BUILD_OK: "1",
|
||||
FAKE_PACKAGE_INFO: previewPackageInfo(),
|
||||
FAKE_FAIL_PUBLISH_DIR: previewPackageDir(7),
|
||||
},
|
||||
preparePreviewFixture,
|
||||
);
|
||||
|
||||
assert.notEqual(result.status, 0, "a failed preview must fail the dry run");
|
||||
assert.match(result.output, /one or more publish payload previews failed/);
|
||||
assert.match(result.output, /fixture preview failure/);
|
||||
for (let index = 1; index <= PREVIEW_PACKAGE_COUNT; index += 1) {
|
||||
assert.match(
|
||||
result.output,
|
||||
new RegExp(`--- ${previewPackageDir(index)} ---`),
|
||||
"every package's log must still replay so the failure is diagnosable",
|
||||
);
|
||||
}
|
||||
assert.doesNotMatch(result.output, /Would create git tag/, "a failed preview must not reach tagging");
|
||||
});
|
||||
@@ -35,8 +35,8 @@ function dryRunJson(args) {
|
||||
return JSON.parse(result.stdout);
|
||||
}
|
||||
|
||||
const SHARD_COUNT = 5;
|
||||
const SERIALIZED_SHARD_COUNT = 5;
|
||||
const SHARD_COUNT = 12;
|
||||
const SERIALIZED_SHARD_COUNT = 9;
|
||||
|
||||
|
||||
test("the serialized shards form a complete, non-overlapping partition", () => {
|
||||
@@ -174,44 +174,17 @@ test("the checked-in manifest loads and covers most of the current suite set", (
|
||||
);
|
||||
});
|
||||
|
||||
test("the measured chat integration cohort does not share a general-server shard", () => {
|
||||
test("the chat integration suite keeps a measured duration for duration-aware fallbacks", () => {
|
||||
// The PR and release matrices both run the chat suite in dedicated
|
||||
// line-sharded lanes, but the plain general-server group (local full runs)
|
||||
// still weighs it into the LPT partition; a median-fallback weight there
|
||||
// would silently overload whichever shard receives it.
|
||||
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
|
||||
const durations = loadShardDurations(durationsManifest);
|
||||
assert.ok(
|
||||
Number.isFinite(durations[chatSuite]),
|
||||
"the full chat cohort must have a measured duration, not the median fallback",
|
||||
);
|
||||
const unsharded = dryRunJson([
|
||||
"--mode",
|
||||
"general",
|
||||
"--group",
|
||||
"general-server",
|
||||
"--shard-index",
|
||||
"0",
|
||||
"--shard-count",
|
||||
"1",
|
||||
]);
|
||||
const shards = partitionGeneralServerSuites(
|
||||
unsharded.selectedGeneralServerSuites,
|
||||
SHARD_COUNT,
|
||||
durations,
|
||||
);
|
||||
const chatShards = shards.filter((shard) => shard.files.includes(chatSuite));
|
||||
assert.equal(
|
||||
chatShards.length,
|
||||
1,
|
||||
"the full chat cohort must run exactly once",
|
||||
);
|
||||
assert.deepEqual(
|
||||
chatShards[0].files,
|
||||
[chatSuite],
|
||||
"its measured cost must reserve one existing shard without other suites",
|
||||
);
|
||||
assert.deepEqual(
|
||||
shards.flatMap((shard) => shard.files).sort(),
|
||||
[...unsharded.selectedGeneralServerSuites].sort(),
|
||||
"duration balancing must not omit or duplicate any general-server suite",
|
||||
);
|
||||
});
|
||||
|
||||
test("the checked-in serialized manifest loads and covers most of the current suite set", () => {
|
||||
@@ -248,10 +221,12 @@ test("the real serialized shard partition is duration-balanced", () => {
|
||||
});
|
||||
|
||||
test("the real shard partition is duration-balanced", () => {
|
||||
// Mirrors the PR matrix: general-server-without-chat across SHARD_COUNT
|
||||
// runners, with the chat suite carried by the dedicated general-chat lanes.
|
||||
const durations = loadShardDurations(durationsManifest);
|
||||
const fallback = defaultSuiteWeight(durations);
|
||||
const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>
|
||||
dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
|
||||
dryRunJson(["--mode", "general", "--group", "general-server-without-chat", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
|
||||
);
|
||||
|
||||
const totals = shards.map((shard) =>
|
||||
@@ -259,8 +234,12 @@ test("the real shard partition is duration-balanced", () => {
|
||||
);
|
||||
const maxTotal = Math.max(...totals);
|
||||
const minTotal = Math.min(...totals);
|
||||
// LPT keeps the spread within the heaviest single suite; use that as the bound.
|
||||
const heaviest = Math.max(...Object.values(durations));
|
||||
// LPT keeps the spread within the heaviest single suite; use that as the
|
||||
// bound. The chat suite runs in its own lanes, so exclude it here.
|
||||
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
|
||||
const heaviest = Math.max(
|
||||
...Object.entries(durations).filter(([file]) => file !== chat).map(([, ms]) => ms),
|
||||
);
|
||||
assert.ok(
|
||||
maxTotal - minTotal <= heaviest,
|
||||
`shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
|
||||
@@ -268,20 +247,23 @@ test("the real shard partition is duration-balanced", () => {
|
||||
});
|
||||
|
||||
|
||||
test("release server shards plus the dedicated chat file cover the original server group exactly", () => {
|
||||
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
|
||||
const shards = Array.from({ length: 10 }, (_, index) => dryRunJson([
|
||||
"--mode", "general", "--group", "general-server-without-chat",
|
||||
"--shard-index", String(index), "--shard-count", "10",
|
||||
]));
|
||||
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
|
||||
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
|
||||
assert.ok(!files.includes(chat));
|
||||
assert.deepEqual([...files, chat].sort(), full.selectedGeneralServerSuites.sort());
|
||||
assert.equal(new Set(files).size, files.length);
|
||||
const defaultRun = dryRunJson([]);
|
||||
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
|
||||
});
|
||||
// 12 mirrors pr-trusted.yml, 10 mirrors release-verify.yml.
|
||||
for (const withoutChatShardCount of [10, 12]) {
|
||||
test(`${withoutChatShardCount} without-chat shards plus the dedicated chat file cover the original server group exactly`, () => {
|
||||
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
|
||||
const shards = Array.from({ length: withoutChatShardCount }, (_, index) => dryRunJson([
|
||||
"--mode", "general", "--group", "general-server-without-chat",
|
||||
"--shard-index", String(index), "--shard-count", String(withoutChatShardCount),
|
||||
]));
|
||||
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
|
||||
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
|
||||
assert.ok(!files.includes(chat));
|
||||
assert.deepEqual([...files, chat].sort(), full.selectedGeneralServerSuites.sort());
|
||||
assert.equal(new Set(files).size, files.length);
|
||||
const defaultRun = dryRunJson([]);
|
||||
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
|
||||
});
|
||||
}
|
||||
|
||||
const lineShardFile = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
|
||||
const caseAt = (line, name) => ({ name, file: lineShardFile, projectName: "@paperclipai/server", location: { line, column: 3 } });
|
||||
|
||||
@@ -1,21 +1,37 @@
|
||||
{
|
||||
"$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 29775184389 and 29769465682, 2026-07-20). The split Smoke Lab weights scale its prior 168s CI duration by the measured local before/after ratio and scenario count; refresh them from the next two real PR runs. Specs missing here get the median weight, so the manifest only needs occasional refreshes: pull the 'Run e2e tests' logs from a recent PR run and sum the Playwright list-reporter durations per spec file.",
|
||||
"$comment": "Per-spec wall-clock durations (ms) for the Playwright e2e lane, used by scripts/e2e-shard.mjs to balance specs across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 35036001734 and 35024948947, 2026-09-15) by diffing consecutive test-completion timestamps in the 'Run e2e tests' logs, attributing each gap to the finishing test's spec — that captures fixture and navigation time the list-reporter durations alone miss. Smoke Lab tests report under tests/e2e/smoke-lab.shared.ts; their time is mapped back to the p1-p4/p5-p7 spec by scenario tag. Specs missing here get the median weight, so the manifest only needs occasional refreshes: regenerate the same way when e2e shard steps drift apart. chat-adapters-ui and agent-chat were split 2026-09-15 into providers/messaging specs; their weights are the per-describe sums from the same two runs. agent-chat halves are weighted by their per-test sums scaled to the measured spec total.",
|
||||
"unit": "ms",
|
||||
"durations": {
|
||||
"tests/e2e/app-not-connected.spec.ts": 26750,
|
||||
"tests/e2e/application-delete-screenshot.spec.ts": 6750,
|
||||
"tests/e2e/applications-crud.spec.ts": 16100,
|
||||
"tests/e2e/apps-dark-mode-shots.spec.ts": 18000,
|
||||
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 16800,
|
||||
"tests/e2e/conference-room-typing-intro.spec.ts": 4900,
|
||||
"tests/e2e/mcp-user-stories.spec.ts": 39000,
|
||||
"tests/e2e/nux-phase4-screenshots.spec.ts": 19650,
|
||||
"tests/e2e/onboarding.spec.ts": 3200,
|
||||
"tests/e2e/pipelines-tutorial-flow.spec.ts": 30198,
|
||||
"tests/e2e/planning-mode-visual-verification.spec.ts": 23250,
|
||||
"tests/e2e/sidebar-takeover.spec.ts": 16650,
|
||||
"tests/e2e/signoff-policy.spec.ts": 9696,
|
||||
"tests/e2e/smoke-lab-p1-p4.spec.ts": 71000,
|
||||
"tests/e2e/smoke-lab-p5-p7.spec.ts": 53000
|
||||
"tests/e2e/acp-stop-continuation.spec.ts": 44152,
|
||||
"tests/e2e/action-test-agent-picker.spec.ts": 10050,
|
||||
"tests/e2e/agent-chat-projects.spec.ts": 159100,
|
||||
"tests/e2e/agent-chat-sessions.spec.ts": 151200,
|
||||
"tests/e2e/agentmail.spec.ts": 8944,
|
||||
"tests/e2e/app-not-connected.spec.ts": 15684,
|
||||
"tests/e2e/application-delete-screenshot.spec.ts": 3585,
|
||||
"tests/e2e/applications-crud.spec.ts": 10924,
|
||||
"tests/e2e/apps-dark-mode-shots.spec.ts": 27798,
|
||||
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 19557,
|
||||
"tests/e2e/archived-company-url.spec.ts": 11149,
|
||||
"tests/e2e/board-attachment-receipts.spec.ts": 102785,
|
||||
"tests/e2e/chat-adapters-ui-messaging.spec.ts": 241600,
|
||||
"tests/e2e/chat-adapters-ui-providers.spec.ts": 200200,
|
||||
"tests/e2e/composer-stop.spec.ts": 57673,
|
||||
"tests/e2e/conference-room-typing-intro.spec.ts": 15196,
|
||||
"tests/e2e/connection-intents.spec.ts": 14451,
|
||||
"tests/e2e/connection-reviews.spec.ts": 141760,
|
||||
"tests/e2e/legacy-failure-continuation.spec.ts": 74050,
|
||||
"tests/e2e/mcp-user-stories.spec.ts": 38899,
|
||||
"tests/e2e/nux-phase4-screenshots.spec.ts": 18776,
|
||||
"tests/e2e/onboarding.spec.ts": 21295,
|
||||
"tests/e2e/paused-composer.spec.ts": 33211,
|
||||
"tests/e2e/pipelines-tutorial-flow.spec.ts": 30545,
|
||||
"tests/e2e/planning-mode-visual-verification.spec.ts": 26193,
|
||||
"tests/e2e/project-repositories.spec.ts": 21634,
|
||||
"tests/e2e/runner-e2e-dashboard.spec.ts": 2028,
|
||||
"tests/e2e/sidebar-takeover.spec.ts": 19034,
|
||||
"tests/e2e/signoff-policy.spec.ts": 12121,
|
||||
"tests/e2e/smoke-lab-p1-p4.spec.ts": 95529,
|
||||
"tests/e2e/smoke-lab-p5-p7.spec.ts": 64328
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
+57
-11
@@ -314,21 +314,67 @@ VERIFY_DELAY_SECONDS="${NPM_PUBLISH_VERIFY_DELAY_SECONDS:-5}"
|
||||
release_info ""
|
||||
if [ "$dry_run" = true ]; then
|
||||
release_info "==> Step 5/7: Previewing publish payloads (--dry-run)..."
|
||||
# Previews are independent per package (each works in its own temp dir and
|
||||
# never touches npm), so run them concurrently in batches. Serial previews
|
||||
# took ~78s of every PR Canary Dry Run job (run 35036001734, 2026-09-15).
|
||||
# The real publish path below stays strictly serial.
|
||||
preview_logs_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-previews.XXXXXX")"
|
||||
preview_batch_size=8
|
||||
preview_pids=()
|
||||
preview_dirs=()
|
||||
preview_logs=()
|
||||
preview_failed=false
|
||||
|
||||
wait_for_preview_batch() {
|
||||
# Guard the expansion: "${preview_pids[@]}" on an empty array trips
|
||||
# `set -u` under bash < 4.4 (the macOS system bash).
|
||||
if [ "${#preview_pids[@]}" -eq 0 ]; then
|
||||
return 0
|
||||
fi
|
||||
for pid in "${preview_pids[@]}"; do
|
||||
if ! wait "$pid"; then
|
||||
preview_failed=true
|
||||
fi
|
||||
done
|
||||
preview_pids=()
|
||||
}
|
||||
|
||||
while IFS=$'\t' read -r pkg_dir _pkg_name _pkg_version; do
|
||||
[ -z "$pkg_dir" ] && continue
|
||||
release_info " --- $pkg_dir ---"
|
||||
cd "$REPO_ROOT/$pkg_dir"
|
||||
publish_tool="$(package_publish_tool)"
|
||||
if [ "$publish_tool" = "npm" ]; then
|
||||
publish_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-package.XXXXXX")"
|
||||
node "$REPO_ROOT/scripts/prepare-bundled-package.mjs" "$REPO_ROOT/$pkg_dir" "$publish_dir"
|
||||
cd "$publish_dir"
|
||||
run_bundled_npm_pack pack --pack-destination "$publish_dir" 2>&1 | tail -3
|
||||
rm -rf "$publish_dir"
|
||||
else
|
||||
pnpm publish --dry-run --no-git-checks --tag "$DIST_TAG" 2>&1 | tail -3
|
||||
preview_log="$preview_logs_dir/$(printf '%03d' "${#preview_logs[@]}")-${pkg_dir//\//_}.log"
|
||||
preview_dirs+=("$pkg_dir")
|
||||
preview_logs+=("$preview_log")
|
||||
(
|
||||
set -euo pipefail
|
||||
cd "$REPO_ROOT/$pkg_dir"
|
||||
publish_tool="$(package_publish_tool)"
|
||||
if [ "$publish_tool" = "npm" ]; then
|
||||
publish_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-package.XXXXXX")"
|
||||
node "$REPO_ROOT/scripts/prepare-bundled-package.mjs" "$REPO_ROOT/$pkg_dir" "$publish_dir"
|
||||
cd "$publish_dir"
|
||||
run_bundled_npm_pack pack --pack-destination "$publish_dir" 2>&1 | tail -3
|
||||
rm -rf "$publish_dir"
|
||||
else
|
||||
pnpm publish --dry-run --no-git-checks --tag "$DIST_TAG" 2>&1 | tail -3
|
||||
fi
|
||||
) > "$preview_log" 2>&1 &
|
||||
preview_pids+=($!)
|
||||
if [ "$(( ${#preview_pids[@]} % preview_batch_size ))" -eq 0 ]; then
|
||||
wait_for_preview_batch
|
||||
fi
|
||||
done <<< "$VERSIONED_PACKAGE_INFO"
|
||||
wait_for_preview_batch
|
||||
|
||||
preview_index=0
|
||||
for pkg_dir in "${preview_dirs[@]}"; do
|
||||
release_info " --- $pkg_dir ---"
|
||||
cat "${preview_logs[$preview_index]}"
|
||||
preview_index=$((preview_index + 1))
|
||||
done
|
||||
rm -rf "$preview_logs_dir"
|
||||
if [ "$preview_failed" = true ]; then
|
||||
release_fail "one or more publish payload previews failed; see the package output above"
|
||||
fi
|
||||
release_info " [dry-run] Would create git tag $tag_name on $CURRENT_SHA"
|
||||
else
|
||||
release_info "==> Step 5/7: Publishing packages to npm..."
|
||||
|
||||
Reference in new issue
Block a user