mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
master
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandPaperclip
|
8df3ee2cf5 |
ci: rebalance serialized tests with current suite durations (#13328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud deployments wait for verified source commits. > - Verification splits serialized server tests across independent runners. > - The shard duration estimates came from August and no longer match current tests. > - Stale estimates put much more work on one runner than the others. > - This PR refreshes the estimates from a complete successful run to balance the existing runners. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time spent waiting for the slowest serialized server-test shard in PR and release verification. **Current behavior** In [Cloud readiness run 34705914878](https://github.com/paperclipai/paperclip/actions/runs/34705914878), the five serialized shards spent 479, 371, 322, 390, and 365 seconds running tests. The recovery suite had a 55-second estimate but now takes about 156 seconds including process overhead. **Proposed behavior** Use fresh per-suite measurements with the existing deterministic duration balancer. Applying the same measured costs to the new assignment gives 385, 385, 386, 385, and 385 seconds. This predicts about 93 seconds less waiting for the slowest shard, before runner/setup overhead. Live CI will confirm the result. **Reason and benefit** Use the existing runners more evenly. No extra runner, test parallelism, cache, timeout, or routing change is needed. **Breaking changes** Suite-to-shard assignments change. The full suite set, assertions, and per-suite process isolation stay the same. **Additional context** Searched related CI and shard PRs. This updates the existing duration manifest, without duplicating a pending sharding implementation. ## What Changed - Refresh all 145 serialized suite weights from the same successful release verification run. - Record source job IDs and the measurement method in the manifest. Durations include process startup, imports, collection, tests, and shutdown. ## Verification - Passed all 30 shard and release-workflow tests: `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Confirmed every measured suite appears exactly once across all five source logs. - Compared old and new assignments using the same measured weights. The maximum fell from 478862ms to 385551ms. - In [PR CI run 34710696242](https://github.com/paperclipai/paperclip/actions/runs/34710696242), all five serialized jobs passed in 7m05s–7m27s including setup. The measured assignment is now balanced in a live run. - The same run passed full typecheck, all grouped tests, native verification, build, release dry run, and browser checks. Local full-suite verification on this base was limited by disk exhaustion; local typecheck and targeted shard tests passed. - Latest-head Greptile is 5/5 with no open findings. Every current-head CI check must be green or intentionally skipped before merge. ## Risks - Individual durations vary with load and future test changes. These estimates affect assignment only; missing or renamed suites receive the existing median weight. - Both PR and release verification read this manifest, so both receive the new assignments. Each suite still runs in its own serialized Vitest process. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
49217aadf0 |
refactor: balance serialized server shards by recorded suite duration (#11528)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100). --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The PR verify workflow gates every pull request; its wall-clock time sets the feedback loop for all contributors > - In a recent successful PR run (actions run 32012408876), the slowest check was "Verify serialized server suites (1/5)" at 337s, while its four sibling shards finished in 212-238s > - The serialized lane assigns suites to shards round-robin over an alphabetical list, so the heavy heartbeat and issues suites cluster on one runner > - The general-server lane already solves this with a duration-aware LPT partition backed by a recorded manifest > - This pull request reuses that partitioner for the serialized lane with a fresh per-suite duration manifest > - The benefit is a balanced serialized matrix: the measured 968s suite total levels to about 194s per shard, which removes about 80-100s from the run's slowest check ## Linked Issues or Issue Description **What existing behavior does this improve?** The `Verify serialized server suites` shard matrix in `.github/workflows/pr.yml` distributes route/authz test suites across five runners. **Subsystem affected** CI / test infrastructure (`scripts/run-vitest-stable.mjs`). **Current behavior** `selectSerializedSuites` assigns suites round-robin (`index % shardCount`) over the alphabetically sorted file list. The heavy suites cluster on shard 1/5. In actions run 32012408876, shard 1/5 spent 291s in its test step while the other shards spent 170-201s, which made that job (337s total) the slowest check of the whole PR run. **Proposed behavior** Partition the serialized suites with the same duration-aware LPT algorithm the general-server lane already uses (`scripts/general-server-shard.mjs`), backed by a new per-suite duration manifest. All five shards then carry about 194s of measured test time. **Reason and benefit** The slowest check bounds PR feedback time. Balancing the serialized matrix removes about 80-100s from that bound without adding runners. **Breaking changes** None. The partition remains deterministic, complete, and non-overlapping; suites missing from the manifest get the median weight. ## What Changed - Added `scripts/serialized-shard-durations.json`: per-suite wall-clock durations (ms) for all 134 serialized suites, sampled from actions run 32012408876 by diffing consecutive per-suite label timestamps in the shard logs (captures vitest spawn overhead, not just reported test time) - `scripts/run-vitest-stable.mjs`: `selectSerializedSuites` now uses the existing LPT partitioner (`selectGeneralServerShard`) with the new manifest instead of round-robin - `scripts/__tests__/run-vitest-stable-shard.test.mjs`: added a manifest-freshness test and a shard-balance test for the serialized lane, mirroring the general-server ones - `.github/workflows/pr.yml`: updated the serialized matrix comment with the new measurement and mechanism ## Verification - `node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs` passes (13 tests), including the existing test that the serialized shards form a complete, non-overlapping partition - Dry-run of all five shards shows estimated totals of 194/194/194/194/193s (round-robin was 276/175/160/172/187s): `node scripts/run-vitest-stable.mjs --mode serialized --shard-index N --shard-count 5 --dry-run` - The `Verify serialized server suites` jobs on this PR run the real partition end to end ## Risks - Low risk. Selection logic only; the vitest invocation per suite is unchanged - A stale manifest degrades gracefully: unknown suites get the median weight, and a dedicated test fails if fewer than half the current suites have recorded durations ## Model Used - Claude (Anthropic), model ID `claude-fable-5`, agentic coding session with tool use (Claude Code / Claude Agent SDK); no extended-thinking mode ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related prior work: #10923 (split serialized tests into five shards), #10925 (general-server duration manifest), #11156 (workspaces-a native shards). Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |