mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
Nicky LeachandClaude Opus 5.5
97953e9b59
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - CI test shards are balanced with per-file durations that a weekly workflow refreshes > - The weekly refresh has failed since 2026-11-02, and the e2e weights have not been measured since 2026-09-15. Today e2e shards 3 and 6 take 7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes > - A single file can also grow until it sets the time of its shard, and nothing tells us about it > - This pull request repairs the refresh, extends it to the e2e and Runner package lanes, and warns about test files over 180 seconds > - The benefit is balanced shards that stay balanced, and early notice of oversized test files ## Linked Issues or Issue Description **What existing behavior does this improve?** The weekly refresh of the test-duration weights that balance CI shards. **Subsystem affected** CI: `scripts/ci/refresh-test-durations.mjs`, `.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright CI reporters. **Current behavior** - The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now refuses output that contains terminal escape sequences. - Most successful `cloud-readiness.yml` master runs are admission-only and have no server jobs. The script took the newest 2 runs, so it would fail with "No server timing jobs found" on most weeks. - The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh and are from 2026-09-15. - No signal tells us when one test file becomes too slow. **Proposed behavior** The refresh runs again, selects master runs that ran the server jobs, refreshes the e2e and Runner package-lane weights from merged PRs, and lists oversized files. CI writes a warning for each test file over 180 seconds. **Reason and benefit** Unbalanced shards add minutes to every PR. The current e2e weights put 472 seconds on the slowest shard. The new weights put about 300 seconds on each shard. **Breaking changes** None. Weights change only shard placement, never which tests run. ## What Changed - Add `scripts/ci/gh-api.mjs`. It always passes `--allow-escape-sequences`, retries without the flag when an older `gh` does not know it, and strips ANSI codes from logs. - `scripts/ci/refresh-test-durations.mjs`: - scans up to 50 master runs and uses the 2 newest that ran the server jobs; - adds the `--dry-run`, `--pr-body=<path>` and `--lanes=server,e2e,runner` options; - validates every lane before it writes a file. If a merged-PR lane has fewer than 2 valid runs, it keeps the current weights and writes a warning. - Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing `pr.yml` run of each merged PR, and it validates: - the PR is merged into `master`; - the merge commit is in the checkout; - the run's head SHA is the PR's final head SHA; - every shard of the run succeeded. It parses Playwright per-spec times and Runner per-file times. - Add `scripts/ci/oversized-test-files.mjs` (Vitest) and `scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub Actions they write a `::warning` for each test file over `PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never fail a job. The root and Runner Vitest configs and the e2e Playwright config load them. - `scripts/ci/test-timing-reporter.mjs`: move the per-file duration calculation into an exported `moduleDurationMs()` helper. The behavior does not change. - `.github/workflows/ci-test-durations.yml` and `.github/ci-duration-pr.md`: - the bot PR can also update the e2e and Runner manifests; - the PR body lists oversized files; - the verify step also runs the new tests and the e2e shard tests. - Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured (before: 30), from merged PRs #19613, #19191 and #19176. The file now uses the shared schema with `sources` and `coverage`. ## Verification - `node --test scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e shard balance check with the new weights. - `node scripts/ci/refresh-test-durations.mjs --dry-run` against real GitHub data, all lanes: | Manifest | Measured | Sources | | --- | --- | --- | | `general-server-shard-durations.json` | 1488/1534 files | master runs 42296943853, 42294279748 | | `serialized-shard-durations.json` | 167/167 files | the same two runs | | `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365, 42297700476, 42296994179 | | Runner `vitest-lane-durations.json` | 310/311 files | the same three PR runs | - e2e placement over 8 shards, with the new medians as the real costs: - with the old weights, the slowest shard has 472 seconds; - with the new weights, every shard has about 300 seconds. - The dry run lists 12 files over 180 seconds. The largest is `server/src/services/workspace-recovery.test.ts`, at 636 seconds. - In this PR, only the e2e weight file changes. The weekly run refreshes the server weights. The Runner manifest is added by #19646. ## Risks - The `--allow-escape-sequences` path ran only in unit tests. The local `gh` did not support the flag, so the real dry run used the fallback. The next weekly run is the first real test of the flag. - The changes to the bot-PR shell steps did not run in Actions yet. - The e2e weights come from 3 PR runs. Real shard times vary more than the estimate. - This PR and #19637 both edit the `reporters` lists in `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The second PR to merge needs a small merge resolution. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
91 lines
4.2 KiB
JSON
91 lines
4.2 KiB
JSON
{
|
|
"$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful .github/workflows/pr.yml runs of PRs merged into master, each pinned to the PR's final head SHA. Per spec: the sum of gaps between consecutive Playwright result lines in 'Run e2e tests' (one worker), so fixture and navigation time count; tests located in a *.shared.ts file are attributed to the shard's matching spec. Median across runs. scripts/e2e-shard.mjs uses these weights only to place specs on shards; they never change which specs run. Missing specs retain their prior weight or the median fallback.",
|
|
"unit": "ms",
|
|
"sources": [
|
|
{
|
|
"pull": 19613,
|
|
"run": 42296374365,
|
|
"attempt": 2,
|
|
"sha": "cd8c9cd07c192832fa180a845577d36fb0bc41c8",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/42296374365"
|
|
},
|
|
{
|
|
"pull": 19191,
|
|
"run": 42297700476,
|
|
"attempt": 1,
|
|
"sha": "36c0400008a12ec74ed65a0e17951f1b47221a02",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/42297700476"
|
|
},
|
|
{
|
|
"pull": 19176,
|
|
"run": 42296994179,
|
|
"attempt": 1,
|
|
"sha": "d0a937768da9be297485323c5b905eb63371ec15",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/42296994179"
|
|
}
|
|
],
|
|
"coverage": {
|
|
"measured": 56,
|
|
"withSetup": 56,
|
|
"inventory": 56
|
|
},
|
|
"durations": {
|
|
"tests/e2e/acp-stop-continuation.spec.ts": 67299,
|
|
"tests/e2e/action-test-agent-picker.spec.ts": 13115,
|
|
"tests/e2e/agent-chat-messages.spec.ts": 23569,
|
|
"tests/e2e/agent-chat-projects.spec.ts": 206156,
|
|
"tests/e2e/agent-chat-sessions.spec.ts": 117218,
|
|
"tests/e2e/agent-configuration.spec.ts": 105794,
|
|
"tests/e2e/agentmail.spec.ts": 27227,
|
|
"tests/e2e/ai-personal-connection-recovery.spec.ts": 12373,
|
|
"tests/e2e/announcement-first-session.spec.ts": 9986,
|
|
"tests/e2e/announcement-placement.spec.ts": 18485,
|
|
"tests/e2e/app-not-connected.spec.ts": 10832,
|
|
"tests/e2e/application-delete-screenshot.spec.ts": 2970,
|
|
"tests/e2e/applications-crud.spec.ts": 23859,
|
|
"tests/e2e/apps-dark-mode-shots.spec.ts": 18910,
|
|
"tests/e2e/apps-prosumer-mcp-flow.spec.ts": 13276,
|
|
"tests/e2e/archived-company-url.spec.ts": 6145,
|
|
"tests/e2e/artifact-tab-arrival.spec.ts": 20087,
|
|
"tests/e2e/artifact-uploads.spec.ts": 28200,
|
|
"tests/e2e/board-attachment-receipts.spec.ts": 112020,
|
|
"tests/e2e/cases-read-only.spec.ts": 9657,
|
|
"tests/e2e/chat-adapters-ui-messaging.spec.ts": 181303,
|
|
"tests/e2e/chat-adapters-ui-providers.spec.ts": 181541,
|
|
"tests/e2e/chunk-load-recovery.spec.ts": 16330,
|
|
"tests/e2e/classic-workspace.spec.ts": 6483,
|
|
"tests/e2e/cloud-auth.spec.ts": 33711,
|
|
"tests/e2e/company-context-refresh.spec.ts": 4128,
|
|
"tests/e2e/completed-task-paused-assignee.spec.ts": 25016,
|
|
"tests/e2e/composer-stop.spec.ts": 74978,
|
|
"tests/e2e/conference-room-typing-intro.spec.ts": 30042,
|
|
"tests/e2e/connection-intents.spec.ts": 23775,
|
|
"tests/e2e/connection-reviews.spec.ts": 134583,
|
|
"tests/e2e/coordinator-inbox-views.spec.ts": 18241,
|
|
"tests/e2e/daytona-connector.spec.ts": 2603,
|
|
"tests/e2e/disconnect-resilience.spec.ts": 200278,
|
|
"tests/e2e/disconnect-write-recovery.spec.ts": 51072,
|
|
"tests/e2e/document-image-gallery.spec.ts": 46344,
|
|
"tests/e2e/legacy-failure-continuation.spec.ts": 89961,
|
|
"tests/e2e/mcp-user-stories.spec.ts": 37417,
|
|
"tests/e2e/new-task-picker-touch.spec.ts": 22514,
|
|
"tests/e2e/nux-phase4-screenshots.spec.ts": 11330,
|
|
"tests/e2e/onboarding.spec.ts": 62933,
|
|
"tests/e2e/paused-composer.spec.ts": 38240,
|
|
"tests/e2e/plan-task-fixture.spec.ts": 7828,
|
|
"tests/e2e/planning-mode-visual-verification.spec.ts": 26325,
|
|
"tests/e2e/project-repositories.spec.ts": 13403,
|
|
"tests/e2e/railway-catalog.spec.ts": 2813,
|
|
"tests/e2e/runner-e2e-dashboard.spec.ts": 4985,
|
|
"tests/e2e/runner-task-creation.spec.ts": 32437,
|
|
"tests/e2e/sidebar-takeover.spec.ts": 9119,
|
|
"tests/e2e/signoff-policy.spec.ts": 52229,
|
|
"tests/e2e/smoke-lab-p1-p4.spec.ts": 22390,
|
|
"tests/e2e/smoke-lab-p5-p7.spec.ts": 16416,
|
|
"tests/e2e/task-loading-frame.spec.ts": 31309,
|
|
"tests/e2e/task-reload.spec.ts": 20988,
|
|
"tests/e2e/text-attachment-tabs.spec.ts": 12758,
|
|
"tests/e2e/unikraft-connector.spec.ts": 2613
|
|
}
|
|
}
|