Files
PaperClipAI/tests/e2e/playwright.config.ts
T
Nicky LeachandClaude Opus 5.5 97953e9b59 fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - CI test shards are balanced with per-file durations that a weekly
workflow refreshes
> - The weekly refresh has failed since 2026-11-02, and the e2e weights
have not been measured since 2026-09-15. Today e2e shards 3 and 6 take
7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes
> - A single file can also grow until it sets the time of its shard, and
nothing tells us about it
> - This pull request repairs the refresh, extends it to the e2e and
Runner package lanes, and warns about test files over 180 seconds
> - The benefit is balanced shards that stay balanced, and early notice
of oversized test files

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The weekly refresh of the test-duration weights that balance CI shards.

**Subsystem affected**

CI: `scripts/ci/refresh-test-durations.mjs`,
`.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright
CI reporters.

**Current behavior**

- The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now
refuses output that contains terminal escape sequences.
- Most successful `cloud-readiness.yml` master runs are admission-only
and have no server jobs. The script took the newest 2 runs, so it would
fail with "No server timing jobs found" on most weeks.
- The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh
and are from 2026-09-15.
- No signal tells us when one test file becomes too slow.

**Proposed behavior**

The refresh runs again, selects master runs that ran the server jobs,
refreshes the e2e and Runner package-lane weights from merged PRs, and
lists oversized files. CI writes a warning for each test file over 180
seconds.

**Reason and benefit**

Unbalanced shards add minutes to every PR. The current e2e weights put
472 seconds on the slowest shard. The new weights put about 300 seconds
on each shard.

**Breaking changes**

None. Weights change only shard placement, never which tests run.

## What Changed

- Add `scripts/ci/gh-api.mjs`. It always passes
`--allow-escape-sequences`, retries without the flag when an older `gh`
does not know it, and strips ANSI codes from logs.
- `scripts/ci/refresh-test-durations.mjs`:
- scans up to 50 master runs and uses the 2 newest that ran the server
jobs;
- adds the `--dry-run`, `--pr-body=<path>` and
`--lanes=server,e2e,runner` options;
- validates every lane before it writes a file. If a merged-PR lane has
fewer than 2 valid runs, it keeps the current weights and writes a
warning.
- Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing
`pr.yml` run of each merged PR, and it validates:
  - the PR is merged into `master`;
  - the merge commit is in the checkout;
  - the run's head SHA is the PR's final head SHA;
  - every shard of the run succeeded.

  It parses Playwright per-spec times and Runner per-file times.
- Add `scripts/ci/oversized-test-files.mjs` (Vitest) and
`scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub
Actions they write a `::warning` for each test file over
`PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never
fail a job. The root and Runner Vitest configs and the e2e Playwright
config load them.
- `scripts/ci/test-timing-reporter.mjs`: move the per-file duration
calculation into an exported `moduleDurationMs()` helper. The behavior
does not change.
- `.github/workflows/ci-test-durations.yml` and
`.github/ci-duration-pr.md`:
  - the bot PR can also update the e2e and Runner manifests;
  - the PR body lists oversized files;
  - the verify step also runs the new tests and the e2e shard tests.
- Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured
(before: 30), from merged PRs #19613, #19191 and #19176. The file now
uses the shared schema with `sources` and `coverage`.

## Verification

- `node --test scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e
shard balance check with the new weights.
- `node scripts/ci/refresh-test-durations.mjs --dry-run` against real
GitHub data, all lanes:

| Manifest | Measured | Sources |
| --- | --- | --- |
| `general-server-shard-durations.json` | 1488/1534 files | master runs
42296943853, 42294279748 |
| `serialized-shard-durations.json` | 167/167 files | the same two runs
|
| `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365,
42297700476, 42296994179 |
| Runner `vitest-lane-durations.json` | 310/311 files | the same three
PR runs |

- e2e placement over 8 shards, with the new medians as the real costs:
  - with the old weights, the slowest shard has 472 seconds;
  - with the new weights, every shard has about 300 seconds.
- The dry run lists 12 files over 180 seconds. The largest is
`server/src/services/workspace-recovery.test.ts`, at 636 seconds.
- In this PR, only the e2e weight file changes. The weekly run refreshes
the server weights. The Runner manifest is added by #19646.

## Risks

- The `--allow-escape-sequences` path ran only in unit tests. The local
`gh` did not support the flag, so the real dry run used the fallback.
The next weekly run is the first real test of the flag.
- The changes to the bot-PR shell steps did not run in Actions yet.
- The e2e weights come from 3 PR runs. Real shard times vary more than
the estimate.
- This PR and #19637 both edit the `reporters` lists in
`vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The
second PR to merge needs a small merge resolution.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:50:02 -08:00

108 lines
5.1 KiB
TypeScript

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { defineConfig } from "@playwright/test";
// Use a dedicated port so e2e tests always start their own server in local_trusted mode,
// even when the dev server is running on :3100 in authenticated mode.
const PORT = Number(process.env.PAPERCLIP_E2E_PORT ?? 3199);
const BASE_URL = `http://127.0.0.1:${PORT}`;
const PAPERCLIP_HOME = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-e2e-home-"));
const PAPERCLIP_INSTANCE_ID = "playwright-e2e";
const PAPERCLIP_CONFIG = path.join(PAPERCLIP_HOME, "instances", PAPERCLIP_INSTANCE_ID, "config.json");
const PAPERCLIP_AGENT_JWT_SECRET = process.env.PAPERCLIP_AGENT_JWT_SECRET ?? "playwright-e2e-agent-jwt-secret";
const PAPERCLIP_DECISION_SIGNING_SECRET =
process.env.PAPERCLIP_DECISION_SIGNING_SECRET ?? "playwright-e2e-decision-signing-secret";
const PAPERCLIP_TOOL_ACTION_SIGNING_SECRET =
process.env.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET ?? "playwright-e2e-tool-action-signing-secret";
const PLAYWRIGHT_CHANNEL = process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL;
process.env.PAPERCLIP_HOME = PAPERCLIP_HOME;
process.env.PAPERCLIP_CONFIG = PAPERCLIP_CONFIG;
// Worker processes reload this config; retain the main process's server path
// for specs that seed historical database state in the throwaway instance.
process.env.PAPERCLIP_E2E_SERVER_CONFIG ??= PAPERCLIP_CONFIG;
// Specs that mint agent JWTs in-process (via createLocalAgentJwt) must derive
// the same per-instance signing key as the webServer, or verification fails
// with a 401 instead of authenticating as the agent.
process.env.PAPERCLIP_INSTANCE_ID = PAPERCLIP_INSTANCE_ID;
process.env.PAPERCLIP_AGENT_JWT_SECRET = PAPERCLIP_AGENT_JWT_SECRET;
process.env.PAPERCLIP_DECISION_SIGNING_SECRET = PAPERCLIP_DECISION_SIGNING_SECRET;
process.env.PAPERCLIP_TOOL_ACTION_SIGNING_SECRET = PAPERCLIP_TOOL_ACTION_SIGNING_SECRET;
export default defineConfig({
testDir: ".",
testMatch: "**/*.spec.ts",
// These suites target dedicated multi-user configurations/ports and are
// intentionally not part of the default local_trusted e2e run.
testIgnore: ["in-feed-native/**", "multi-user.spec.ts", "multi-user-authenticated.spec.ts"],
timeout: 60_000,
retries: 0,
// All specs share one throwaway server, and several toggle instance-level
// state (the `enableConferenceRoomChat` experimental flag) that changes
// which UI variant renders. Run files serially so a flag flip in one spec
// can't change the wizard/thread under another spec mid-flight.
workers: 1,
use: {
baseURL: BASE_URL,
headless: true,
screenshot: "only-on-failure",
trace: "on-first-retry",
},
projects: [
{
name: "chromium",
use: {
browserName: "chromium",
...(PLAYWRIGHT_CHANNEL ? { channel: PLAYWRIGHT_CHANNEL } : {}),
},
},
],
// The webServer directive bootstraps a throwaway instance and then starts it.
// `onboard --yes --run` works in a non-interactive temp PAPERCLIP_HOME.
webServer: {
cwd: path.resolve(import.meta.dirname, "../.."),
// Exercise the shipped UI. Source-checkout onboarding otherwise enables
// Vite middleware: every reload traverses thousands of modules, including
// service-worker-intercepted requests, before React can even start.
// Build the server's first-choice static directory so a prior package build
// cannot shadow the UI under test with stale server/ui-dist assets.
command: "pnpm --filter @paperclipai/ui build --outDir ../server/ui-dist --emptyOutDir && node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts onboard --yes --run",
url: `${BASE_URL}/api/health`,
// Always boot a dedicated throwaway instance for e2e so browser tests
// never attach to the developer's active Paperclip home/server.
reuseExistingServer: false,
timeout: 120_000,
stdout: "pipe",
stderr: "pipe",
env: {
...process.env,
NODE_ENV: "test",
PAPERCLIP_UI_DEV_MIDDLEWARE: "false",
NODE_OPTIONS: `${process.env.NODE_OPTIONS ?? ""} --import=${path.resolve(import.meta.dirname, "fixtures/agent-chat-github.mjs")} --import=${path.resolve(import.meta.dirname, "fixtures/ai-connection-provider.mjs")}`,
PORT: String(PORT),
PAPERCLIP_OPEN_ON_LISTEN: "false",
PAPERCLIP_API_URL: BASE_URL,
PAPERCLIP_HOME,
PAPERCLIP_INSTANCE_ID,
PAPERCLIP_CONFIG,
PAPERCLIP_AGENT_JWT_SECRET,
PAPERCLIP_DECISION_SIGNING_SECRET,
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET,
PAPERCLIP_BIND: "loopback",
PAPERCLIP_DEPLOYMENT_MODE: "local_trusted",
PAPERCLIP_DEPLOYMENT_EXPOSURE: "private",
},
},
outputDir: "./test-results",
reporter: [
["list"], ["html", { open: "never", outputFolder: "./playwright-report" }],
...(process.env.GITHUB_ACTIONS === "true"
? [[path.resolve(import.meta.dirname, "../../scripts/ci/oversized-test-files-playwright.mjs")] as [string]]
: []),
...(process.env.PAPERCLIP_CI_PROFILE === "true"
? [[path.resolve(import.meta.dirname, "../../scripts/ci-profile/playwright-reporter.mjs")] as [string]]
: []),
],
});