DottaandPaperclip
52e846f2f3
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its CI must catch regressions without running unrelated work on every pull request. > - The existing task planner selects package lanes, but each selected lane still runs all its test files. > - Vitest can find related imports, but that graph misses tests that read files or launch processes. > - This pull request combines the real Vitest graph with conservative dependency checks before CI allocates test runners. > - The result runs fewer test files and can allocate fewer jobs while retaining tests with unknown dependencies. ## Linked Issues or Issue Description Refs #18214. The repository and PR search found no duplicate file-selection implementation. **What existing behavior does this improve?** The general and serialized Vitest lanes in pull request CI. **Current behavior** An affected package schedules its complete test lane and its fixed shard count. **Proposed behavior** For source-only diffs, select related tests plus conservative filesystem, process, network and opaque consumers. Repartition those files within their existing lane owners. Keep the complete original plan when selection cannot be trusted. **Reason and benefit** Avoid unrelated test execution and allocate fewer runners when the selected workload fits fewer existing job budgets. Report file counts and job counts separately. **Breaking changes** No product or API changes. The set of PR test files and jobs changes for eligible diffs. Full CI and the kill switch retain the original coverage. ## What Changed - Add a separate selection job on the gate-approved test fleet. The metadata gate does not load PR code. - Inspect real Vitest/Vite dependency edges. Retain opaque consumers and unchanged downstream projects. Respect only complete local mock factories. Reject unknown setup dependencies and source mutations during dependency setup. - Declare `oxc-parser` directly for AST inspection. This exact version is already a transitive dependency in the checked-in lockfile; the direct declaration avoids relying on pnpm's internal dependency layout. - Produce a merge-SHA-bound manifest with exact lane ownership and duration-aware partitions. Preserve serialized execution, chat shards and native Runner ownership. - Execute exact Vitest test specifications. Positional substring filters cannot accidentally run a similarly named file from another shard. - Use the original matrix if selection fails or times out. Fail selected jobs if their published artifact is missing or invalid. - Save the uploaded artifact ID in the executable plan. Failed-job retries reuse that artifact even when GitHub increments the attempt number. - Hash selected assignments into Turbo cache inputs. A subset cannot satisfy a full run or another subset. - Add graph, workflow, real-runner and real-Turbo qualification. The regression fixture proves a filesystem contract fails even when raw Vitest selection misses it. - Document selection limits, audit artifacts and `CI_VITEST_SELECTION=off`. ## Verification - All 138 affected routing, sharding, workflow, manifest, real Vitest and real Turbo checks passed under Node 24 on the final implementation. Additional release workflow checks passed. Cases cover opaque setup, remocking, generated inputs, Node's builtin loader, artifact retries, and exact execution of similarly named files. - Repository-wide `pnpm -r typecheck` and `pnpm build` passed before the clean rebase onto current master. The full remote typecheck and build are required on the current head. - `actionlint` passed for both changed workflows. `git diff --check` passed. - The Linux qualification run passed and selected 699 of 911 workspace test files for an Inbox change. UI graph construction took 16.1 seconds. This still uses two jobs. Synthetic partition coverage proves a 15-of-120 selection uses two of twelve jobs without dropping or duplicating files. - Both selected workspace shards passed locally. After tightening execution to exact specifications, the real colliding UI test ran as exactly one file with two passing assertions; the real server auth middleware test ran as exactly one file with 28 passing assertions. - The server probe selected 1,158 of 1,260 general files and 161 of 163 serialized files for an auth middleware change. Those conservative selections retain 12 general and nine serialized jobs. The server's temporary-home setup exception is pinned to reviewed content hashes; changed setup fails closed. - Full `pnpm test:run` has not been run locally. The full PR test matrix is required because this change modifies global CI inputs. Final-head CI is pending at runner allocation, and both qualification jobs are queued. Other recent repository workflows are also queued. Superseded runs of this PR were cancelled; this PR remains in draft until the required checks execute. - Greptile reviewed final commit `31b484fc5b412da245cffb52a8ac8977228c5651` at 5/5 with no unresolved findings. The earlier qualification run passed; it does not replace checks on the final commit. ## Risks - Dependency analysis adds setup and graph time. Fewer files do not guarantee a shorter critical path. CI summaries expose the actual file and job counts. - Static dependency graphs cannot describe all runtime inputs. Unknown inputs retain tests or use the original plan. The pure-package allowlist requires conservative review. - The trusted workflow runs from master. This PR exercises the new selector in the qualification workflow; ordinary PR selection starts after merge. Set `CI_VITEST_SELECTION=off`, use `full-ci`, or select full/shadow mode to disable narrowing. - The lockfile remains managed by the existing repository actions and is not changed here. ## Model Used OpenAI Codex, based on GPT-6, with repository tools and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
5.7 KiB
Vitest dependency selection experiment
The production selector is documented in doc/DEVELOPING.md. This probe remains
a diagnostic for the raw import graph, without the production retention rules.
This probe measures Vitest's real import graph against the current checkout. It does not change the CI matrix or authorize skipping tests. It loads Vitest configurations and Vite plugins, so run it in a normal test environment, never in the trusted metadata-only routing job.
Install the workspace dependencies and prepare the same shared build outputs used by CI:
pnpm install --frozen-lockfile
node scripts/ensure-plugin-build-deps.mjs
node --test scripts/ci/vitest-impact-report.integration.test.mjs
Probe a hypothetical change (the file need not actually be modified):
node scripts/ci/vitest-impact-report.mjs \
--project @paperclipai/ui --file ui/src/pages/Inbox.tsx \
--output /tmp/inbox-impact.json
Or compare a base revision with the checked-out HEAD. This requires a clean tracked working tree. For a PR, check out the tested merge commit and supply the event's base SHA, as the existing CI planner does:
node scripts/ci/vitest-impact-report.mjs \
--project @paperclipai/ui --project @paperclipai/server \
--base BASE_SHA --output /tmp/pr-impact.json
Every requested project is analyzed separately to bound memory usage. Imports across package boundaries are followed within each graph, but unrequested projects are not assessed. Runner's separate Vitest configurations, browser E2E, Rust, typechecking, and builds are outside this experiment. The JSON records selected and omitted test files, project inventory, startup and graph times, and the comparison revisions. A failed or timed-out probe is not an empty plan.
Initial observations
Measured on 5c807d11fa with Vitest 5.0.3, macOS, and Node 26.4.0. These are
hypothetical single-file changes against that revision, not replays of historical
PR checkouts. CI uses Node 24 and different hardware.
| Changed file | Assessed project | Selected / total test files |
|---|---|---|
ui/src/pages/Inbox.tsx |
UI | 6 / 844 |
ui/src/components/MarkdownBody.tsx |
UI | 135 / 844 |
server/src/middleware/auth.ts |
Server | 45 / 1,428 |
server/src/services/run-failure-report.ts |
Server | 787 / 1,428 |
packages/shared/src/issue-references.ts |
Shared only | 5 / 96 |
docs/api/issues.md |
Server | 0 / 1,428 — known false negative |
Cold graph construction took 27.43 seconds for UI, 25.68 seconds for server, and 2.79 seconds for shared. Subsequent queries on the same graph took about 0.05 seconds; fresh CI jobs must budget for cold construction. The six Inbox files ran 81 assertions successfully in 20.92 seconds with four workers. These figures exclude dependency installation and shared package builds.
For a local execution comparison, the complete UI project passed all 10,043 assertions in 844 files in 361.50 seconds with the same four-worker setting. The initial Inbox graph plus selected execution took about 48 seconds, versus six minutes for the full project. This is a local experiment, not a measured change in GitHub CI latency. Later runs benefit from filesystem/transform caches.
The CLI's Git-diff path was also exercised against the actual HEAD^..HEAD
change for PR #18321. Across its eleven changed files, the server graph selected
1,138 / 1,428 files and the shared graph selected 3 / 96. Other projects were
not assessed. Broad shared/service imports can therefore preserve most of a
suite; the Inbox reduction should not be generalized to every PR.
Why the raw graph is not yet a safe execution plan
There are concrete missing edges in this repository:
ui/src/pages/apps/chat/chat-ui-contract.test.tsreadsInbox.tsxthroughreadFileSync. It is absent from the six-file selection even though changes to Inbox can break its assertions.server/src/__tests__/paperclip-skill-utils.test.tsreadsdocs/api/issues.mdthroughfs.readFile. Changing that document selects zero server tests.
A deliberate negative control confirmed the first missing edge: temporarily
append a comment containing reason: "retry_failed_run" to Inbox.tsx, which
violates the contract test's source assertion without changing runtime behavior.
The fresh graph still selected six files. Running those six plus the omitted
contract produced six passing files and exactly one failing assertion, in
chat-ui-contract.test.ts (92 assertions passed, one failed). The original source
was restored immediately afterward. The full baseline had passed this contract.
The integration test uses actual Vitest graph traversal to verify transitive imports, aliases, literal dynamic imports, direct test edits, and the missing filesystem edge. Test modules deliberately throw if executed: graph discovery must not execute the tests. Passing this test documents the limitation; it does not prove that skipped repository tests are irrelevant.
A production rollout needs explicit dependencies or an always-run lane for filesystem/subprocess contract tests; broad fallbacks for configuration, setup, lockfiles, deleted files, unknown inputs, and selector failures; and comparison against full-suite failures. Source-only changes also need those protections: the Inbox example is not a documentation-only edge case. Once qualified, compute the selected test list before constructing the matrix and partition that list, preserving the existing serialized, chat, and native-runner ownership rules.
Verification for this experiment covers the real-graph integration test, CLI
probes, the complete UI project, and the deliberate negative control. Repository
wide typechecking, build, server execution, and the complete test:run suite
were not run; this is a report-only experiment, not a production CI rollout.