6 Commits

Include renames

Author	SHA1	Message	Date
Nicky LeachandClaude Opus 5.5	79c724b292	ci: run the Runner support suite as a required PR check (#19537) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner support suite (`pnpm test:e2e:runner:unit`) checks the code behind paid Runner runs: workflow security policy, evidence oracles, cost receipts and controller recovery. > - Only a manual Docker Runner check ran this suite. PR CI did not run it. > - So changes on master broke 16 of its tests, and nobody saw it. #19527 and #19529 fix those tests. > - This pull request runs the suite in PR CI and makes the `verify` check require it. > - The benefit is that a change that breaks the suite fails on its own pull request, before merge. ## Linked Issues or Issue Description Refs #19527 and #19529 (both merged; they repair the suite). Also accounts for #19533, which added the sandbox provider job. **Merge order:** #19527 and #19529 have merged, so the suite now passes on master. This PR can merge. **What existing behavior does this improve?** PR CI for `pr-trusted.yml`. **Current behavior** `pnpm test:e2e:runner:unit` runs only in `docker-runner-check.yml`, by manual dispatch. Master changes broke 12–13 tests by Nov 3 and 16 by Nov 6, with no signal on any pull request. **Proposed behavior** A required "Runner support suite" job runs the suite on pull requests. **Reason and benefit** Drift fails on the pull request that causes it. Authors update deliberate pins (profile versions, suite hashes, workflow policy) in the same pull request. ## What Changed - `pr-trusted.yml`: new job `verify_runner_e2e_support` ("Runner support suite"). It uses the same steps as other trusted jobs: checkout without credentials, setup-node directly before pnpm, the npm overrides, the shared `setup-ci-dependencies` restore, and CI profiling. Then it runs `pnpm test:e2e:runner:unit`. It has no secrets. Timeout: 20 minutes. - `pr-trusted.yml`, job `verify`: the job now needs the new job and checks its result. A failed, cancelled or wrongly skipped run fails `verify`. - `turbo.json`: new routing task `ci:runner-e2e-support`. Its inputs are `server/**`, `cli/**`, `packages/**` and `ui/**`. Changes to `tests/**`, `scripts/**` and `.github/**` already force full CI, so they also run the job. - `.github/ci-task-matrix.json`: one lane for the new job. - Tests: `ci-plan.test.mjs`, `ci-profile.test.mjs`, `e2e-shard.test.mjs` and `workflow-security.test.ts` now include the new job. A UI component change now selects this one Runner task. The suite imports UI modules and checks that catalog-listed UI files stay tracked in git, so UI changes can break it. - CI profiling: when `PAPERCLIP_CI_PROFILE` is on, the suite's Vitest config adds `scripts/ci-profile/vitest-reporter.mjs`, and `run-unit.mjs` runs the Node tests through `scripts/ci-profile/node-test.mjs`. So the job uploads per-test timings, like the root and Runner package suites. The launcher loads the profiling helper only after its "Vitest is not installed" check. - `tests/runner-e2e/README.md`: says that PR CI runs the suite, and that deliberate pin changes go in the same pull request. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/ci/*.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs`: 830 passed, 3 failed. The same 3 fail on the base branch without this change. They need local tools that are not on this Mac. - A local run with `PAPERCLIP_CI_PROFILE=true` wrote per-test and per-file timing records for Vitest and for Node. Without profiling, output and exit codes do not change. Both `unit-launcher.test.mjs` tests pass. - The aggregate test now covers the new job: `verify` rejects it when it fails, is cancelled, or is skipped while selected. - The pnpm bootstrap policy tests in `workflow-security.test.ts` pass with 10 setups in `pr-trusted.yml`. - On this PR's merged tree (`80b526d697`, master plus this PR): `pnpm test:e2e:runner:unit` passed, Vitest 320 files (6,830 tests) and node:test 373, in 58s. It also passes on master `4c1f629c31`. - Suite runtime on a 10-core Mac: 43–63s. With `--maxWorkers=4`: Vitest 118s, about 490s of CPU in total. On a 4-vCPU GitHub runner I expect about 3–4 minutes for the tests and 1–2 minutes for setup. The job runs in parallel with the longer test shards, so it should not lengthen CI. - `pr.yml` calls `pr-trusted.yml@master`. This PR's own CI does not run the new job. The first real CI timing comes from the first pull request after merge. ## Risks - More runner minutes: one extra job on most pull requests (all except documentation-only ones). - The suite has not run on Linux CI recently: old `docker-runner-check` logs have expired. A Linux-only failure would show up on the first pull requests after merge. The job can be removed from the `verify` list quickly if needed. - #19229 removes `version: 9.15.4` from the pnpm steps in this file. Whichever PR merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use (shell, file edits, git, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 11:50:42 -08:00
Nicky LeachandClaude Opus 5.5	3b3037aab5	fix(ci): run the sandbox provider tests in required CI (#19533) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox provider plugins (Daytona, E2B, Cloudflare, Kubernetes, Modal, and others) run agents in remote sandboxes > - The providers are outside the root pnpm workspace, so `pnpm test:run` and the CI test lanes never run their Vitest suites > - paperclipai/paperclip#18839 typechecks those tests, but a test that fails at runtime still fails nothing > - The providers have about 1,040 tests, and they finish in about 15 seconds on Linux > - This pull request adds a required `Sandbox provider tests` job that runs them when a change can affect them > - The benefit is that a broken provider, plugin SDK, or shared contract fails CI on the pull request instead of in a release or a paid end-to-end run ## Linked Issues or Issue Description Refs #18839 **What happened?** No CI job runs the sandbox provider tests. `runner-full-stack-e2e.yml` builds the Daytona plugin for a paid nightly run, but it does not run any provider's `pnpm test`. **Expected behavior** A pull request that can break a provider runs that provider's tests, and a failure blocks `ci / verify`. **Steps to reproduce** Add `expect(1).toBe(2)` to a test in `packages/plugins/sandbox-providers/e2b/src`. Open a pull request. CI stays green. ## What Changed - Added `scripts/test-sandbox-providers.mjs` and its tests. The script installs the providers through `typecheckTestPrograms` with no programs, because that function already owns the provider install and the plugin SDK link. Then it runs each provider's `pnpm test`, four at a time, and prints each provider's output as one block. It fails if a suite fails or if no provider has a test script. - The script skips `cloudflare/bridge-template`, because Cloudflare's Vitest config already runs those tests. - Added the `sandbox_provider_tests` job to `.github/workflows/pr-trusted.yml`, and made `ci / verify` require its result. - Added the `//#ci:sandbox-providers` task to `turbo.json` and its lane to `.github/ci-task-matrix.json`. The CI planner runs the job only when a change touches `packages/plugins/sandbox-providers/**`, `packages/plugins/sdk/**`, or `packages/shared/**`. `packages/shared` is the plugin SDK's only workspace dependency. Global CI inputs, such as `scripts/**`, `.github/**`, `pnpm-lock.yaml`, or any `package.json`, still select every job. - The job turns off core dumps. A Daytona capacity test kills a process at a file-size limit inside the exported directory. If core dumps are on, the core file joins the export, and the test sees 12 files instead of 11. - Updated the job lists in `scripts/__tests__/e2e-shard.test.mjs` and the job count in `.github/scripts/tests/ci-profile.test.mjs`. Added planner tests to `.github/scripts/tests/ci-plan.test.mjs`. One of these tests fails if the plugin SDK gets a new workspace dependency that is not in the job's inputs. - Added the new test file to `test:release-registry`, and documented the script in `doc/DEVELOPING.md`. ## Verification - `node scripts/test-sandbox-providers.mjs` on macOS: 8 of 9 suites pass in 31.6 s, including the install. Kubernetes fails 12 `file-sync` tests, because macOS has no `/proc/self/fd`. Some Daytona tests skip, because they need GNU `tar` and coreutils. - The same script in a Linux container (`node:24`, a non-root user, core dumps off): all 9 suites pass, with 1,012 tests passed and 26 skipped. The script took 15 s after the install. - Kubernetes: 205 passed. - Daytona: 590 passed and 16 skipped. - With core dumps on, Daytona's `preserve literal capacity repair` test fails. The export includes a `core` file. With `--ulimit core=0`, it passes. - To look for timing failures under load, I ran the whole script 12 more times in the container, limited to 2 CPUs (8 runs) and 1 CPU (4 runs). All 12 runs passed. They took 45 to 76 s on 2 CPUs and 73 to 102 s on 1 CPU. - `ubuntu-latest` has `zstd`, and my container did not, so some Daytona tests skipped in those runs. With `zstd` installed, Daytona passed with 598 tests and 8 skipped, and only the live tests skipped. One earlier `zstd` run failed while the stress runs used the same CPUs, and I did not capture its output. The next run passed. - `node --test` on every test that reads `pr-trusted.yml`, the CI planner, or the task matrix: 816 passed. These include `.github/scripts/tests/*.test.mjs`, `scripts/__tests__/e2e-shard.test.mjs`, `scripts/ci/test-selection-workflow.test.mjs`, and the new tests. - `pnpm run test:release-registry` on macOS: 251 of 255 pass, including the new tests. The 4 failures are `wait_for_npm_package_versions` tests in `scripts/release-lib.test.mjs`. They fail the same way on master on the same machine, and CI runs this suite on Linux. - `actionlint` 1.7.7 reports no errors on the changed workflow. - I called `planTasks` from `.github/scripts/ci-plan.mjs` with one changed file at a time. `server/src/index.ts` and `ui/src/App.tsx` leave `sandbox_provider_tests` off. `packages/plugins/sandbox-providers/e2b/src/plugin.ts` and `packages/shared/src/index.ts` turn it on. This pull request cannot run its own new job. `pr.yml` calls `pr-trusted.yml@master`, so the job first runs on pull requests after this one merges. After merge, check the first pull request that changes a provider or `packages/shared`. ## Risks - **First real run is after merge.** I measured the time on Linux in Docker, not on a CI runner. CI runners are about 2 times slower than my machine, so I expect about 2 to 3 minutes for the job, including the dependency restore. The job runs in parallel with the other jobs, and the slowest required job takes about 13 minutes. So the job should not make CI slower. - **Open provider pull requests.** After this merges, open pull requests that change a provider run its tests. If those tests fail, `ci / verify` blocks them until they are fixed. - **`packages/shared` changes often.** Any change there runs this job. A Linux-only provider failure would block those pull requests. All suites pass on Linux today. - **One unexplained Daytona failure.** One Daytona run with `zstd` failed under CPU contention, and I did not capture its output. Some Daytona tests have short deadlines. If the job is flaky after merge, quarantine the failing test or revert the job, and do not leave `ci / verify` red. - **Path rules can be too narrow.** If a provider starts to import another workspace package, a change to that package does not run this job. The plugin SDK dependency test covers the SDK. It does not cover a provider that imports a workspace package directly. Today no provider does. - **Overlap with paperclipai/paperclip#19489 and paperclipai/paperclip#19522.** Both pull requests change the provider install in `scripts/typecheck-test-programs.mjs`. This pull request does not edit that file. It calls the same install function, so it uses their install changes when they merge. All three pull requests edit the same section of `doc/DEVELOPING.md`, so a small text conflict is possible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI, Docker). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:34:03 -08:00
DottaandPaperclip	e40de0c9e8	perf(ci): refresh test weights and narrow graph dependencies (#18464) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, reliable CI feedback. > - Diff-aware CI selects test files and partitions them into runner jobs. > - Stale cost estimates put several slow suites on the same runner. Broad imports also couple small tests to large modules. > - This PR refreshes test costs, runs independent graph discovery in parallel, and narrows three dependency boundaries. > - Selection safeguards and CODEOWNER-only AWS routing stay in force. ## Linked Issues or Issue Description Refs #18398. This follows the audit of [PR #18427's verification run](https://github.com/paperclipai/paperclip/actions/runs/41169978784). The audit found 403 selected general-server files without timing weights. Three suites measured at 232s, 122s and 116s shared one shard because each received a roughly 2s fallback weight. Local graph discovery across four projects took 43.7s serially and 21.7s with two workers. A reverse-order repeat took 37.7s serially and 22.6s with two workers. Both comparisons produced identical selection results. This is a local measurement, not a claim about production CI speed. ## What Changed - Refresh general-server costs for 1,285 of 1,309 current files and serialized costs for all 163 files from two successful protected-master Cloud readiness runs. Retain prior costs for unmeasured files, including dedicated chat and native-runner suites. - Record passing server file costs, including collection, setup and environment preparation. A weekly hosted workflow validates trusted source runs and opens a timing-only PR. It tests shard coverage, pauses while a refresh PR is open, and never merges automatically. - Discover at most two project graphs concurrently in isolated, heap-limited Node processes. Cache repeated inventory reads within one selection. Worker errors and timeouts retain the full task plan. - Fetch only the merge commit and parents for the selector. A real shallow-clone regression proves the complete diff still includes more than 300 files, renames and deletions. - Share the identical log viewer and run details between both AgentDetail pages. The viewer test imports the shared implementation and runs each of its 13 distinct cases once. - Extract the pure API error class while preserving the API client's public exports. Import delivery work signals through their existing narrow DB export. ## Verification - PASS: `pnpm -r typecheck` and `pnpm build`. - PASS: 284 CI selection, timing, shallow-diff, sharding and routing tests. - PASS: 74 further cache, routing and graph-proof tests. - PASS: 94 targeted UI assertions across the viewer, run invocation, API errors/client and progress helpers. All 76 assertions in the 12 server suites made independently selectable also pass. - PASS: `pnpm check:token-gates`, module-boundary checks, YAML parsing and `git diff --check`. - PASS: [Full PR CI and required `ci / verify` aggregate](https://github.com/paperclipai/paperclip/actions/runs/41206683345), including all 12 general-server shards, all nine serialized shards, workspace groups, chat suites, Runner lanes and build checks. - PASS: [GitHub hosted real-workspace selector and cache/routing qualification](https://github.com/paperclipai/paperclip/actions/runs/41206683205). - PASS: Both Greptile and Paperclip reviews are 5/5 with no actionable findings or inline comments on `7dc65c51b8`. The PR has no merge conflicts. - PASS: The production refresh parser accepted all 107 structured file-cost records from a real successful server CI job, including setup/loading cost. - Local `pnpm test:run` completed all six chat shards successfully. The duplicate monolithic local run was stopped after the complete remote suite passed; it was not completed locally. The focused log-viewer command improved from 18.84s (26 duplicate variant cases) to 8.49s (all 13 distinct behavior cases). The dependency audit confirms 12 server files and the API-error test no longer need conservative always-run retention. Repartitioning the audited PR with the new costs changes the slowest shard from 810s to 591s of summed historical test execution (1,124 of 1,142 files measured). This is a workload simulation, not measured new CI wall time. ## Risks - Two graph processes use more peak memory. Each has a 2 GiB heap limit and a three-minute timeout. Any failure preserves the original task plan. Real workspace qualification runs in GitHub CI. - Historical logs contain test/hook costs only. New reporter records add loading/setup costs after merge. The manifests identify measurement coverage and source runs. These weights change placement, never permission to skip tests. - Runner load adds timing noise. The refresh uses medians and requires at least 90% measurement coverage in each lane. - GitHub may require approval to run checks on a PR created with `GITHUB_TOKEN`. Refreshes remain reviewable and do not merge themselves. - The shared viewer is a mechanical extraction. Both pages retain their existing exports, and the behavior assertions remain covered. AWS runner authorization is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, reasoning, code execution and GitHub tools. The exact deployment ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and six chat shards; complete suite passed in CI as detailed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 11:05:12 -05:00
DottaandPaperclip	3b84c89885	fix(ci): use complete merge diffs and allow audited documentation (#18362) ... ## Thinking Path > - Paperclip manages AI agents and needs reliable checks for each contribution. > - CI selects affected suites and then affected Vitest files before allocating runners. > - Recent runs compared a tested merge against a stale event base. > - That included unrelated master changes and sometimes reached the compare API file cap. > - Ordinary documentation also forced every suite and invalidated test caches. > - This PR reads complete merge trees and exempts a small set of audited prose inputs. > - Tests retain conservative coverage when the diff or an input is uncertain. ## Linked Issues or Issue Description Refs #18348 and #18214. This is a follow-up to the merged affected-test selector. Related CI performance work: #16513 and #16500 address setup and suite execution. This PR changes diff collection and input classification. ## What Changed - Fetch the exact tested merge and its parents into a temporary bare repository. Read the complete tree diff against the actual first parent. Validate the second parent against the event PR head. Never execute PR code in the routing job. - Carry the immutable base, merge and PR-head identities into the plan. Validate the same complete diff again before file selection. - Use one prose policy across the planner, Vitest selector and task cache adapter. Exempt only `doc/DEVELOPING.md`, `doc/DATABASE.md`, and Markdown under `doc/plans/`. - Keep the uncached typecheck and CLI guidance guard for the first two documents. Keep policy checks for historical plans. Keep existing coverage for all other docs, runtime skills, fixtures and unknown inputs. - Extend candidate-workflow qualification to exercise mixed source and prose changes. ## Verification - Passed: 15 planner and Git routing tests, including a 353-file diff, stale base, rename, deletion, wrong parent and fetch-failure cases. - Passed: 83 routing, Vitest graph, exact selection, workflow and real Turbo cache checks. - Passed: `pnpm -r typecheck` and `pnpm build`. - Full local `pnpm test:run` was attempted and stopped after repeated embedded PostgreSQL startup failures in native-session and Muse tests. It did not pass or complete. The exact cause remains unverified. The changed files do not include those tests or their database setup. Linux CI results are recorded below. - Passed: real mixed Inbox/source/prose graph qualification selected 699/911 workspace test files, preserving the source-only selection. - Passed: both GitHub CI qualification jobs, including real mixed source/prose Vitest selection. Greptile reviewed commit `10ac6f5ec9` at 5/5 with no findings. - First full CI attempt: the [chat shard](https://github.com/paperclipai/paperclip/actions/runs/41111003419/job/124253342684) received a runner shutdown signal. The [Runner shard](https://github.com/paperclipai/paperclip/actions/runs/41111003419/job/124253341271) failed the unchanged Antigravity authentication success assertion. All 11 cases in that file passed when run directly. That Runner lane does not use the changed selector/cache path. The exact assertion-failure cause remains unverified. The chat shard passed on the one retry. The authentication assertion also passed on retry, but the same Runner shard failed a different unchanged test: `native-distribution-integrity.test.ts:242`, "retains spawned native bytes until the exact child exits", exceeded 5000ms. The [retry log](https://github.com/paperclipai/paperclip/actions/runs/41111003419/job/124259174201) preserves the failure. The Runner shard and required verify aggregate are still red; all other checks passed. All 33 native-distribution tests then passed in a direct local run. This shows recovery in isolation, not the cause of the Linux timeout. No further blind reruns were issued. - Replayed 15 real recent PR merge commits through a blob-filtered bare fetch. Nine needed a corrected base. The four API comparisons that returned 300 filenames became complete diffs of 3, 57, 66 and 54 files. The 3-file case (#18326) becomes eligible for graph selection. Most other samples still correctly use full CI because they include inputs outside this narrow prose policy. - Cache proof: prose edits preserve a product-test cache hit; changing runtime API documentation invalidates it. ## Risks - Prose exceptions must be removed or revised if product code starts reading those files. The allowlist is deliberately small. Runtime docs and skills remain covered. - A network or Git failure keeps the full-suite fallback. Unexpected merge topology also keeps full CI. Stack and force-full policies remain in place. - This does not claim an end-to-end speedup yet. Many mixed-content PRs still require full CI. The routing fetch reads only shallow commit/tree objects; it does not check out or execute PR files. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and code execution. The exact served model ID and context-window size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 14:29:39 -05:00
DottaandPaperclip	52e846f2f3	perf(ci): select affected Vitest files before allocating test runners (#18348) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its CI must catch regressions without running unrelated work on every pull request. > - The existing task planner selects package lanes, but each selected lane still runs all its test files. > - Vitest can find related imports, but that graph misses tests that read files or launch processes. > - This pull request combines the real Vitest graph with conservative dependency checks before CI allocates test runners. > - The result runs fewer test files and can allocate fewer jobs while retaining tests with unknown dependencies. ## Linked Issues or Issue Description Refs #18214. The repository and PR search found no duplicate file-selection implementation. **What existing behavior does this improve?** The general and serialized Vitest lanes in pull request CI. **Current behavior** An affected package schedules its complete test lane and its fixed shard count. **Proposed behavior** For source-only diffs, select related tests plus conservative filesystem, process, network and opaque consumers. Repartition those files within their existing lane owners. Keep the complete original plan when selection cannot be trusted. **Reason and benefit** Avoid unrelated test execution and allocate fewer runners when the selected workload fits fewer existing job budgets. Report file counts and job counts separately. **Breaking changes** No product or API changes. The set of PR test files and jobs changes for eligible diffs. Full CI and the kill switch retain the original coverage. ## What Changed - Add a separate selection job on the gate-approved test fleet. The metadata gate does not load PR code. - Inspect real Vitest/Vite dependency edges. Retain opaque consumers and unchanged downstream projects. Respect only complete local mock factories. Reject unknown setup dependencies and source mutations during dependency setup. - Declare `oxc-parser` directly for AST inspection. This exact version is already a transitive dependency in the checked-in lockfile; the direct declaration avoids relying on pnpm's internal dependency layout. - Produce a merge-SHA-bound manifest with exact lane ownership and duration-aware partitions. Preserve serialized execution, chat shards and native Runner ownership. - Execute exact Vitest test specifications. Positional substring filters cannot accidentally run a similarly named file from another shard. - Use the original matrix if selection fails or times out. Fail selected jobs if their published artifact is missing or invalid. - Save the uploaded artifact ID in the executable plan. Failed-job retries reuse that artifact even when GitHub increments the attempt number. - Hash selected assignments into Turbo cache inputs. A subset cannot satisfy a full run or another subset. - Add graph, workflow, real-runner and real-Turbo qualification. The regression fixture proves a filesystem contract fails even when raw Vitest selection misses it. - Document selection limits, audit artifacts and `CI_VITEST_SELECTION=off`. ## Verification - All 138 affected routing, sharding, workflow, manifest, real Vitest and real Turbo checks passed under Node 24 on the final implementation. Additional release workflow checks passed. Cases cover opaque setup, remocking, generated inputs, Node's builtin loader, artifact retries, and exact execution of similarly named files. - Repository-wide `pnpm -r typecheck` and `pnpm build` passed before the clean rebase onto current master. The full remote typecheck and build are required on the current head. - `actionlint` passed for both changed workflows. `git diff --check` passed. - The Linux qualification run passed and selected 699 of 911 workspace test files for an Inbox change. UI graph construction took 16.1 seconds. This still uses two jobs. Synthetic partition coverage proves a 15-of-120 selection uses two of twelve jobs without dropping or duplicating files. - Both selected workspace shards passed locally. After tightening execution to exact specifications, the real colliding UI test ran as exactly one file with two passing assertions; the real server auth middleware test ran as exactly one file with 28 passing assertions. - The server probe selected 1,158 of 1,260 general files and 161 of 163 serialized files for an auth middleware change. Those conservative selections retain 12 general and nine serialized jobs. The server's temporary-home setup exception is pinned to reviewed content hashes; changed setup fails closed. - Full `pnpm test:run` has not been run locally. The full PR test matrix is required because this change modifies global CI inputs. Final-head CI is pending at runner allocation, and both qualification jobs are queued. Other recent repository workflows are also queued. Superseded runs of this PR were cancelled; this PR remains in draft until the required checks execute. - Greptile reviewed final commit `31b484fc5b412da245cffb52a8ac8977228c5651` at 5/5 with no unresolved findings. The earlier qualification run passed; it does not replace checks on the final commit. ## Risks - Dependency analysis adds setup and graph time. Fewer files do not guarantee a shorter critical path. CI summaries expose the actual file and job counts. - Static dependency graphs cannot describe all runtime inputs. Unknown inputs retain tests or use the original plan. The pure-package allowlist requires conservative review. - The trusted workflow runs from master. This PR exercises the new selector in the qualification workflow; ordinary PR selection starts after merge. Set `CI_VITEST_SELECTION=off`, use `full-ci`, or select full/shadow mode to disable narrowing. - The lockfile remains managed by the existing repository actions and is not changed here. ## Model Used OpenAI Codex, based on GPT-6, with repository tools and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 13:16:29 -05:00
DottaandPaperclip	f5dcc505c0	perf(ci): select affected suites and cache task results (#18214) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Pull request CI checks the server, UI, packages, Runner, and release paths. > - An ordinary PR starts 48 jobs even when its diff affects only a small part of the repository. > - Repeated pushes also repeat successful tests with unchanged inputs. > - This PR selects affected suites before runner allocation and caches deterministic task results with Turbo. > - Required checks validate the plan. Unknown inputs and failed comparisons keep full verification. > - The benefit is fewer runner jobs and less repeated work, with full post-merge verification retained. ## Linked Issues or Issue Description **What existing behavior does this improve?** The GitHub Actions PR workflow repeats unrelated suites for each push. **Subsystem affected** Cross-cutting CI routing, test execution, and required-check aggregation. **Current behavior** An ordinary UI component change starts the complete 48-job PR workflow. Existing dependency and compiler caches do not reuse successful test results. **Proposed behavior** A trusted planner reads the tested merge diff and task inputs. A UI component change such as `ui/src/components/Sidebar.tsx` selects 18 jobs, including build and browser coverage. Successful general, serialized, and standalone Rust tasks can reuse unchanged results across pushes in the same PR. **Reason and benefit** Reduce unrelated jobs and repeated test execution. The 48-to-18 comparison measures planned job count. Remote wall-clock savings have not yet been measured. **Breaking changes** No application API changes. Required `verify` and `e2e` names remain stable. `verify` also checks serialized suites and the canary dry run. ## What Changed - Added explicit task inputs and a pinned Turbo version. An execution adapter calls existing package scripts without adding script aliases. - Added trusted diff planning and dynamic matrices with complete fallback matrices. - Added PR-scoped task result caches. External-state and revision-stamped checks remain uncached. - Added and created the `full-ci` override label, plus repository modes `enabled`, `shadow`, and `full`. Forced and shadow runs bypass result caches. - Added candidate-workflow qualification, planner and aggregate tests, and a real Turbo cache fixture. - Kept the repository-wide CLI guidance guard in the uncached typecheck lane so UI-only diffs retain this security check. - Documented rollout controls, input maintenance, and local inspection commands. ## Verification - Passed 537 focused CI policy, routing, matrix, cache, and shard tests, plus 146 existing release workflow tests. - Passed actionlint for all four changed or added workflows and `git diff --check`. - Passed `pnpm -r typecheck` and `pnpm build` on the implementation base. - Checked real repository task plans for existing general-test and native Rust commands. A real server partition passed, then restored from cache: Turbo task time was 20.022 seconds for execution and 151 ms for the cache hit, excluding wrapper/setup overhead. - The real Turbo fixture exercises the execution adapter and proves source, argument, toolchain-scope, and environment invalidation, result restoration, and retry of failed tasks. - Stopped the serial local `pnpm test:run` after over 30 minutes in favor of the complete parallel GitHub suite. A local parallel database-lane attempt had PostgreSQL startup errors and timeouts. The direct migration fixture passed, then all 273 database tests passed with one local worker. This recovery does not establish the exact cause of the earlier startup failures. The full GitHub PR suite and candidate qualification passed on commit `0bf07ae071c9fe7903edc604d44b5a175727fed0`. Greptile is 5/5 with no unresolved review threads. - All 40 CLI guidance safety tests passed with the exact uncached workflow command. - Verified that master rulesets require only `ci / verify` and `ci / e2e`; both names remain unchanged. - Trusted workflows load from master. The added GitHub-hosted qualification workflow exercises this candidate planner before merge. ## Risks - Changes to global inputs, including task definitions and execution helpers, force execution and disable cache restore/save for the full PR diff. Missing a cross-package filesystem read could omit a suite. Inputs include known server reads of UI files and shared tests of UI assets. Unknown paths, malformed or truncated diffs, and definition download failures select all suites. - Task result reuse requires complete inputs. Keys include Git source files, scripts, resolved dependencies, environment, toolchain, checkout path, and unclassified source files. Generated build artifacts are excluded from input hashes. Registry smoke, browser, and revision-stamped tasks remain uncached. - Maintainers must update task inputs when suites start reading new paths. `shadow` reports selection while running all work. `full` and `full-ci` disable selection and result reuse. - The implementation becomes active after merge. Full post-merge verification remains unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool use, and code execution. The session does not expose a verifiable exact runtime model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:56:28 -05:00