PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-06 14:02:36 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	a194166017	perf(ci): repair dependency caches and parallelize transport fixtures (#18883 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect changes to its native Runner and control plane. > - Recent Runner jobs restored a Rust cache but still rebuilt dependencies for 74–92 seconds. > - New workspace install hooks disabled the materialized JavaScript dependency cache. > - A single transport file also kept 326 test cases on one worker. > - This pull request corrects the cache inputs, replays known install hooks, and separates independent transport test groups. > - The same checks can finish sooner without removing test cases or widening runner access. ## Linked Issues or Issue Description Refs #18794 and #18544. Related dependency-action version update: #17698; this change fixes the cache input contract used by the currently pinned action. What happened? Recent PR jobs such as #18783 and #18825 restored an exact Rust dependency cache from October 16. The virtual workspace's development profile changed on October 30. Rust Cache 2.9.2 hashes member manifests but omits that virtual manifest. Rebuilding cannot replace an immutable exact cache entry. The JavaScript cache also rejected the new server and Runner Copilot postinstall hooks. Expected behavior Changing a Cargo profile selects a new dependency cache. An exact JavaScript dependency restore still performs required workspace install hooks. Independent transport fixtures use the existing file workers while retaining all assertions and real deadlines. Steps to reproduce Inspect the Rust dependency cache hit and following `build:rust` in the recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs` on the previous revision to see `cacheable: false`. Run the original transport file to measure the serial test work. ## What Changed - Add the virtual workspace manifest and ancestor Cargo configuration to all shared Rust dependency keys. Keep reader/writer identities aligned and preserve master-only cache writes and CODEOWNERS-only AWS routing. - Recognize only the exact reviewed root, server, and Runner postinstall commands. Include the server/Runner installer scripts in the dependency key. Replay workspace postinstall hooks after a materialized restore. Unknown lifecycle hooks still disable reuse. - Move attachment recovery, warm attachment, live adoption, and suspension fixtures into separate files. Keep the large event-backlog cases in the original file to avoid starting another expensive fixture group at the same time. Share five stateless test helpers outside production source. - Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache producer and a fresh-host consumer. Require exact dependency/workspace hits and reject any Cargo unit that is rebuilt. Retain the JSON build evidence. Existing setup qualification checks real JavaScript cache restores. - Schedule the ten measured files over 50 seconds first, in approximate duration order. Package CI uses up to four workers, bounded by available CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15% memory. Preserve Vitest selection and shard membership and default ordering for other files. Job count and AWS authorization are unchanged. - Scope the optional paired server benchmark to changes in its measured inputs. Cache-only changes qualify exact restores and run the shared package tests using restored dependencies. - Update the cache and test documentation and the existing native cache contract guard. ## Verification - All 586 `.github/scripts/tests/.test.mjs` policy tests passed, including cache trust and runner routing guards. Actionlint passed for all three changed workflows. - All 169 original top-level transport test definitions remain byte-identical. The original file passed all 326 cases in 317.99 seconds on this macOS host. A clean preliminary split passed the exact same named cases in 183.01 seconds with the same four-worker cap. - Preserve the first failed experiment: its parallel run overlapped the policy suite and timed out in the 1,024-event settled backlog case at 60 seconds; 325 cases passed. A clean repeat passed, but that does not establish the earlier failure's sole cause. The final layout keeps the backlog cases in the original file. The final layout passed the same 326 cases in 186.35 seconds (41.4% faster than the 317.99-second baseline). The original test bodies and all timeouts remain unchanged. - Fresh-host Rust qualification passed: all 308 compilation units were fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)). JavaScript setup dropped from 46 seconds cold to 23 seconds with exact dependency/build restores; all 1,052 shared-package cases passed against the restored tree ([final-head job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)). The previous qualification also passed (43 to 19 seconds) and confirmed all three postinstall hooks replayed. These are qualification results, not an end-to-end PR timing claim. - Initial full AWS package run passed 4,907 cases with 10 existing skips. The original-file remainder fell to 118.7 seconds, but the 100-second deadline file started last, leaving the full package at 417.43 seconds. Starting only three long files first passed 4,911 cases (including four new sequencer guards) in 422.54 seconds, so reordering alone did not improve full-package time. Summed file execution was 1,202 seconds across three busy workers. The final four-worker configuration passed 4,912 cases, with the same 10 existing skips, in 324.99 seconds, a 23.1% reduction* from 422.54 seconds. The five added cases guard ordering, selection, sharding, and path handling. Peak memory was 17.95% of the 16 GiB runner; average CPU was 49.14% ([final-head package job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)). - Local workspace typecheck and build passed. The complete local `pnpm test:run` passed all six chat partitions but reported failures in one existing OAuth redirect case and one gateway authority case in the general-server group. Both focused repetitions passed (three OAuth cases and one gateway case), and a complete repeat of both files passed all 593 cases in 42.89 seconds; that does not explain the original failures. The broad local run was stopped after 61 minutes once full final-head CI had passed; its remaining general/workspace/serialized work was not completed locally, and it is not reported as a pass. Original failure logs were retained. Full final-head CI provides the complete suite validation. Sequencer selection/shard guards and the 11-case transport suspension smoke passed. The corrected release workflow guard passed all 71 tests. - The first CI run failed only the policy guard that still expected the old native cache key; downstream aggregate checks also failed. That guard now checks the new profile/config prefix and existing host/target/protocol suffix. A second policy run found that the new sequencer test imported Vitest in the dependency-free policy lane; it now runs in the Runner package lane, where all four cases pass. Greptile reviewed final head `8011f84189` at 5/5 with no actionable findings and confirmed this issue fixed. Final-head policy, typecheck, build, and review passed. At 23:52:46 UTC on November 3, several AWS jobs were terminated together: the Runner package process received SIGTERM and browser logs recorded a runner shutdown signal. The failed/cancelled jobs passed on the same-head retry, including required `ci / verify`; no assertion or timeout was changed in response. The final concurrency adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable findings. Final-head required CI, including `ci / verify`, passed ([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)). Local timings exclude install, build, cache transfer, and CI queue time. ## Risks A new Rust key needs one successful canonical master build before ordinary PRs can hit it. The fresh-host qualification uses separate PR-scoped entries. Parallel test files can increase I/O contention; case execution inside each file, timeouts, and assertions are unchanged. Two extracted suites exceed 1,000 lines because they retain existing multi-stage recovery fixtures unchanged; one parameterized fixture alone is 1,059 lines. No production execution logic, lockfile, credentials, runner trust gates, or required check is removed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, command execution, and GitHub inspection. The host does not expose a more precise model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks passed; the broad-run failures and repetitions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 20:17:08 -06:00
DottaandPaperclip	f5dcc505c0	perf(ci): select affected suites and cache task results (#18214 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Pull request CI checks the server, UI, packages, Runner, and release paths. > - An ordinary PR starts 48 jobs even when its diff affects only a small part of the repository. > - Repeated pushes also repeat successful tests with unchanged inputs. > - This PR selects affected suites before runner allocation and caches deterministic task results with Turbo. > - Required checks validate the plan. Unknown inputs and failed comparisons keep full verification. > - The benefit is fewer runner jobs and less repeated work, with full post-merge verification retained. ## Linked Issues or Issue Description What existing behavior does this improve? The GitHub Actions PR workflow repeats unrelated suites for each push. Subsystem affected Cross-cutting CI routing, test execution, and required-check aggregation. Current behavior An ordinary UI component change starts the complete 48-job PR workflow. Existing dependency and compiler caches do not reuse successful test results. Proposed behavior A trusted planner reads the tested merge diff and task inputs. A UI component change such as `ui/src/components/Sidebar.tsx` selects 18 jobs, including build and browser coverage. Successful general, serialized, and standalone Rust tasks can reuse unchanged results across pushes in the same PR. Reason and benefit Reduce unrelated jobs and repeated test execution. The 48-to-18 comparison measures planned job count. Remote wall-clock savings have not yet been measured. Breaking changes No application API changes. Required `verify` and `e2e` names remain stable. `verify` also checks serialized suites and the canary dry run. ## What Changed - Added explicit task inputs and a pinned Turbo version. An execution adapter calls existing package scripts without adding script aliases. - Added trusted diff planning and dynamic matrices with complete fallback matrices. - Added PR-scoped task result caches. External-state and revision-stamped checks remain uncached. - Added and created the `full-ci` override label, plus repository modes `enabled`, `shadow`, and `full`. Forced and shadow runs bypass result caches. - Added candidate-workflow qualification, planner and aggregate tests, and a real Turbo cache fixture. - Kept the repository-wide CLI guidance guard in the uncached typecheck lane so UI-only diffs retain this security check. - Documented rollout controls, input maintenance, and local inspection commands. ## Verification - Passed 537 focused CI policy, routing, matrix, cache, and shard tests, plus 146 existing release workflow tests. - Passed actionlint for all four changed or added workflows and `git diff --check`. - Passed `pnpm -r typecheck` and `pnpm build` on the implementation base. - Checked real repository task plans for existing general-test and native Rust commands. A real server partition passed, then restored from cache: Turbo task time was 20.022 seconds for execution and 151 ms for the cache hit, excluding wrapper/setup overhead. - The real Turbo fixture exercises the execution adapter and proves source, argument, toolchain-scope, and environment invalidation, result restoration, and retry of failed tasks. - Stopped the serial local `pnpm test:run` after over 30 minutes in favor of the complete parallel GitHub suite. A local parallel database-lane attempt had PostgreSQL startup errors and timeouts. The direct migration fixture passed, then all 273 database tests passed with one local worker. This recovery does not establish the exact cause of the earlier startup failures. The full GitHub PR suite and candidate qualification passed on commit `0bf07ae071c9fe7903edc604d44b5a175727fed0`. Greptile is 5/5 with no unresolved review threads. - All 40 CLI guidance safety tests passed with the exact uncached workflow command. - Verified that master rulesets require only `ci / verify` and `ci / e2e`; both names remain unchanged. - Trusted workflows load from master. The added GitHub-hosted qualification workflow exercises this candidate planner before merge. ## Risks - Changes to global inputs, including task definitions and execution helpers, force execution and disable cache restore/save for the full PR diff. Missing a cross-package filesystem read could omit a suite. Inputs include known server reads of UI files and shared tests of UI assets. Unknown paths, malformed or truncated diffs, and definition download failures select all suites. - Task result reuse requires complete inputs. Keys include Git source files, scripts, resolved dependencies, environment, toolchain, checkout path, and unclassified source files. Generated build artifacts are excluded from input hashes. Registry smoke, browser, and revision-stamped tasks remain uncached. - Maintainers must update task inputs when suites start reading new paths. `shadow` reports selection while running all work. `full` and `full-ci` disable selection and result reuse. - The implementation becomes active after merge. Full post-merge verification remains unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool use, and code execution. The session does not expose a verifiable exact runtime model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:56:28 -05:00
DottaandPaperclip	7ec6fd0972	perf(ci): remove repeated test setup and fixture work (#16429 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI must check changes without making contributors wait for repeated setup. > - A complete CI profile showed 158 serialized Vitest launches and repeated dependency installs. > - A large fixture also transferred 40,000 files to prove two different size boundaries. > - This PR removes repeated controller, install, build, transfer, and wait work. > - It keeps isolated test processes and all useful regression boundaries. > - Paired command benchmarks check speed and test inventories before we claim a benefit. ## Linked Issues or Issue Description Refs #16423 for the complete CI profiling data path. No duplicate optimization PR was found. Current behavior Each serialized file starts a new Vitest controller. Each CI lane restores the pnpm store and installs dependencies. Large Git scanning and staging share one 40,000-file fixture. A readiness regression waits over a minute for the default database idle deadline. Proposed behavior Use one controller per serialized shard, with a fresh fork for each file. Restore exact installed dependencies and certified shared/SDK build outputs when inputs match. Keep the normal install and build paths on a miss. Separate the 32 MiB Git scan proof from staging above command argument limits. Cross a real database idle deadline with a one-second fixture setting. Reason and benefit The paired Linux CI benchmark reduced the complete 566-case serialized shard from 303.43s to 246.96s (18.6%). The Git/database fixture command fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration reduced setup from 56s to 19s (66.1%). These are measured command/setup times. They do not establish a controlled reduction in the full workflow's queue-to-completion time. Breaking changes None to application behavior. Tests still run in full. Test results are not cached. Cache misses preserve the existing lockfile resolution fallback. ## What Changed - Batch serialized files into one isolated, single-worker Vitest invocation per shard. Give each worker its own home, config and instance ID. - Add a real two-order isolation check for processes, environment, globals, timers and filesystem state. - Add exact dependency keys for all manifests, lockfile, workspace settings, patches, installer, runtime, platform, libc, pnpm, store and checkout path. Cache only installed package directories, not source fixtures. - Add a separate content key for the shared/SDK dependency graph. Retain the existing source/output fingerprint verifier and rebuild path. Only an exact canonical master push can populate normal caches. Stale lockfile fallback runs cannot publish entries. - Keep four real Git filename lanes above 32 MiB. Use a separate 4,000-file staging/deletion case above 2 MiB. Check every staged file, unusual names, cleanup and bounded commands. - Configure a real one-second PostgreSQL idle timeout in the readiness fixture. Assert that the backend connection changes and that another connection can write the parent row. - Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It compares the tested merge tree with its actual first parent on one runner. It also removes installed outputs and qualifies a real cache restore using temporary PR-scoped entries. ## Verification - Local Node 24.19.0: unchanged serialized shard 255.69s; candidate 231.44s; all 566 cases pass in both. - Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate scan 18.58s, staging 13.41s, database case 2.80s. All three candidate cases pass. - Negative control: moving readiness back inside the real parent-lock transaction fails the shortened regression at the parent-write assertion. The production source was restored and is absent from the diff. - Worker isolation check passes in both file orders. - 565 CI policy, partition and workflow regression checks pass with local HTTP/IPC access enabled. The first sandboxed attempt was denied listen access; it was not a code failure. - `actionlint` passes for the three changed workflows. - [All 48 main CI jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045) on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full typecheck, tests, build, browser E2E and release checks. All 158 serialized files and 3,250 cases passed through 9 controller invocations. Full typecheck/test/build were verified in CI; they were not all repeated locally. - [Paired Linux benchmark passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127) on Node 24.21.0. Its baseline is `57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is `f68f1b32966090f11647502e31852130c20237cf`. The artifact contains results, exact case inventories, setup times and raw logs. Serialized case identities match exactly. All three fixture contracts execute in both variants. \| Measured work \| Baseline \| Candidate \| Reduction \| \|---\|---:\|---:\|---:\| \| Complete serialized shard, 566 cases \| 303.43s \| 246.96s \| 18.6% \| \| Git/database fixture command \| 156.54s \| 33.78s \| 78.4% \| \| Dependency/build setup, cold vs exact hit \| 56s \| 19s \| 66.1% \| - [Honeycomb profile export succeeded on its second attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2). The first attempt failed on GitHub HTTP 503 while downloading an artifact, before export. Only the export was retried; tests were not repeated. The temporary PR-scoped qualification caches were removed after validation, reclaiming 3.34 GB. Timing artifacts remain available. - Greptile is 5/5 on the current head, with no unresolved review threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the corrected reporter paths and paired comparison passed. Earlier runs were superseded while this was fixed. A copy-batching experiment did not speed up the Git fixture and was replaced by the two-boundary fixture split. ## Risks - A persistent controller retains transform state longer. The local shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must confirm all shards fit the runner budget. - Materialized caches are larger than metadata caches. The Linux benchmark measured real archive transfer and validation after removing installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes cannot publish cache entries. Exact misses run installation. Canonical cache warming requires a frozen-lockfile install; the existing stale-lockfile fallback remains uncached. - The PR uses the trusted reusable workflow from master. The paired benchmark directly exercises the candidate dependency action; normal workflow adoption still requires this PR to merge. This PR does not change the trust references. The completed main workflow took 11m02 including queue time. No controlled full-workflow latency improvement is claimed before this workflow adopts the cache change. - Only deterministic dependency/build outputs are reused. Broad dependency-based test omission and caching successful test results are deliberately excluded from this change. - The real Runner lease minimum and cold-admission waits remain. They protect protocol boundaries and were not shortened by weakening production limits. ## Model Used OpenAI GPT-6 through Codex, with reasoning, shell execution and code tools. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 09:23:47 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

a194166017

perf(ci): repair dependency caches and parallelize transport fixtures (#18883 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Required CI checks protect changes to its native Runner and control
plane.
> - Recent Runner jobs restored a Rust cache but still rebuilt
dependencies for 74–92 seconds.
> - New workspace install hooks disabled the materialized JavaScript
dependency cache.
> - A single transport file also kept 326 test cases on one worker.
> - This pull request corrects the cache inputs, replays known install
hooks, and separates independent transport test groups.
> - The same checks can finish sooner without removing test cases or
widening runner access.

## Linked Issues or Issue Description

Refs #18794 and #18544. Related dependency-action version update:
#17698; this change fixes the cache input contract used by the currently
pinned action.

**What happened?**

Recent PR jobs such as #18783 and #18825 restored an exact Rust
dependency cache from October 16. The virtual workspace's development
profile changed on October 30. Rust Cache 2.9.2 hashes member manifests
but omits that virtual manifest. Rebuilding cannot replace an immutable
exact cache entry. The JavaScript cache also rejected the new server and
Runner Copilot postinstall hooks.

**Expected behavior**

Changing a Cargo profile selects a new dependency cache. An exact
JavaScript dependency restore still performs required workspace install
hooks. Independent transport fixtures use the existing file workers
while retaining all assertions and real deadlines.

**Steps to reproduce**

Inspect the Rust dependency cache hit and following `build:rust` in the
recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs`
on the previous revision to see `cacheable: false`. Run the original
transport file to measure the serial test work.

## What Changed

- Add the virtual workspace manifest and ancestor Cargo configuration to
all shared Rust dependency keys. Keep reader/writer identities aligned
and preserve master-only cache writes and CODEOWNERS-only AWS routing.
- Recognize only the exact reviewed root, server, and Runner postinstall
commands. Include the server/Runner installer scripts in the dependency
key. Replay workspace postinstall hooks after a materialized restore.
Unknown lifecycle hooks still disable reuse.
- Move attachment recovery, warm attachment, live adoption, and
suspension fixtures into separate files. Keep the large event-backlog
cases in the original file to avoid starting another expensive fixture
group at the same time. Share five stateless test helpers outside
production source.
- Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache
producer and a fresh-host consumer. Require exact dependency/workspace
hits and reject any Cargo unit that is rebuilt. Retain the JSON build
evidence. Existing setup qualification checks real JavaScript cache
restores.
- Schedule the ten measured files over 50 seconds first, in approximate
duration order. Package CI uses up to four workers, bounded by available
CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs
and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15%
memory. Preserve Vitest selection and shard membership and default
ordering for other files. Job count and AWS authorization are unchanged.
- Scope the optional paired server benchmark to changes in its measured
inputs. Cache-only changes qualify exact restores and run the shared
package tests using restored dependencies.
- Update the cache and test documentation and the existing native cache
contract guard.

## Verification

- All 586 `.github/scripts/tests/*.test.mjs` policy tests passed,
including cache trust and runner routing guards. Actionlint passed for
all three changed workflows.
- All 169 original top-level transport test definitions remain
byte-identical. The original file passed all 326 cases in 317.99 seconds
on this macOS host. A clean preliminary split passed the exact same
named cases in 183.01 seconds with the same four-worker cap.
- Preserve the first failed experiment: its parallel run overlapped the
policy suite and timed out in the 1,024-event settled backlog case at 60
seconds; 325 cases passed. A clean repeat passed, but that does not
establish the earlier failure's sole cause. The final layout keeps the
backlog cases in the original file. The final layout passed the same 326
cases in 186.35 seconds (41.4% faster than the 317.99-second baseline).
The original test bodies and all timeouts remain unchanged.
- Fresh-host Rust qualification passed: all 308 compilation units were
fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head
run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)).
JavaScript setup dropped from 46 seconds cold to 23 seconds with exact
dependency/build restores; all 1,052 shared-package cases passed against
the restored tree ([final-head
job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)).
The previous qualification also passed (43 to 19 seconds) and confirmed
all three postinstall hooks replayed. These are qualification results,
not an end-to-end PR timing claim.
- Initial full AWS package run passed 4,907 cases with 10 existing
skips. The original-file remainder fell to 118.7 seconds, but the
100-second deadline file started last, leaving the full package at
417.43 seconds. Starting only three long files first passed 4,911 cases
(including four new sequencer guards) in 422.54 seconds, so reordering
alone did not improve full-package time. Summed file execution was 1,202
seconds across three busy workers. The final four-worker configuration
passed **4,912 cases, with the same 10 existing skips**, in **324.99
seconds**, a **23.1% reduction** from 422.54 seconds. The five added
cases guard ordering, selection, sharding, and path handling. Peak
memory was 17.95% of the 16 GiB runner; average CPU was 49.14%
([final-head package
job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)).
- Local workspace typecheck and build passed. The complete local `pnpm
test:run` passed all six chat partitions but reported failures in one
existing OAuth redirect case and one gateway authority case in the
general-server group. Both focused repetitions passed (three OAuth cases
and one gateway case), and a complete repeat of both files passed all
593 cases in 42.89 seconds; that does not explain the original failures.
The broad local run was stopped after 61 minutes once full final-head CI
had passed; its remaining general/workspace/serialized work was not
completed locally, and it is not reported as a pass. Original failure
logs were retained. Full final-head CI provides the complete suite
validation. Sequencer selection/shard guards and the 11-case transport
suspension smoke passed. The corrected release workflow guard passed all
71 tests.
- The first CI run failed only the policy guard that still expected the
old native cache key; downstream aggregate checks also failed. That
guard now checks the new profile/config prefix and existing
host/target/protocol suffix. A second policy run found that the new
sequencer test imported Vitest in the dependency-free policy lane; it
now runs in the Runner package lane, where all four cases pass. Greptile
reviewed final head `8011f84189` at 5/5 with no actionable findings and
confirmed this issue fixed. Final-head policy, typecheck, build, and
review passed. At 23:52:46 UTC on November 3, several AWS jobs were
terminated together: the Runner package process received SIGTERM and
browser logs recorded a runner shutdown signal. The failed/cancelled
jobs passed on the same-head retry, including required `ci / verify`; no
assertion or timeout was changed in response. The final concurrency
adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable
findings. Final-head required CI, including `ci / verify`, passed
([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)).
Local timings exclude install, build, cache transfer, and CI queue time.

## Risks

A new Rust key needs one successful canonical master build before
ordinary PRs can hit it. The fresh-host qualification uses separate
PR-scoped entries. Parallel test files can increase I/O contention; case
execution inside each file, timeouts, and assertions are unchanged. Two
extracted suites exceed 1,000 lines because they retain existing
multi-stage recovery fixtures unchanged; one parameterized fixture alone
is 1,059 lines. No production execution logic, lockfile, credentials,
runner trust gates, or required check is removed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, command
execution, and GitHub inspection. The host does not expose a more
precise model snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — targeted checks passed;
the broad-run failures and repetitions are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-03 20:17:08 -06:00

DottaandPaperclip

f5dcc505c0

perf(ci): select affected suites and cache task results (#18214 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request CI checks the server, UI, packages, Runner, and release
paths.
> - An ordinary PR starts 48 jobs even when its diff affects only a
small part of the repository.
> - Repeated pushes also repeat successful tests with unchanged inputs.
> - This PR selects affected suites before runner allocation and caches
deterministic task results with Turbo.
> - Required checks validate the plan. Unknown inputs and failed
comparisons keep full verification.
> - The benefit is fewer runner jobs and less repeated work, with full
post-merge verification retained.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The GitHub Actions PR workflow repeats unrelated suites for each push.

**Subsystem affected**

Cross-cutting CI routing, test execution, and required-check
aggregation.

**Current behavior**

An ordinary UI component change starts the complete 48-job PR workflow.
Existing dependency and compiler caches do not reuse successful test
results.

**Proposed behavior**

A trusted planner reads the tested merge diff and task inputs. A UI
component change such as `ui/src/components/Sidebar.tsx` selects 18
jobs, including build and browser coverage. Successful general,
serialized, and standalone Rust tasks can reuse unchanged results across
pushes in the same PR.

**Reason and benefit**

Reduce unrelated jobs and repeated test execution. The 48-to-18
comparison measures planned job count. Remote wall-clock savings have
not yet been measured.

**Breaking changes**

No application API changes. Required `verify` and `e2e` names remain
stable. `verify` also checks serialized suites and the canary dry run.

## What Changed

- Added explicit task inputs and a pinned Turbo version. An execution
adapter calls existing package scripts without adding script aliases.
- Added trusted diff planning and dynamic matrices with complete
fallback matrices.
- Added PR-scoped task result caches. External-state and
revision-stamped checks remain uncached.
- Added and created the `full-ci` override label, plus repository modes
`enabled`, `shadow`, and `full`. Forced and shadow runs bypass result
caches.
- Added candidate-workflow qualification, planner and aggregate tests,
and a real Turbo cache fixture.
- Kept the repository-wide CLI guidance guard in the uncached typecheck
lane so UI-only diffs retain this security check.
- Documented rollout controls, input maintenance, and local inspection
commands.

## Verification

- Passed 537 focused CI policy, routing, matrix, cache, and shard tests,
plus 146 existing release workflow tests.
- Passed actionlint for all four changed or added workflows and `git
diff --check`.
- Passed `pnpm -r typecheck` and `pnpm build` on the implementation
base.
- Checked real repository task plans for existing general-test and
native Rust commands. A real server partition passed, then restored from
cache: Turbo task time was 20.022 seconds for execution and 151 ms for
the cache hit, excluding wrapper/setup overhead.
- The real Turbo fixture exercises the execution adapter and proves
source, argument, toolchain-scope, and environment invalidation, result
restoration, and retry of failed tasks.
- Stopped the serial local `pnpm test:run` after over 30 minutes in
favor of the complete parallel GitHub suite. A local parallel
database-lane attempt had PostgreSQL startup errors and timeouts. The
direct migration fixture passed, then all 273 database tests passed with
one local worker. This recovery does not establish the exact cause of
the earlier startup failures. The full GitHub PR suite and candidate
qualification passed on commit
`0bf07ae071c9fe7903edc604d44b5a175727fed0`. Greptile is 5/5 with no
unresolved review threads.
- All 40 CLI guidance safety tests passed with the exact uncached
workflow command.
- Verified that master rulesets require only `ci / verify` and `ci /
e2e`; both names remain unchanged.
- Trusted workflows load from master. The added GitHub-hosted
qualification workflow exercises this candidate planner before merge.

## Risks

- Changes to global inputs, including task definitions and execution
helpers, force execution and disable cache restore/save for the full PR
diff. Missing a cross-package filesystem read could omit a suite. Inputs
include known server reads of UI files and shared tests of UI assets.
Unknown paths, malformed or truncated diffs, and definition download
failures select all suites.
- Task result reuse requires complete inputs. Keys include Git source
files, scripts, resolved dependencies, environment, toolchain, checkout
path, and unclassified source files. Generated build artifacts are
excluded from input hashes. Registry smoke, browser, and
revision-stamped tasks remain uncached.
- Maintainers must update task inputs when suites start reading new
paths. `shadow` reports selection while running all work. `full` and
`full-ci` disable selection and result reuse.
- The implementation becomes active after merge. Full post-merge
verification remains unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool
use, and code execution. The session does not expose a verifiable exact
runtime model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-29 16:56:28 -05:00

DottaandPaperclip

7ec6fd0972

perf(ci): remove repeated test setup and fixture work (#16429 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI must check changes without making contributors wait for repeated
setup.
> - A complete CI profile showed 158 serialized Vitest launches and
repeated dependency installs.
> - A large fixture also transferred 40,000 files to prove two different
size boundaries.
> - This PR removes repeated controller, install, build, transfer, and
wait work.
> - It keeps isolated test processes and all useful regression
boundaries.
> - Paired command benchmarks check speed and test inventories before we
claim a benefit.

## Linked Issues or Issue Description

Refs #16423 for the complete CI profiling data path. No duplicate
optimization PR was found.

**Current behavior**

Each serialized file starts a new Vitest controller. Each CI lane
restores the pnpm store and installs dependencies. Large Git scanning
and staging share one 40,000-file fixture. A readiness regression waits
over a minute for the default database idle deadline.

**Proposed behavior**

Use one controller per serialized shard, with a fresh fork for each
file. Restore exact installed dependencies and certified shared/SDK
build outputs when inputs match. Keep the normal install and build paths
on a miss. Separate the 32 MiB Git scan proof from staging above command
argument limits. Cross a real database idle deadline with a one-second
fixture setting.

**Reason and benefit**

The paired Linux CI benchmark reduced the complete 566-case serialized
shard from 303.43s to 246.96s (18.6%). The Git/database fixture command
fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration
reduced setup from 56s to 19s (66.1%). These are measured command/setup
times. They do not establish a controlled reduction in the full
workflow's queue-to-completion time.

**Breaking changes**

None to application behavior. Tests still run in full. Test results are
not cached. Cache misses preserve the existing lockfile resolution
fallback.

## What Changed

- Batch serialized files into one isolated, single-worker Vitest
invocation per shard. Give each worker its own home, config and instance
ID.
- Add a real two-order isolation check for processes, environment,
globals, timers and filesystem state.
- Add exact dependency keys for all manifests, lockfile, workspace
settings, patches, installer, runtime, platform, libc, pnpm, store and
checkout path. Cache only installed package directories, not source
fixtures.
- Add a separate content key for the shared/SDK dependency graph. Retain
the existing source/output fingerprint verifier and rebuild path. Only
an exact canonical master push can populate normal caches. Stale
lockfile fallback runs cannot publish entries.
- Keep four real Git filename lanes above 32 MiB. Use a separate
4,000-file staging/deletion case above 2 MiB. Check every staged file,
unusual names, cleanup and bounded commands.
- Configure a real one-second PostgreSQL idle timeout in the readiness
fixture. Assert that the backend connection changes and that another
connection can write the parent row.
- Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It
compares the tested merge tree with its actual first parent on one
runner. It also removes installed outputs and qualifies a real cache
restore using temporary PR-scoped entries.

## Verification

- Local Node 24.19.0: unchanged serialized shard 255.69s; candidate
231.44s; all 566 cases pass in both.
- Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate
scan 18.58s, staging 13.41s, database case 2.80s. All three candidate
cases pass.
- Negative control: moving readiness back inside the real parent-lock
transaction fails the shortened regression at the parent-write
assertion. The production source was restored and is absent from the
diff.
- Worker isolation check passes in both file orders.
- 565 CI policy, partition and workflow regression checks pass with
local HTTP/IPC access enabled. The first sandboxed attempt was denied
listen access; it was not a code failure.
- `actionlint` passes for the three changed workflows.
- [All 48 main CI jobs
passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045)
on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full
typecheck, tests, build, browser E2E and release checks. All 158
serialized files and 3,250 cases passed through 9 controller
invocations. Full typecheck/test/build were verified in CI; they were
not all repeated locally.
- [Paired Linux benchmark
passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127)
on Node 24.21.0. Its baseline is
`57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is
`f68f1b32966090f11647502e31852130c20237cf`. The artifact contains
results, exact case inventories, setup times and raw logs. Serialized
case identities match exactly. All three fixture contracts execute in
both variants.

  | Measured work | Baseline | Candidate | Reduction |
  |---|---:|---:|---:|
  | Complete serialized shard, 566 cases | 303.43s | 246.96s | 18.6% |
  | Git/database fixture command | 156.54s | 33.78s | 78.4% |
  | Dependency/build setup, cold vs exact hit | 56s | 19s | 66.1% |

- [Honeycomb profile export succeeded on its second
attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2).
The first attempt failed on GitHub HTTP 503 while downloading an
artifact, before export. Only the export was retried; tests were not
repeated. The temporary PR-scoped qualification caches were removed
after validation, reclaiming 3.34 GB. Timing artifacts remain available.
- Greptile is 5/5 on the current head, with no unresolved review
threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the
corrected reporter paths and paired comparison passed. Earlier runs were
superseded while this was fixed. A copy-batching experiment did not
speed up the Git fixture and was replaced by the two-boundary fixture
split.

## Risks

- A persistent controller retains transform state longer. The local
shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must
confirm all shards fit the runner budget.
- Materialized caches are larger than metadata caches. The Linux
benchmark measured real archive transfer and validation after removing
installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes
cannot publish cache entries. Exact misses run installation. Canonical
cache warming requires a frozen-lockfile install; the existing
stale-lockfile fallback remains uncached.
- The PR uses the trusted reusable workflow from master. The paired
benchmark directly exercises the candidate dependency action; normal
workflow adoption still requires this PR to merge. This PR does not
change the trust references. The completed main workflow took 11m02
including queue time. No controlled full-workflow latency improvement is
claimed before this workflow adopts the cache change.
- Only deterministic dependency/build outputs are reused. Broad
dependency-based test omission and caching successful test results are
deliberately excluded from this change.
- The real Runner lease minimum and cold-admission waits remain. They
protect protocol boundaries and were not shortened by weakening
production limits.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, shell execution and code
tools. The runtime does not expose a more specific deployment ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-15 09:23:47 -05:00