6 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	676d3ec8d4	perf(ci): reuse package downloads and overlap runtime checks (#16474) ... ## Thinking Path > - Paperclip contributors wait for CI before delivering changes. > - Build still finishes last after #16460. > - CI already downloads the public npm graph while the build runs. > - Bundled-package preparation ignores that cache and downloads the same archives again. > - Reuse the existing cache and overlap independent runtime checks while preserving fresh installs, integrity, offline lifecycle and admission verification. > - Measure staging and the full workflow separately. ## Linked Issues or Issue Description Refs #16460 and #16434. This is a focused follow-up to merged #16460, now based directly on master. **Current behavior:** The Build workflow warms `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`, but `startBundledInstalls` inherits the host's default npm cache. Server staging took 31.17 seconds in the latest passing run, before packing and the clean consumer install. **Proposed behavior:** Pass the verifier environment, including the explicitly configured warm cache, to the three existing staging children. Without an explicit shared cache, keep the usual producer npm configuration. **Alternatives considered:** Copying an installed dependency tree would bypass normal fresh installation. Adding jobs would add setup cost. This change reuses only npm's integrity-checked download cache. It does not remove coverage or change product behavior and does not overlap a roadmap feature. Open PR title search found no equivalent cache wiring change. ## What Changed - Initialize the existing consumer cache before package staging and expose it to producer npm only when explicitly configured. - Pass that environment to the existing bundled-install children. - Extend the child orchestration check to verify cache propagation and document the cache boundary. - Start Pi setup after the offline lifecycle, in its own download sandbox, while Codex and Grok probes run in separate offline sandboxes. Join successful Pi setup before lock verification and Pi admission. Name and clean up only the verifier-owned setup container. ## Verification - 33 focused public-install, closure and child-orchestration tests pass on Node 24.21.0. - A real server dependency staging experiment used `prepare-bundled-package.mjs`, its actual manifests, provider overrides and release patches. Built application payloads were omitted from this dependency-only measurement. - Cold cache: **16.139s**. After the actual CI warm-up script: **6.980s**. A fully warm producer cache took **6.127s**. These are local sequential measurements, not a CI wall-clock claim; filesystem warmth and registry timing vary. - All **28,329 files/links**, including both lockfiles, were byte-identical between the cold output and the actual-workflow-warm output. - The warm-up itself took 49.631s; CI already runs it during Rust restore/build. This change adds no warm-up operation. - First follow-up run [passed all 48 jobs](https://github.com/paperclipai/paperclip/actions/runs/38791277425), and Greptile gave the cache-only head 5/5. Server staging fell **31.172s → 17.565s** and total staging **70.169s → 54.869s**. But complete CI was **8m20s**, slower than the parent's 8m04s; longer Codex/Pi probe times outweighed that saving. This is not a full-run speed improvement claim. - The next revision overlaps the independent probes instead of serializing Pi setup behind them. All 46 focused sandbox, staging, closure and process-supervision tests pass, including failed background children, parent exit and cancellation. Pre-retarget head `590873a44` passes all 48 jobs and Greptile 5/5 with no review findings. After #16460 merged, head `fe4f8d38e` incorporates master and retargets this PR directly to master. Its diff remains only four optimization/documentation/test files; 46 focused checks pass again. Fresh full CI and review are pending. The unchanged parent conflict-resolution run passed all 48 jobs in 8m04s: https://github.com/paperclipai/paperclip/actions/runs/38789785045. Final paired runs against the same updated master, both on `m7i-flex.xlarge`: | Measurement | Parent #16460 | This PR | | --- | ---: | ---: | | Bundled server staging | 22.220s | 14.202s | | All package staging | 61.277s | 48.304s | | Clean consumer install | 67.617s | 73.490s | | Pi setup remaining wait | 25.428s (serial setup) | 20.719s | | Build job | 385s | 368s | | Build start after workflow creation | 23s | 136s | | Entire workflow, including admission and aggregates | 452s (7m32s) | 552s (9m12s) | [Parent run](https://github.com/paperclipai/paperclip/actions/runs/38792983313), [final run](https://github.com/paperclipai/paperclip/actions/runs/38792986392). Build execution improved by **17s (4.4%)** in this pair. The total workflow did **not** improve: it waited 113 seconds longer to start Build, and other lanes/aggregates also contribute. The overlapping setup ran for 25.667s while Codex/Grok took 4.946s, leaving 20.719s to wait. Actual installed Codex, Grok and Pi checks all passed with zero provider calls. These are individual samples with filesystem/network/CPU variance, not a claimed steady-state average. The earlier slower cache-only run is retained above. No tests were removed and no jobs were added. ## Risks Shared-cache contention can reduce the benefit if warm-up is still running. npm handles concurrent content-addressed cache writers and checks integrity; misses still download normally. Two runtime containers can now overlap and contend for host resources; the full CI run measures that tradeoff. They retain separate network policies and per-container bounds. Pi writes only its provider assets and a private receipt; the other probes inspect Codex/Grok. No installed tree, lock, test result or build output is reused. The consumer still starts empty and performs the full lifecycle and runtime checks. Normal unconfigured producer cache behavior is preserved. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, execution and GitHub tools. The serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no private ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [ ] All Paperclip CI gates are green (prior head passed; rerunning after master merge) - [ ] Greptile is 5/5 with no open P2s, recommendations or follow-ups (prior head 5/5; current head pending) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 15:16:52 -05:00
DottaandPaperclip	307eb16c0c	perf(ci): avoid duplicate extraction in public install checks (#16460) ... ## Thinking Path > - Paperclip manages AI agents for work. > - Contributors wait for CI before they can deliver changes. > - After #16434, Build is the longest CI job. > - Its clean public npm install spends about 90 seconds installing packed artifacts. > - npm extracts local archives to read metadata before it installs them. > - Serve metadata from those same packed artifacts, then let normal npm verify and install their bytes once. > - Preserve the offline lifecycle and runtime checks, and measure the complete workflow. ## Linked Issues or Issue Description Refs #16434 and #16429. **Current behavior:** The Linux Build check installs every staged package by local tarball path. npm's file fetcher extracts an archive to obtain its manifest. Installation then extracts it again. In the last green run, Build took 408 seconds; consumer installation took 89.7 seconds. The full workflow took 8m06s. **Proposed behavior:** Read manifests directly from the packed archives. Serve those manifests and their SHA-512-bound tarballs from a temporary registry on container loopback. Use normal npm installation into the empty consumer. Other packages still resolve from the public npm registry. **Alternatives:** More parallel jobs would add setup and transfer work. Skipping the install or runtime probes would remove useful coverage. This change removes duplicate extraction instead. It does not add a product feature or duplicate a roadmap item. ## What Changed - Build a registry catalog from the actual tarball manifests and archive hashes. - Run the short-lived registry and npm in the existing unprivileged download container. No host port is exposed. - Preserve public dependency tarball URLs, archive integrity checks, disabled initial lifecycle scripts, the offline rebuild, lock checks and all Codex, Grok and Pi probes. - Close the registry before offline verification. Keep native ARM and Mac install qualification unchanged. - Add real npm parity and failure tests. Record metadata preparation time in the existing timing receipt. - Fix an existing policy false positive: the filename `docker-compose-secrets.test.mjs` is not a workflow credential. Inspect GitHub expressions and retain rejection of Honeycomb keys, dot/bracket secret reads and `toJSON(secrets)`. ## Verification - 31 focused install-contract checks pass on Node 24.21.0. All 16 targeted CI profile-policy tests also pass. - Real npm file and registry installs produce identical installed files, bundled dependencies, optional dependencies and executable links. Both defer lifecycle execution; offline rebuild works after the registry closes and preserves the lock. - npm rejects an archive whose gzip header changes while its extracted payload remains valid. The CLI propagates the failure. - The permanent real-npm parity test now includes non-catalog `is-number@7.0.0`. Both paths install it from its official npmjs tarball, and the test asserts the lock URL. Fetches have a 10-second bound and no retries. This addresses the first Greptile finding. A separate live probe also confirmed public provenance. - The first local run hit the host sandbox's loopback `EPERM`; the same checks passed with local listener access. An initial corrupted-gzip negative fixture hit decompression failure before integrity verification; the final fixture preserves valid gzip and specifically proves integrity rejection. An old command-shape assertion was updated for the new entry point without removing its ordering checks. - The [first CI run](https://github.com/paperclipai/paperclip/actions/runs/38778371043) passed all test lanes and Build, but the existing credential-guard false positive failed policy and both aggregates. Its 7m25s full duration is **not** counted as a passing performance sample. The guard repair and public-download regression are on the new head. - The real Linux install passed all 18 packed packages, the lifecycle sentinel, offline rebuild, unchanged lock, official Codex payload/lease checks, Grok missing/present checks and Pi admission. The receipt confirms `artifactTransport: loopback-registry`. No provider calls ran. - Compared with the previous green run, consumer install fell **89.72s → 66.29s** (26.1%), including 1.87s to prepare metadata. Build fell **408s → 374s**. Both Build hosts were `m7i-flex.xlarge`; these are separate runs, so registry/cache and upstream changes remain confounders. - The base advanced since the previous 8m06s run, including runtime and test changes. Treat that duration as historical context, not an identical-source control. - Pre-conflict head `d8f9de1c` [passes all 48 workflow jobs](https://github.com/paperclipai/paperclip/actions/runs/38779549190), including both required branch-protection checks. Greptile is **5/5** on this head; its review thread is resolved. The non-required Superagent Security Scan is still running. | Measurement | Previous green run | Final green run | Observed change | | --- | ---: | ---: | ---: | | Full CI, workflow creation through final aggregate | 8m06s | **7m35s** | **31s shorter (6.4%)** | | Build job execution | 408s | **389s** | **19s shorter** | | Clean consumer install, including catalog preparation | 89.72s | **69.69s** | **20.03s less (22.3%)** | The final catalog preparation took 1.87s and is included in the install figure. Both Build hosts were `m7i-flex.xlarge`. Build began 42 seconds after workflow creation instead of 53 seconds in the historical run, so faster queue/setup admission explains part of the 31-second total. Upstream code also advanced; these measurements are not a controlled causal estimate of the complete gain. The two candidate install measurements (66.29s and 69.69s) support the narrower reduction in install work. No tests or runtime probes were removed. - Conflict-resolution head `ea8287b81` [passes all 48 workflow jobs](https://github.com/paperclipai/paperclip/actions/runs/38789785045), including both required gates. Greptile re-reviewed this exact head at 5/5, with the sole earlier thread resolved. The conflict was the independent filename-policy fix; the stronger expression guard is retained. This run took **8m04s** overall, Build 423s, consumer installation 80.56s and bundled-server staging 31.17s. This confirms meaningful run variance: the earlier 7m35s is a sample, not a guaranteed new wall-clock duration. - A second independent master update touched the same credential test. Final head `1c740b73e` resolves that conflict while retaining the same stronger guard. [All 48 jobs pass](https://github.com/paperclipai/paperclip/actions/runs/38792983313); Greptile re-reviewed this exact head at 5/5. This run took **7m32s** overall, Build **385s**, consumer install **67.62s**. Auto-merge is enabled but GitHub requires code-owner approval. ## Risks The initial install now uses registry package specs instead of file specs. The parity test checks the resulting installed bytes. Full CI must still exercise real bundled provider graphs and all offline probes. The temporary consumer lock contains loopback artifact URLs; it is not exported or reused for another installation. Native release consumers retain their existing artifact install path. Queue, host and cache variance limit attribution from separate workflow runs. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, execution and GitHub tools. The serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no private ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green (48 jobs and both required gates on conflict-resolution head) - [x] Greptile is 5/5 with no open P2s, recommendations or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 14:52:15 -05:00
DottaandPaperclip	68a09dcb5f	perf(ci): shorten the test execution critical path (#16434) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors wait for the full PR workflow before they can deliver fixes. > - Server shards still execute independent files in series, and a single Runner file serializes two long real-clock waits. > - Faster work outside the critical path does not prove a faster workflow. > - Overlap independent files while preserving isolation, test cases and real production deadlines. > - Compare complete baseline and candidate workflows, separating queue delay from execution cost. ## Linked Issues or Issue Description Refs #16429. Related duration refresh: #12577. **What existing behavior does this improve?** The elapsed time from PR CI creation to the last required check. **Current behavior** General server shards use one worker. One Runner transport file takes about 333 seconds on Linux, including over 100 seconds in two real-clock regressions. These are the longest verification lanes after #16429. **Proposed behavior** Run independent general-server files in at most two isolated forks. Move the two intact Runner deadline regressions into a separate file to overlap their waits in the existing pool. Run two chat partitions per existing job, each with its own temporary home and embedded PostgreSQL. Preserve serial execution within files and every assertion and timing boundary. ## What Changed - Qualify both serial and parallel forks with real worker processes, isolated homes/configuration, timers, globals and environment. - Require both parallel workers to arrive before either exits. - Use two isolated forks for sharded general-server tests; retain one worker with an explicit shared test database. Other test groups keep their existing concurrency. - Extract the lease-expiry and Pi cold-admission regressions verbatim into a separate Runner test file; retain real waits and process/lease assertions. - Split each sharded chat job into two independent invocations; preserve serial shared-database execution. Await both children before failing the parent. Prove disjoint coverage and failure propagation. - Update the paired Linux benchmark to compare a complete general-server shard with identical case inventories. - Document concurrency and deadline-test placement. No production behavior, workflow topology or runner count changes. - Allow 15 seconds for snapshot integration cases that cold-start multiple real TypeScript workers. On the repeat, one-worker cases took 2.5–3 seconds, so queued or sequential starts could exceed the default five seconds. Preserve all cleanup, queue-release and filesystem assertions; these cases do not enforce a production latency deadline. ## Verification All 48 jobs passed on scheduling head `5d718e18` in [attempt 1](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1). The [bounded repeat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/2) failed two existing snapshot-worker tests at their default five-second timeout. Its shorter duration is **not** counted as a successful performance sample. The latest head `3c894462` adds a targeted test-budget repair and [passes all 48 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/38768433532) in **8m06s** (workflow creation to final required workflow job). This is **70 seconds / 12.6% shorter** than the 9m16s intermediate run, but the earlier 9m23s result remains part of the evidence: a stable average improvement is not established. Both branch-protection checks (`ci / verify` and `ci / e2e`) pass. The non-required Superagent Security Scan is still running; external PR scans are separate from this workflow duration. | Measurement | Before | After | What it establishes | | --- | ---: | ---: | --- | | General-server shard, paired on one Linux host | 430.94s | 292.99s | 32.0% faster; identical 1,047 case inventory | | Chat job 1 | 460s | 211s | Same full chat inventory spread across isolated partitions | | Chat job 2 | 408s | 274s | Serial cases within each child, separate PostgreSQL instances | | Chat job 3 | 408s | 216s | Total chat runner time fell from 1,276s to 701s | | Full PR workflow, first chat candidate | 9m16s | 9m23s | Not an end-to-end win in this pair: unchanged Build rose from 365s to 465s | | Full PR workflow, final repaired head | 9m16s | 8m06s | All 48 jobs pass; 70 seconds faster observed, not a controlled causal estimate | - Latest [paired benchmark](https://github.com/paperclipai/paperclip/actions/runs/38763885223) passes. Earlier paired server measurements were 33.3% and 37.4% faster. Unchanged fixture timing noise is not attributed to this PR. - All **1,337 chat case identities and outcomes match exactly** across [before-chat](https://github.com/paperclipai/paperclip/actions/runs/38760937988) and [after-chat](https://github.com/paperclipai/paperclip/actions/runs/38763885288/attempts/1), without duplicates or removals. Whole-suite records rose from 44,563 to 44,574: three new launcher checks and eight upstream warm-owner tests. Remaining name differences were generated fixture IDs/timestamps/PIDs, not removed tests. - The final run used `m7i-flex.xlarge` for Build (408s), with all chat jobs completing in 238s / 318s / 280s. It used the same actual merge base `afe5b341` as the first chat candidate. Optional profiling was disabled after the bounded measurement experiment; the full 48-job correctness workflow still ran. - The fleet used `m7i.xlarge` for the earlier Build and `m6a.xlarge` for the first chat candidate Build. Several unchanged compile/install/probe phases slowed together. This does not prove hardware alone caused the difference, and full-run attribution must account for runner/cache variance and the upstream base change. - Original full baseline: [all 48 jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38753547876/attempts/2), 12m07s, including a server job that queued over five minutes. It is not a clean basis for attributing the whole elapsed-time reduction to this PR. An interrupted baseline and two misrouted candidate attempts were excluded and preserved; routing/trust rules were not weakened. - Local: 32 sharding/launcher checks pass, including disjoint coverage, shared-database fallback and a failed child that waits for its sibling. Real paired chat partitions passed 416 cases in 217.63s on independent embedded databases. - Both extracted Runner deadline regressions pass unchanged with real waits. An initial local attempt rejected a Node executable under the controller home; the identical binary in an owned non-home path passed without weakening that guard. - Real worker-isolation proof, actionlint and diff checks pass. Latest-head Greptile is **5/5**, with no review threads or actionable findings. - The eight snapshot-worker cases pass locally with a private `PAPERCLIP_HOME`. The first direct local invocation failed to open the default-home lock database under the sandbox; no production guard was changed. ## Risks Parallel files and paired chat databases may expose shared-state assumptions or increase peak memory. Cases stay serial within each invocation, and explicit shared databases keep the serial path. Complete server coverage and the full 48-job CI workflow pass. Check external required PR scans before merge; the full-workflow measurements are observational and do not establish a steady 12.6% speedup. Queueing, cache state and runner variance must be distinguished from execution improvements. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and GitHub tools. The exact serving model suffix and context capacity are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (48 jobs; both required branch-protection checks pass; non-required external security scan pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 12:22:59 -05:00
DottaandPaperclip	7ec6fd0972	perf(ci): remove repeated test setup and fixture work (#16429) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI must check changes without making contributors wait for repeated setup. > - A complete CI profile showed 158 serialized Vitest launches and repeated dependency installs. > - A large fixture also transferred 40,000 files to prove two different size boundaries. > - This PR removes repeated controller, install, build, transfer, and wait work. > - It keeps isolated test processes and all useful regression boundaries. > - Paired command benchmarks check speed and test inventories before we claim a benefit. ## Linked Issues or Issue Description Refs #16423 for the complete CI profiling data path. No duplicate optimization PR was found. **Current behavior** Each serialized file starts a new Vitest controller. Each CI lane restores the pnpm store and installs dependencies. Large Git scanning and staging share one 40,000-file fixture. A readiness regression waits over a minute for the default database idle deadline. **Proposed behavior** Use one controller per serialized shard, with a fresh fork for each file. Restore exact installed dependencies and certified shared/SDK build outputs when inputs match. Keep the normal install and build paths on a miss. Separate the 32 MiB Git scan proof from staging above command argument limits. Cross a real database idle deadline with a one-second fixture setting. **Reason and benefit** The paired Linux CI benchmark reduced the complete 566-case serialized shard from 303.43s to 246.96s (18.6%). The Git/database fixture command fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration reduced setup from 56s to 19s (66.1%). These are measured command/setup times. They do not establish a controlled reduction in the full workflow's queue-to-completion time. **Breaking changes** None to application behavior. Tests still run in full. Test results are not cached. Cache misses preserve the existing lockfile resolution fallback. ## What Changed - Batch serialized files into one isolated, single-worker Vitest invocation per shard. Give each worker its own home, config and instance ID. - Add a real two-order isolation check for processes, environment, globals, timers and filesystem state. - Add exact dependency keys for all manifests, lockfile, workspace settings, patches, installer, runtime, platform, libc, pnpm, store and checkout path. Cache only installed package directories, not source fixtures. - Add a separate content key for the shared/SDK dependency graph. Retain the existing source/output fingerprint verifier and rebuild path. Only an exact canonical master push can populate normal caches. Stale lockfile fallback runs cannot publish entries. - Keep four real Git filename lanes above 32 MiB. Use a separate 4,000-file staging/deletion case above 2 MiB. Check every staged file, unusual names, cleanup and bounded commands. - Configure a real one-second PostgreSQL idle timeout in the readiness fixture. Assert that the backend connection changes and that another connection can write the parent row. - Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It compares the tested merge tree with its actual first parent on one runner. It also removes installed outputs and qualifies a real cache restore using temporary PR-scoped entries. ## Verification - Local Node 24.19.0: unchanged serialized shard 255.69s; candidate 231.44s; all 566 cases pass in both. - Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate scan 18.58s, staging 13.41s, database case 2.80s. All three candidate cases pass. - Negative control: moving readiness back inside the real parent-lock transaction fails the shortened regression at the parent-write assertion. The production source was restored and is absent from the diff. - Worker isolation check passes in both file orders. - 565 CI policy, partition and workflow regression checks pass with local HTTP/IPC access enabled. The first sandboxed attempt was denied listen access; it was not a code failure. - `actionlint` passes for the three changed workflows. - [All 48 main CI jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045) on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full typecheck, tests, build, browser E2E and release checks. All 158 serialized files and 3,250 cases passed through 9 controller invocations. Full typecheck/test/build were verified in CI; they were not all repeated locally. - [Paired Linux benchmark passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127) on Node 24.21.0. Its baseline is `57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is `f68f1b32966090f11647502e31852130c20237cf`. The artifact contains results, exact case inventories, setup times and raw logs. Serialized case identities match exactly. All three fixture contracts execute in both variants. | Measured work | Baseline | Candidate | Reduction | |---|---:|---:|---:| | Complete serialized shard, 566 cases | 303.43s | 246.96s | 18.6% | | Git/database fixture command | 156.54s | 33.78s | 78.4% | | Dependency/build setup, cold vs exact hit | 56s | 19s | 66.1% | - [Honeycomb profile export succeeded on its second attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2). The first attempt failed on GitHub HTTP 503 while downloading an artifact, before export. Only the export was retried; tests were not repeated. The temporary PR-scoped qualification caches were removed after validation, reclaiming 3.34 GB. Timing artifacts remain available. - Greptile is 5/5 on the current head, with no unresolved review threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the corrected reporter paths and paired comparison passed. Earlier runs were superseded while this was fixed. A copy-batching experiment did not speed up the Git fixture and was replaced by the two-boundary fixture split. ## Risks - A persistent controller retains transform state longer. The local shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must confirm all shards fit the runner budget. - Materialized caches are larger than metadata caches. The Linux benchmark measured real archive transfer and validation after removing installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes cannot publish cache entries. Exact misses run installation. Canonical cache warming requires a frozen-lockfile install; the existing stale-lockfile fallback remains uncached. - The PR uses the trusted reusable workflow from master. The paired benchmark directly exercises the candidate dependency action; normal workflow adoption still requires this PR to merge. This PR does not change the trust references. The completed main workflow took 11m02 including queue time. No controlled full-workflow latency improvement is claimed before this workflow adopts the cache change. - Only deterministic dependency/build outputs are reused. Broad dependency-based test omission and caching successful test results are deliberately excluded from this change. - The real Runner lease minimum and cold-admission waits remain. They protect protocol boundaries and were not shortened by weakening production limits. ## Model Used OpenAI GPT-6 through Codex, with reasoning, shell execution and code tools. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 09:23:47 -05:00
DottaandPaperclip	7420af26fc	fix(ci): finish large Honeycomb profile exports (#16423) ... ## Thinking Path > - Paperclip uses pull request CI to check changes before merge. > - The merged CI profiler records a selected full PR suite in Honeycomb. > - A real 48-job profile exposed an export failure after Honeycomb accepted only part of the trace. > - The same run exposed three fixture subprocesses that looked like incomplete test launches, one direct reporter without a launcher marker, and one overlong test name. > - The exporter needs to handle temporary ingest rejection without sending accepted spans again. > - This pull request fixes those cases and documents the trace display limit. > - The benefit is a complete, checkable CI waterfall with test-file timings. ## Linked Issues or Issue Description **What happened?** The first live `profile-ci` run on #16423 completed all 48 jobs, but the post-run exporter stopped when Honeycomb rejected part of a batch. The initial trace had no root span. Its coverage flag also reported four unfinished invocations and one invalid timing record. **Expected behavior** The exporter should pace large uploads, retry only temporarily rejected events, and publish the root after every child is accepted. Completed reporters should count as complete. Long test names should not erase test timing. **Steps to reproduce** 1. Add `profile-ci` to a CI CODEOWNER PR with the merged profiler. 2. Run the full PR suite, as in [run 38732093469](https://github.com/paperclipai/paperclip/actions/runs/38732093469). 3. Inspect the separate [failed exporter run](https://github.com/paperclipai/paperclip/actions/runs/38733393370) and the trace coverage fields. **Paperclip version or commit** `12612b06eb4332844ac4c8770a1fb6c6b7eccbc9` (PR run head, based on merged #16378). ## What Changed - Pace Honeycomb uploads and retry only transiently rejected events from each batch. - Preserve overlong test timings with bounded names and count complete reporter pairs without a launcher record. - Keep nested Vitest fixture runs from inheriting CI profiling and creating false coverage gaps. - Clarify ordinary CI access, large-trace display limits, and export behavior in the operator guide. ## Verification - [Full PR suite](https://github.com/paperclipai/paperclip/actions/runs/38732093469/attempts/1): 48 of 48 jobs succeeded; 10m 29s wall time. The selected jobs used the AWS public PR fleet. The protected exporter used a separate GitHub-hosted runner. - [Clean Honeycomb file waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace?trace_id=01ed0c541b0442e613a030ffb6dc8d9b&trace_start_ts=1792065955&trace_end_ts=1792066584): 48 jobs, 822 steps, 2,622 file spans, and `ci.profile_complete=true`. It was reconstructed from that run's 45 timing artifacts after identifying three nested fixture records. The full trace also recorded 44,497 tests; Honeycomb's chart omits some deep test spans because it displays at most 32,000 spans per trace. - The original exporter stopped after a partially accepted batch. A local replay of that batch succeeded. I used the authorized local test-environment key to complete the original upload and create the file-level view. No key was added to this branch or PR. - `node --test .github/scripts/tests/ci-profile.test.mjs scripts/ci-profile/reporters.test.mjs`: 18 passed. - `pnpm exec vitest run server/src/__tests__/vitest-chat-shards.test.ts`: 1 passed. - `git diff --check`: passed. - The local `pnpm -r typecheck` and profiled fixture rerun could not start: the package-manager shim could not verify the `pnpm@9.15.4` registry signature because its fetches failed. `pnpm test:run` and `pnpm build` were not run locally; hosted CI is pending. - The updated exporter has not yet run from protected `master`; this PR remains draft until CI and review finish. ## Risks - A 48,000-span export now takes about three extra minutes due to pacing. Retry limits still fail a persistently rejected batch and keep the root unpublished. - Honeycomb may retain duplicates from an already partial export. The clean file-level evidence trace uses a separate trace ID. - The API key stays only in the protected `ci-profiling` environment. PR jobs write timing artifacts but never receive it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — see `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, GitHub API, and browser inspection. The runtime does not expose the exact serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 07:51:02 -05:00
DottaandPaperclip	6acec4266e	feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 06:59:50 -05:00