4 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	1c9ca5829f	perf(ci): overlap Runner workloads and cache debug workspace builds (#18794) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect its server and Runner contracts. > - Recent full runs waited on general server files and the final Runner lane. > - That Runner lane ran package tests before native server tests. > - PR jobs also rebuilt unchanged debug workspace binaries. > - This pull request overlaps those workloads and reuses exact trusted build inputs. > - The benefit is a shorter critical path with the same test coverage and AWS authorization. ## Linked Issues or Issue Description Refs #18544, #18773, #18774, #18798, #18799. **What existing behavior does this improve?** The wall time of required PR tests and the precision of affected-test selection. **Current behavior** General server and native files use two workers. The second Runner Vitest lane runs its package shard and then the native server group. Each Runner job compiles debug workspace units again. A Vite raw import also retains its consumer for unrelated source edits. **Proposed behavior** Use three isolated workers for general server files and four for native files. Reuse the two Runner slots for a complete package suite and a complete native server suite. Restore all debug binaries only on an exact input key. Record raw imports as explicit file inputs. ## What Changed - Separate the Runner package and native server workloads without adding CI jobs. Preserve the old command arguments for master-pinned workflows, local commands, and other shard counts. - Increase file workers while retaining fork isolation, serial cases, and the shared-database guard. - Add an exact debug workspace cache. Derive binary targets from Cargo. Include source, fixtures, generated inputs, manifests, compiler identity, flags, Cargo configuration, and checkout path in its key. Discard incomplete restores. Only verified master pushes can save it. - Keep older PR checkouts working when they lack the new cache helper. - Track Vite `?raw` imports as repository file inputs. Keep unknown loaders, untracked files, and opaque effects conservative. - Add cache and coverage guards and document the behavior. AWS runner authorization is unchanged. - Keep paired test benchmarking available when install hooks correctly disable materialized dependency reuse. The old benchmark asserted a cache hit in that safe fallback. - Wait for active dispatch work to finish before asserting idle reads. Analyze the two 16,400-row database fixtures before their queries so they have current planner statistics. Suppress only PostgreSQL notices inside the cleanup transaction to avoid logging every cascaded table; warnings/errors remain visible. Row counts, real queries, assertions, database reset, and timeouts remain unchanged. ## Verification - 196 affected CI tests passed, including routing, coverage, selection failure fallbacks, shared-database serialization, and existing/new cache guards. Final worker/cache adjustments also passed their focused checks. - `pnpm build` and `pnpm -r typecheck` passed locally. Full test coverage is being verified by the Linux CI profile; `pnpm test:run` was not repeated as a full serial local run. The paired CI benchmark passed. Both paired benchmarks passed, and full CI passed again after integrating newer master commits. The external Paperclip Review app is the only pending status. - Actual debug artifacts remained fresh after simulating checkout timestamps: Cargo completed in 0.82 seconds; the content key was unchanged. This is local reuse evidence, not a measured GitHub cache transfer. - Same real UI inventory for a retention-test-only change: 861 files; retained selection fell from 413 to 401. All 12 omitted files are raw-input consumers; edits to their inputs remain covered by regression tests. - The first local worker comparison is invalid. Development dependency links disappeared during its second run. The two-worker baseline also had two macOS path assertions in company-skills. Neither result is counted as a successful speedup. Dependencies were restored. A local repeat was interrupted before producing results. The completed Linux comparison below supplies the valid paired evidence. - [Paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41671244016): baseline and candidate ran the same 113 general-server files and 1,777 cases on one host. Wall time fell from 381.29s to 346.89s (9.0%). Both exited successfully and file/case inventories matched exactly. The fixture command took 31.77s versus 28.65s. This is one sequential pair, excluding setup, queueing and the full CI critical path. - [Repeat paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41675496862): the same 113 files and 1,974 cases took 485.06s in the baseline versus 368.54s in the candidate (24.0%). File and case inventories again matched exactly. The baseline includes newer master changes than the first pair, so these are two independent comparisons, not repeated timings of an identical tree. - Initial full AWS profile: package suite passed 4,891 tests (289 files, one skipped); native suite passed 172 tests (9 files, 12 pre-existing platform/feature skips). Native execution took 239.06 seconds; the older sample ran 165 native tests in 391.29 seconds. The package lane now takes 11m01 including setup and debug compilation. This is an indicative cross-run comparison, not a paired full-CI speedup claim. - The initial run failed two UI fixtures, one serialized inbox settlement assertion, and one dispatch idle assertion. The two UI failures reproduced locally. The fixes preserve authorization, admission, and zero-idle-read assertions. All four affected files passed locally: 48/48 tests. Token gates passed. All four also passed in the next AWS profile. That profile passed every other workload but timed out the 16,400-row keyed attention fixture at its existing 180s limit; this is accounted for by the explicit fixture-statistics change. Local focused comparisons pass both large cases before and after; their roughly 33s test execution is unchanged locally, so this is not claimed as an additional speed gain. A full local file run passed both large cases (7.57s / 18.05s) but hit the unchanged 30s teardown deadline on its first test while emitting thousands of cascade notices. Cleanup now suppresses those notices within its transaction; the complete file then passed 34/34 in 107.72s. The latest-head AWS run passed the previously failing shard: the keyed bulk case took 40.25s (previously exceeded 180s), the keyless case took 8.02s, and the complete attention file took 89.61s. The 16,400-row fixtures and original timeouts remain intact. - [Full CI profile before master integration](https://github.com/paperclipai/paperclip/actions/runs/41675497699) passed on commit `5858956d4ac2c99c911ad2447b63e1f8fc84e31e`, including all tests, build, typecheck and final verification. Greptile reviewed that exact commit at 5/5; the only actionable review thread is resolved. The separate Paperclip Review app is still queued. - [Final merged-head CI](https://github.com/paperclipai/paperclip/actions/runs/41678956312) passed on `499aa206a6e03bf8f3249692ea731367a000591d`: 48 successful jobs and one intentionally skipped file-selection job. Backup checkpoint qualification and legacy security status passed. Greptile reviewed this exact commit at 5/5. The PR is conflict-free; only the separate Paperclip Review app remains queued. - Integrated current master after #18798 and #18799 merged the inbox and UI fixture fixes separately. Preserved master's inbox settlement fix and verified its entire file (6/6 tests). Those three files are no longer part of this PR's diff. Removed the temporary `profile-ci` label after both successful benchmark pairs; the normal planner still schedules every job because the CI configuration changes are global inputs. - Second AWS profile: native tests took 229.97s with the same 172 passes / 12 skips; package tests took 427.44s with 4,891 passes / 10 skips. The complete package job took 10m44, including setup and 79s debug compilation. - The PR uses the trusted workflow from master. Candidate worker/layout code runs now; workflow-level debug restores begin after merge and the first successful master cache writer. ## Risks - Extra workers can expose load-sensitive tests. The final full CI profile passed, but one run is not a long-term flake-rate measurement. - The cache is an optimization only. Misses rebuild, and test commands still execute. A new Cargo workspace member requires a cache-contract review. - Compiler alternatives remain experiments. Required TypeScript checks are unchanged. Typechecker experiment: three sequential checks per tool on identical built projects, with TypeScript 7.0.2, Bun 1.4.4, and tsrs 0.20.1-ts7.1.0-beta. Median seconds (TypeScript / Bun / tsrs): shared 0.429 / 0.083 / 0.157; UI 2.554 / 0.390 / 0.740; server 9.949 / 1.194 / 1.782. TypeScript and Bun passed all three projects. All tools caught the same four deliberate assignment, property, nullability, and generic-constraint errors. tsrs reported one additional server TS2345 diagnostic in the run-dispatch adapter, so it did not pass the project comparison. These local compiler times exclude generation and build steps; they are not full CI savings. ## Model Used OpenAI GPT-6 Codex. Tool use and code execution assisted this change. The exact deployment identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:17:53 -06:00
DottaandPaperclip	54cc522e86	feat(runner): add gated native Antigravity ACP execution (#18188) ... ## Thinking Path > - Paperclip manages agents and their work. > - Native Runner must bind each provider turn to the selected task, account, model, and instructions. > - Antigravity needs native ACP admission and physical process ownership across continuation. > - A saved conversation must not reuse an old process owner or silently switch its effective model. > - This pull request adds the native candidate behind exact operator admission. > - The default release gate remains closed while qualification is pending. ## Linked Issues or Issue Description Builds on merged #18187. Product account setup and the candidate producer are in dependent #18189. The repeatable browser journey is in #18190. Refs: #12503, #14938; those proposals use legacy CLI adapters. **Agent or provider** Google Antigravity ACP 1.4.3. **Why this adapter is useful** Users need to set up an Antigravity agent and run real work through their managed Google subscription. **How the agent is invoked** Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission is required while qualification remains pending. ## What Changed - Add native Antigravity ACP model and instruction admission, selected-account homes, and fenced refresh ownership. - Carry exact candidate authority from the control plane through Runner, Rust, the sidecar, and each current run. Clamp daemon lifetime to the absolute permission expiry, including replacement owners. - Preserve a rotated OAuth refresh token before later identity verification can fail, including failed or cancelled startup admission. Make private lease-close retries safe. - Bound native probe retirement and retry the exact original owner; do not delete credential homes before physical proof. - Preserve Gemini API-key recovery when shared Google subscription capability is added. - Resume the same tagged native conversation with fresh per-turn process ownership and physical native retirement. - Include native manifests in isolated Rust image stages, release cache keys, and dev rebuild detection. The workflow changes only extend existing cache input lists. - Keep current master provider behavior and append Grok 13, Pi 35, and Copilot 84 input identities. Preserve prior identities, including saved Copilot 82 and 83 runs. ## Verification - Runner TypeScript main checks, profile generators, Rust formatting, and offline workspace checks pass. The native backend integration target passes 28 tests. - The complete transport and cleanup group passes 387 tests without unhandled rejections. These checks use local scripted children and actual process and port ownership. - Saved-profile decoding, native-execution parsing, installation, and selected-credential controls pass 208 focused tests. All 109 pre-existing profile identity files remain byte-for-byte unchanged. Copilot, Grok, and shared TypeScript/Rust generation checks pass. - The final recovery fixture passes all seven cases. After the master rebase, the combined recovery and owned-Node-runtime group passes all 10 cases. Each expiry case sets one 45-second absolute grant before startup. It includes the deliberate six-second cold delay, four existing eight-second command budgets, and active-turn headroom. The test proves an active turn before expiry and then checks the daemon, owned process group, child processes, and all three native port fences. The 60-second test deadline does not renew provider authority. - The consolidated model-discovery and pending-release group passes 22 tests. Google credential compatibility and auth recovery pass 25 tests, including existing Gemini API-key recovery. The full product settings test group remains intact in #18189. - Earlier failures remain in their original CI runs and local records. They exposed the Copilot 82 decoder gap, an expired five-second fixture bootstrap ticket, and a fixture grant consumed before activation on loaded CI. Each repair has a separate result. Production ticket limits and admission behavior are unchanged by the final test commits. - An unrelated decision-expiry check failed in [the earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/40941986698/job/123688668997). Its owning code is unchanged from the base. The unchanged suite passes 20 tests on isolated PostgreSQL. A controlled database rollback reproduces the earlier wake: clean startup schedules the 120-second decision expiry, while rollback reconciliation leaves a 30-second continuation-coalescing wake. The decision stays open, with no wake delivery or error. The fresh final-head server shard passes; the failed result is preserved. - Final native commit `ec6bc7e4d77f5ae0cc4946dbba6c662e95a87f5b` passed `ci / verify` and `ci / e2e`. Fresh Greptile and Paperclip reviews both score 5/5, with no open findings. Antigravity retains the recorded native `.5` closure and exact low-model profile. This PR does not claim global or cross-platform qualification. ## Risks - The new transport carries private managed credentials. Exact ownership, account generation, model, instructions, and native retirement are fail-closed. - Only local macOS ARM64 has a verified complete Antigravity runtime. Default Antigravity task execution remains disabled without current trusted operator admission. The Google server sign-in lane and production candidate producer ship in #18189; this native slice does not expose the subscription catalog choice. - Shared launch inputs change provider identities. New identities are append-only; existing recorded identities remain available. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:36:12 -05:00
DottaandPaperclip	d855e550ad	Finish Copilot native runtime on current master (#15701) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner controls provider processes, permissions and task completion. > - Copilot needs its pinned distribution, native message identities and accepted semantic completion receipts. > - The release must preserve saved-token access and task lifecycle guarantees. > - This pull request will keep the current completion and recovery fixes while adding the Copilot runtime. > - The benefit is a reviewable part of the Copilot production release. ## Linked Issues or Issue Description Refs #14434. Supersedes the runtime work in #15560. This is the first PR in the stack; #15703, #15705 and #15706 follow. Related alternative adapter work: #15881. This change uses the shared Runner path. ## What Changed - Pin Copilot CLI 1.0.88 and bind the integrated implementation as profile45. - Keep native instruction, permission, receipt, process ownership and cancellation boundaries. - Preserve current master completion continuation and remote verification behavior. Other provider release identities stay unchanged. ## Verification Final reviewed source [`4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5`](https://github.com/paperclipai/paperclip/commit/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5) includes master `cde944ed75648adfb569e86133adc8b118b55dd6`. The combined four-part PR tree matches it except for the CI-owned lockfile. Every prior public PR head remains an ancestor. Released master migrations remain unchanged; Copilot adds migration361. No unrelated provider is enabled. Copilot CLI1.0.88 and profile47 retain digest `sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`. The pending qualification marker is removed. The [qualification record](https://github.com/paperclipai/paperclip/blob/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5/doc/plans/2026-10-20-copilot-qualification-v47.json) reports **20/20 Product cells,154 assertions;7/7 protocol cases,46 assertions** against frozen installed application `c2616370800f5f6df45246b0890ba35a7cd2faab`. Both environments use the exact authenticated model `gpt-5.6-luna`. Every passing Product case confirms cleanup. Automatic behavior retries are disabled. Original failed attempts remain recorded. The ordinary installed remote journey passed saved-token selection, authenticated model discovery, normal UI save/reopen, metadata-only Test, task completion, accepted receipt and an exact19-byte browser download. An open model catalog caused no new discovery requests during117seconds. Actual installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token browser entry was not exercised; the encrypted connection was created through the normal API. The original Daytona file attempt timed out during delayed semantic dispatch. Its precise stalled layer remains unresolved. Independent ordinary delivery and the other19 cells passed before one instrumented recheck passed8/8 with unchanged source,prompt,assertions and deadlines. No causal latency repair is claimed. After current-master reconciliation,129 connection tests,59 runner tests,72 shared eval tests and88 recovery/workspace tests pass. Migration generation/safety, unchanged profile identity and UI token gates pass. Prior required CI was green. Fresh exact-head required CI and E2E are green for all four updated heads. Each has a fresh Greptile5/5 review and zero unresolved review threads. The [final normal-package cloud build](https://github.com/paperclipai/paperclip/actions/runs/39414239704) is running. Recursive typecheck,full CI test shards,build and all8 browser shards pass. These checks cover the final source; no additional local full-suite pass is claimed. The [frozen image build](https://github.com/paperclipai/paperclip/actions/runs/39334401638) and [original normal package build](https://github.com/paperclipai/paperclip/actions/runs/39334407471) passed. The exact image is published and independently retrieved anonymously. No separate full-image scan is claimed. All18 original normal archives were scanned with zero selected credentials. Docker builds run in the cloud. Provider USD remains unknown and GitHub-attributed. **Qualification is complete on the recorded frozen build. Normal protected merge, automatic master canary publication and final shipped-package local/Daytona canaries remain. No production deployment is claimed.** ## Risks Shared Runner behavior can affect active tasks. Exact profile admission rejects stale runtime identities. The final campaign and installed canaries remain release gates. ## Model Used OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this task. Used for code changes, review and tool-assisted verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 06:31:47 -05:00
3 people	8227485372	perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its trusted PR workflow builds the Runner's Rust binary in five jobs: Typecheck + Release Registry, Build, Canary Dry Run, and the `rust` and `vitest 2/2` lanes of Verify Paperclip Runner. The `Build` job is the slowest check in almost every green run (461s, 489s and 515s in runs 38180574768, 38182717052 and 38183149920 on 2026-10-11). > - Each of those jobs restores master's `release-runner-v2` dependency cache. That cache holds only third-party crates. `rust-cache` removes workspace crates when it saves, and a fresh checkout invalidates cargo's mtime fingerprints for workspace crates in any case. So cargo compiles `paperclip-runner-core` in release mode from scratch on every run. The job logs show one `Compiling` line and 28s to 49s for it. > - 18 of the 239 commits on `master` in the seven days before this change touched the crate's inputs. The other 221 compiled the same sources to the same binary again. > - This pull request adds a second cache that holds only that crate's release artifacts (about 70 MB). The key is made from the compiler, the checkout root and a hash of every input the build reads. The master Rust lane writes it. A pull request whose merge tree has byte-identical inputs restores it and marks the artifacts newer than the checkout, so cargo keeps them. Every other pull request misses and compiles as before. > - The benefit is about 28s to 49s less on `Build`, which is the run's critical path, and the same on Typecheck, Canary Dry Run and two Runner lanes, for about nine of ten pull requests. ## Linked Issues or Issue Description No GitHub issue exists for this. The description below follows the enhancement template. **What existing behavior does this improve?** The time the trusted PR workflow spends to compile the Runner's own crate on every run. **Subsystem affected** CI / build **Current behavior** Every PR job that builds the Runner binary compiles `paperclip-runner-core` in release mode from scratch. The compile takes 28s to 49s on the `runs-on` fleet (one `Compiling paperclip-runner-core` line, then `Finished release profile` in runs 38169436532 to 38183149920 on 2026-10-11). The shared Rust cache cannot help: it holds only third-party crates, and a fresh checkout gives every source a new mtime, which makes cargo discard the crate's fingerprints. **Proposed behavior** The master Rust lane in `release-verify.yml` saves the crate's release artifacts under a key made from the compiler, the checkout root and a hash of every input the release build reads. The PR jobs that build the binary restore that entry read-only. On a hit, a script sets the mtime of every restored artifact to now, so cargo sees every source as older than the build and keeps the artifacts. `pnpm build` then prints `Finished release profile` in under a second for that crate. On a miss, cargo compiles the crate exactly as before. **Reason and benefit** About 28s to 49s off `Build`, the slowest check in almost every green run, and the same off the Typecheck, Canary Dry Run and two Verify Paperclip Runner lanes. The saving applies to every pull request whose merge tree does not change the crate's inputs, which was 221 of 239 `master` commits in the week before this change. **Breaking changes** None. The checks are unchanged. A miss or an eviction costs the compile time it costs today and nothing else. **Additional context** This is a workflow change. `pr.yml` calls `pr-trusted.yml@master`, so this pull request's own run uses the current reader steps, and the writer runs only on a master push. The saving becomes visible on the first PR runs after the merge and after the next master push has written the entry. ## What Changed - `scripts/runner-release-crate-cache.mjs` (new): the one place that defines the cached path list (`CACHED_PATHS`), the hashed inputs (`HASHED_INPUTS`) and the files a restore must contain (`REQUIRED`). The `scope` subcommand prints the platform, the `rustc --version --verbose` release, commit and host, and a hash of the checkout root. The `mark-current` subcommand sets the mtime of every restored artifact to now and prints a `paperclip.runner-release-crate.v1` JSON line with the file count and bytes. If the restored set lacks any of the seven files cargo's freshness check reads or checks for (the fingerprint hash and dep-info of each unit, the lib's rlib and rmeta, the binary under `deps`), it removes the restored files, prints `action: discarded` naming the missing file, and exits 0, so cargo compiles as on a miss. - `.github/workflows/pr-trusted.yml`: after the `release-runner-v2` restore in `typecheck_release_registry`, `build` and `canary_dry_run`, and in the `rust` and `vitest 2/2` lanes of `verify_paperclip_runner` (new matrix key `release_crate: true`), three steps: compute the key scope, `actions/cache/restore` the crate under `runner-release-crate-v1-<scope>-<hashFiles(...)>` with no `restore-keys`, and run `mark-current` when `cache-hit == 'true'`. The `hashFiles` list covers the toolchain pin, the manifests and lockfile, the crate sources, the protocol tree and the three fixture trees the test modules include (two under the Runner package and `tests/runner-e2e/fixtures` at the repository root). - `.github/workflows/release-verify.yml`: in the Rust lane of an own-repository master push, after the debug warm-up: compute the key scope, look the key up (`lookup-only: true`), and on a miss run `cargo clean --release -p paperclip-runner-core` and `pnpm build:binary` (the exact command the readers run, so the entry carries no test-profile variants), then `actions/cache/save` the same path list under the same key. - `scripts/__tests__/runner-release-crate-cache.test.mjs` (new, 10 tests, in `test:release-registry`): the path list and hashed inputs are the expected ones and exist; every `include_str!` and `include_bytes!` in the crate sources resolves under a hashed input, whether its argument is a literal or a `concat!` of literals, `env!("CARGO_MANIFEST_DIR")` and macro metavariables, and any other form fails the test by name (this guard found the `test/fixtures` tree during development and the `tests/runner-e2e/fixtures` tree in review); the parser itself is tested on a synthetic crate for each supported form and six rejected ones; every PR reader restores after the dependency cache and marks before its build command with the script's exact path list and key and no `restore-keys`; the Verify lanes are gated by `release_crate`; the writer runs after the checks with the lookup, the clean and build, and the save under the readers' exact path list and key; the scope changes with the compiler and the checkout root and survives a missing `rustc`; `mark-current` makes every artifact newer than the checkout, leaves a dependency's artifacts alone, discards a restore missing any one of the seven required files (one case each, checking the named reason), and still marks a restore missing any one of the seven files cargo does not need. ## Review fixes Greptile's two findings on the first commit were both valid and are fixed in the second commit: - **Incomplete caches looked usable.** `REQUIRED` checked four files, but cargo's freshness check needs seven. Measured with cargo 1.97.1 on the real crate (restore, touch the artifacts, delete one file, build): without the lib's fingerprint hash, dep-info, rlib or rmeta it recompiles the crate (24s to 27s); without the bin's fingerprint hash, dep-info or the binary under `deps` it relinks (1.5s); without either `.json` fingerprint record, either rustc `.d` file, the uplifted binary or its `.d` it reports the crate fresh in 0.15s and recreates the uplifted binary from `deps`, and the staging step then copies it. `REQUIRED` now lists exactly the seven files, the glob for the binary under `deps` excludes the `.d` beside it, and the test removes each of the seven in turn (discarded, naming it) and each of the other seven in turn (still marked). - **Composed include paths escaped the guard.** The regex matched only `include_str!("literal")` and skipped `include_str!(concat!(...))`, which the crate uses four times (30 of 34 includes were checked). The guard now parses every include argument and resolves the `concat!` forms; one of them, in `provider_events.rs`, reads `tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json` at the repository root, outside every hashed input. It is test-only, so today's release binary does not read it, but the key is meant to cover every include whatever its build, so `tests/runner-e2e/fixtures/**` (9 files, 80 KB) joins `HASHED_INPUTS` and the six key embeddings in the two workflows. With the first commit's input list, the new guard fails on exactly that include. - `doc/RELEASE-AUTOMATION-SETUP.md`: a "Runner release crate cache" section under the Runner verification dependency cache, with the key, the writer, the readers and the trust boundary. - `package.json`: the new test file in `test:release-registry`. ## Verification - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: 10 tests pass, three consecutive runs. - The second commit, with the real toolchain on the real crate (`rustc 1.97.1`, cold build 2m20s, the crate's artifacts 74 MB): every source touched to a fresh-checkout mtime and the archived artifacts restored, cargo compiles (`Compiling paperclip-runner-core`, 24.8s and 25.1s, the two controls); restored and marked by `mark-current`, `Finished release profile in 0.16s` with no `Compiling`, and the same on a second build. Then, for each of the fourteen files cargo wrote for the two units, one at a time: restored, marked and that file deleted, cargo recompiles without any of the seven `REQUIRED` files and reports fresh without any of the other seven; restored with that file deleted and run through `mark-current`, the seven required ones print `action: discarded` naming the file and the others print `marked`. A restore holding only the seven required files builds fresh in 0.15s, cargo recreates the uplifted binary, and `stage-runner-binary.mjs` copies it into `dist/bin`. - `node --test '.github/scripts/tests/*.test.mjs'`: 378 of 381 pass. The 3 failures are in `docker-canary-promotion.test.mjs` and fail the same way on an unmodified `master` checkout in this sandbox (the stand-in docker writes no `calls.jsonl`). - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs`: 99 of 101 pass; the 2 failures spawn `pnpm`, which is not on this sandbox's PATH, and fail the same way on `master`. - Both workflows parse as YAML. No job has a duplicate step name or step id. - The mechanism, with the real toolchain (`rustc 1.97.1`, the package's pin) on a copy of the crate: a cold `cargo build --release -p paperclip-runner-core --bin paperclip-runnerd` (2m23s here). Then the artifact set was archived with `tar`, removed, every source re-copied so each had a new mtime, and the set restored: - restored, not marked: `Compiling paperclip-runner-core`, 24.6s (the control; this is what CI does today); - restored and marked by `mark-current`: `Finished release profile in 0.15s`, no `Compiling`; a second build is the same; - one Rust source edited after a marked restore: `Compiling`, 24.0s; - one protocol schema edited after a marked restore: `Compiling`, 23.9s (the protocol files enter cargo's dep-info as absolute paths and are hashed into the key); - the rlib removed from a restored set: `mark-current` prints `action: discarded, reason: incomplete: the lib rlib is missing` and removes the rest, and cargo compiles. - Not run here: the workflow steps themselves. `pr.yml` pins `pr-trusted.yml@master`, so this pull request's own `Build` job runs the current reader steps, and the writer runs only on a master push. The first PR runs after the merge and after the next master push show the `Restore the Runner release crate` step's result and the `paperclip.runner-release-crate.v1` line in their `Build` logs. ## Risks - Correctness: a reader can only restore artifacts that master built from inputs byte-identical to its own merge tree, because every input the build reads is in the key, and the key has no prefix match. A pull request cannot write. The trust boundary is unchanged: master writes, a pull request restores master's entry. - An input the build reads that is outside the hashed patterns would let a stale artifact pass. The release dep-info lists the crate sources and the protocol schemas and fixtures; all are hashed. The guard test resolves every `include_str!` and `include_bytes!` in the crate, literal or `concat!`, and fails if one lands outside the hashed inputs or uses a form it cannot resolve, so a new include form cannot slip past it by not matching a regex. - GitHub-hosted runners check out under `/home/runner/work`, the `runs-on` fleets under `/home/runner/_work`. Cargo records the included protocol files as absolute paths, so an entry from one root is useless under the other. The checkout root is in the key, so the other root misses instead of restoring something cargo then discards. - A cache that does not keep mtimes on restore changes nothing: the artifacts are then already newer than the checkout, and `mark-current` sets them to now in any case. - Cost: one new entry of about 70 MB per distinct input set, written only by the master Rust lane on a miss, plus one `cargo clean` and `build:binary` (about 40s to 50s) on that lane when the inputs changed. The readers pay one restore of about 70 MB (two to three seconds at the fleet's observed restore speed) and a sub-second `mark-current`. - An incomplete entry (a writer that saved a partial set) is discarded by `mark-current` and costs one compile, the same as a miss. ## Model Used - Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1), extended thinking, with tool use (shell, file editing, GitHub CLI, a local Rust toolchain for the experiment) in a Paperclip agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-12 11:25:31 -07:00