2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	d855e550ad	Finish Copilot native runtime on current master (#15701) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner controls provider processes, permissions and task completion. > - Copilot needs its pinned distribution, native message identities and accepted semantic completion receipts. > - The release must preserve saved-token access and task lifecycle guarantees. > - This pull request will keep the current completion and recovery fixes while adding the Copilot runtime. > - The benefit is a reviewable part of the Copilot production release. ## Linked Issues or Issue Description Refs #14434. Supersedes the runtime work in #15560. This is the first PR in the stack; #15703, #15705 and #15706 follow. Related alternative adapter work: #15881. This change uses the shared Runner path. ## What Changed - Pin Copilot CLI 1.0.88 and bind the integrated implementation as profile45. - Keep native instruction, permission, receipt, process ownership and cancellation boundaries. - Preserve current master completion continuation and remote verification behavior. Other provider release identities stay unchanged. ## Verification Final reviewed source [`4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5`](https://github.com/paperclipai/paperclip/commit/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5) includes master `cde944ed75648adfb569e86133adc8b118b55dd6`. The combined four-part PR tree matches it except for the CI-owned lockfile. Every prior public PR head remains an ancestor. Released master migrations remain unchanged; Copilot adds migration361. No unrelated provider is enabled. Copilot CLI1.0.88 and profile47 retain digest `sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`. The pending qualification marker is removed. The [qualification record](https://github.com/paperclipai/paperclip/blob/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5/doc/plans/2026-10-20-copilot-qualification-v47.json) reports **20/20 Product cells,154 assertions;7/7 protocol cases,46 assertions** against frozen installed application `c2616370800f5f6df45246b0890ba35a7cd2faab`. Both environments use the exact authenticated model `gpt-5.6-luna`. Every passing Product case confirms cleanup. Automatic behavior retries are disabled. Original failed attempts remain recorded. The ordinary installed remote journey passed saved-token selection, authenticated model discovery, normal UI save/reopen, metadata-only Test, task completion, accepted receipt and an exact19-byte browser download. An open model catalog caused no new discovery requests during117seconds. Actual installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token browser entry was not exercised; the encrypted connection was created through the normal API. The original Daytona file attempt timed out during delayed semantic dispatch. Its precise stalled layer remains unresolved. Independent ordinary delivery and the other19 cells passed before one instrumented recheck passed8/8 with unchanged source,prompt,assertions and deadlines. No causal latency repair is claimed. After current-master reconciliation,129 connection tests,59 runner tests,72 shared eval tests and88 recovery/workspace tests pass. Migration generation/safety, unchanged profile identity and UI token gates pass. Prior required CI was green. Fresh exact-head required CI and E2E are green for all four updated heads. Each has a fresh Greptile5/5 review and zero unresolved review threads. The [final normal-package cloud build](https://github.com/paperclipai/paperclip/actions/runs/39414239704) is running. Recursive typecheck,full CI test shards,build and all8 browser shards pass. These checks cover the final source; no additional local full-suite pass is claimed. The [frozen image build](https://github.com/paperclipai/paperclip/actions/runs/39334401638) and [original normal package build](https://github.com/paperclipai/paperclip/actions/runs/39334407471) passed. The exact image is published and independently retrieved anonymously. No separate full-image scan is claimed. All18 original normal archives were scanned with zero selected credentials. Docker builds run in the cloud. Provider USD remains unknown and GitHub-attributed. **Qualification is complete on the recorded frozen build. Normal protected merge, automatic master canary publication and final shipped-package local/Daytona canaries remain. No production deployment is claimed.** ## Risks Shared Runner behavior can affect active tasks. Exact profile admission rejects stale runtime identities. The final campaign and installed canaries remain release gates. ## Model Used OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this task. Used for code changes, review and tool-assisted verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 06:31:47 -05:00
3 people	8227485372	perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its trusted PR workflow builds the Runner's Rust binary in five jobs: Typecheck + Release Registry, Build, Canary Dry Run, and the `rust` and `vitest 2/2` lanes of Verify Paperclip Runner. The `Build` job is the slowest check in almost every green run (461s, 489s and 515s in runs 38180574768, 38182717052 and 38183149920 on 2026-10-11). > - Each of those jobs restores master's `release-runner-v2` dependency cache. That cache holds only third-party crates. `rust-cache` removes workspace crates when it saves, and a fresh checkout invalidates cargo's mtime fingerprints for workspace crates in any case. So cargo compiles `paperclip-runner-core` in release mode from scratch on every run. The job logs show one `Compiling` line and 28s to 49s for it. > - 18 of the 239 commits on `master` in the seven days before this change touched the crate's inputs. The other 221 compiled the same sources to the same binary again. > - This pull request adds a second cache that holds only that crate's release artifacts (about 70 MB). The key is made from the compiler, the checkout root and a hash of every input the build reads. The master Rust lane writes it. A pull request whose merge tree has byte-identical inputs restores it and marks the artifacts newer than the checkout, so cargo keeps them. Every other pull request misses and compiles as before. > - The benefit is about 28s to 49s less on `Build`, which is the run's critical path, and the same on Typecheck, Canary Dry Run and two Runner lanes, for about nine of ten pull requests. ## Linked Issues or Issue Description No GitHub issue exists for this. The description below follows the enhancement template. **What existing behavior does this improve?** The time the trusted PR workflow spends to compile the Runner's own crate on every run. **Subsystem affected** CI / build **Current behavior** Every PR job that builds the Runner binary compiles `paperclip-runner-core` in release mode from scratch. The compile takes 28s to 49s on the `runs-on` fleet (one `Compiling paperclip-runner-core` line, then `Finished release profile` in runs 38169436532 to 38183149920 on 2026-10-11). The shared Rust cache cannot help: it holds only third-party crates, and a fresh checkout gives every source a new mtime, which makes cargo discard the crate's fingerprints. **Proposed behavior** The master Rust lane in `release-verify.yml` saves the crate's release artifacts under a key made from the compiler, the checkout root and a hash of every input the release build reads. The PR jobs that build the binary restore that entry read-only. On a hit, a script sets the mtime of every restored artifact to now, so cargo sees every source as older than the build and keeps the artifacts. `pnpm build` then prints `Finished release profile` in under a second for that crate. On a miss, cargo compiles the crate exactly as before. **Reason and benefit** About 28s to 49s off `Build`, the slowest check in almost every green run, and the same off the Typecheck, Canary Dry Run and two Verify Paperclip Runner lanes. The saving applies to every pull request whose merge tree does not change the crate's inputs, which was 221 of 239 `master` commits in the week before this change. **Breaking changes** None. The checks are unchanged. A miss or an eviction costs the compile time it costs today and nothing else. **Additional context** This is a workflow change. `pr.yml` calls `pr-trusted.yml@master`, so this pull request's own run uses the current reader steps, and the writer runs only on a master push. The saving becomes visible on the first PR runs after the merge and after the next master push has written the entry. ## What Changed - `scripts/runner-release-crate-cache.mjs` (new): the one place that defines the cached path list (`CACHED_PATHS`), the hashed inputs (`HASHED_INPUTS`) and the files a restore must contain (`REQUIRED`). The `scope` subcommand prints the platform, the `rustc --version --verbose` release, commit and host, and a hash of the checkout root. The `mark-current` subcommand sets the mtime of every restored artifact to now and prints a `paperclip.runner-release-crate.v1` JSON line with the file count and bytes. If the restored set lacks any of the seven files cargo's freshness check reads or checks for (the fingerprint hash and dep-info of each unit, the lib's rlib and rmeta, the binary under `deps`), it removes the restored files, prints `action: discarded` naming the missing file, and exits 0, so cargo compiles as on a miss. - `.github/workflows/pr-trusted.yml`: after the `release-runner-v2` restore in `typecheck_release_registry`, `build` and `canary_dry_run`, and in the `rust` and `vitest 2/2` lanes of `verify_paperclip_runner` (new matrix key `release_crate: true`), three steps: compute the key scope, `actions/cache/restore` the crate under `runner-release-crate-v1-<scope>-<hashFiles(...)>` with no `restore-keys`, and run `mark-current` when `cache-hit == 'true'`. The `hashFiles` list covers the toolchain pin, the manifests and lockfile, the crate sources, the protocol tree and the three fixture trees the test modules include (two under the Runner package and `tests/runner-e2e/fixtures` at the repository root). - `.github/workflows/release-verify.yml`: in the Rust lane of an own-repository master push, after the debug warm-up: compute the key scope, look the key up (`lookup-only: true`), and on a miss run `cargo clean --release -p paperclip-runner-core` and `pnpm build:binary` (the exact command the readers run, so the entry carries no test-profile variants), then `actions/cache/save` the same path list under the same key. - `scripts/__tests__/runner-release-crate-cache.test.mjs` (new, 10 tests, in `test:release-registry`): the path list and hashed inputs are the expected ones and exist; every `include_str!` and `include_bytes!` in the crate sources resolves under a hashed input, whether its argument is a literal or a `concat!` of literals, `env!("CARGO_MANIFEST_DIR")` and macro metavariables, and any other form fails the test by name (this guard found the `test/fixtures` tree during development and the `tests/runner-e2e/fixtures` tree in review); the parser itself is tested on a synthetic crate for each supported form and six rejected ones; every PR reader restores after the dependency cache and marks before its build command with the script's exact path list and key and no `restore-keys`; the Verify lanes are gated by `release_crate`; the writer runs after the checks with the lookup, the clean and build, and the save under the readers' exact path list and key; the scope changes with the compiler and the checkout root and survives a missing `rustc`; `mark-current` makes every artifact newer than the checkout, leaves a dependency's artifacts alone, discards a restore missing any one of the seven required files (one case each, checking the named reason), and still marks a restore missing any one of the seven files cargo does not need. ## Review fixes Greptile's two findings on the first commit were both valid and are fixed in the second commit: - **Incomplete caches looked usable.** `REQUIRED` checked four files, but cargo's freshness check needs seven. Measured with cargo 1.97.1 on the real crate (restore, touch the artifacts, delete one file, build): without the lib's fingerprint hash, dep-info, rlib or rmeta it recompiles the crate (24s to 27s); without the bin's fingerprint hash, dep-info or the binary under `deps` it relinks (1.5s); without either `.json` fingerprint record, either rustc `.d` file, the uplifted binary or its `.d` it reports the crate fresh in 0.15s and recreates the uplifted binary from `deps`, and the staging step then copies it. `REQUIRED` now lists exactly the seven files, the glob for the binary under `deps` excludes the `.d` beside it, and the test removes each of the seven in turn (discarded, naming it) and each of the other seven in turn (still marked). - **Composed include paths escaped the guard.** The regex matched only `include_str!("literal")` and skipped `include_str!(concat!(...))`, which the crate uses four times (30 of 34 includes were checked). The guard now parses every include argument and resolves the `concat!` forms; one of them, in `provider_events.rs`, reads `tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json` at the repository root, outside every hashed input. It is test-only, so today's release binary does not read it, but the key is meant to cover every include whatever its build, so `tests/runner-e2e/fixtures/**` (9 files, 80 KB) joins `HASHED_INPUTS` and the six key embeddings in the two workflows. With the first commit's input list, the new guard fails on exactly that include. - `doc/RELEASE-AUTOMATION-SETUP.md`: a "Runner release crate cache" section under the Runner verification dependency cache, with the key, the writer, the readers and the trust boundary. - `package.json`: the new test file in `test:release-registry`. ## Verification - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: 10 tests pass, three consecutive runs. - The second commit, with the real toolchain on the real crate (`rustc 1.97.1`, cold build 2m20s, the crate's artifacts 74 MB): every source touched to a fresh-checkout mtime and the archived artifacts restored, cargo compiles (`Compiling paperclip-runner-core`, 24.8s and 25.1s, the two controls); restored and marked by `mark-current`, `Finished release profile in 0.16s` with no `Compiling`, and the same on a second build. Then, for each of the fourteen files cargo wrote for the two units, one at a time: restored, marked and that file deleted, cargo recompiles without any of the seven `REQUIRED` files and reports fresh without any of the other seven; restored with that file deleted and run through `mark-current`, the seven required ones print `action: discarded` naming the file and the others print `marked`. A restore holding only the seven required files builds fresh in 0.15s, cargo recreates the uplifted binary, and `stage-runner-binary.mjs` copies it into `dist/bin`. - `node --test '.github/scripts/tests/*.test.mjs'`: 378 of 381 pass. The 3 failures are in `docker-canary-promotion.test.mjs` and fail the same way on an unmodified `master` checkout in this sandbox (the stand-in docker writes no `calls.jsonl`). - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs`: 99 of 101 pass; the 2 failures spawn `pnpm`, which is not on this sandbox's PATH, and fail the same way on `master`. - Both workflows parse as YAML. No job has a duplicate step name or step id. - The mechanism, with the real toolchain (`rustc 1.97.1`, the package's pin) on a copy of the crate: a cold `cargo build --release -p paperclip-runner-core --bin paperclip-runnerd` (2m23s here). Then the artifact set was archived with `tar`, removed, every source re-copied so each had a new mtime, and the set restored: - restored, not marked: `Compiling paperclip-runner-core`, 24.6s (the control; this is what CI does today); - restored and marked by `mark-current`: `Finished release profile in 0.15s`, no `Compiling`; a second build is the same; - one Rust source edited after a marked restore: `Compiling`, 24.0s; - one protocol schema edited after a marked restore: `Compiling`, 23.9s (the protocol files enter cargo's dep-info as absolute paths and are hashed into the key); - the rlib removed from a restored set: `mark-current` prints `action: discarded, reason: incomplete: the lib rlib is missing` and removes the rest, and cargo compiles. - Not run here: the workflow steps themselves. `pr.yml` pins `pr-trusted.yml@master`, so this pull request's own `Build` job runs the current reader steps, and the writer runs only on a master push. The first PR runs after the merge and after the next master push show the `Restore the Runner release crate` step's result and the `paperclip.runner-release-crate.v1` line in their `Build` logs. ## Risks - Correctness: a reader can only restore artifacts that master built from inputs byte-identical to its own merge tree, because every input the build reads is in the key, and the key has no prefix match. A pull request cannot write. The trust boundary is unchanged: master writes, a pull request restores master's entry. - An input the build reads that is outside the hashed patterns would let a stale artifact pass. The release dep-info lists the crate sources and the protocol schemas and fixtures; all are hashed. The guard test resolves every `include_str!` and `include_bytes!` in the crate, literal or `concat!`, and fails if one lands outside the hashed inputs or uses a form it cannot resolve, so a new include form cannot slip past it by not matching a regex. - GitHub-hosted runners check out under `/home/runner/work`, the `runs-on` fleets under `/home/runner/_work`. Cargo records the included protocol files as absolute paths, so an entry from one root is useless under the other. The checkout root is in the key, so the other root misses instead of restoring something cargo then discards. - A cache that does not keep mtimes on restore changes nothing: the artifacts are then already newer than the checkout, and `mark-current` sets them to now in any case. - Cost: one new entry of about 70 MB per distinct input set, written only by the master Rust lane on a miss, plus one `cargo clean` and `build:binary` (about 40s to 50s) on that lane when the inputs changed. The readers pay one restore of about 70 MB (two to three seconds at the fleet's observed restore speed) and a sub-second `mark-current`. - An incomplete entry (a writer that saved a partial set) is discarded by `mark-current` and costs one compile, the same as a miss. ## Model Used - Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1), extended thinking, with tool use (shell, file editing, GitHub CLI, a local Rust toolchain for the experiment) in a Paperclip agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-12 11:25:31 -07:00