5 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	f1a394bd30	feat(runner): add Grok Build through native ACP (#13882) ... ## Thinking Path > - Paperclip manages AI agents and governs their work. > - Its native runner uses structured provider protocols for sessions and tools. > - Grok Build supports ACP over stdio, but the runner did not expose it. > - Native execution requires company-scoped credentials, verified identities, and permission gates. > - This change adds Grok through ACPX for local and Daytona execution. > - Subscription login and explicit API-key execution have separate credential paths. > - Qualification grades real tool outcomes, durable state, and browser workflows. ## Linked Issues or Issue Description Refs #13845, #13847, #13850, #13878, #13901, #13973, #13977, #13979. Add **Grok Build** to `paperclip_runner` with `provider: "acpx"`, `acpxAgent: "grok"`, and model `grok-4.7`. Existing legacy Grok agents keep their adapter. Merge the three companion fixes (#13973, #13977, #13979) before treating the integrated Product qualification as deployed behavior. ## What Changed - Synchronize shared, TypeScript, Rust, server, validation, and UI provider contracts. - Run Grok native ACP stdio through ACPX and the authenticated Paperclip MCP bridge. Verify the pinned executable and exact ACP model identity. - Prefer company subscription login. Support an explicit company-secret API key without automatic paid fallback. Fence refresh and copyback to the same account and remove private runtime credentials after containment. - Preserve selected permissions, cancellation, durable session identity, resume, and restart recovery. Keep unsupported steering and goals unavailable. Preserve missing usage and cost as unknown. - Package checksum-verified Grok Build 1.0.13 for Daytona with an immutable, signed image built on EC2. - Add deterministic admission, protocol, permissions, identity, credential, failure, and cleanup checks. Add the maintained 39-case protocol roster and separate subscription/API Product profiles. - Fix live-test findings in reasoning events, reloads, idle-owner retirement, credential-home cleanup, expired-login model discovery, launcher pinning, and rerun evidence selection. - Align control-plane state readers with the transport's 64 MiB bound while retaining identity, ownership, lifecycle, and size rejection checks. - Stabilize two asynchronous CI assertions while retaining actual outcome and filesystem-evidence checks. ## Verification Current integration head `f114948376056fe0b6b34c1496ae8667b59daa63` includes master `3447609d2247e75e55d91493dda91a608364f672` (2026-09-28). Two master advances during verification overlapped the eval catalog; the final merge preserves Grok qualification, completion updates, and bounded API-response reading in all 348 cells. All 77 focused catalog/eval/workflow tests pass. Both native stack layers (#14397) are mergeable, and both exact-head Greptile reviews are 5/5 with successful security scans and no unresolved review threads. All current-head CI is green: 56 successful checks/statuses and four intentional skips ([CI run](https://github.com/paperclipai/paperclip/actions/runs/36447097232)). Trunk code-owner requirements remain enforced. The review summary’s non-blocking saved-asset offset classification note concerns code already merged in #14301; those runtime files are identical to master and outside this stack’s diff. Historical live evidence below retains its original source revisions. Earlier integration checkpoint: `24fc9b94ca0afb21ccdc8d26dbb2e4b258ad72cb`. Refreshed against master `0f14d2612`, preserving Grok qualification alongside the new accounting and lifecycle suites. All 124 focused catalog, evidence, and service-worker checks pass. The current base workflow includes the explicitly selected public-install verification lane; follow-up #14024 supplies its verifier script. CI at that earlier checkpoint was green (56 successful checks/statuses, four intentional skips), and the review is 5/5 with no unresolved findings. Prior feature CI at `fd73f0a9b1ecdf4094685054028df71739ddc3e1` passed ([run 36148259902](https://github.com/paperclipai/paperclip/actions/runs/36148259902)); that is historical evidence, not a current-head result. Paid Product measurements use frozen integrated source `2d939a92b21dcaf5c77c88b54d96784d2ddd0699`, which combines the feature with #13973, #13977, and #13979. That source passed all 52 CI checks and clean 5/5 review. Later master syncs incorporate upstream changes. Their checks remain separate from these pinned live measurements. | Check | Result and source-pinned report | | --- | --- | | Subscription protocol roster | [39/39 first attempts; 206 assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-36046839612-1/index.html), runtime `bc6833f7`, evals `92bb4b8c` | | API protocol roster | [39/39 first attempts; 206 assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-35926577007-1/index.html), runtime `4a1061c8`, evals `3213dbec` | | Subscription full Product matrix | [16/16 first attempts; 144 assertions; cleanup passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36096908572-1/index.html), source `2d939a92` | | Subscription core repetitions | 18/18: tool use, planning approval, and Stop/resume each passed three times in local and Daytona profiles. The full matrix contains repetition one; [repeat two](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36104551060-1/index.html) and [repeat three](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36108100404-1/index.html) each passed 6/6. Total: 28 unique subscription attempts at `2d939a92`. | | API smoke and question continuation | [4/4 first attempts; cleanup passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36147315401-1/index.html), both environments at `2d939a92` | | Historical API Product coverage | [16/16 full matrix](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35875144860-1/index.html) and 18/18 core repetitions at `4a1061c8`; retained as measurements of that revision | | Native Daytona proof | Three subscription and three API MCP/permissions/resume runs passed at `bc6833f7`. Three expired-login admission and fenced refresh checks passed without inference. All test sandboxes were removed. | | Inspectable artifacts and UI | Current-source screenshots verify planning approval, direct Ask completion, question continuation after controller restart, and two downloadable project revisions. The project downloads pass 12 and 18 tests; all 40 independent artifact oracle checks pass. | | Provider-free checks | 116 eval-validator tests, 39 Grok definitions, and 359 enabled/external campaign cells pass. Continuation regressions above 2 MiB and 16 MiB failed before their fixes; 32 focused recovery/ownership/size checks pass. | The 32 unique current-source Product attempts have no failures, retries, or skipped cells, and all cleanup checks pass. Whole-workflow timing, model identity, image and provider-pack provenance, attempts, and accounting coverage are retained in the canonical reports. The report publisher's conservative `complete=false` flag is preserved; independent audits verify the exact selected source catalog and immutable result rows. Pins: Grok Build `1.0.13 (5e9a58528b76)`, ACPX `0.13.1`, ACP model `grok-4.7`. Linux binary SHA-256: `edf79521581bb5e6b95abef848491a6a742e860da3e237ebe86a280d30dce4c1`. Launcher SHA-256: `f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55`. Image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:76b24edfd850219e949418b19e4ceba690e84d51d199ade426e484953329b5e9`. Image build source is `4196a4cd`, recorded separately from application source `2d939a92`; each campaign verifies the image signature and provider pack. Original failed campaigns remain available: [continuation bound](https://github.com/paperclipai/paperclip/actions/runs/36057718059), [scheduler/event capture](https://github.com/paperclipai/paperclip/actions/runs/36071063537), and [startup cleanup plus EC2 interruption](https://github.com/paperclipai/paperclip/actions/runs/36080870743). They retain their original grades. No Docker or Rust builds ran on the developer laptop for these follow-ups. ## Risks Merge packaging follow-up #14024 with this base before public release. The follow-up replaces the private Grok bridge package with a built-in launcher and makes the native binary an explicit sandbox prerequisite. Three separate, reviewed fixes are part of the tested integrated behavior: #13973 serializes task-run admission; #13977 captures complete event evidence; #13979 durably reconciles failed Daytona creation. Each has green CI and clean 5/5 review. Failed-create recovery has 277 plugin tests, 92 SDK tests, host-runtime recovery tests, and a real Daytona lost-deletion-receipt proof. The live proof uses a private file for journal persistence; database durability is covered by host tests. Worker death before delivery of a failure envelope remains outside that recovery mechanism. Subscription fixtures stage an authorized company login; interactive browser sign-in is not qualified. Local Product profiles ran on EC2 Linux. The temporary subscription credential was removed from the protected GitHub environment after all subscription audits, with absence verified. Runtime homes and refresh copyback remain ownership-fenced. Protocol results remain pinned to their original revisions; they are not relabeled as tests of the latest feature commit. New binary/model versions require qualification. Missing token usage and model cost remain unknown; runtime estimates do not establish a full bill. Automatic paid Grok scheduling remains disabled pending separate reviewed enablement. The 64 MiB bound can increase memory use for verbose sessions, and larger files still fail closed. No automatic legacy-agent migration occurs. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-28 13:54:36 -05:00
3 people	3c2bc4f546	ci: halve the isolated native Runner check by seeding it from the public master build cache (#13736) ... ## What Recurring CI health check (PAP-31): on the most recent fully-green PR run (`cb703ac`, run [35519542997](https://github.com/paperclipai/paperclip/actions/runs/35519542997)), the slowest check was **Compile isolated native Runner** at **452s** — ahead of the largest test shards (379s). Every run recompiled the full Rust dependency tree from zero, even though `docker.yml` already refreshes a **public** `mode=max` BuildKit cache (`ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}`) on every master push, containing exactly these layers. This PR seeds **only the baseline build** with that registry cache, via anonymous pull. **Measured on this PR's own CI (which exercises the seeded path): the check completed in 123s, down from 452s — a 73% reduction, ~5.5 minutes saved per run.** ## Thinking Path Cost breakdown of the 452s from the job log: `cargo chef cook` dependency compile 214.7s, local cache export 43.1s, runner-core build 37.0s, metadata proof layer 36.8s, `cargo install cargo-chef` 36.4s, rebuild-verification build ~65s, setup/teardown ~20s. The dependency compile and toolchain layers are identical to what the production `docker.yml` build already caches publicly on every master push, so recompiling them here bought no signal — the check's real assertions live in the *verification* build, not the baseline. A first attempt used `actions/cache` plus a master `push` trigger, but the CI bot's GitHub App lacks `workflows` permission; the registry-cache approach is strictly better anyway (shared across PRs immediately, no 10GB Actions-cache quota pressure, no workflow change). ## What Changed - `scripts/check-docker-runner-cache.sh`: the baseline build now adds `--cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-{amd64|arm64}` (selected by host arch). `RUNNER_CHECK_SEED_CACHE` overrides the ref, or set it empty to force the old cold path. The script header documents the anonymous external read. - `.github/workflows/docker-runner-check.yml` (comment-only): the stale "no external cache" note now describes the anonymous GHCR seed and the verification build's local-cache-only isolation. This was pushed in a follow-up commit with workflow-edit permissions; the original CI-bot token could not touch workflow files. No Dockerfile stages or verification assertions changed. ## Verification - This PR's own `Compile isolated native Runner` check runs the seeded path (the script is in the workflow's trigger paths): **passed in 123s** vs the 452s baseline. - The rebuild-verification semantics are untouched: it still runs on a **fresh builder** importing **only the local cache exported by this run's baseline**, so it proves exactly what it proved before — that the runner image rebuilds reproducibly from this run's own exported layers. - Verified `ghcr.io/paperclipai/paperclip:buildcache-amd64` is anonymously readable (unauthenticated manifest pull succeeds), so the check gains no credential or secret dependency. ## Risks - **Stale or missing seed cache:** if the GHCR ref is unreachable, private, or garbage-collected, BuildKit logs a warning and falls back to the pre-PR cold compile — the check gets slower, never wrong. `RUNNER_CHECK_SEED_CACHE=""` restores the cold path explicitly. - **Cache trust:** the seed only accelerates the *baseline* build; the verification build still runs on a fresh builder against only this run's locally exported cache, so a stale or poisoned registry cache cannot make verification pass spuriously. The ref lives under `ghcr.io/paperclipai/*`, written only by repo CI on master pushes. ## Model Used Claude Fable 5 (`claude-fable-5`) via Paperclip agent **Bender (Fable)**, issue PAP-31. --------- Co-authored-by: Bender (Fable) <bender-fable@paperclip.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-09-21 20:22:48 -07:00
Devin FoleyandPaperclip	44f6312cd8	fix(ci): reuse one available Cloud registry cache (#13334) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud needs a verified image for each merged source commit. > - Fresh builders restore compiled native dependencies from registry caches. > - The current workflow imports up to eleven historical cache manifests at once. > - Live builds missed native layers that a fresh builder reused from one manifest. > - This PR selects the nearest available cache and tests reuse across fresh builders. ## Linked Issues or Issue Description Refs #13329 and #13330. A search of open cache PRs found no duplicate of this change. **What existing behavior does this improve?** Remote Docker cache reuse on fresh Cloud image builders. **Current behavior** [Cloud run 34714483272](https://github.com/paperclipai/paperclip/actions/runs/34714483272/job/103609096836) imported the previous cache manifest successfully but rebuilt `cargo-chef` and Rust dependencies. The dependency compile took 3m43s. The preceding image build had already exported those layers. A controlled [fresh-builder diagnostic](https://github.com/paperclipai/paperclip/actions/runs/34715336530) used the same source and registry cache. The single-manifest job reused both layers immediately. The multiple-manifest job rebuilt them and failed the cache assertion. Both jobs used GitHub-hosted runners with read-only access. **Proposed behavior** Inspect cache manifests in first-parent order and import only the nearest available one. Keep full-SHA cache exports, the ten-commit search bound, and the legacy fallback. If caches cannot be read, permit a cold build. **Reason and benefit** Avoid the observed cache misses without changing image contents or builder sizes. Expected savings include about four minutes of native tool/dependency compilation when those inputs are unchanged. The final merge-to-deployable gain still needs a post-merge measurement. **Breaking changes** No image, artifact, deployment, or runner-routing contract changes. ## What Changed - Select one available ancestor cache after Docker login and Buildx setup. - Preserve separate writable cache tags for each full source SHA. - Test cache ordering, missing caches, registry errors, and workflow integration. - Add the selector tests to the existing release-registry suite. - Export a local test cache, remove the first builder, and verify a source rebuild on a fresh builder. - Document cache selection and the stronger Docker check. ## Verification - Passed 456 focused workflow, routing, readiness, preview-artifact, and cache-selector tests. - Passed shell syntax, ShellCheck for the changed probe, actionlint workflow validation, and `git diff --check`. actionlint's shell checks were disabled for the workflow validation because unchanged migration-label commands trigger existing SC2012 notes. - The fresh-builder registry diagnostic proves the single-cache behavior. The [permanent two-builder probe passed](https://github.com/paperclipai/paperclip/actions/runs/34715771048/job/103612624090), including a changed real binary and dependency-declaration invalidation. - Passed all 35 latest-head checks (green or intentionally skipped), including full typecheck, test, build, and browser suites in [PR CI run 34715771217](https://github.com/paperclipai/paperclip/actions/runs/34715771217). - The real selector CLI inspected registry metadata and chose the nearest available ancestor cache. - Fresh Greptile review is 5/5 with no open findings. The PR title was corrected to meet the source-change naming rule; the review check passed after that correction. - Local full-suite runs and Docker builds are unavailable because the local Docker daemon is unresponsive after disk exhaustion. CI provides the Linux verification. ## Risks - Missing or unreadable caches cause a slower cold build. The selector logs that condition and preserves image publication. - Inspecting several missing ancestors adds lookup time. Each lookup has a ten-second timeout and the search is bounded. - The Docker test now exports a local cache. It removes the first builder before starting the second to release disk space, then cleans up its builders and files. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-12 13:20:18 -07:00
Devin FoleyandPaperclip	7435b2ee9c	ci: cache compiled Docker Rust dependencies separately from source (#13329) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys images that contain the native Rust Runner. > - The image already builds that Runner before copying ordinary app source. > - A Rust source change still invalidates its entire compiled dependency layer. > - Compiled dependencies can survive source changes when their recipe is unchanged. > - This PR adds a separate locked dependency build before compiling the real workspace. ## Linked Issues or Issue Description Refs #13195. A search of related Docker and Cargo cache PRs found no duplicate dependency-recipe change. **What existing behavior does this improve?** Docker image build time after Rust source or embedded protocol changes. **Current behavior** The `runner-build` stage compiles dependencies and workspace code in one layer. In Cloud readiness run 34698143548, that stage took about 3m48s when its cache was unavailable. **Proposed behavior** Generate a recipe with pinned cargo-chef 0.1.73. Build locked release dependencies in `runner-deps`, then copy and compile real Rust source and embedded protocol inputs in `runner-build`. Source edits can reuse the dependency layer from the existing registry cache. **Reason and benefit** Reduce dependency recompilation during source changes and merge bursts. Expected savings are roughly 2–4 minutes when the old native layer would miss but dependency layers are available. Full cold builds also pay for the recipe tool installation. Ordinary app-only cache hits gain little from this change. **Breaking changes** None to the shipped application or image tags. The recipe tool and compiled dependencies remain in build stages. ## What Changed - Install a pinned recipe generator with its locked dependencies and the existing package-owned compiler. - Add recipe planning and compiled dependency stages. Use the same release profile, package, binary, and lockfile enforcement as the real native build. - Remove generated source stubs before copying actual source. Preserve protocol inputs, timestamp normalization, binary staging, and application checks. - Add Docker cache wiring regressions and update the Docker cache documentation. - Run a two-build probe in Docker Runner check. It requires dependency reuse, changed real binary metadata after a source edit, and a changed recipe after a dependency declaration edit. It uses a disposable tracked-source context and exports only small metadata files. ## Verification - Passed all five Docker build-stamp and dependency-cache tests with `pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`. - Passed the local ARM64 `docker buildx build --target runner-build --progress plain`. Local Docker then hit storage errors during a runtime probe; cache invalidation verification continues on GitHub-hosted Linux. - Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`, and `git diff --check`. - Passed a [Linux AMD64 cache probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199) against the PR source: dependencies compiled in 3m49s for the baseline and were `CACHED` after a source edit; real source compilation took about 37 seconds. Binary metadata changed and dependency declaration changes altered the recipe. The permanent probe is also running in latest-head Docker Runner check. - Passed latest-head [Docker Runner check](https://github.com/paperclipai/paperclip/actions/runs/34711145160), including the permanent source/dependency invalidation probe. - Passed full [PR verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2): typecheck, all grouped tests, native verification, build, release dry run, and browser checks. One unrelated signoff-policy browser test failed waiting for a heartbeat run on attempt 1; only that failed shard and dependent checks were retried, and passed. - Latest-head Greptile is 5/5 with no unresolved findings. Full local tests/build were limited by local disk exhaustion; Linux CI completed those checks. ## Risks - The two-build CI probe has a 20-minute job limit to cover the cold build and source rebuild. It adds no AWS routing. - A fully cold build must install cargo-chef and populate the dependency layer. Both become reusable registry layers; no Actions cache is added. - The recipe and final build must keep the same compiler, build profile, package, binary, and directory layout. A source-change rebuild probe checks real cache reuse and binary invalidation. - Dependency or compiler changes still require rebuilding dependencies. Existing image verification and full-SHA publication gates remain unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-12 11:57:32 -07:00
Devin FoleyandPaperclip	59d74b68b2	ci: cache the native Runner in a separate Docker stage (#13195) ... Compile the native Runner from its complete Cargo and protocol inputs in a separate cached Docker stage. Preserve Cargo validation and generated-contract checks during the normal application build, normalize input timestamps across checkouts, and compile the isolated target in PR CI. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-09-10 20:36:55 -07:00