From 02a7964775a8d2dc0a4b5dbaee08740edcaf98e7 Mon Sep 17 00:00:00 2001 From: Dotta Date: Wed, 7 Oct 2026 08:12:08 -0500 Subject: [PATCH] docs(hermes): record paid acceptance and qualification limits Co-Authored-By: Paperclip --- doc/plans/2026-10-06-hermes-native-runner.md | 145 +++++++++++++++++-- 1 file changed, 132 insertions(+), 13 deletions(-) diff --git a/doc/plans/2026-10-06-hermes-native-runner.md b/doc/plans/2026-10-06-hermes-native-runner.md index a8acf7d26fa..a2c08970c11 100644 --- a/doc/plans/2026-10-06-hermes-native-runner.md +++ b/doc/plans/2026-10-06-hermes-native-runner.md @@ -1,7 +1,8 @@ # Hermes native runner implementation Status (updated 2026-10-07): implementation candidate; **not qualified**. -Branch: `codex/hermes-native-runner`. +Current branch: `codex/hermes-qualification`; stacked on +`codex/hermes-native-runner` and `codex/hermes-routines`. ## Accepted outcome @@ -21,7 +22,8 @@ Keep the existing Hermes local/gateway adapters compatible. - [x] Focused TS/Python tests and production-path macOS execution with a deterministic model server. - [x] Complete final Rust/regression checks and resolve or classify failures. - [ ] Browser acceptance, Linux/Daytona execution and connection-method qualification. -- [ ] Required repository checks and reviewable PR. +- [x] Reviewable draft PR stack, green CI and fresh Greptile 5/5 on the implementation heads. +- [ ] Complete the final local aggregate test invocation and release qualification. ## Qualification evidence @@ -58,7 +60,8 @@ because fixtures pass. Native cron and gateway messaging are disabled. - Pending Hermes choice in existing runner configuration; existing connection, model, permission and transcript components. Ten Product E2E candidate cells - (five local, five Daytona) are registered but have not been run live. + (five local, five Daytona) are registered. Four local cases have passing paid + attempts; the complete release matrix has not passed. ## Evidence and outstanding release gates @@ -74,7 +77,7 @@ Hermes uses the new shared ACP profile manifest. The shared extension fix also updates Cursor's ACPX patch attestation and profile identity to revision 15; the Cursor usage, delegation and model-selection package contracts pass. -Post-rebase checks: +Historical post-rebase checks (superseded where newer results appear below): | Check | Result | | --- | --- | @@ -106,7 +109,7 @@ installed package (not workspace imports). The candidate provider-pack materializer also verifies the copied runtime. Both native execution targets are still pending real model and product qualification. -Reproduced runtime closure SHA-256: +Historical pre-review runtime closure SHA-256: | Target | Closure digest | | --- | --- | @@ -125,22 +128,25 @@ Release blockers remain explicit: including refresh/revocation/concurrent ownership, permission prompts, questions across reconnect, steering/queue/stop, remote recovery, cross-task learned skills, routine firing and cost attribution. -4. Obtain a green aggregate CI run and review the implementation before - promoting the candidate. Default production selection remains disabled. +4. Complete the full live qualification before promoting the candidate. + Implementation CI and review are green; they do not replace live proof. + Default production selection remains disabled. ## Review handoff -The routine service binding is a separate eight-file commit on +The routine service binding is a separate review on `codex/hermes-routines`. The native integration is stacked on it on `codex/hermes-native-runner`, within the 100-file review limit. Changes are committed. The generated root lockfile is excluded as required by the -repository; the Daytona Dockerfile pins the twice-reproduced resolved lock. +repository. The Daytona Dockerfile requires the caller to provide the SHA-256 +of its resolved lock and verifies the copied lock before installing dependencies. The user subsequently authorized release qualification and PR verification. The routine PR is [#15434](https://github.com/paperclipai/paperclip/pull/15434). The native integration is stacked in [#15435](https://github.com/paperclipai/paperclip/pull/15435). Both are drafts; -review and CI are running. Qualification fixes use `codex/hermes-qualification` +their current implementation heads have green CI and fresh Greptile 5/5. +Qualification fixes use `codex/hermes-qualification` to retain the under-100-file limit for each review. ## Paid qualification, 2026-10-07 @@ -211,15 +217,128 @@ paths and re-signs the changed macOS library with a deterministic ad-hoc signature. Independently provisioned interpreter paths produce identical closures; this is distribution proof, not live Linux sandbox qualification. -| Final target | Closure digest | +| Historical review target | Closure digest | | --- | --- | | macOS arm64 | `4c89b24335e1869a9faba6996a4e82979337ee5f850d792ab41a838e79afdce3` | | Linux amd64 | `f9919bd2e812e86e81ecd964d6e1961bb68f96e2154f816c554f31c4f1d78211` | The qualification stack is [#15436](https://github.com/paperclipai/paperclip/pull/15436). Runtime -qualification fixtures and this implementation/evidence record belong to that -PR to keep the native integration review below 100 files. The new shared ACPX +qualification workflows, follow-up fixes and this implementation/evidence record +belong to that PR. The native runtime fixtures belong to #15435. Each review +remains below 100 files. The new shared ACPX patch has an explicit Cursor profile revision 16, preserving historical revision decoding. The Docker lock digest is twice reproduced; the root lock file remains owned by the repository's lock bot. + +### Current qualification record, 2026-10-07 + +Hermes remains **pending qualification**. All three draft implementation PRs +have passing CI and fresh Greptile 5/5 at these +heads: routines `3745f3c5a46bda7778ee132682d1b7ae23b088c1`, native +`6a7a006b0738558a4abb1c030f2b7b11f5afea2c`, and qualification +`09195bb8c8bc564eaa3f5061a7d6b5d685e708a4`. A later native review also found a +standalone image command without the required resolved-lock digest. Both image +guides were corrected in `0dea682031f8e35631faee7a05519ed4dce80d07`, their shell +syntax and checksum ordering were checked, and the addressed thread was +resolved. The qualification commits were rebased onto that fix. Subsequent +documentation heads require fresh checks and review before handoff. + +Paid planning exposed two integration defects. Assigned plan-document and +task-title tools were rejected by the native read-only guard before the +controller could apply its task-mode authority. Structured MCP results also +appeared as `null` in the transcript. Assigned workflow tools now reach the +existing controller authorization and configured permission check; native +commands and file writes remain denied in planning mode. Tool results retain +their actual output. A pre-dispatch denial synthesizes a failed call using +Hermes's authoritative call ID exactly once, including overlapping calls. + +Managed Hermes restoration now loads the validated native history without +emitting it again as new ACP transcript output. Paperclip owns the persisted +transcript. Standalone, non-negotiated ACP clients retain native history replay. +Missing or unreadable native history still fails restoration. + +Current runtime closure SHA-256 (fresh provisioning reproduced both): + +| Target | Closure digest | +| --- | --- | +| macOS arm64 | `970f0c48b905d17e616a3b75ef28d91a0e28afba8dbb3218628cd02b3b1e709c` | +| Linux amd64 | `15fc9631318d50a2aafa9c566410b4d486265fb3e58a7981fd0c2525a5fb8f7a` | + +Current focused verification: 46 TypeScript permission/sandbox/configuration/ +installation tests, 20 pinned Python bridge tests, and both native production +fixtures pass. The ACPX fixture additionally checks planning round trips, +visible write denials, strict cancellation recovery and missing-history +rejection. These fixtures use a deterministic model endpoint. The full Rust +workspace ran 652 passing test executions with two ignored. Repository +typecheck, build, protocol generation checks and UI token gates pass. + +The local aggregate test invocation finished with 16,084 passing tests, one +failed test, 216 skipped tests and two failed suite setups. Both suite setups +failed during embedded PostgreSQL bootstrap before their assertions ran. The +real 40,000-file Git streaming test reached its existing five-minute deadline. +These failures remain failures of that invocation; focused reruns and green +sharded CI do not rewrite it as a pass. +The isolated rerun passed both database suites (eight tests). The Git fixture +still reached its five-minute deadline. Its test and shared Git implementation +are unchanged from the source baseline; Linux PR CI passes that coverage. The +local Git timeout remains an explicit verification limitation. + +Paid browser attempts used the managed OpenRouter account and exact model +`deepseek/deepseek-v4-flash-0731`, on macOS arm64, through Chromium, the isolated +Paperclip server/database, Rust Runnerd, ACPX and native Hermes: + +| Case | Passing campaign | Source provenance | Duration | Cleanup | +| --- | --- | --- | --- | --- | +| Hello/completion | `hermes-local-paid-20261007-first` | Report source null; observed checkout `2796b80a9` with image inputs in progress | 47.63 s | Pass | +| Question/resume | `hermes-local-paid-20261007-question-replay-fix-retry2` | Recorded `74b692bbd8c98bbeda4c39cf8327680245ac2cbf` | 106.90 s | Pass | +| File edit/validation | `hermes-local-paid-20261007-remaining-continuity` | Recorded `0ba0d75511cf9fdf1fa4b21d9e900503078f3620` | 122.28 s | Pass | +| Plan/approve/complete | `hermes-local-paid-20261007-plan-policy-fix` | Recorded `09195bb8c8bc564eaa3f5061a7d6b5d685e708a4` | 134.19 s | Pass | +| Structured question/controller restart/resume | `hermes-local-paid-20261007-restart-resource-retry` | Report source null: invocation used the wrong source-variable names; observed checkout `09195bb8c8bc564eaa3f5061a7d6b5d685e708a4` | 118.72 s | Pass | + +All five registered local cases have passing attempts across multiple heads. +This is not a complete final-head campaign or the full requested release +matrix. The restart case proves persistence of the pending question across a +controller restart, submission of its answer, native session reuse, and task +completion. Its resumed run reported 64,166 input and 281 output tokens. The +planning completion run reported 62,268 input, 478 output and 37,888 cached +input tokens. Their interrupted first runs have incomplete usage receipts. +Cost remains unpriced; missing cost is not a zero-cost execution. + +The preceding restart attempt failed during test-database bootstrap, before +any model request. Clearing only four confirmed user-owned, unattached, +56-byte shared-memory segments with dead creators allowed the unchanged test +to run. Earlier session-open timeout and transcript failures remain retained +in their own campaign results. No deadline or oracle was weakened. + +Private numeric-only budget receipts show the qualification key's $5 hard +limit still has $4.559226202 remaining after these attempts. The shared-key +usage change is an aggregate ceiling, not exact attribution to individual +runs. Sanitized results remain under `tests/runner-e2e/results/`; private native +history, credentials and hidden reasoning are not published as artifacts. + +The clean installed-package root/evals/testing conformance check passed on +`09195bb8c8bc564eaa3f5061a7d6b5d685e708a4`. It uses offline packed runtime +dependencies, including the reviewed ACPX patch, rather than unmodified +registry dependencies. The installed package's shipped provisioner reproduced +the current macOS closure. Both native fixtures passed from installed compiled +code, using the published release Runnerd artifact: 29.03 seconds for the Rust +path and 146.75 seconds for the ACPX path. Only fixture import locations and the +explicit Runnerd artifact path were adapted; assertions and deadlines stayed +unchanged. This is clean-package transport proof with a simulated model. + +The protected paid workflow now provisions Python, bubblewrap, and the verified +Hermes closure only for an explicit Hermes selection, before credentials enter +the paid step. It must land on `master` before its trusted dispatch can run +paid Linux/Daytona campaigns. No paid remote campaign has been dispatched. +An available development host permits bubblewrap but is Linux arm64; it does +not satisfy the requested Linux amd64 target. The emulated local Linux amd64 +container provides provisioning proof and still fails the namespace gate. + +Outstanding release proof includes the other API providers, subscriptions, +custom protocols, Bedrock, real vision input, credential refresh/revocation and +concurrent ownership, live steering/queue/stop, lower permission modes, +cross-task memory/learned skills, routine firing and deduplication, and actual +Linux amd64/Daytona execution and restoration. Exact approved Connection names +and a compatible remote execution environment are still needed. The existing +standalone Hermes local/gateway adapters retain their contracts.