From ec40b1eb324ca298d47233792b210ee4cd27df5b Mon Sep 17 00:00:00 2001 From: Dotta Date: Thu, 8 Oct 2026 11:42:45 -0500 Subject: [PATCH] fix(hermes): retain the owned billing receipt during Stop Co-Authored-By: Paperclip --- doc/connections/AI-CONNECTIONS.md | 8 ---- doc/evals.md | 32 --------------- doc/plans/2026-10-06-hermes-native-runner.md | 37 ++++++++++++++++++ packages/paperclip-runner/docs/hermes.md | 4 ++ .../acpx/codex-runtime-adapter.test.ts | 39 +++++++++++++++++++ .../src/drivers/acpx/codex-runtime-adapter.ts | 21 +++++++++- 6 files changed, 100 insertions(+), 41 deletions(-) diff --git a/doc/connections/AI-CONNECTIONS.md b/doc/connections/AI-CONNECTIONS.md index 0625720641e..af955ba2bf3 100644 --- a/doc/connections/AI-CONNECTIONS.md +++ b/doc/connections/AI-CONNECTIONS.md @@ -617,14 +617,6 @@ fingerprint; secrets travel through the ephemeral credential channel. Implemente projections remain subject to per-method live qualification. See the [Hermes runner contract](../../packages/paperclip-runner/docs/hermes.md). -The Hermes runner's optional per-turn billing receipt currently recognizes -provider-reported OpenRouter `usage.cost` only on its selected managed Chat -Completions route. Account attribution still comes from the authoritative -connection projection. Token totals, displayed native price estimates, and -cumulative session costs do not prove billed spending. Incomplete wire attempts -retain known positive subtotals and leave missing charges unpriced; other Hermes -providers and protocols remain pending billing qualification. - OpenRouter connections without an explicit model list automatically load its public [model catalog](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties), ordered with `sort=most-popular`. New-agent setup and agent settings share this diff --git a/doc/evals.md b/doc/evals.md index d67d4c3b66a..77b5ac15e23 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -4,31 +4,9 @@ The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER- is a Product E2E workflow for fresh subscription/API-key/gateway connections, with attended login and independent artifact checks against local or staging targets. -The explicit-only [Hermes managed API suite](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only) -checks native completion through five selected managed API accounts on local -and Daytona, with independent account/model attribution. Its registered cells -remain pending until their live evidence passes; its API-only scope leaves the -subscription and lifecycle gates separate. -The launcher records clean Git source before credential loading for Hermes -candidates and rejects a mismatched requested SHA. Its source receipt and normal -result metadata identify the controller; they do not qualify runtime behavior. - -The same guide registers the separate explicit-only `hermes-bedrock-connections` -suite for one exact `us-east-1` inference profile on local and Daytona. Its two -cells remain pending. They verify the explicitly selected personal grant and -public region/protocol/model routing before and after native completion, using -a short-lived AWS bearer credential and the existing budget/evidence pipeline. - Paperclip has two live eval families with different questions, owners, and evidence. Choose the family before selecting a model, profile, or case. -The explicit-only `hermes-native-interactions` Product E2E suite adds a native -three-question batch on local and Daytona. It checks browser reconnect within -the active run, exact native callback delivery, late duplicate rejection and a -final reply derived from the returned reviewer text. The registered cells remain -pending; assigned Paperclip question tools do not satisfy this native oracle. -See the [Hermes fixture contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only). - The explicit-only [native instruction consolidation comparison](plans/2026-10-03-native-completion-consolidation.md) uses six Product E2E cells per source variant. It measures the completion constraint reduction separately from the earlier native tool-description @@ -447,13 +425,3 @@ See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmati Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately. The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills. - -The native interaction suite also declares one local `native-question-batch-stop` -cell. It leaves a complete native callback unanswered, retains the pending -request, clicks the browser Stop control, and independently requires the exact -cancelled callback/turn and audited public Stop acknowledgement. The task must -remain In Progress, a late answer must receive HTTP 409, and the public per-turn -process owner and observed descendants must retire before cleanup and remain -retired through cleanup. Cancelled-run account attribution and complete reported -OpenRouter billing are required. This is a separate pending release gate; remote -Stop still requires its own retirement observer and actual Daytona proof. diff --git a/doc/plans/2026-10-06-hermes-native-runner.md b/doc/plans/2026-10-06-hermes-native-runner.md index 0609cf2eb98..6e8d764c34a 100644 --- a/doc/plans/2026-10-06-hermes-native-runner.md +++ b/doc/plans/2026-10-06-hermes-native-runner.md @@ -1416,3 +1416,40 @@ fixture attempts remain separate: one checked the late answer prematurely and failed during shutdown; another placed the workspace inside its isolated host home and was rejected before launch. Neither is live-provider proof. A fresh paid browser Stop campaign and the remaining release gates are still required. + + +### 2026-10-08 native Stop projection and terminal receipt fixes + +The original `759f7bfccd` paid Stop attempt remains failed and is not regraded. +Its native callback and turn cancelled, its audited browser Stop returned HTTP +200, and all nine observed process identities retired. The card stayed pending, +generic stranded-work recovery later blocked the task, and the cancelled run's +cost stayed unpriced. No held reservation is released from these fixes. + +The server now projects the exact native input cancellation into its question +card, checking company, task, agent, run, session, runner, turn, item and the +committed creation. The pending-card compare-and-update cannot overwrite an +answer, replay cannot create another expiration, and the transaction locks the +task before the card. Both local and durable PRP observation use this projection. +Unrelated historical questions and provider-loss handoffs retain their lifecycle. +Recovery checks deliberate operator Stop before escalating an uninvokable +assignee, so a later agent pause cannot turn that Stop into stranded work. + +The ACP boundary had discarded Hermes's final usage notification once Stop +aborted its control signal. Only that admitted prompt's final usage can now +cross the cancelled boundary, with its exact native session and negotiated turn +token, before prompt settlement. Requests and other activity remain closed. +Stream/process closure ends the receipt window and cannot reopen it. The native +wire ledger and completeness rules are unchanged; missing charges stay unknown. + +Credential-free regression evidence: the paused-agent recovery test first failed, +then its full suite passed 79 tests. Native card cancellation first failed seven +checks, then the final suite passed 38 tests, including nine identity denials, +replay, late-answer rejection, preserved historical answers and a concurrent +answer/Stop race. The billing-boundary regression first failed because no receipt +reached its owner; the affected ACP/usage suites passed all 110 tests after repair. +Direct server and Runner TypeScript checks passed. An intermediate implementation +scope error is retained as a failed test attempt; it is not qualification proof. +No local Docker, Rust build, secret access or paid provider call was used for this +repair. A fresh committed-source browser Stop attempt and complete release gates +are still required. The PR stack remains draft and Hermes remains pending. diff --git a/packages/paperclip-runner/docs/hermes.md b/packages/paperclip-runner/docs/hermes.md index 6d05b6a1ddd..29e57eec61d 100644 --- a/packages/paperclip-runner/docs/hermes.md +++ b/packages/paperclip-runner/docs/hermes.md @@ -116,6 +116,10 @@ extend the task's execution deadline. selected OpenRouter billing identity and the current native turn before using it for accounting. Cumulative session cost and native model-price estimates cannot supply this receipt. +- During Stop, requests and steering close immediately. The same admitted + Hermes prompt may still deliver its final usage notification with the exact + native session and turn token before prompt settlement. Stream or process + closure ends that receipt window. Missing charges remain unpriced. - Failed, interrupted, unsupported asynchronous and background delegated work leave measurement totals incomplete. Known positive reported subtotals survive, but missing charges remain unpriced. Explicit reported zero is accepted only for diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts index dbeba649296..5836e1a1504 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts @@ -29,6 +29,45 @@ const HANDLE: AcpRuntimeHandle = { }; describe("Codex ACPX runtime adapter", () => { + it("accepts only the owning Hermes terminal usage after Stop and before prompt settlement", async () => { + const pending = pendingExtensionTurn("turn-1"); + const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn); + let created!: AcpRuntimeOptions; + const options = openOptions(fakeCommand()); + options.profile = { ...options.profile, agent: "hermes" }; + const port = await openCodexAcpxRuntime(options, { + createRegistry: () => registry(), createStore: () => store(), createRuntime: value => { created = value; return runtime; }, + }); + const notification = vi.fn(); + const request = vi.fn(); + const turn = port.startTurn({ text: "Work", requestId: "turn-1", onExtensionRequest: request, onExtensionNotification: notification }); + const identity = { version: 1, sessionId: "backend-1", turnToken: "00000000-0000-0000-0000-000000000001" }; + created.onExtensionNotification!("_hermes/turn_started", identity); + await turn.cancel(); + created.onExtensionNotification!("_hermes/delegation", identity); + await expect(created.onExtensionRequest!("_hermes/ask_questions", identity, + { requestId: 0, signal: new AbortController().signal })).rejects.toThrow("admitted active turn"); + for (const foreign of [{ ...identity, sessionId: "foreign" }, { ...identity, sessionId: "agent-1" }, { ...identity, turnToken: "00000000-0000-0000-0000-000000000002" }]) { + expect(() => created.onExtensionNotification!("_hermes/usage", foreign)).not.toThrow(); + } + expect(notification).not.toHaveBeenCalled(); + const receipt = { ...identity, tokens: "reported", cost: "unavailable", billing: { + schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD", + complete: true, requestCount: 1, reportedRequestCount: 1, amountUsd: 0.0042, amountUsdExact: "0.004200000", + } }; + created.onExtensionNotification!("_hermes/usage", receipt); + expect(notification).toHaveBeenCalledExactlyOnceWith("_hermes/usage", receipt); + expect(request).not.toHaveBeenCalled(); + await turn.closeStream(); + await turn.cancel(); + created.onExtensionNotification!("_hermes/usage", receipt); + expect(notification).toHaveBeenCalledTimes(1); + pending.settle(); await turn.result; + created.onExtensionNotification!("_hermes/usage", receipt); + expect(notification).toHaveBeenCalledTimes(1); + await port.close({ reason: "terminal billing boundary verified" }); + }); + it("rejects forged permission session identifiers before delegating or applying full-auto policy", async () => { const pending = pendingExtensionTurn("turn-1"); const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn); diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts index 51a0d542d7f..c1b22411645 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts @@ -101,6 +101,8 @@ interface AcpxRuntimeExtensionBoundary { active: AcpxRuntimeExtensionTurn | null; sessionIds: Set; steering: AcpxRuntimeSteeringCapability | null; + cancelledReceiptOwner: AcpxRuntimeExtensionTurn | null; + receiptAdmissionClosed: boolean; } class AcpxRuntimeCloseTimeoutError extends Error { @@ -278,6 +280,7 @@ export async function openQualifiedAcpxRuntime( const extensionNotifications = new Set(extensionProfile.extensionNotifications); const extensionBoundary: AcpxRuntimeExtensionBoundary = { agent: options.profile.agent, active: null, sessionIds: new Set(), steering: null, + cancelledReceiptOwner: null, receiptAdmissionClosed: false, }; const ownsExtensionTurn = (active: AcpxRuntimeExtensionTurn, params: Record): boolean => extensionBoundary.active === active && !active.signal.aborted @@ -354,7 +357,15 @@ export async function openQualifiedAcpxRuntime( active.nativeTurnToken = params.turnToken; return; } - if (!extensionNotifications.has(method) || !active?.onNotification || !ownsExtensionTurn(active, params)) return; + // Stop revokes requests and activity immediately, but the native prompt + // still settles its owned work and billing. Admit only its final usage + // notification, with the exact negotiated session/token, until result + // settlement. Stream/process closure never opens this receipt window. + const cancelledUsage = options.profile.agent === "hermes" && method === "_hermes/usage" + && active !== null && extensionBoundary.cancelledReceiptOwner === active && extensionBoundary.active === active + && params.sessionId === active.sessionId && params.version === 1 + && !!active.nativeTurnToken && params.turnToken === active.nativeTurnToken; + if (!extensionNotifications.has(method) || !active?.onNotification || (!ownsExtensionTurn(active, params) && !cancelledUsage)) return; if (options.profile.agent === "hermes" && (params.version !== 1 || !active.nativeTurnToken || params.turnToken !== active.nativeTurnToken)) throw new Error("Hermes activity belongs to a stale turn"); active.onNotification(method, params.sessionId === undefined ? { ...params, sessionId: active.sessionId } : params); }, @@ -1204,6 +1215,8 @@ function runtimePort( attemptNumber: number; }; }): Promise { + extensionBoundary.receiptAdmissionClosed = true; + extensionBoundary.cancelledReceiptOwner = null; extensionBoundary.active?.controller.abort(new Error("ACPX runtime is closing")); if (runtimeClosed) return; if ( @@ -1399,7 +1412,10 @@ function runtimePort( onRequest: input.onExtensionRequest, onNotification: input.onExtensionNotification, }; extensionBoundary.active = extensionTurn; + let receiptWindowClosed = false; const releaseExtensionTurn = (): void => { + receiptWindowClosed = true; + if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null; controller.abort(new Error("ACPX extension turn expired")); if (extensionBoundary.active === extensionTurn) extensionBoundary.active = null; }; @@ -1450,10 +1466,13 @@ function runtimePort( ...guarded, result, cancel: (input) => { + if (!extensionBoundary.receiptAdmissionClosed && !receiptWindowClosed && extensionBoundary.active === extensionTurn) extensionBoundary.cancelledReceiptOwner = extensionTurn; controller.abort(new Error("ACPX extension turn cancelled")); return guarded.cancel(input); }, closeStream: (input) => { + receiptWindowClosed = true; + if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null; controller.abort(new Error("ACPX extension stream closed")); return guarded.closeStream(input); },