fix(hermes): retain the owned billing receipt during Stop

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-08 12:28:13 -05:00
1 parent d4eee95165
commit ec40b1eb32
6 files changed
+100 -41

No files matched your search

-8
View File
@@ -617,14 +617,6 @@ fingerprint; secrets travel through the ephemeral credential channel. Implemente
projections remain subject to per-method live qualification. See the
[Hermes runner contract](../../packages/paperclip-runner/docs/hermes.md).
The Hermes runner's optional per-turn billing receipt currently recognizes
provider-reported OpenRouter `usage.cost` only on its selected managed Chat
Completions route. Account attribution still comes from the authoritative
connection projection. Token totals, displayed native price estimates, and
cumulative session costs do not prove billed spending. Incomplete wire attempts
retain known positive subtotals and leave missing charges unpriced; other Hermes
providers and protocols remain pending billing qualification.
OpenRouter connections without an explicit model list automatically load its public
[model catalog](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties),
ordered with `sort=most-popular`. New-agent setup and agent settings share this
-32
View File
@@ -4,31 +4,9 @@ The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER-
is a Product E2E workflow for fresh subscription/API-key/gateway connections,
with attended login and independent artifact checks against local or staging targets.
The explicit-only [Hermes managed API suite](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only)
checks native completion through five selected managed API accounts on local
and Daytona, with independent account/model attribution. Its registered cells
remain pending until their live evidence passes; its API-only scope leaves the
subscription and lifecycle gates separate.
The launcher records clean Git source before credential loading for Hermes
candidates and rejects a mismatched requested SHA. Its source receipt and normal
result metadata identify the controller; they do not qualify runtime behavior.
The same guide registers the separate explicit-only `hermes-bedrock-connections`
suite for one exact `us-east-1` inference profile on local and Daytona. Its two
cells remain pending. They verify the explicitly selected personal grant and
public region/protocol/model routing before and after native completion, using
a short-lived AWS bearer credential and the existing budget/evidence pipeline.
Paperclip has two live eval families with different questions, owners, and
evidence. Choose the family before selecting a model, profile, or case.
The explicit-only `hermes-native-interactions` Product E2E suite adds a native
three-question batch on local and Daytona. It checks browser reconnect within
the active run, exact native callback delivery, late duplicate rejection and a
final reply derived from the returned reviewer text. The registered cells remain
pending; assigned Paperclip question tools do not satisfy this native oracle.
See the [Hermes fixture contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only).
The explicit-only [native instruction consolidation comparison](plans/2026-10-03-native-completion-consolidation.md)
uses six Product E2E cells per source variant. It measures the completion
constraint reduction separately from the earlier native tool-description
@@ -447,13 +425,3 @@ See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmati
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills.
The native interaction suite also declares one local `native-question-batch-stop`
cell. It leaves a complete native callback unanswered, retains the pending
request, clicks the browser Stop control, and independently requires the exact
cancelled callback/turn and audited public Stop acknowledgement. The task must
remain In Progress, a late answer must receive HTTP 409, and the public per-turn
process owner and observed descendants must retire before cleanup and remain
retired through cleanup. Cancelled-run account attribution and complete reported
OpenRouter billing are required. This is a separate pending release gate; remote
Stop still requires its own retirement observer and actual Daytona proof.
@@ -1416,3 +1416,40 @@ fixture attempts remain separate: one checked the late answer prematurely and
failed during shutdown; another placed the workspace inside its isolated host
home and was rejected before launch. Neither is live-provider proof. A fresh
paid browser Stop campaign and the remaining release gates are still required.
### 2026-10-08 native Stop projection and terminal receipt fixes
The original `759f7bfccd` paid Stop attempt remains failed and is not regraded.
Its native callback and turn cancelled, its audited browser Stop returned HTTP
200, and all nine observed process identities retired. The card stayed pending,
generic stranded-work recovery later blocked the task, and the cancelled run's
cost stayed unpriced. No held reservation is released from these fixes.
The server now projects the exact native input cancellation into its question
card, checking company, task, agent, run, session, runner, turn, item and the
committed creation. The pending-card compare-and-update cannot overwrite an
answer, replay cannot create another expiration, and the transaction locks the
task before the card. Both local and durable PRP observation use this projection.
Unrelated historical questions and provider-loss handoffs retain their lifecycle.
Recovery checks deliberate operator Stop before escalating an uninvokable
assignee, so a later agent pause cannot turn that Stop into stranded work.
The ACP boundary had discarded Hermes's final usage notification once Stop
aborted its control signal. Only that admitted prompt's final usage can now
cross the cancelled boundary, with its exact native session and negotiated turn
token, before prompt settlement. Requests and other activity remain closed.
Stream/process closure ends the receipt window and cannot reopen it. The native
wire ledger and completeness rules are unchanged; missing charges stay unknown.
Credential-free regression evidence: the paused-agent recovery test first failed,
then its full suite passed 79 tests. Native card cancellation first failed seven
checks, then the final suite passed 38 tests, including nine identity denials,
replay, late-answer rejection, preserved historical answers and a concurrent
answer/Stop race. The billing-boundary regression first failed because no receipt
reached its owner; the affected ACP/usage suites passed all 110 tests after repair.
Direct server and Runner TypeScript checks passed. An intermediate implementation
scope error is retained as a failed test attempt; it is not qualification proof.
No local Docker, Rust build, secret access or paid provider call was used for this
repair. A fresh committed-source browser Stop attempt and complete release gates
are still required. The PR stack remains draft and Hermes remains pending.
+4
View File
@@ -116,6 +116,10 @@ extend the task's execution deadline.
selected OpenRouter billing identity and the current native turn before
using it for accounting. Cumulative session cost and native model-price
estimates cannot supply this receipt.
- During Stop, requests and steering close immediately. The same admitted
Hermes prompt may still deliver its final usage notification with the exact
native session and turn token before prompt settlement. Stream or process
closure ends that receipt window. Missing charges remain unpriced.
- Failed, interrupted, unsupported asynchronous and background delegated work
leave measurement totals incomplete. Known positive reported subtotals survive, but
missing charges remain unpriced. Explicit reported zero is accepted only for
@@ -29,6 +29,45 @@ const HANDLE: AcpRuntimeHandle = {
};
describe("Codex ACPX runtime adapter", () => {
it("accepts only the owning Hermes terminal usage after Stop and before prompt settlement", async () => {
const pending = pendingExtensionTurn("turn-1");
const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn);
let created!: AcpRuntimeOptions;
const options = openOptions(fakeCommand());
options.profile = { ...options.profile, agent: "hermes" };
const port = await openCodexAcpxRuntime(options, {
createRegistry: () => registry(), createStore: () => store(), createRuntime: value => { created = value; return runtime; },
});
const notification = vi.fn();
const request = vi.fn();
const turn = port.startTurn({ text: "Work", requestId: "turn-1", onExtensionRequest: request, onExtensionNotification: notification });
const identity = { version: 1, sessionId: "backend-1", turnToken: "00000000-0000-0000-0000-000000000001" };
created.onExtensionNotification!("_hermes/turn_started", identity);
await turn.cancel();
created.onExtensionNotification!("_hermes/delegation", identity);
await expect(created.onExtensionRequest!("_hermes/ask_questions", identity,
{ requestId: 0, signal: new AbortController().signal })).rejects.toThrow("admitted active turn");
for (const foreign of [{ ...identity, sessionId: "foreign" }, { ...identity, sessionId: "agent-1" }, { ...identity, turnToken: "00000000-0000-0000-0000-000000000002" }]) {
expect(() => created.onExtensionNotification!("_hermes/usage", foreign)).not.toThrow();
}
expect(notification).not.toHaveBeenCalled();
const receipt = { ...identity, tokens: "reported", cost: "unavailable", billing: {
schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
complete: true, requestCount: 1, reportedRequestCount: 1, amountUsd: 0.0042, amountUsdExact: "0.004200000",
} };
created.onExtensionNotification!("_hermes/usage", receipt);
expect(notification).toHaveBeenCalledExactlyOnceWith("_hermes/usage", receipt);
expect(request).not.toHaveBeenCalled();
await turn.closeStream();
await turn.cancel();
created.onExtensionNotification!("_hermes/usage", receipt);
expect(notification).toHaveBeenCalledTimes(1);
pending.settle(); await turn.result;
created.onExtensionNotification!("_hermes/usage", receipt);
expect(notification).toHaveBeenCalledTimes(1);
await port.close({ reason: "terminal billing boundary verified" });
});
it("rejects forged permission session identifiers before delegating or applying full-auto policy", async () => {
const pending = pendingExtensionTurn("turn-1");
const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn);
@@ -101,6 +101,8 @@ interface AcpxRuntimeExtensionBoundary {
active: AcpxRuntimeExtensionTurn | null;
sessionIds: Set<string>;
steering: AcpxRuntimeSteeringCapability | null;
cancelledReceiptOwner: AcpxRuntimeExtensionTurn | null;
receiptAdmissionClosed: boolean;
}
class AcpxRuntimeCloseTimeoutError extends Error {
@@ -278,6 +280,7 @@ export async function openQualifiedAcpxRuntime(
const extensionNotifications = new Set(extensionProfile.extensionNotifications);
const extensionBoundary: AcpxRuntimeExtensionBoundary = {
agent: options.profile.agent, active: null, sessionIds: new Set(), steering: null,
cancelledReceiptOwner: null, receiptAdmissionClosed: false,
};
const ownsExtensionTurn = (active: AcpxRuntimeExtensionTurn, params: Record<string, unknown>): boolean =>
extensionBoundary.active === active && !active.signal.aborted
@@ -354,7 +357,15 @@ export async function openQualifiedAcpxRuntime(
active.nativeTurnToken = params.turnToken;
return;
}
if (!extensionNotifications.has(method) || !active?.onNotification || !ownsExtensionTurn(active, params)) return;
// Stop revokes requests and activity immediately, but the native prompt
// still settles its owned work and billing. Admit only its final usage
// notification, with the exact negotiated session/token, until result
// settlement. Stream/process closure never opens this receipt window.
const cancelledUsage = options.profile.agent === "hermes" && method === "_hermes/usage"
&& active !== null && extensionBoundary.cancelledReceiptOwner === active && extensionBoundary.active === active
&& params.sessionId === active.sessionId && params.version === 1
&& !!active.nativeTurnToken && params.turnToken === active.nativeTurnToken;
if (!extensionNotifications.has(method) || !active?.onNotification || (!ownsExtensionTurn(active, params) && !cancelledUsage)) return;
if (options.profile.agent === "hermes" && (params.version !== 1 || !active.nativeTurnToken || params.turnToken !== active.nativeTurnToken)) throw new Error("Hermes activity belongs to a stale turn");
active.onNotification(method, params.sessionId === undefined ? { ...params, sessionId: active.sessionId } : params);
},
@@ -1204,6 +1215,8 @@ function runtimePort(
attemptNumber: number;
};
}): Promise<void> {
extensionBoundary.receiptAdmissionClosed = true;
extensionBoundary.cancelledReceiptOwner = null;
extensionBoundary.active?.controller.abort(new Error("ACPX runtime is closing"));
if (runtimeClosed) return;
if (
@@ -1399,7 +1412,10 @@ function runtimePort(
onRequest: input.onExtensionRequest, onNotification: input.onExtensionNotification,
};
extensionBoundary.active = extensionTurn;
let receiptWindowClosed = false;
const releaseExtensionTurn = (): void => {
receiptWindowClosed = true;
if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null;
controller.abort(new Error("ACPX extension turn expired"));
if (extensionBoundary.active === extensionTurn) extensionBoundary.active = null;
};
@@ -1450,10 +1466,13 @@ function runtimePort(
...guarded,
result,
cancel: (input) => {
if (!extensionBoundary.receiptAdmissionClosed && !receiptWindowClosed && extensionBoundary.active === extensionTurn) extensionBoundary.cancelledReceiptOwner = extensionTurn;
controller.abort(new Error("ACPX extension turn cancelled"));
return guarded.cancel(input);
},
closeStream: (input) => {
receiptWindowClosed = true;
if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null;
controller.abort(new Error("ACPX extension stream closed"));
return guarded.closeStream(input);
},