mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 01:41:42 +02:00
fix(hermes): retain per-turn reported OpenRouter billing
Capture pinned SDK wire attempts, carry closed versioned receipts through ACPX and PRP, and bind reported subtotals to native turn accounting. Keep incomplete attempts unpriced and require settled cost plus budget health in the live OpenRouter oracle. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
01855199ad
commit
e5afbd7107
38 files changed
+1286
-48
No files matched your search
@@ -617,6 +617,14 @@ fingerprint; secrets travel through the ephemeral credential channel. Implemente
|
||||
projections remain subject to per-method live qualification. See the
|
||||
[Hermes runner contract](../../packages/paperclip-runner/docs/hermes.md).
|
||||
|
||||
The Hermes runner's optional per-turn billing receipt currently recognizes
|
||||
provider-reported OpenRouter `usage.cost` only on its selected managed Chat
|
||||
Completions route. Account attribution still comes from the authoritative
|
||||
connection projection. Token totals, displayed native price estimates, and
|
||||
cumulative session costs do not prove billed spending. Incomplete wire attempts
|
||||
retain known positive subtotals and leave missing charges unpriced; other Hermes
|
||||
providers and protocols remain pending billing qualification.
|
||||
|
||||
OpenRouter connections without an explicit model list automatically load its public
|
||||
[model catalog](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties),
|
||||
ordered with `sort=most-popular`. New-agent setup and agent settings share this
|
||||
|
||||
@@ -864,3 +864,46 @@ that digest and reports the actual dirty state instead of calling the working
|
||||
tree clean. Other edits, staged replacements, missing/mismatched approval,
|
||||
deletions and symlinks remain rejected. All eighteen source-admission tests and
|
||||
Product E2E TypeScript pass. The root lockfile itself remains unchanged.
|
||||
|
||||
### 2026-10-07 reported-cost qualification work
|
||||
|
||||
The source/approved-lock correction at `805d787b` has 54 passing checks, two
|
||||
intentional skips, a fresh Greptile 5/5 and no open review threads. The live Mac
|
||||
budget pause above remains historical evidence, not a successful billing result.
|
||||
|
||||
The managed bridge now negotiates optional v1 wire billing for the selected
|
||||
OpenRouter Chat Completions account. It observes the pinned synchronous SDK
|
||||
without changing its requests or loop. Every inference attempt enters a
|
||||
turn-owned ledger, including SDK retries and synchronous auxiliary calls.
|
||||
Completed response `usage.cost` amounts become an exact nine-decimal USD
|
||||
subtotal. Missing/failed/interrupted attempts, unsupported asynchronous calls
|
||||
and background delegation keep settlement incomplete. Positive known spend
|
||||
survives; absent charges never become free work. Other provider/protocol paths
|
||||
retain unavailable billed cost. Credential/header values do not enter receipts.
|
||||
|
||||
The optional closed receipt crosses the ACP extension, both runner drivers,
|
||||
Rust PRP normalization, replay and controller accounting. The controller binds
|
||||
it to the selected biller and current turn, keeps cumulative estimates separate,
|
||||
and rebuilds totals without duplicate charging on replay. Compaction resets of
|
||||
native counters cannot replace wire totals. No budget safeguard was relaxed.
|
||||
The public OpenRouter completion oracle now independently requires reported
|
||||
settled cost and healthy company/agent budget state before cleanup.
|
||||
|
||||
Credential-free proof passed: 34 pinned Python tests, 63 focused Runner Vitest
|
||||
tests, 16 Rust event tests, all 608 native executor tests, 28 Node protocol/build
|
||||
checks and 87 source/API/Bedrock oracle tests. Runner, server and Product E2E
|
||||
TypeScript and Rust formatting passed. Both Mac production transport fixtures
|
||||
passed with the freshly built runner: Rust/image/semantic completion in 33.043
|
||||
seconds and native streaming/restore/tools/questions/planning/memory/steering/
|
||||
stop in 82.158 seconds. These use a deterministic loopback model and no
|
||||
credentials; they are not paid billing or full-stack browser proof.
|
||||
|
||||
The reviewed closure changes only the bridge and added billing module. Both
|
||||
previous platform manifests were authenticated against their committed pins;
|
||||
the task-owned Mac asset was verified file by file before and after the update.
|
||||
The new Mac closure is `9f1af058a963305ccd5c4f1555f2828458f72677ceacea2de456ac503f7212de`;
|
||||
the derived Linux closure is `dd918ec15f5bd8025f3d01c4f2849eac4de8d59e237a520607f9c5f988a94645`.
|
||||
Fresh Linux materialization, current-head CI/review, live reported-cost browser
|
||||
proof and broader original release gates remain required. Existing cloud images
|
||||
and prior results keep their old runtime/source identities. No local Docker,
|
||||
paid inference or additional reservation was used for these offline checks.
|
||||
@@ -106,8 +106,22 @@ extend the task's execution deadline.
|
||||
- Restored history is recorded without becoming new assistant text or live tools.
|
||||
- Restoration requires nonempty native history and follows the native
|
||||
compaction chain. Missing history and failed persistence are errors.
|
||||
- Usage is a per-prompt delta. Native price calculations are labeled estimates;
|
||||
missing receipts and unverified billed cost are unavailable, not zero.
|
||||
- Usage is a per-prompt delta. Clients negotiate `billingReceipts: 1` inside
|
||||
`_meta.paperclipHermes`; the optional closed `paperclip.usage.billing/v1`
|
||||
receipt remains replay-compatible with older events.
|
||||
- For the managed OpenRouter Chat Completions route, the bridge observes the
|
||||
pinned SDK's wire responses and sums provider-reported `usage.cost`, including
|
||||
retries and auxiliary synchronous calls. The receipt carries request counts,
|
||||
completeness and an exact nine-decimal USD amount. Paperclip binds it to the
|
||||
selected OpenRouter billing identity and the current native turn before
|
||||
using it for accounting. Cumulative session cost and native model-price
|
||||
estimates cannot supply this receipt.
|
||||
- Failed, interrupted, unsupported asynchronous and background delegated work
|
||||
leave settlement incomplete. Known positive reported subtotals survive, but
|
||||
missing charges remain unpriced. Explicit reported zero is accepted only for
|
||||
a complete receipt. Other provider/protocol routes retain their existing
|
||||
unavailable billed-cost behavior. Complete live billing, including delegated
|
||||
work, remains a qualification gate; these checks do not qualify a provider.
|
||||
|
||||
The managed execution middleware applies planning/read-only, protected-path and
|
||||
permission policy to inline, concurrent and delegated native tools. Assigned
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
{
|
||||
"path": "schemas/event-v2.schema.json",
|
||||
"id": "https://paperclip.dev/schemas/prp/v2/event.schema.json",
|
||||
"sha256": "9dcf149bdb8e10041ca2212037231e7ceff11be08a0b8ee5dbd0f21974daa294"
|
||||
"sha256": "f37013b70d768076473bef1cf864c7aa5979f543b389fb69ac77be010bb2122e"
|
||||
},
|
||||
{
|
||||
"path": "schemas/event-v3.schema.json",
|
||||
@@ -60,7 +60,7 @@
|
||||
{
|
||||
"path": "schemas/event.schema.json",
|
||||
"id": "https://paperclip.dev/schemas/prp/v1/event.schema.json",
|
||||
"sha256": "d48916630976ff7e850b6e2d7d94131761fda322f5d8a05212651ec46b0db19d"
|
||||
"sha256": "24f026d2513d80a7ff938df4e18cd131e77b70357369bc629e6925408435bc20"
|
||||
},
|
||||
{
|
||||
"path": "schemas/fixture.schema.json",
|
||||
@@ -135,7 +135,7 @@
|
||||
{
|
||||
"path": "schemas/usage.schema.json",
|
||||
"id": "https://paperclip.dev/schemas/prp/v1/usage.schema.json",
|
||||
"sha256": "34b1478d2054f898d64b54b62a9bb738670087893bb1e5967da541539d2daf36"
|
||||
"sha256": "998e8217adcff554e38fc697a7f408dfffe08dd59760bb248fb3c75c8f4217dc"
|
||||
},
|
||||
{
|
||||
"path": "schemas/workspace-diff.schema.json",
|
||||
|
||||
@@ -46,6 +46,8 @@
|
||||
"debug": { "type": "object", "additionalProperties": true }
|
||||
},
|
||||
"allOf": [
|
||||
{ "if": { "properties": { "eventType": { "const": "usage.reported" } } }, "then": { "properties": { "payload": { "type": "object", "properties": { "billing": { "$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing" } } } } } },
|
||||
{ "if": { "properties": { "payload": { "type": "object", "required": ["kind", "usage"], "properties": { "kind": { "const": "usage" }, "usage": { "type": "object", "required": ["billing"] } } } } }, "then": { "properties": { "payload": { "type": "object", "properties": { "usage": { "type": "object", "properties": { "billing": { "$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing" } } } } } } } },
|
||||
{
|
||||
"if": { "properties": { "eventType": { "enum": ["session.goal.snapshot", "session.goal.updated"] } } },
|
||||
"then": {
|
||||
|
||||
@@ -69,6 +69,8 @@
|
||||
"debug": { "type": "object", "additionalProperties": true }
|
||||
},
|
||||
"allOf": [
|
||||
{ "if": { "properties": { "eventType": { "const": "usage.reported" } } }, "then": { "properties": { "payload": { "type": "object", "properties": { "billing": { "$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing" } } } } } },
|
||||
{ "if": { "properties": { "payload": { "type": "object", "required": ["kind", "usage"], "properties": { "kind": { "const": "usage" }, "usage": { "type": "object", "required": ["billing"] } } } } }, "then": { "properties": { "payload": { "type": "object", "properties": { "usage": { "type": "object", "properties": { "billing": { "$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing" } } } } } } } },
|
||||
{ "if": { "properties": { "eventType": { "const": "plan.updated" } } }, "then": { "properties": { "payload": { "$ref": "https://paperclip.dev/schemas/prp/v1/provider-event.schema.json#/$defs/plan" } } } },
|
||||
{ "if": { "properties": { "eventType": { "enum": ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"] } } }, "then": { "properties": { "payload": { "$ref": "https://paperclip.dev/schemas/prp/v1/provider-event.schema.json#/$defs/toolExecution" } } } },
|
||||
{ "if": { "properties": { "eventType": { "enum": ["research.started", "research.progressed", "research.completed"] } } }, "then": { "properties": { "payload": { "$ref": "https://paperclip.dev/schemas/prp/v1/provider-event.schema.json#/$defs/research" } } } },
|
||||
|
||||
@@ -11,10 +11,27 @@
|
||||
"providerRequestId": { "type": ["string", "null"], "maxLength": 240 },
|
||||
"cumulative": { "$ref": "#/$defs/measurement" },
|
||||
"runDeltaAvailable": { "type": "boolean" },
|
||||
"billing": { "$ref": "#/$defs/billing" },
|
||||
"runDelta": { "$ref": "#/$defs/measurement" }
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"$defs": {
|
||||
"billing": {
|
||||
"type": "object",
|
||||
"required": ["schema", "source", "biller", "currency", "complete", "requestCount", "reportedRequestCount", "amountUsd", "amountUsdExact"],
|
||||
"properties": {
|
||||
"schema": { "const": "paperclip.usage.billing/v1" },
|
||||
"source": { "const": "provider_reported" },
|
||||
"biller": { "const": "openrouter" },
|
||||
"currency": { "const": "USD" },
|
||||
"complete": { "type": "boolean" },
|
||||
"requestCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
|
||||
"reportedRequestCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
|
||||
"amountUsd": { "type": "number", "minimum": 0, "maximum": 1000000 },
|
||||
"amountUsdExact": { "type": "string", "pattern": "^(0|[1-9][0-9]{0,6})\\.[0-9]{9}$" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"measurement": {
|
||||
"type": "object",
|
||||
"required": ["inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "activeSeconds", "requests", "providerCostUsd"],
|
||||
|
||||
@@ -669,6 +669,51 @@ fn measurement(value: &Value) -> Value {
|
||||
})
|
||||
}
|
||||
|
||||
/// Optional, closed per-turn billing authority; legacy placeholder cost stays unknown.
|
||||
fn provider_usage_billing(value: Option<&Value>) -> Option<Value> {
|
||||
let value = value?;
|
||||
let object = value.as_object()?;
|
||||
if object.len() != 9
|
||||
|| value["schema"] != "paperclip.usage.billing/v1"
|
||||
|| value["source"] != "provider_reported"
|
||||
|| value["biller"] != "openrouter"
|
||||
|| value["currency"] != "USD"
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let complete = value["complete"].as_bool()?;
|
||||
let requests = value["requestCount"].as_u64()?;
|
||||
let reported = value["reportedRequestCount"].as_u64()?;
|
||||
let amount = value["amountUsd"].as_f64()?;
|
||||
let exact = value["amountUsdExact"].as_str()?;
|
||||
let (whole, fraction) = exact.split_once('.')?;
|
||||
if whole.is_empty()
|
||||
|| whole.len() > 7
|
||||
|| (whole.len() > 1 && whole.starts_with('0'))
|
||||
|| !whole.bytes().all(|c| c.is_ascii_digit())
|
||||
|| fraction.len() != 9
|
||||
|| !fraction.bytes().all(|c| c.is_ascii_digit())
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let units = whole
|
||||
.parse::<u64>()
|
||||
.ok()?
|
||||
.checked_mul(1_000_000_000)?
|
||||
.checked_add(fraction.parse::<u64>().ok()?)?;
|
||||
if requests > 9_007_199_254_740_991
|
||||
|| reported > requests
|
||||
|| (complete && reported != requests)
|
||||
|| units > 1_000_000_000_000_000
|
||||
|| !amount.is_finite()
|
||||
|| amount != units as f64 / 1_000_000_000.0
|
||||
|| (reported == 0 && amount != 0.0)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
Some(value.clone())
|
||||
}
|
||||
|
||||
/// Converts Codex app-server notifications into provider-neutral PRP events.
|
||||
/// Provider-native envelopes are consumed here and never cross the PRP boundary.
|
||||
pub fn normalize_codex_notification(method: &str, params: &Value) -> Vec<NormalizedProviderEvent> {
|
||||
@@ -1144,6 +1189,7 @@ fn normalize_acpx_status(
|
||||
) -> Vec<NormalizedProviderEvent> {
|
||||
let tag = string(payload.get("tag"));
|
||||
if tag == "usage_update" {
|
||||
let billing = provider_usage_billing(payload.get("billing"));
|
||||
let breakdown = payload.get("breakdown").unwrap_or(&Value::Null);
|
||||
let cache_read = breakdown
|
||||
.get("cachedReadTokens")
|
||||
@@ -1208,23 +1254,24 @@ fn normalize_acpx_status(
|
||||
),
|
||||
"activeSeconds": 0.0,
|
||||
"requests": 1,
|
||||
"providerCostUsd": 0.0,
|
||||
"providerCostUsd": billing.as_ref().map(|receipt| receipt["amountUsd"].clone()).unwrap_or(json!(0.0)),
|
||||
});
|
||||
let mut usage_payload = json!({
|
||||
"provider": "acpx",
|
||||
"model": payload.get("model").and_then(Value::as_str).map(|value| bounded_text(value, 240)),
|
||||
"providerSessionId": Value::Null,
|
||||
"providerRequestId": Value::Null,
|
||||
"cumulative": cumulative,
|
||||
"runDeltaAvailable": run_delta_available,
|
||||
"runDelta": run_delta,
|
||||
});
|
||||
if let Some(billing) = billing {
|
||||
usage_payload["billing"] = billing;
|
||||
}
|
||||
return vec![NormalizedProviderEvent {
|
||||
event_type: "usage.reported".to_owned(),
|
||||
priority: EventPriority::P0,
|
||||
payload: json!({
|
||||
"provider": "acpx",
|
||||
"model": payload
|
||||
.get("model")
|
||||
.and_then(Value::as_str)
|
||||
.map(|value| bounded_text(value, 240)),
|
||||
"providerSessionId": Value::Null,
|
||||
"providerRequestId": Value::Null,
|
||||
"cumulative": cumulative,
|
||||
"runDeltaAvailable": run_delta_available,
|
||||
"runDelta": run_delta,
|
||||
}),
|
||||
payload: usage_payload,
|
||||
}];
|
||||
}
|
||||
if tag == "current_mode_update" {
|
||||
|
||||
@@ -287,6 +287,39 @@ fn maps_usage_and_review_status_but_ignores_inventory_updates() {
|
||||
.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retains_only_closed_per_turn_reported_billing_without_reusing_cumulative_cost() {
|
||||
let billing = json!({"schema":"paperclip.usage.billing/v1","source":"provider_reported", "biller":"openrouter", "currency":"USD",
|
||||
"complete":true,"requestCount":2,"reportedRequestCount":2,"amountUsd":0.0042,"amountUsdExact":"0.004200000"});
|
||||
let status = json!({"type":"status","tag":"usage_update","billing":billing,
|
||||
"breakdown":{"inputTokens":12,"outputTokens":4,"thoughtTokens":0,"cachedReadTokens":2,"cachedWriteTokens":0},
|
||||
"cost":{"amount":999}});
|
||||
let usage = normalize(AcpxRuntimeEventKind::Status, status.clone());
|
||||
assert_eq!(usage[0].payload["billing"], billing);
|
||||
assert_eq!(usage[0].payload["runDelta"]["providerCostUsd"], 0.0042);
|
||||
assert_eq!(usage[0].payload["cumulative"]["providerCostUsd"], 999.0);
|
||||
let mut partial = status.clone();
|
||||
partial["billing"]["complete"] = json!(false);
|
||||
partial["billing"]["reportedRequestCount"] = json!(1);
|
||||
assert_eq!(
|
||||
normalize(AcpxRuntimeEventKind::Status, partial)[0].payload["billing"]["complete"],
|
||||
false
|
||||
);
|
||||
for (key, value) in [
|
||||
("source", json!("estimate")),
|
||||
("currency", json!("EUR")),
|
||||
("amountUsdExact", json!("0.004200001")),
|
||||
("reportedRequestCount", json!(1)),
|
||||
("credential", json!("private")),
|
||||
] {
|
||||
let mut invalid = status.clone();
|
||||
invalid["billing"][key] = value;
|
||||
let projected = normalize(AcpxRuntimeEventKind::Status, invalid);
|
||||
assert!(projected[0].payload.get("billing").is_none());
|
||||
assert_eq!(projected[0].payload["runDelta"]["providerCostUsd"], 0.0);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn does_not_claim_missing_or_partial_usage_breakdowns_are_exact() {
|
||||
for payload in [
|
||||
|
||||
@@ -56,7 +56,7 @@ for item in (source / ".venv/lib/python3.12/site-packages").iterdir():
|
||||
elif item.suffix not in (".pyc", ".pyo"):
|
||||
shutil.copy2(item, site / item.name)
|
||||
copy_tree(source, destination / "app", (".venv", ".git", ".github", "tests", "website", "docs", ".pytest_cache"))
|
||||
for name in ("bridge.py", "policy.py", "tool_process.py", "entry.py", "version.json"):
|
||||
for name in ("bridge.py", "billing.py", "policy.py", "tool_process.py", "entry.py", "version.json"):
|
||||
shutil.copy2(provider / name, destination / name)
|
||||
# Installed RECORD and direct_url metadata can contain mutable build locations.
|
||||
for item in destination.rglob("*.dist-info/direct_url.json"):
|
||||
|
||||
@@ -93,6 +93,7 @@ test('pinned Hermes streams images and semantic completion through Rust PRP and
|
||||
attachments: [{ schema: 'paperclip.user_attachment.v1', kind: 'image', name: 'pixel.png', mediaType: 'image/png', data: imageData }],
|
||||
});
|
||||
bundle = createRunnerdCodexTransport({
|
||||
runnerBinary: process.env.PAPERCLIP_RUNNER_BINARY,
|
||||
provider: 'acpx', acpxAgent: 'hermes', acpxCandidateProfile: 'hermes', acpxPermissionMode: 'approve-all',
|
||||
stateDirectory: join(root, 'runner'), acpxRuntimeDirectory: join(root, 'runtime'), runtimeContext,
|
||||
lifecyclePolicy: { mode: 'per_turn', idleTimeoutMs: null }, turnStartTimeoutMs: 90_000,
|
||||
|
||||
@@ -2,16 +2,42 @@
|
||||
// This is transport proof, not live model or product qualification.
|
||||
// PAPERCLIP_HERMES_QUALIFY=1 node --test scripts/qualify-hermes-runtime.test.mjs
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import { mkdir, mkdtemp, rm, readFile, readdir } from "node:fs/promises";
|
||||
import { createServer } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import test from "node:test";
|
||||
import { AcpxRuntimeHost } from "../dist/drivers/acpx/runtime-host.js";
|
||||
import { openCodexAcpxRuntime } from "../dist/drivers/acpx/codex-runtime-adapter.js";
|
||||
import { acpxProfileClientCapabilities } from "../dist/drivers/acpx/profile-extensions.js";
|
||||
import { acpxProviderSessionIdentity } from "../dist/drivers/acpx/recovery-identity.js";
|
||||
import { HERMES_CLOSURES } from "../dist/drivers/acpx/hermes-distributions.js";
|
||||
import { verifyHermesRuntimeFiles } from "../dist/drivers/acpx/hermes-setup-integrity.js";
|
||||
|
||||
test("pinned Python qualifies Hermes bridge and wire billing without credentials", {
|
||||
skip: process.env.PAPERCLIP_HERMES_QUALIFY !== "1", timeout: 60_000,
|
||||
}, async t => {
|
||||
const packageRoot = fileURLToPath(new URL("../", import.meta.url));
|
||||
const platform = `${process.platform}-${process.arch}`;
|
||||
const runtime = join(packageRoot, "provider-assets/hermes", platform);
|
||||
assert.ok(HERMES_CLOSURES[platform], "Unqualified Hermes execution platform");
|
||||
await verifyHermesRuntimeFiles(runtime, HERMES_CLOSURES[platform]);
|
||||
const home = await mkdtemp(join(tmpdir(), "paperclip-hermes-unit-"));
|
||||
t.after(() => rm(home, { recursive: true, force: true }));
|
||||
const code = `import sys, unittest\nsys.path[:0] = ${JSON.stringify([join(packageRoot, "src/providers/hermes"), join(runtime, "app")])}\n` +
|
||||
"suite = unittest.defaultTestLoader.loadTestsFromNames(['test_bridge', 'test_billing'])\n" +
|
||||
"sys.exit(0 if unittest.TextTestRunner().run(suite).wasSuccessful() else 1)\n";
|
||||
const output = await promisify(execFile)(join(runtime, "python/bin/python3.12"), ["-I", "-B", "-c", code], {
|
||||
timeout: 30_000, maxBuffer: 64 * 1024,
|
||||
env: { PATH: process.env.PATH, HOME: home, HERMES_HOME: join(home, "hermes") },
|
||||
});
|
||||
assert.match(output.stderr, /Ran \d+ tests[\s\S]*\bOK\b/);
|
||||
t.diagnostic(output.stderr.trim());
|
||||
});
|
||||
|
||||
test("pinned Hermes streams through the production ACPX host using a no-auth local connection", {
|
||||
skip: process.env.PAPERCLIP_HERMES_QUALIFY !== "1", timeout: 180_000,
|
||||
|
||||
@@ -17,6 +17,7 @@ import type {
|
||||
|
||||
import { acpxProfileClientCapabilities, bindAcpxExtensionTurn, validateAcpxRichEvent, createAcpxProfileExtensionAdapter, type AcpxExtensionInput } from "../drivers/acpx/profile-extensions.js";
|
||||
import type { PaperclipQuestionSet } from "../contracts/question-set.js";
|
||||
import { readProviderUsageBilling, type ProviderUsageBilling } from "../contracts/usage-billing.js";
|
||||
import { createAcpxToolEventNormalizer, createGrokMessageNormalizer } from "../provider-events.js";
|
||||
import { parseNativeUserAttachments, validateNativeUserMessageSize } from "../contracts/user-attachments.js";
|
||||
import { parseNativeRuntimeContext } from "../contracts/runtime-context.js";
|
||||
@@ -366,9 +367,14 @@ async function dispatch(
|
||||
turnId = currentTurnId;
|
||||
turnControls.begin(currentTurnId);
|
||||
let runtimeTurn: AcpxRuntimeTurn;
|
||||
let billing: ProviderUsageBilling | undefined;
|
||||
const extensions = bindAcpxExtensionTurn({
|
||||
adapter: createAcpxProfileExtensionAdapter(openParams!.agent, {
|
||||
workspacePath: openParams!.workingDirectory, sessionId: activeHost.identity().backendSessionId, turnId: currentTurnId,
|
||||
onBilling: receipt => {
|
||||
if (billing) throw new Error("Hermes supplied more than one terminal billing receipt");
|
||||
billing = receipt;
|
||||
},
|
||||
}),
|
||||
active: () => turnId === currentTurnId && host === activeHost,
|
||||
sessionId: activeHost.identity().backendSessionId,
|
||||
@@ -401,7 +407,7 @@ async function dispatch(
|
||||
turnId = null;
|
||||
throw error;
|
||||
}
|
||||
void pumpTurn(currentTurnId, runtimeTurn, activeHost, usageBefore, extensions.drain, activity, toolEvidence);
|
||||
void pumpTurn(currentTurnId, runtimeTurn, activeHost, usageBefore, extensions.drain, activity, toolEvidence, () => billing);
|
||||
// Warm sessions may defer initialize until their first prompt. Publish only
|
||||
// the capabilities of that live initialized connection, never old disk state.
|
||||
await runtimeTurn.promptStarted;
|
||||
@@ -614,6 +620,7 @@ async function pumpTurn(
|
||||
drainExtensions: () => Promise<void>,
|
||||
activity: AcpxActivityAdapter,
|
||||
toolEvidence?: AcpxToolEvidence,
|
||||
readBilling?: () => ProviderUsageBilling | undefined,
|
||||
): Promise<void> {
|
||||
let terminal: Record<string, unknown>;
|
||||
try {
|
||||
@@ -648,6 +655,7 @@ async function pumpTurn(
|
||||
usageAfter,
|
||||
runtimeTurn.requestId,
|
||||
openParams?.agent,
|
||||
readBilling?.(),
|
||||
);
|
||||
if (usage) {
|
||||
const estimate = acpxUsageEstimateNotice(usage, `${currentTurnId}:usage-estimate`);
|
||||
@@ -996,6 +1004,7 @@ function sanitizeRuntimeEvent(event: AcpRuntimeEvent): Record<string, unknown> {
|
||||
};
|
||||
}
|
||||
if (event.type === "status") {
|
||||
const billing = readProviderUsageBilling(record(event).billing);
|
||||
return boundedSidecarValue({
|
||||
type: "status",
|
||||
text: boundedOptionalText(event.text, "", 4_000),
|
||||
@@ -1003,6 +1012,7 @@ function sanitizeRuntimeEvent(event: AcpRuntimeEvent): Record<string, unknown> {
|
||||
used: safeNonNegativeNumber(event.used),
|
||||
size: safeNonNegativeNumber(event.size),
|
||||
...safeUsage(event.cost, event.breakdown),
|
||||
...(billing ? { billing } : {}),
|
||||
});
|
||||
}
|
||||
if (event.type === "tool_call") {
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseProviderUsageBilling, readProviderUsageBilling } from "./usage-billing.js";
|
||||
import { validatePrpEvent } from "../protocol/replay-contract.js";
|
||||
|
||||
export const billingFixture = {
|
||||
schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: true, requestCount: 2, reportedRequestCount: 2, amountUsd: 0.0042, amountUsdExact: "0.004200000",
|
||||
} as const;
|
||||
|
||||
describe("versioned per-turn billing authority", () => {
|
||||
it("retains positive partial spend and explicit zero without promoting unknown price", () => {
|
||||
expect(parseProviderUsageBilling(billingFixture)).toEqual(billingFixture);
|
||||
expect(parseProviderUsageBilling({ ...billingFixture, complete: false, reportedRequestCount: 1 }).complete).toBe(false);
|
||||
expect(parseProviderUsageBilling({ ...billingFixture, amountUsd: 0, amountUsdExact: "0.000000000" }).complete).toBe(true);
|
||||
expect(readProviderUsageBilling(undefined)).toBeNull();
|
||||
});
|
||||
it.each([
|
||||
{ schema: "paperclip.usage.billing/v2" }, { source: "rate_card" }, { biller: "other" }, { currency: "EUR" },
|
||||
{ complete: "true" }, { requestCount: 1 }, { reportedRequestCount: 3 }, { reportedRequestCount: 1 },
|
||||
{ amountUsd: -1 }, { amountUsd: Infinity }, { amountUsd: 1_000_001 }, { amountUsdExact: "0.5" },
|
||||
{ amountUsdExact: "00.004200000" }, { amountUsdExact: "0.004200001" }, { unexpected: "private" },
|
||||
])("rejects malformed or conflicting billing %j", mutation => {
|
||||
expect(() => parseProviderUsageBilling({ ...billingFixture, ...mutation })).toThrow();
|
||||
});
|
||||
it("replays existing PRP usage and validates the optional closed billing version", () => {
|
||||
const measurement = { inputTokens: 10, outputTokens: 4, cacheReadTokens: 0, cacheWriteTokens: 0, activeSeconds: 0, requests: 1, providerCostUsd: 0 };
|
||||
const event = { schema: "paperclip.prp.event.v1", sourceEventId: "runner:1", sourceSeq: 1,
|
||||
sourceInstanceId: "runner", sourceKind: "runner", runId: "run", normalizedSessionId: "session",
|
||||
turnId: "turn", itemId: "usage", eventType: "usage.reported", schemaVersion: 1, priority: 0,
|
||||
emittedAt: "2026-10-08T00:00:00.000Z", payload: { provider: "acpx", model: "exact-model", cumulative: measurement, runDelta: measurement } };
|
||||
expect(validatePrpEvent(event).ok).toBe(true);
|
||||
expect(validatePrpEvent({ ...event, payload: { ...event.payload, billing: billingFixture } }).ok).toBe(true);
|
||||
expect(validatePrpEvent({ ...event, payload: { ...event.payload, billing: { ...billingFixture, credential: "private" } } }).ok).toBe(false);
|
||||
expect(validatePrpEvent({ ...event, eventType: "item.completed", payload: { kind: "usage", usage: null } }).ok).toBe(true);
|
||||
expect(validatePrpEvent({ ...event, eventType: "item.completed", payload: { kind: "usage", usage: { billing: billingFixture } } }).ok).toBe(true);
|
||||
expect(validatePrpEvent({ ...event, eventType: "item.completed", payload: { kind: "usage", usage: { billing: { ...billingFixture, credential: "private" } } } }).ok).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,33 @@
|
||||
/** Optional versioned PRP price authority. Absence preserves legacy unknown cost. */
|
||||
export interface ProviderUsageBilling {
|
||||
schema: "paperclip.usage.billing/v1";
|
||||
source: "provider_reported";
|
||||
biller: "openrouter";
|
||||
currency: "USD";
|
||||
complete: boolean;
|
||||
requestCount: number;
|
||||
reportedRequestCount: number;
|
||||
amountUsd: number;
|
||||
amountUsdExact: string;
|
||||
}
|
||||
|
||||
export function parseProviderUsageBilling(value: unknown): ProviderUsageBilling {
|
||||
const invalid = (): never => { throw new Error("Invalid provider billing receipt"); };
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) invalid();
|
||||
const r = value as Record<string, unknown>;
|
||||
if (Object.keys(r).some(key => !["schema", "source", "biller", "currency", "complete", "requestCount", "reportedRequestCount", "amountUsd", "amountUsdExact"].includes(key))
|
||||
|| r.schema !== "paperclip.usage.billing/v1" || r.source !== "provider_reported" || r.biller !== "openrouter"
|
||||
|| r.currency !== "USD" || typeof r.complete !== "boolean"
|
||||
|| typeof r.requestCount !== "number" || !Number.isSafeInteger(r.requestCount) || r.requestCount < 0
|
||||
|| typeof r.reportedRequestCount !== "number" || !Number.isSafeInteger(r.reportedRequestCount) || r.reportedRequestCount < 0
|
||||
|| r.reportedRequestCount > r.requestCount || (r.complete && r.reportedRequestCount !== r.requestCount)
|
||||
|| typeof r.amountUsd !== "number" || !Number.isFinite(r.amountUsd) || r.amountUsd < 0 || r.amountUsd > 1_000_000
|
||||
|| typeof r.amountUsdExact !== "string" || !/^(0|[1-9][0-9]{0,6})\.[0-9]{9}$/.test(r.amountUsdExact)
|
||||
|| Number(r.amountUsdExact) !== r.amountUsd
|
||||
|| (r.reportedRequestCount === 0 && r.amountUsd !== 0)) invalid();
|
||||
return { ...r } as unknown as ProviderUsageBilling;
|
||||
}
|
||||
|
||||
export function readProviderUsageBilling(value: unknown): ProviderUsageBilling | null {
|
||||
try { return parseProviderUsageBilling(value); } catch { return null; }
|
||||
}
|
||||
@@ -87,6 +87,7 @@ import {
|
||||
|
||||
import { AcpxTurnControlLedger, parseAcpxTurnControl, type AcpxTurnControlMode } from "./turn-controls.js";
|
||||
import { acpxUsageEstimateNotice, persistedAcpxTurnUsage } from "./usage-accounting.js";
|
||||
import type { ProviderUsageBilling } from "../../contracts/usage-billing.js";
|
||||
|
||||
const MAX_BUFFERED_EVENTS = 512;
|
||||
const TERMINAL_EVENT_RESERVE = 3;
|
||||
@@ -890,9 +891,14 @@ class CodexAcpxSession implements HarnessSession {
|
||||
this.#emit("turn.submitted", { text: input.message.text }, { turnId });
|
||||
this.#emit("turn.accepted", { turnId }, { turnId });
|
||||
this.#emit("turn.started", { status: "inProgress" }, { turnId });
|
||||
let billing: ProviderUsageBilling | undefined;
|
||||
const extensions = bindAcpxExtensionTurn({
|
||||
adapter: createAcpxProfileExtensionAdapter(this.#agent, {
|
||||
workspacePath: this.#input.workingDirectory, sessionId: this.#host.identity().backendSessionId, turnId,
|
||||
onBilling: receipt => {
|
||||
if (billing) throw new Error("Hermes supplied more than one terminal billing receipt");
|
||||
billing = receipt;
|
||||
},
|
||||
}),
|
||||
active: () => this.#activeTurnId === turnId && !this.#closingStarted,
|
||||
sessionId: this.#host.identity().backendSessionId,
|
||||
@@ -937,7 +943,7 @@ class CodexAcpxSession implements HarnessSession {
|
||||
);
|
||||
throw error;
|
||||
}
|
||||
const pump = this.#pumpTurn(turnId, turn, extensions.drain, usageBefore, activity, toolEvidence);
|
||||
const pump = this.#pumpTurn(turnId, turn, extensions.drain, usageBefore, activity, toolEvidence, () => billing);
|
||||
this.#activePump = pump;
|
||||
void pump
|
||||
.finally(() => {
|
||||
@@ -1442,7 +1448,7 @@ class CodexAcpxSession implements HarnessSession {
|
||||
.catch(() => undefined);
|
||||
}
|
||||
|
||||
async #pumpTurn(turnId: string, turn: AcpxRuntimeTurn, drainExtensions: () => Promise<void>, usageBefore: unknown, activity: AcpxActivityAdapter, toolEvidence?: AcpxToolEvidence): Promise<void> {
|
||||
async #pumpTurn(turnId: string, turn: AcpxRuntimeTurn, drainExtensions: () => Promise<void>, usageBefore: unknown, activity: AcpxActivityAdapter, toolEvidence?: AcpxToolEvidence, readBilling?: () => ProviderUsageBilling | undefined): Promise<void> {
|
||||
try {
|
||||
let index = 0;
|
||||
const normalizeToolEvent =
|
||||
@@ -1463,7 +1469,7 @@ class CodexAcpxSession implements HarnessSession {
|
||||
const notice = activity.usageNotice?.(usageBefore, usageAfter, turn.requestId, turnId);
|
||||
if (notice) { validateAcpxRichEvent(notice); this.#emit(notice.eventType, notice.payload, { turnId, itemId: notice.itemId }); }
|
||||
} catch { /* Partial native diagnostics are optional, never settlement authority. */ }
|
||||
const receipt = persistedAcpxTurnUsage(usageBefore, usageAfter, turn.requestId, this.#agent);
|
||||
const receipt = persistedAcpxTurnUsage(usageBefore, usageAfter, turn.requestId, this.#agent, readBilling?.());
|
||||
if (receipt) {
|
||||
this.#mapRuntimeEvent(receipt as unknown as AcpRuntimeEvent, turnId, ++index);
|
||||
const estimate = acpxUsageEstimateNotice(receipt, `${turnId}:usage-estimate`);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/** Reviewed native closures. Setup cannot adopt a digest from downloaded files. */
|
||||
export const HERMES_CLOSURES: Readonly<Record<string, string>> = Object.freeze({
|
||||
"darwin-arm64": "690b3b84a543b04a47a6859969bd613ced61d68776d80b57c85ca424bde31829",
|
||||
"linux-x64": "6166fadd24dae41b9fdfd994e6c7bc129e9317ed252b7dabdd777b691ab47771",
|
||||
"darwin-arm64": "9f1af058a963305ccd5c4f1555f2828458f72677ceacea2de456ac503f7212de",
|
||||
"linux-x64": "dd918ec15f5bd8025f3d01c4f2849eac4de8d59e237a520607f9c5f988a94645",
|
||||
});
|
||||
@@ -41,4 +41,17 @@ describe("Hermes native extensions", () => {
|
||||
expect(events[0]?.payload.category).toBe("hermes_usage_provenance");
|
||||
expect(JSON.stringify(events)).toContain("unverified");
|
||||
});
|
||||
it("passes structured billing to the owned accounting callback separately from display events", async () => {
|
||||
const receipts: unknown[] = [];
|
||||
const owned = createAcpxProfileExtensionAdapter("hermes", { sessionId: "session", turnId: "turn", workspacePath: "/workspace",
|
||||
onBilling: receipt => { receipts.push(receipt); } })!;
|
||||
const billing = { schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: true, requestCount: 2, reportedRequestCount: 2, amountUsd: 0.0042, amountUsdExact: "0.004200000" };
|
||||
const events = await owned.notification("_hermes/usage", { version: 1, sessionId: "session", tokens: "reported", cost: "unavailable", billing });
|
||||
events.forEach(validateAcpxRichEvent);
|
||||
expect(receipts).toEqual([billing]);
|
||||
expect(events[0]?.payload.summary).toContain("OpenRouter reports");
|
||||
await expect(owned.notification("_hermes/usage", { version: 1, sessionId: "other", tokens: "reported", cost: "unavailable", billing })).rejects.toThrow();
|
||||
await expect(adapter().notification("_hermes/usage", { version: 1, sessionId: "session", tokens: "reported", cost: "unavailable", billing })).rejects.toThrow("not negotiated");
|
||||
});
|
||||
});
|
||||
@@ -5,6 +5,7 @@ import { parsePaperclipQuestionSet, parsePaperclipQuestionResponse, type Papercl
|
||||
import { isCanonicalProviderEventType, type CanonicalProviderEvent } from "../../provider-events.js";
|
||||
import { validatePrpEvent } from "../../protocol/replay-contract.js";
|
||||
import type { QualifiedAcpxAgent } from "./qualified-profiles.js";
|
||||
import { parseProviderUsageBilling, type ProviderUsageBilling } from "../../contracts/usage-billing.js";
|
||||
|
||||
export const ACPX_CANONICAL_INPUT_METHODS = [
|
||||
"elicitation/create", "cursor/ask_question", "cursor/create_plan", "_hermes/ask_questions",
|
||||
@@ -31,6 +32,7 @@ export interface AcpxProfileExtensionContext {
|
||||
workspacePath: string;
|
||||
sessionId: string;
|
||||
turnId: string;
|
||||
onBilling?(receipt: ProviderUsageBilling): void;
|
||||
}
|
||||
|
||||
/** Provider branches install their closed, pinned adapters here after qualification research. */
|
||||
@@ -81,13 +83,21 @@ export function createAcpxProfileExtensionAdapter(
|
||||
throw new Error("Invalid Hermes usage provenance");
|
||||
}
|
||||
const estimated = params.cost === "estimated";
|
||||
const billing = params.billing === undefined ? null : parseProviderUsageBilling(params.billing);
|
||||
if (billing) {
|
||||
if (!context.onBilling) throw new Error("Hermes billing receipts were not negotiated");
|
||||
context.onBilling(billing);
|
||||
}
|
||||
const itemId = `${context.turnId}:hermes-usage`;
|
||||
return [{ eventType: "provider.notice.recorded", itemId, payload: {
|
||||
schema: "paperclip.provider.notice.v1", noticeId: itemId, severity: "info", category: "hermes_usage_provenance", scope: "turn",
|
||||
recoverable: true, userActionable: false,
|
||||
summary: estimated ? `Hermes estimates this turn at $${(params.estimatedUsd as number).toFixed(6)}. Billing cost is unverified.`
|
||||
summary: billing ? `OpenRouter reports $${billing.amountUsd.toFixed(6)}${billing.complete ? " for this turn." : "; remaining billing is unavailable."}`
|
||||
: estimated ? `Hermes estimates this turn at $${(params.estimatedUsd as number).toFixed(6)}. Billing cost is unverified.`
|
||||
: "Hermes billing cost is unavailable.",
|
||||
details: [{ name: "Token usage", value: String(params.tokens) }, { name: "Cost source", value: estimated ? "Hermes model pricing estimate" : "Unavailable" },
|
||||
details: [{ name: "Token usage", value: String(params.tokens) }, { name: "Cost source", value: billing ? "OpenRouter response usage.cost" : estimated ? "Hermes model pricing estimate" : "Unavailable" },
|
||||
...(billing ? [{ name: "Reported USD", value: billing.amountUsdExact }, { name: "Billing complete", value: String(billing.complete) },
|
||||
{ name: "Reported requests", value: `${billing.reportedRequestCount}/${billing.requestCount}` }] : []),
|
||||
...(estimated ? [{ name: "Estimated USD", value: String(params.estimatedUsd) }] : [])],
|
||||
} }];
|
||||
}
|
||||
@@ -99,7 +109,7 @@ export function createAcpxProfileExtensionAdapter(
|
||||
}
|
||||
export function acpxProfileClientCapabilities(agent: QualifiedAcpxAgent): Record<string, unknown> {
|
||||
if (agent === "cursor") return structuredClone(CURSOR_CLIENT_CAPABILITIES);
|
||||
return agent === "hermes" ? { _meta: { paperclipHermes: { version: 1 } } } : {};
|
||||
return agent === "hermes" ? { _meta: { paperclipHermes: { version: 1, billingReceipts: 1 } } } : {};
|
||||
}
|
||||
|
||||
/** Reject an oversized approval document; never silently approve a truncated revision. */
|
||||
|
||||
@@ -170,3 +170,23 @@ describe("Pi prompt accounting authority", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("owned Hermes wire billing", () => {
|
||||
const billingFixture = { schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: true, requestCount: 2, reportedRequestCount: 2, amountUsd: 0.0042, amountUsdExact: "0.004200000" } as const;
|
||||
const after = { lastRequestId: "active", requestTokenUsage: { current: { input_tokens: 10, output_tokens: 4,
|
||||
cache_read_input_tokens: 0, cache_creation_input_tokens: 0, thought_tokens: 0 } } };
|
||||
it("joins billing only to the exact current Hermes receipt and keeps cumulative price separate", () => {
|
||||
expect(persistedAcpxTurnUsage({}, after, "active", "hermes", billingFixture)).toMatchObject({ billing: billingFixture, cost: undefined });
|
||||
expect(persistedAcpxTurnUsage({}, after, "stale", "hermes", billingFixture)).toBeNull();
|
||||
expect(persistedAcpxTurnUsage({}, after, "active", "pi", billingFixture)?.billing).toBeUndefined();
|
||||
expect(persistedAcpxTurnUsage({}, { ...after, requestTokenUsage: { first: {}, second: {} } }, "active", "hermes", billingFixture)).toBeNull();
|
||||
});
|
||||
it("retains billing when the native token receipt is unavailable without making its breakdown complete", () => {
|
||||
const usage = persistedAcpxTurnUsage({}, { lastRequestId: "active", requestTokenUsage: {} }, "active", "hermes", {
|
||||
...billingFixture, complete: false, reportedRequestCount: 1,
|
||||
});
|
||||
expect(usage?.billing).toMatchObject({ complete: false, amountUsd: 0.0042 });
|
||||
expect((usage?.breakdown as Record<string, unknown>).inputTokens).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import type { CanonicalProviderEvent } from "../../provider-events.js";
|
||||
import type { QualifiedAcpxAgent } from "./qualified-profiles.js";
|
||||
import { parseProviderUsageBilling, type ProviderUsageBilling } from "../../contracts/usage-billing.js";
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
@@ -40,6 +41,7 @@ export function persistedAcpxTurnUsage(
|
||||
after: unknown,
|
||||
requestId: string,
|
||||
agent: QualifiedAcpxAgent | null = null,
|
||||
billing?: ProviderUsageBilling,
|
||||
): Record<string, unknown> | null {
|
||||
const current = record(after);
|
||||
if (current.lastRequestId !== requestId) return null;
|
||||
@@ -48,8 +50,9 @@ export function persistedAcpxTurnUsage(
|
||||
const added = Object.keys(receipts).filter(
|
||||
(key) => !Object.hasOwn(previousReceipts, key),
|
||||
);
|
||||
if (added.length !== 1) return null;
|
||||
const usage = record(receipts[added[0]!]);
|
||||
if (added.length !== 1 && !(agent === "hermes" && billing && added.length === 0)) return null;
|
||||
const usage = added.length === 1 ? record(receipts[added[0]!]) : {};
|
||||
const boundBilling = agent === "hermes" && billing ? parseProviderUsageBilling(billing) : null;
|
||||
const piReceipt = agent === "pi" ? record(usage.paperclip_pi) : {};
|
||||
const piReceiptVerified = piReceipt.provenance === "assistant_message_receipts"
|
||||
|| piReceipt.provenance === "assistant_message_and_compaction_receipts";
|
||||
@@ -62,6 +65,7 @@ export function persistedAcpxTurnUsage(
|
||||
// Pi calculates cost from catalog prices, not billing receipts. Never feed
|
||||
// this estimate into the authoritative/cumulative provider spend channel.
|
||||
cost: agent === "pi" || agent === "hermes" ? undefined : current.usageCost,
|
||||
...(boundBilling ? { billing: boundBilling } : {}),
|
||||
...(piReceiptVerified ? { usageProvenance: `pi_${piReceipt.provenance}` } : {}),
|
||||
...(estimate === undefined ? {} : { pricingEstimateUsd: estimate }),
|
||||
breakdown: {
|
||||
|
||||
@@ -15,6 +15,7 @@ export * from "./contracts/question-set.js";
|
||||
export * from "./contracts/runtime-context.js";
|
||||
export * from "./contracts/types.js";
|
||||
export * from "./contracts/user-attachments.js";
|
||||
export * from "./contracts/usage-billing.js";
|
||||
export * from "./backends/harness-driver-backend.js";
|
||||
export * from "./drivers/dot/dot-harness-driver.js";
|
||||
export { describeRunnerdNativeSessionBackend } from "./backends/codex-native-backend.js";
|
||||
|
||||
@@ -1834,6 +1834,7 @@ export function rehydrateRunnerdUsageNotification(
|
||||
// authority beside the counters so ACPX/managed driver projections and
|
||||
// durable replay cannot certify an incomplete delta as a free receipt.
|
||||
runDeltaComplete: rawParams.runDeltaAvailable === true,
|
||||
...(rawParams.billing === undefined ? {} : { billing: rawParams.billing }),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -2999,12 +2999,66 @@ export const usageSchema = {
|
||||
"runDeltaAvailable": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"billing": {
|
||||
"$ref": "#/$defs/billing"
|
||||
},
|
||||
"runDelta": {
|
||||
"$ref": "#/$defs/measurement"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"$defs": {
|
||||
"billing": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schema",
|
||||
"source",
|
||||
"biller",
|
||||
"currency",
|
||||
"complete",
|
||||
"requestCount",
|
||||
"reportedRequestCount",
|
||||
"amountUsd",
|
||||
"amountUsdExact"
|
||||
],
|
||||
"properties": {
|
||||
"schema": {
|
||||
"const": "paperclip.usage.billing/v1"
|
||||
},
|
||||
"source": {
|
||||
"const": "provider_reported"
|
||||
},
|
||||
"biller": {
|
||||
"const": "openrouter"
|
||||
},
|
||||
"currency": {
|
||||
"const": "USD"
|
||||
},
|
||||
"complete": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"requestCount": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 9007199254740991
|
||||
},
|
||||
"reportedRequestCount": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 9007199254740991
|
||||
},
|
||||
"amountUsd": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1000000
|
||||
},
|
||||
"amountUsdExact": {
|
||||
"type": "string",
|
||||
"pattern": "^(0|[1-9][0-9]{0,6})\\.[0-9]{9}$"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"measurement": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
@@ -4420,6 +4474,68 @@ export const eventSchema = {
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"eventType": {
|
||||
"const": "usage.reported"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"billing": {
|
||||
"$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"kind",
|
||||
"usage"
|
||||
],
|
||||
"properties": {
|
||||
"kind": {
|
||||
"const": "usage"
|
||||
},
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"billing"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"billing": {
|
||||
"$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
@@ -5106,6 +5222,68 @@ export const eventV2Schema = {
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"eventType": {
|
||||
"const": "usage.reported"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"billing": {
|
||||
"$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"kind",
|
||||
"usage"
|
||||
],
|
||||
"properties": {
|
||||
"kind": {
|
||||
"const": "usage"
|
||||
},
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"billing"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"payload": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"billing": {
|
||||
"$ref": "https://paperclip.dev/schemas/prp/v1/usage.schema.json#/$defs/billing"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
|
||||
@@ -4,4 +4,4 @@
|
||||
import equalModule from "ajv/dist/runtime/equal.js";
|
||||
import ucs2LengthModule from "ajv/dist/runtime/ucs2length.js";
|
||||
|
||||
"use strict";export const fixtureValidator: unknown = validate20;const schema31 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/fixture.schema.json","title":"PRP scripted replay fixture","type":"object","required":["schema","fixtureVersion","protocolVersion","name","description","identity","capabilities","commands","events","result"],"properties":{"schema":{"const":"paperclip.prp.fixture.v1"},"fixtureVersion":{"const":1},"protocolVersion":{"enum":[1,2,3]},"name":{"type":"string","minLength":1,"maxLength":120},"description":{"type":"string","minLength":1,"maxLength":1000},"identity":{"$ref":"https://paperclip.dev/schemas/prp/v1/identity.schema.json"},"capabilities":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/capabilities.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/capabilities.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/capabilities.schema.json"}]},"commands":{"type":"array","items":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/command.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/command.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/command.schema.json"}]}},"events":{"type":"array","minItems":1,"items":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/event.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/event.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/event.schema.json"}]}},"requests":{"type":"array","items":{"$ref":"https://paperclip.dev/schemas/prp/v1/request.schema.json"}},"result":{"$ref":"https://paperclip.dev/schemas/prp/v1/result.schema.json"}},"additionalProperties":true};const schema41 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/capabilities.schema.json","title":"PRP negotiated capabilities","type":"object","required":["schema","sessionReusePolicy","driver","steer","interrupt","resume","runtimeRequests","structuredResult","typedEvents"],"properties":{"schema":{"const":"paperclip.prp.capabilities.v1"},"sessionReusePolicy":{"enum":["new_per_run","reuse_per_issue","reuse_per_workspace"]},"driver":{"type":"object","required":["kind","version"],"properties":{"kind":{"type":"string","minLength":1,"maxLength":80},"version":{"type":"string","minLength":1,"maxLength":80}},"additionalProperties":true},"steer":{"type":"boolean"},"interrupt":{"type":"boolean"},"resume":{"type":"boolean"},"runtimeRequests":{"type":"boolean"},"structuredResult":{"type":"boolean"},"typedEvents":{"type":"boolean"},"typedEventFamilies":{"type":"array","maxItems":64,"items":{"type":"object","required":["family","version","availability","detailLevel"],"properties":{"family":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[a-z][a-z0-9_.-]*$"},"version":{"const":1},"availability":{"enum":["available","unsupported","policy_disabled"]},"detailLevel":{"enum":["summary","structured"]}},"additionalProperties":false}},"semanticTools":{"type":"object","required":["schema","schemaVersion","operations"],"properties":{"schema":{"const":"paperclip.prp.semantic_tools.v1"},"schemaVersion":{"const":1},"operations":{"type":"array","items":{"type":"object","required":["operationId","version","availability","redactionDisposition"],"properties":{"operationId":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[A-Za-z0-9][A-Za-z0-9._:-]*$"},"version":{"const":1},"availability":{"enum":["available","denied","unsupported"]},"redactionDisposition":{"const":"digest_only"},"requiredClaims":{"type":"array","items":{"type":"string","minLength":1,"maxLength":160},"uniqueItems":true},"reasonCode":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[a-z][a-z0-9_.:-]*$"}},"additionalProperties":true}}},"additionalProperties":true},"unsupported":{"type":"array","items":{"type":"string","minLength":1},"uniqueItems":true}},"additionalProperties":true};const schema47 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/command.schema.json","title":"PRP runner command","type":"object","required":["schema","commandId","controllerSeq","type","issuedAt","payload"],"properties":{"schema":{"const":"paperclip.prp.command.v1"},"commandId":{"type":"string","minLength":1,"maxLength":160},"controllerSeq":{"type":"integer","minimum":1},"type":{"enum":["run.prepare","run.attach","session.open","turn.start","turn.steer","turn.interrupt","turn.stop","request.resolve","interaction.receipt","semantic_tool.result","session.snapshot","session.close","session.budget.increase","session.destroy","run.cancel","runner.drain","runner.suspend","runner.shutdown"]},"issuedAt":{"type":"string","format":"date-time"},"deadlineAt":{"type":"string","format":"date-time"},"precondition":{"type":"object","properties":{"runnerState":{"type":"array","items":{"type":"string"}},"runState":{"type":"array","items":{"type":"string"}},"sessionState":{"type":"array","items":{"type":"string"}},"activeTurnId"Line truncated
|
||||
"use strict";export const fixtureValidator: unknown = validate20;const schema31 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/fixture.schema.json","title":"PRP scripted replay fixture","type":"object","required":["schema","fixtureVersion","protocolVersion","name","description","identity","capabilities","commands","events","result"],"properties":{"schema":{"const":"paperclip.prp.fixture.v1"},"fixtureVersion":{"const":1},"protocolVersion":{"enum":[1,2,3]},"name":{"type":"string","minLength":1,"maxLength":120},"description":{"type":"string","minLength":1,"maxLength":1000},"identity":{"$ref":"https://paperclip.dev/schemas/prp/v1/identity.schema.json"},"capabilities":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/capabilities.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/capabilities.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/capabilities.schema.json"}]},"commands":{"type":"array","items":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/command.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/command.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/command.schema.json"}]}},"events":{"type":"array","minItems":1,"items":{"oneOf":[{"$ref":"https://paperclip.dev/schemas/prp/v1/event.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v2/event.schema.json"},{"$ref":"https://paperclip.dev/schemas/prp/v3/event.schema.json"}]}},"requests":{"type":"array","items":{"$ref":"https://paperclip.dev/schemas/prp/v1/request.schema.json"}},"result":{"$ref":"https://paperclip.dev/schemas/prp/v1/result.schema.json"}},"additionalProperties":true};const schema41 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/capabilities.schema.json","title":"PRP negotiated capabilities","type":"object","required":["schema","sessionReusePolicy","driver","steer","interrupt","resume","runtimeRequests","structuredResult","typedEvents"],"properties":{"schema":{"const":"paperclip.prp.capabilities.v1"},"sessionReusePolicy":{"enum":["new_per_run","reuse_per_issue","reuse_per_workspace"]},"driver":{"type":"object","required":["kind","version"],"properties":{"kind":{"type":"string","minLength":1,"maxLength":80},"version":{"type":"string","minLength":1,"maxLength":80}},"additionalProperties":true},"steer":{"type":"boolean"},"interrupt":{"type":"boolean"},"resume":{"type":"boolean"},"runtimeRequests":{"type":"boolean"},"structuredResult":{"type":"boolean"},"typedEvents":{"type":"boolean"},"typedEventFamilies":{"type":"array","maxItems":64,"items":{"type":"object","required":["family","version","availability","detailLevel"],"properties":{"family":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[a-z][a-z0-9_.-]*$"},"version":{"const":1},"availability":{"enum":["available","unsupported","policy_disabled"]},"detailLevel":{"enum":["summary","structured"]}},"additionalProperties":false}},"semanticTools":{"type":"object","required":["schema","schemaVersion","operations"],"properties":{"schema":{"const":"paperclip.prp.semantic_tools.v1"},"schemaVersion":{"const":1},"operations":{"type":"array","items":{"type":"object","required":["operationId","version","availability","redactionDisposition"],"properties":{"operationId":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[A-Za-z0-9][A-Za-z0-9._:-]*$"},"version":{"const":1},"availability":{"enum":["available","denied","unsupported"]},"redactionDisposition":{"const":"digest_only"},"requiredClaims":{"type":"array","items":{"type":"string","minLength":1,"maxLength":160},"uniqueItems":true},"reasonCode":{"type":"string","minLength":1,"maxLength":160,"pattern":"^[a-z][a-z0-9_.:-]*$"}},"additionalProperties":true}}},"additionalProperties":true},"unsupported":{"type":"array","items":{"type":"string","minLength":1},"uniqueItems":true}},"additionalProperties":true};const schema47 = {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://paperclip.dev/schemas/prp/v1/command.schema.json","title":"PRP runner command","type":"object","required":["schema","commandId","controllerSeq","type","issuedAt","payload"],"properties":{"schema":{"const":"paperclip.prp.command.v1"},"commandId":{"type":"string","minLength":1,"maxLength":160},"controllerSeq":{"type":"integer","minimum":1},"type":{"enum":["run.prepare","run.attach","session.open","turn.start","turn.steer","turn.interrupt","turn.stop","request.resolve","interaction.receipt","semantic_tool.result","session.snapshot","session.close","session.budget.increase","session.destroy","run.cancel","runner.drain","runner.suspend","runner.shutdown"]},"issuedAt":{"type":"string","format":"date-time"},"deadlineAt":{"type":"string","format":"date-time"},"precondition":{"type":"object","properties":{"runnerState":{"type":"array","items":{"type":"string"}},"runState":{"type":"array","items":{"type":"string"}},"sessionState":{"type":"array","items":{"type":"string"}},"activeTurnId"Line truncated
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
sessionGoalSchema,
|
||||
stopReasonSchema,
|
||||
terminalSchema,
|
||||
usageSchema,
|
||||
} from "./generated/schema-bundle.js";
|
||||
import {
|
||||
eventValidator as standaloneEventV1Validator,
|
||||
@@ -40,8 +41,9 @@ type EventReferences = [
|
||||
typeof stopReasonSchema,
|
||||
typeof terminalSchema,
|
||||
typeof resultSchema,
|
||||
typeof usageSchema,
|
||||
];
|
||||
type EventV2References = [typeof sessionGoalSchema];
|
||||
type EventV2References = [typeof sessionGoalSchema, typeof usageSchema];
|
||||
type CapabilitiesV2References = [typeof sessionGoalSchema];
|
||||
export type PrpIdentity = FromSchema<typeof identitySchema>;
|
||||
export type PrpCapabilities = FromSchema<typeof capabilitiesSchema>;
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
"""Turn-owned OpenRouter wire receipts for the pinned synchronous SDK transport.
|
||||
|
||||
Observe bytes without changing requests or Hermes's loop. Retry attempts,
|
||||
compaction and delegated calls share the ledger; incomplete requests stay unknown.
|
||||
No request headers, prompts, response text or credentials enter the receipt.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP
|
||||
|
||||
MAX_REQUESTS = 10000
|
||||
MAX_FRAME = 1024 * 1024
|
||||
MAX_BODY = 8 * MAX_FRAME
|
||||
_active = None
|
||||
_active_lock = threading.Lock()
|
||||
|
||||
|
||||
def _count(value):
|
||||
return value if isinstance(value, int) and not isinstance(value, bool) and 0 <= value <= 9007199254740991 else None
|
||||
|
||||
|
||||
def _usage(value):
|
||||
if not isinstance(value, dict):
|
||||
return None, None
|
||||
prompt, output = _count(value.get("prompt_tokens")), _count(value.get("completion_tokens"))
|
||||
details = value.get("prompt_tokens_details") or {}
|
||||
if not isinstance(details, dict):
|
||||
return None, None
|
||||
read, write = _count(details.get("cached_tokens", 0)), _count(details.get("cache_write_tokens", 0))
|
||||
tokens = None
|
||||
if None not in (prompt, output, read, write) and prompt >= read + write:
|
||||
tokens = (prompt - read - write, output, read, write)
|
||||
raw_cost = value.get("cost")
|
||||
cost = None
|
||||
if isinstance(raw_cost, (int, Decimal)) and not isinstance(raw_cost, bool):
|
||||
try:
|
||||
amount = Decimal(raw_cost)
|
||||
if amount.is_finite() and 0 <= amount <= 1000000:
|
||||
cost = int((amount * 1000000000).quantize(Decimal(1), rounding=ROUND_HALF_UP))
|
||||
except InvalidOperation:
|
||||
pass
|
||||
return tokens, cost
|
||||
|
||||
|
||||
class TurnBilling:
|
||||
def __init__(self):
|
||||
self._lock = threading.RLock()
|
||||
self._requests = []
|
||||
self._closed = False
|
||||
self._overflow = False
|
||||
self._children = set()
|
||||
|
||||
def child(self, identity, started):
|
||||
with self._lock:
|
||||
if started:
|
||||
self._children.add(identity)
|
||||
if len(self._children) > MAX_REQUESTS:
|
||||
self._overflow = True
|
||||
else:
|
||||
if identity not in self._children:
|
||||
self._overflow = True
|
||||
self._children.discard(identity)
|
||||
|
||||
def incomplete(self):
|
||||
with self._lock:
|
||||
self._overflow = True
|
||||
|
||||
def begin(self, authorized=True):
|
||||
with self._lock:
|
||||
if self._closed or len(self._requests) >= MAX_REQUESTS:
|
||||
self._overflow = True
|
||||
return None
|
||||
receipt = WireReceipt(self, authorized)
|
||||
self._requests.append(receipt)
|
||||
return receipt
|
||||
|
||||
def finish(self, owned_work_complete=True):
|
||||
with self._lock:
|
||||
self._closed = True
|
||||
completed = [r for r in self._requests if r.valid and r.done]
|
||||
priced = [r for r in completed if r.cost is not None]
|
||||
tokenized = [r for r in completed if r.tokens is not None]
|
||||
scope_complete = owned_work_complete and not self._overflow and not self._children
|
||||
cost = sum(r.cost for r in priced)
|
||||
if cost > 1000000000000000:
|
||||
raise ValueError("Hermes reported turn cost exceeds its receipt bound")
|
||||
complete = scope_complete and len(priced) == len(self._requests)
|
||||
token_complete = scope_complete and len(tokenized) == len(self._requests)
|
||||
totals = tuple(sum(r.tokens[i] for r in tokenized) for i in range(4))
|
||||
if any(value > 9007199254740991 for value in totals):
|
||||
token_complete = False
|
||||
return {
|
||||
"schema": "paperclip.usage.billing/v1", "source": "provider_reported",
|
||||
"biller": "openrouter", "currency": "USD", "complete": complete,
|
||||
"requestCount": len(self._requests), "reportedRequestCount": len(priced),
|
||||
"amountUsd": cost / 1000000000,
|
||||
"amountUsdExact": f"{cost // 1000000000}.{cost % 1000000000:09d}",
|
||||
}, totals if token_complete else None
|
||||
|
||||
|
||||
class WireReceipt:
|
||||
def __init__(self, owner, authorized):
|
||||
self.owner = owner
|
||||
self.valid = authorized
|
||||
self.done = False
|
||||
self.tokens = None
|
||||
self.cost = None
|
||||
self._seen_usage = False
|
||||
self._buffer = bytearray()
|
||||
self._data = []
|
||||
self._frame_size = 0
|
||||
|
||||
def _document(self, data):
|
||||
try:
|
||||
value = json.loads(data, parse_float=Decimal)
|
||||
except (ValueError, UnicodeError):
|
||||
self.valid = False
|
||||
return
|
||||
if not isinstance(value, dict):
|
||||
self.valid = False
|
||||
return
|
||||
usage = value.get("usage")
|
||||
if usage is not None:
|
||||
tokens, cost = _usage(usage)
|
||||
if self._seen_usage and (tokens != self.tokens or cost != self.cost):
|
||||
self.valid = False
|
||||
self._seen_usage = True
|
||||
self.tokens, self.cost = tokens, cost
|
||||
if value.get("error") is not None:
|
||||
self.valid = False
|
||||
|
||||
def json(self, body):
|
||||
with self.owner._lock:
|
||||
if len(body) > MAX_BODY:
|
||||
self.valid = False
|
||||
else:
|
||||
self._document(body)
|
||||
self.done = True
|
||||
|
||||
def feed(self, chunk):
|
||||
with self.owner._lock:
|
||||
if not self.valid or self.done:
|
||||
return
|
||||
self._buffer.extend(chunk)
|
||||
while b"\n" in self._buffer:
|
||||
line, _, rest = self._buffer.partition(b"\n")
|
||||
self._buffer = bytearray(rest)
|
||||
line = line.rstrip(b"\r")
|
||||
if not line:
|
||||
data = b"\n".join(self._data)
|
||||
self._data.clear()
|
||||
self._frame_size = 0
|
||||
if data == b"[DONE]":
|
||||
self.done = True
|
||||
self._buffer.clear()
|
||||
return
|
||||
if data:
|
||||
self._document(data)
|
||||
elif line.startswith(b"data:"):
|
||||
data = line[5:].lstrip(b" ")
|
||||
self._frame_size += len(data)
|
||||
if self._frame_size > MAX_FRAME:
|
||||
self.valid = False
|
||||
self._data.clear()
|
||||
self._buffer.clear()
|
||||
return
|
||||
self._data.append(bytes(data))
|
||||
if len(self._buffer) > MAX_FRAME:
|
||||
self.valid = False
|
||||
self._buffer.clear()
|
||||
|
||||
def eof(self):
|
||||
with self.owner._lock:
|
||||
# A truncated SSE frame is not a complete response, even when a
|
||||
# previous frame happened to contain usage.
|
||||
if not self._buffer and not self._data:
|
||||
self.done = True
|
||||
self._buffer.clear()
|
||||
self._data.clear()
|
||||
|
||||
def failed(self):
|
||||
with self.owner._lock:
|
||||
self.valid = False
|
||||
self._buffer.clear()
|
||||
self._data.clear()
|
||||
|
||||
|
||||
def start_turn(agent):
|
||||
global _active
|
||||
# Only the native selected OpenRouter Chat Completions route is qualified
|
||||
# for billed usage.cost. Custom protocols and model-price estimates are not.
|
||||
enabled = getattr(agent, "provider", None) == "openrouter" and getattr(agent, "api_mode", None) == "chat_completions"
|
||||
with _active_lock:
|
||||
if _active is not None:
|
||||
raise ValueError("Hermes billing already belongs to an active turn")
|
||||
_active = TurnBilling() if enabled else None
|
||||
return _active
|
||||
|
||||
|
||||
def close_turn(ledger):
|
||||
global _active
|
||||
with _active_lock:
|
||||
if _active is ledger:
|
||||
_active = None
|
||||
|
||||
|
||||
def fence_background_dispatch(dispatch):
|
||||
def run(*args, **kwargs):
|
||||
with _active_lock:
|
||||
ledger = _active
|
||||
if ledger is not None:
|
||||
# A background unit can start a child after the parent settles.
|
||||
# Keep its observed subtotal, but do not certify full settlement
|
||||
# before the per-turn process lifecycle has stopped that unit.
|
||||
ledger.incomplete()
|
||||
return dispatch(*args, **kwargs)
|
||||
return run
|
||||
|
||||
|
||||
def install_billing_capture():
|
||||
import httpx
|
||||
import os
|
||||
if getattr(httpx.Client.send, "_paperclip_billing", False):
|
||||
return
|
||||
original_send, original_bytes = httpx.Client.send, httpx.Response.iter_bytes
|
||||
original_async_send = httpx.AsyncClient.send
|
||||
|
||||
def inference_request(request):
|
||||
return request.method == "POST" and request.url.path.rstrip("/").endswith(("/chat/completions", "/responses", "/messages"))
|
||||
|
||||
def send(client, request, *args, **kwargs):
|
||||
with _active_lock:
|
||||
ledger = _active
|
||||
# Include every real wire attempt, including retries beneath the SDK,
|
||||
# rather than only the final response accepted by the agent loop.
|
||||
receipt = None
|
||||
if ledger is not None and inference_request(request):
|
||||
key = os.environ.get("OPENROUTER_API_KEY")
|
||||
authorized = bool(key) and request.url.scheme == "https" and request.url.host == "openrouter.ai" \
|
||||
and request.url.port in (None, 443) and request.url.path == "/api/v1/chat/completions" \
|
||||
and not request.url.query and not request.url.username and not request.url.password \
|
||||
and request.headers.get("authorization") == "Bearer " + key
|
||||
receipt = ledger.begin(authorized)
|
||||
try:
|
||||
response = original_send(client, request, *args, **kwargs)
|
||||
except BaseException:
|
||||
if receipt is not None:
|
||||
receipt.failed()
|
||||
raise
|
||||
if receipt is not None:
|
||||
response.extensions["paperclip_billing"] = receipt
|
||||
if response.status_code != 200 or response.history or response.url != request.url:
|
||||
receipt.failed()
|
||||
elif response.is_stream_consumed:
|
||||
receipt.json(response.content)
|
||||
return response
|
||||
|
||||
def iter_bytes(response, *args, **kwargs):
|
||||
receipt = response.extensions.get("paperclip_billing")
|
||||
stream = response.headers.get("content-type", "").split(";", 1)[0].strip() == "text/event-stream"
|
||||
try:
|
||||
for chunk in original_bytes(response, *args, **kwargs):
|
||||
if receipt is not None and stream:
|
||||
receipt.feed(chunk)
|
||||
yield chunk
|
||||
except GeneratorExit:
|
||||
# The SDK stops at [DONE], before exhausting the HTTP iterator.
|
||||
# Only a fully observed final receipt can survive that close.
|
||||
raise
|
||||
except BaseException:
|
||||
if receipt is not None:
|
||||
receipt.failed()
|
||||
raise
|
||||
else:
|
||||
if receipt is not None:
|
||||
if stream:
|
||||
receipt.eof()
|
||||
elif response.is_stream_consumed:
|
||||
receipt.json(response.content)
|
||||
|
||||
send._paperclip_billing = True
|
||||
httpx.Client.send, httpx.Response.iter_bytes = send, iter_bytes
|
||||
|
||||
async def async_send(client, request, *args, **kwargs):
|
||||
# Native inference uses the pinned synchronous SDK. A new asynchronous
|
||||
# path must not silently turn an unobserved request into free work.
|
||||
with _active_lock:
|
||||
ledger = _active
|
||||
if ledger is not None and inference_request(request):
|
||||
ledger.begin(False)
|
||||
return await original_async_send(client, request, *args, **kwargs)
|
||||
|
||||
httpx.AsyncClient.send = async_send
|
||||
@@ -22,6 +22,7 @@ from acp_adapter.events import _send_update, make_step_cb
|
||||
from acp_adapter.tools import build_tool_start, build_tool_complete
|
||||
from policy import authorize_tool, REPORTING_TOOLS
|
||||
from tool_process import tool_policy, install_tool_process_policy
|
||||
from billing import close_turn, fence_background_dispatch, install_billing_capture, start_turn
|
||||
|
||||
EXTENSION_VERSION = 1
|
||||
ANSWER_MAX_LENGTH = 65536
|
||||
@@ -219,6 +220,8 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
self._negotiated = False
|
||||
self._dispatch_original = None
|
||||
self._usage_before = None
|
||||
self._billing = None
|
||||
self._billing_negotiated = False
|
||||
|
||||
def _build_model_state(self, state):
|
||||
from acp.schema import ModelInfo, SessionModelState
|
||||
@@ -242,10 +245,12 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
capabilities = kwargs.get("client_capabilities")
|
||||
meta = getattr(capabilities, "field_meta", None) or {}
|
||||
self._negotiated = meta.get("paperclipHermes", {}).get("version") == EXTENSION_VERSION
|
||||
self._billing_negotiated = self._negotiated and meta.get("paperclipHermes", {}).get("billingReceipts") == 1
|
||||
response = await super().initialize(*args, **kwargs)
|
||||
response.agent_capabilities.field_meta = {"paperclipHermes": {
|
||||
"version": EXTENSION_VERSION, "steering": True, "questions": True,
|
||||
"queuedFollowUp": False, "strictRestore": True,
|
||||
**({"billingReceipts": 1} if self._billing_negotiated else {}),
|
||||
}}
|
||||
return response
|
||||
|
||||
@@ -305,6 +310,7 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
if state is None:
|
||||
raise ValueError("Hermes session is unavailable")
|
||||
self._usage_before = usage_snapshot(state.agent)
|
||||
self._billing = start_turn(state.agent) if self._billing_negotiated else None
|
||||
state.agent._paperclip_usage_receipts = []
|
||||
state.agent._paperclip_cost_receipts = []
|
||||
await self._conn.ext_notification("hermes/turn_started", {
|
||||
@@ -312,6 +318,8 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
})
|
||||
return await super().prompt(prompt, session_id, **kwargs)
|
||||
finally:
|
||||
close_turn(self._billing)
|
||||
self._billing = None
|
||||
self._active = None
|
||||
self._usage_before = None
|
||||
for future in tuple(self._pending_questions):
|
||||
@@ -331,14 +339,25 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
raise ValueError("Hermes turn accounting is unavailable")
|
||||
after = usage_snapshot(state.agent)
|
||||
receipts = state.agent._paperclip_usage_receipts
|
||||
response.usage = turn_usage(before, after, receipts)
|
||||
billing = None
|
||||
if self._billing is not None:
|
||||
billing, tokens = self._billing.finish()
|
||||
# This includes auxiliary and rejected/truncated responses that
|
||||
# native accepted-response counters do not cover. Missing wire
|
||||
# receipts cannot fall back to a smaller accepted-response total.
|
||||
response.usage = None if tokens is None else acp.schema.Usage(
|
||||
input_tokens=tokens[0], output_tokens=tokens[1], cached_read_tokens=tokens[2],
|
||||
cached_write_tokens=tokens[3], thought_tokens=0, total_tokens=sum(tokens))
|
||||
else:
|
||||
response.usage = turn_usage(before, after, receipts)
|
||||
costs = state.agent._paperclip_cost_receipts
|
||||
estimated = bool(costs) and all(costs) and response.usage is not None
|
||||
estimated = billing is None and bool(costs) and all(costs) and response.usage is not None
|
||||
await conn.ext_notification("hermes/usage", {
|
||||
"version": EXTENSION_VERSION, "sessionId": session_id, "turnToken": self._active[1],
|
||||
"tokens": "reported" if response.usage is not None else "unavailable",
|
||||
"cost": "estimated" if estimated else "unavailable",
|
||||
**({"estimatedUsd": after["estimated_cost_usd"] - before["estimated_cost_usd"]} if estimated else {}),
|
||||
**({"billing": billing} if billing is not None else {}),
|
||||
})
|
||||
return response
|
||||
|
||||
@@ -412,10 +431,14 @@ class ManagedHermesACPAgent(HermesACPAgent):
|
||||
# native content (including diffs) and the original result together.
|
||||
_send_update(conn, session_id, loop, update.model_copy(update={"raw_output": result_text}))
|
||||
|
||||
billing = self._billing
|
||||
|
||||
def progress(event, name=None, preview=None, args=None, **details):
|
||||
child = details.get("child_session_id") or details.get("subagent_id")
|
||||
if event not in {"subagent.start", "subagent.progress", "subagent.tool", "subagent.complete"} or not isinstance(child, str):
|
||||
return
|
||||
if billing is not None and event in {"subagent.start", "subagent.complete"}:
|
||||
billing.child(child, event == "subagent.start")
|
||||
future = asyncio.run_coroutine_threadsafe(conn.ext_notification("hermes/delegation", {
|
||||
"version": EXTENSION_VERSION, "sessionId": session_id, "turnToken": token,
|
||||
"event": event, "childId": child[:160],
|
||||
@@ -513,6 +536,9 @@ def main():
|
||||
|
||||
def configure_managed_runtime():
|
||||
install_tool_process_policy()
|
||||
install_billing_capture()
|
||||
from tools import delegate_tool_dispatch
|
||||
delegate_tool_dispatch._dispatch_background = fence_background_dispatch(delegate_tool_dispatch._dispatch_background)
|
||||
from agent import turn_response_check
|
||||
original_usage = turn_response_check.record_response_usage
|
||||
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
import asyncio
|
||||
import json
|
||||
import threading
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
from openai import OpenAI
|
||||
from billing import MAX_FRAME, TurnBilling, close_turn, fence_background_dispatch, install_billing_capture, start_turn
|
||||
|
||||
|
||||
def document(cost="0.0042"):
|
||||
return ('{"id":"fixture-response","model":"fixture-model","choices":[],"usage":'
|
||||
'{"prompt_tokens":20,"completion_tokens":5,"prompt_tokens_details":'
|
||||
'{"cached_tokens":3,"cache_write_tokens":2},"cost":' + cost + '}}').encode()
|
||||
|
||||
|
||||
def sse(body):
|
||||
return b'data: {"choices":[{"delta":{"content":"private fixture text"},"index":0}]}\n\n' \
|
||||
+ b"data: " + body + b"\n\ndata: [DONE]\n\n"
|
||||
|
||||
|
||||
class Accounting(unittest.TestCase):
|
||||
def test_every_chunk_boundary_preserves_reported_cost_and_disjoint_cache_tokens(self):
|
||||
wire = sse(document())
|
||||
for boundary in range(len(wire) + 1):
|
||||
ledger = TurnBilling(); receipt = ledger.begin()
|
||||
receipt.feed(wire[:boundary]); receipt.feed(wire[boundary:])
|
||||
billed, tokens = ledger.finish()
|
||||
self.assertEqual(tokens, (15, 5, 3, 2))
|
||||
self.assertEqual(billed["amountUsdExact"], "0.004200000")
|
||||
self.assertTrue(billed["complete"])
|
||||
self.assertNotIn("private", json.dumps(billed))
|
||||
|
||||
def test_auxiliary_and_concurrent_requests_are_included_once_each(self):
|
||||
ledger = TurnBilling()
|
||||
def call():
|
||||
receipt = ledger.begin(); receipt.json(document("0.000000001"))
|
||||
receipt.json(document("0.000000001")) # a reread is not another wire request
|
||||
threads = [threading.Thread(target=call) for _ in range(12)]
|
||||
for thread in threads: thread.start()
|
||||
for thread in threads: thread.join()
|
||||
billed, tokens = ledger.finish()
|
||||
self.assertEqual(billed["amountUsdExact"], "0.000000012")
|
||||
self.assertEqual(billed["requestCount"], 12)
|
||||
self.assertEqual(tokens, (180, 60, 36, 24))
|
||||
|
||||
def test_pending_failed_retry_and_background_work_never_certify_a_subtotal(self):
|
||||
for scenario in ("pending", "failed", "background"):
|
||||
ledger = TurnBilling(); ledger.begin().json(document())
|
||||
if scenario != "background":
|
||||
pending = ledger.begin()
|
||||
if scenario == "failed": pending.failed()
|
||||
billed, tokens = ledger.finish(owned_work_complete=scenario != "background")
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertEqual(billed["amountUsdExact"], "0.004200000")
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
def test_explicit_zero_and_no_request_are_distinct_from_missing_price(self):
|
||||
ledger = TurnBilling(); ledger.begin().json(document("0"))
|
||||
billed, tokens = ledger.finish()
|
||||
self.assertTrue(billed["complete"])
|
||||
self.assertEqual(tokens, (15, 5, 3, 2))
|
||||
self.assertEqual(billed["amountUsdExact"], "0.000000000")
|
||||
billed, tokens = TurnBilling().finish()
|
||||
self.assertTrue(billed["complete"])
|
||||
self.assertEqual(tokens, (0, 0, 0, 0))
|
||||
ledger = TurnBilling(); ledger.begin().json(document("null"))
|
||||
billed, tokens = ledger.finish()
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertEqual(tokens, (15, 5, 3, 2))
|
||||
|
||||
def test_invalid_costs_missing_usage_and_conflicting_receipts_are_unpriced(self):
|
||||
for body in [document(value) for value in ("-1", "true", '"0.1"', "NaN", "1e900", "1000001")]:
|
||||
ledger = TurnBilling(); ledger.begin().json(body)
|
||||
self.assertFalse(ledger.finish()[0]["complete"])
|
||||
for body in (b"{}", b"not-json", b'{"error":{"message":"private"}}'):
|
||||
ledger = TurnBilling(); ledger.begin().json(body)
|
||||
self.assertFalse(ledger.finish()[0]["complete"])
|
||||
ledger = TurnBilling(); receipt = ledger.begin()
|
||||
receipt.json(document()); receipt.json(document("0.01"))
|
||||
self.assertFalse(ledger.finish()[0]["complete"])
|
||||
|
||||
def test_truncation_and_oversized_frames_are_unpriced(self):
|
||||
for wire in (b"data: " + document() + b"\n", b"data: " + b"x" * (MAX_FRAME + 1)):
|
||||
ledger = TurnBilling(); receipt = ledger.begin(); receipt.feed(wire); receipt.eof()
|
||||
billed, tokens = ledger.finish()
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
|
||||
class Transport(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
install_billing_capture()
|
||||
install_billing_capture() # idempotent, including the async fallback
|
||||
|
||||
def setUp(self):
|
||||
self.env = patch.dict("os.environ", {"OPENROUTER_API_KEY": "synthetic-fixture-only"})
|
||||
self.env.start()
|
||||
self.ledger = start_turn(SimpleNamespace(provider="openrouter", api_mode="chat_completions"))
|
||||
|
||||
def tearDown(self):
|
||||
close_turn(self.ledger)
|
||||
self.env.stop()
|
||||
|
||||
def test_real_pinned_sdk_stream_keeps_text_and_cost_without_an_agent_callback(self):
|
||||
body = document().replace(b'"choices":[]', b'"choices":[{"delta":{},"index":0,"finish_reason":"stop"}]')
|
||||
transport = httpx.MockTransport(lambda request: httpx.Response(200, headers={"content-type": "text/event-stream"},
|
||||
stream=httpx.ByteStream(sse(body))))
|
||||
with httpx.Client(transport=transport) as client, OpenAI(http_client=client, api_key="synthetic-fixture-only", base_url="https://openrouter.ai/api/v1") as sdk:
|
||||
with sdk.chat.completions.create(model="fixture-model", messages=[{"role": "user", "content": "private fixture prompt"}], stream=True) as stream:
|
||||
chunks = list(stream)
|
||||
self.assertEqual(chunks[0].choices[0].delta.content, "private fixture text")
|
||||
self.assertEqual(self.ledger.finish()[0]["amountUsdExact"], "0.004200000")
|
||||
|
||||
def test_sdk_retry_is_a_second_attempt_and_preserves_unknown_cost(self):
|
||||
calls = []
|
||||
def reply(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(429, json={"error": {"message": "retry fixture"}}, headers={"retry-after-ms": "1"}) if len(calls) == 1 \
|
||||
else httpx.Response(200, content=document())
|
||||
with httpx.Client(transport=httpx.MockTransport(reply)) as client, OpenAI(http_client=client, api_key="synthetic-fixture-only", base_url="https://openrouter.ai/api/v1", max_retries=1) as sdk:
|
||||
sdk.chat.completions.create(model="fixture-model", messages=[])
|
||||
billed, tokens = self.ledger.finish()
|
||||
self.assertEqual(billed["requestCount"], 2)
|
||||
self.assertEqual(billed["reportedRequestCount"], 1)
|
||||
self.assertEqual(billed["amountUsdExact"], "0.004200000")
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
def test_account_or_endpoint_changes_cannot_mint_billing_for_the_selected_account(self):
|
||||
for url, key in [("https://other.test/v1/chat/completions", "synthetic-fixture-only"),
|
||||
("https://openrouter.ai/api/v1/chat/completions", "another-fixture-account"),
|
||||
("http://openrouter.ai/api/v1/chat/completions", "synthetic-fixture-only")]:
|
||||
with httpx.Client(transport=httpx.MockTransport(lambda request: httpx.Response(200, content=document()))) as client:
|
||||
client.post(url, headers={"authorization": "Bearer " + key})
|
||||
billed, tokens = self.ledger.finish()
|
||||
self.assertEqual(billed["requestCount"], 3)
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertEqual(billed["reportedRequestCount"], 0)
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
def test_future_async_inference_is_unknown_instead_of_silently_free(self):
|
||||
async def call():
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(lambda request: httpx.Response(200, content=document()))) as client:
|
||||
await client.post("https://openrouter.ai/api/v1/chat/completions", headers={"authorization": "Bearer synthetic-fixture-only"})
|
||||
asyncio.run(call())
|
||||
billed, tokens = self.ledger.finish()
|
||||
self.assertEqual(billed["requestCount"], 1)
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
def test_delayed_background_dispatch_is_fenced_before_a_child_can_start(self):
|
||||
calls = []
|
||||
dispatch = fence_background_dispatch(lambda value: calls.append(value) or "native dispatch result")
|
||||
self.assertEqual(dispatch("batch"), "native dispatch result")
|
||||
self.assertEqual(calls, ["batch"])
|
||||
billed, tokens = self.ledger.finish()
|
||||
self.assertFalse(billed["complete"])
|
||||
self.assertIsNone(tokens)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -8,6 +8,7 @@ from unittest.mock import AsyncMock, Mock, patch
|
||||
from acp.schema import ClientCapabilities
|
||||
from bridge import ManagedHermesACPAgent, ManagedSessionManager, ChannelClient, native_answers, question_set, install_no_auth_transport, turn_usage, USAGE_COUNTERS
|
||||
from policy import authorize_tool
|
||||
from billing import TurnBilling
|
||||
|
||||
|
||||
class Accounting(unittest.TestCase):
|
||||
@@ -163,6 +164,39 @@ class Controls(unittest.IsolatedAsyncioTestCase):
|
||||
self.bridge._negotiated = True
|
||||
self.bridge._active = ("session", "turn")
|
||||
|
||||
async def test_reported_wire_usage_includes_calls_missing_from_native_accepted_response_counters(self):
|
||||
self.state.agent = SimpleNamespace(**{"session_" + key: 0 for key in USAGE_COUNTERS},
|
||||
_paperclip_usage_receipts=[True], _paperclip_cost_receipts=[True])
|
||||
# Native session counters can reset during compaction. Complete wire
|
||||
# receipts remain scoped to this turn instead of subtracting sessions.
|
||||
self.bridge._usage_before = dict.fromkeys(USAGE_COUNTERS, 100)
|
||||
self.bridge._billing = TurnBilling()
|
||||
for _ in range(2):
|
||||
self.bridge._billing.begin().json(b'{"usage":{"prompt_tokens":20,"completion_tokens":5,"cost":0.0042}}')
|
||||
conn = SimpleNamespace(ext_notification=AsyncMock())
|
||||
with patch("bridge.HermesACPAgent._finish_turn", new=AsyncMock(return_value=SimpleNamespace(usage=None))):
|
||||
response = await self.bridge._finish_turn(self.state, "session", conn, {}, None, True)
|
||||
self.assertEqual(response.usage.input_tokens, 40)
|
||||
self.assertEqual(response.usage.output_tokens, 10)
|
||||
params = conn.ext_notification.call_args.args[1]
|
||||
self.assertEqual(params["turnToken"], "turn")
|
||||
self.assertEqual(params["billing"]["amountUsdExact"], "0.008400000")
|
||||
self.assertEqual(params["billing"]["reportedRequestCount"], 2)
|
||||
self.assertEqual(params["cost"], "unavailable") # no smaller model-price estimate
|
||||
|
||||
async def test_missing_wire_usage_cannot_fall_back_to_the_smaller_native_receipt(self):
|
||||
self.state.agent = SimpleNamespace(**{"session_" + key: 0 for key in USAGE_COUNTERS},
|
||||
_paperclip_usage_receipts=[True], _paperclip_cost_receipts=[True])
|
||||
self.bridge._usage_before = dict.fromkeys(USAGE_COUNTERS, 0)
|
||||
self.bridge._billing = TurnBilling()
|
||||
self.bridge._billing.begin().json(b'{"usage":{"prompt_tokens":20,"completion_tokens":5,"cost":0.0042}}')
|
||||
self.bridge._billing.begin() # interrupted request never returned usage
|
||||
conn = SimpleNamespace(ext_notification=AsyncMock())
|
||||
with patch("bridge.HermesACPAgent._finish_turn", new=AsyncMock(return_value=SimpleNamespace(usage=None))):
|
||||
response = await self.bridge._finish_turn(self.state, "session", conn, {}, None, True)
|
||||
self.assertIsNone(response.usage)
|
||||
self.assertFalse(conn.ext_notification.call_args.args[1]["billing"]["complete"])
|
||||
|
||||
async def test_steering_is_bound_to_live_session_and_turn(self):
|
||||
for session, token in [("other", "turn"), ("session", "old")]:
|
||||
response = await self.bridge.ext_method("hermes/steer", {
|
||||
|
||||
@@ -323,6 +323,7 @@ import {
|
||||
buildRemoteCodexLauncherCommand,
|
||||
mayUsePreinstalledRunnerArtifact,
|
||||
nativeUsageCostUsd,
|
||||
createNativeTurnAccounting,
|
||||
nativeUsageBiller,
|
||||
normalizeNativeUsage,
|
||||
resolveNativeBilling,
|
||||
@@ -1001,6 +1002,45 @@ describe("native incomplete-bootstrap evidence", () => {
|
||||
});
|
||||
|
||||
describe("native provider usage normalization", () => {
|
||||
const hermes = { kind: "acpx", agent: "hermes", model: "fixture-model" } as NativeExecutionInput["provider"];
|
||||
const billing = { schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: true, requestCount: 2, reportedRequestCount: 2, amountUsd: 0.25, amountUsdExact: "0.250000000" };
|
||||
const priced = { runDelta: { inputTokens: 20, outputTokens: 5, providerCostUsd: 0.25 }, billing };
|
||||
it("requires exact selected-account price authority for Hermes instead of a scalar or cumulative cost", () => {
|
||||
expect(nativeUsageCostUsd(priced, hermes, "openrouter")).toBe(0.25);
|
||||
for (const selectedBiller of [undefined, "unknown", "anthropic"]) expect(nativeUsageCostUsd(priced, hermes, selectedBiller)).toBeUndefined();
|
||||
for (const candidate of [{ ...priced, billing: undefined }, { ...priced, billing: { ...billing, source: "rate_card" } },
|
||||
{ ...priced, runDelta: { ...priced.runDelta, providerCostUsd: 1 } }]) {
|
||||
expect(nativeUsageCostUsd(candidate, hermes, "openrouter")).toBeUndefined();
|
||||
}
|
||||
expect(nativeUsageCostUsd({ ...priced, billing: { ...billing, complete: false, reportedRequestCount: 1 } }, hermes, "openrouter")).toBe(0.25);
|
||||
const zero = { ...priced, runDelta: { ...priced.runDelta, providerCostUsd: 0 }, billing: { ...billing, amountUsd: 0, amountUsdExact: "0.000000000" } };
|
||||
expect(nativeUsageCostUsd(zero, hermes, "openrouter")).toBe(0);
|
||||
expect(nativeUsageCostUsd({ ...zero, billing: { ...zero.billing, complete: false } }, hermes, "openrouter")).toBeUndefined();
|
||||
});
|
||||
it("rebuilds exact per-turn price totals across replay and preserves unknown earlier work", () => {
|
||||
const accounting = createNativeTurnAccounting(hermes, "openrouter");
|
||||
let sequence = 0;
|
||||
const event = (eventType: string, turnId: string, payload: object = {}) => ({
|
||||
eventType, turnId, payload, sourceSeq: ++sequence, sourceInstanceId: "provider",
|
||||
}) as unknown as PrpEvent;
|
||||
accounting.observe(event("turn.started", "first"));
|
||||
const price = event("item.completed", "first", { kind: "usage", usage: priced });
|
||||
accounting.observe(price); accounting.observe(price);
|
||||
expect(accounting.snapshot().complete).toBe(false);
|
||||
accounting.observe(event("turn.completed", "first"));
|
||||
expect(accounting.snapshot().usage?.accountingCostUsdExact).toBe("0.250000000");
|
||||
accounting.observe(event("turn.started", "second"));
|
||||
accounting.observe(event("item.completed", "second", { kind: "usage", usage: {
|
||||
runDelta: { inputTokens: 10, outputTokens: 2, providerCostUsd: 999 },
|
||||
} }));
|
||||
accounting.observe(event("turn.completed", "second"));
|
||||
const snapshot = accounting.snapshot();
|
||||
expect(snapshot.complete).toBe(true);
|
||||
expect(snapshot.usage).toMatchObject({ accountingCostUsdExact: "0.250000000", accountingCostIncomplete: true,
|
||||
billing: { complete: false, amountUsd: 0.25 } });
|
||||
expect(nativeUsageCostUsd(snapshot.usage, hermes, "openrouter")).toBe(0.25);
|
||||
});
|
||||
it.each([
|
||||
{}, { inputTokens: 1 }, { outputTokens: 1 },
|
||||
{ inputTokens: 0, outputTokens: -1 }, { inputTokens: 0.5, outputTokens: 1 },
|
||||
@@ -5515,6 +5555,46 @@ describe("native terminal-turn accounting", () => {
|
||||
if (scenario === "late_update") expect(result.usage).toMatchObject({ inputTokens: 32, outputTokens: 8 });
|
||||
});
|
||||
|
||||
it.each(["reported", "partial", "legacy", "wrong_account", "zero"] as const)("settles bound Hermes billing through the executor (%s)", async scenario => {
|
||||
const previous = process.env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION;
|
||||
process.env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION = JSON.stringify([{ agent: "hermes", model: "fixture-model" }]);
|
||||
try {
|
||||
const onUsage = vi.fn(async (_receipt: import("@paperclipai/adapter-utils").AdapterUsageCheckpoint) => {});
|
||||
const amount = scenario === "zero" ? 0 : 0.25;
|
||||
const billing = { schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: scenario !== "partial", requestCount: 2, reportedRequestCount: scenario === "partial" ? 1 : 2,
|
||||
amountUsd: amount, amountUsdExact: amount.toFixed(9) };
|
||||
const notification = (await import("@paperclipai/paperclip-runner/live")).rehydrateRunnerdUsageNotification({
|
||||
runDeltaAvailable: true, runDelta: { inputTokens: 20, outputTokens: 5, cacheReadTokens: 3, cacheWriteTokens: 2, providerCostUsd: amount },
|
||||
...(scenario === "legacy" ? {} : { billing }),
|
||||
}, "session", "turn");
|
||||
const event = (eventType: string, sourceSeq: number, payload: object = {}) => ({ eventType, turnId: "turn", payload,
|
||||
sourceSeq, sourceInstanceId: "provider", emittedAt: new Date().toISOString() }) as unknown as PrpEvent;
|
||||
state.execute.mockReset().mockImplementationOnce(async () => {
|
||||
await accountingEvents.committed!(event("turn.started", 1));
|
||||
const usage = event("item.completed", 2, { kind: "usage", usage: notification.tokenUsage });
|
||||
await accountingEvents.committed!(usage);
|
||||
expect(onUsage.mock.calls.at(-1)![0].complete).toBe(false);
|
||||
await accountingEvents.committed!(event("turn.completed", 3));
|
||||
await accountingEvents.duplicate!(usage);
|
||||
return { result: { summary: "cancelled" }, terminal: { runTerminalState: "cancelled" }, turnId: "turn",
|
||||
normalizedSessionId: "session", providerSessionId: null, driverKind: "test", driverVersion: "1", nativeEventCount: 3,
|
||||
highestContiguousSourceSeq: 3, usage: null };
|
||||
});
|
||||
const known = !["legacy", "wrong_account"].includes(scenario);
|
||||
const result = await executePaperclipNativeSession({ db: leaseDb(), runnerInstanceId: "runner", onUsage,
|
||||
billingIdentity: { provider: "deepseek", biller: scenario === "wrong_account" ? "anthropic" : "openrouter", billingType: "metered_api" },
|
||||
execution: { ...execution, provider: { kind: "acpx", agent: "hermes", model: "fixture-model", permissionMode: "approve-all", profile: QUALIFIED_ACPX_PROFILES.hermes } } as NativeExecutionInput });
|
||||
expect(result).toMatchObject({ usageComplete: true, costUsd: known ? amount : null,
|
||||
costStatus: known && scenario !== "partial" ? "reported" : "unpriced" });
|
||||
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ complete: true, costUsd: result.costUsd, costStatus: result.costStatus }));
|
||||
if (known) expect(result.pricingProvenance).toMatchObject({ source: "provider_reported", version: "hermes-openrouter-wire/v1" });
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION;
|
||||
else process.env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION = previous;
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["committed", "duplicate"] as const)("saves completed usage before returning to result commitment (%s)", async mode => {
|
||||
const onUsage = vi.fn(async (_receipt: import("@paperclipai/adapter-utils").AdapterUsageCheckpoint) => {});
|
||||
const failure = new Error("stopped after result commit, before executor returns");
|
||||
|
||||
@@ -4,6 +4,7 @@ import { dotRunnerBroker } from "../dot-runner-broker.js";
|
||||
import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js";
|
||||
import { CURSOR_DISTRIBUTION_PINS, QUALIFIED_ACPX_PROFILES, QUALIFIED_ACPX_VERSION } from "../../vendor/paperclip-runner/index.js";
|
||||
import { isProviderMode } from "../../vendor/paperclip-runner/index.js";
|
||||
import { readProviderUsageBilling } from "../../vendor/paperclip-runner/index.js";
|
||||
import { bundledRemoteProviderPackManifestPath, bundledRemoteRunnerBinary } from "../../vendor/paperclip-runner/index.js";
|
||||
import { nativeRetryCancellationEligible, rethrowNativeCancellationLockConflict, assertCancellationRequest, cancellationIntentId as callerCancellationIntentId, cancellationRequestId } from "./native-cancellation-request.js";
|
||||
import { readNativePlanWait } from "./native-plan-wait.js";
|
||||
@@ -7977,7 +7978,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
let nativeAccountingComplete = false;
|
||||
// OpenCode and ACPX report their current turn; Codex reports a cumulative run delta.
|
||||
// Rebuild per-turn state from the durable event log when a controller resumes.
|
||||
const turnAccounting = ["opencode", "acpx"].includes(input.execution.provider.kind) ? createNativeTurnAccounting(input.execution.provider) : null;
|
||||
const selectedBiller = resolveNativeBilling(input.execution.provider, input.runnerEnvironment, input.billingIdentity).biller;
|
||||
const turnAccounting = ["opencode", "acpx"].includes(input.execution.provider.kind) ? createNativeTurnAccounting(input.execution.provider, selectedBiller) : null;
|
||||
if (turnAccounting) {
|
||||
const history = await input.db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, input.execution.binding.companyId), eq(heartbeatRunEvents.runId, input.execution.binding.runId),
|
||||
@@ -7988,20 +7990,22 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
}
|
||||
const persistAccountingUsage = async (usage: Record<string, unknown> | null, complete: boolean) => {
|
||||
const accountingUsage = normalizeNativeUsage(usage, { inputIncludesCacheReads: input.execution.provider.kind === "codex" });
|
||||
const accountingCost = accountingUsage ? nativeUsageCostUsd(usage, input.execution.provider) : undefined;
|
||||
const accountingCost = accountingUsage ? nativeUsageCostUsd(usage, input.execution.provider, selectedBiller) : undefined;
|
||||
complete = complete && accountingUsage !== undefined;
|
||||
const costStatus = complete && usage?.accountingCostIncomplete !== true
|
||||
? input.execution.provider.kind === "aws_agentcore" ? "estimated" as const : undefined
|
||||
? input.execution.provider.kind === "aws_agentcore" ? "estimated" as const : nativeHermesPriceEvidence(usage, input.execution.provider, selectedBiller) ? "reported" as const : undefined
|
||||
: "unpriced" as const;
|
||||
const billing = resolveNativeBilling(input.execution.provider, input.runnerEnvironment, input.billingIdentity);
|
||||
const costUsdExact = typeof usage?.accountingCostUsdExact === "string" ? usage.accountingCostUsdExact : undefined;
|
||||
await input.onUsage?.({ usage: accountingUsage, ...billing, complete, usageBasis: "per_run", costUsdExact,
|
||||
model: input.execution.provider.model ?? "unknown", costUsd: accountingCost ?? null, costStatus });
|
||||
model: input.execution.provider.model ?? "unknown", costUsd: accountingCost ?? null, costStatus,
|
||||
...nativeHermesPriceEvidence(usage, input.execution.provider, selectedBiller) });
|
||||
if (!input.onUsage) await input.db.update(heartbeatRuns).set({
|
||||
costAccountingPending: true,
|
||||
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({
|
||||
...accountingUsage, ...billing, accountingReceiptReady: complete, model: input.execution.provider.model ?? "unknown",
|
||||
costUsd: accountingCost ?? null, costUsdExact: costUsdExact ?? null, costStatus: costStatus ?? null, usageSource: "per_run",
|
||||
...nativeHermesPriceEvidence(usage, input.execution.provider, selectedBiller),
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(heartbeatRuns.id, input.execution.binding.runId), eq(heartbeatRuns.companyId, input.execution.binding.companyId), isNull(heartbeatRuns.costAccountedAt)));
|
||||
};
|
||||
@@ -9501,12 +9505,13 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
model: input.execution.provider.model,
|
||||
usage: normalizeNativeUsage(native.usage, { inputIncludesCacheReads: input.execution.provider.kind === "codex" }),
|
||||
costUsd: input.execution.provider.kind !== "openai_dot" && normalizeNativeUsage(native.usage, { inputIncludesCacheReads: input.execution.provider.kind === "codex" })
|
||||
? nativeUsageCostUsd(native.usage, input.execution.provider) ?? null : null,
|
||||
? nativeUsageCostUsd(native.usage, input.execution.provider, selectedBiller) ?? null : null,
|
||||
costUsdExact: typeof native.usage?.accountingCostUsdExact === "string" ? native.usage.accountingCostUsdExact : undefined,
|
||||
costStatus: nativeAccountingComplete && native.usage?.accountingCostIncomplete !== true
|
||||
? input.execution.provider.kind === "aws_agentcore" ? "estimated" : undefined
|
||||
? input.execution.provider.kind === "aws_agentcore" ? "estimated" : nativeHermesPriceEvidence(native.usage, input.execution.provider, selectedBiller) ? "reported" : undefined
|
||||
: "unpriced",
|
||||
usageComplete: nativeAccountingComplete,
|
||||
...nativeHermesPriceEvidence(native.usage, input.execution.provider, selectedBiller),
|
||||
usageBasis: "per_run",
|
||||
nativeFinalization: finalization,
|
||||
};
|
||||
@@ -9575,7 +9580,7 @@ export function nativeUsageBiller(provider: NativeExecutionInput["provider"]): s
|
||||
return "openai";
|
||||
}
|
||||
|
||||
function createNativeTurnAccounting(provider: NativeExecutionInput["provider"]) {
|
||||
export function createNativeTurnAccounting(provider: NativeExecutionInput["provider"], selectedBiller?: string | null) {
|
||||
const turns = new Map<string, { usage: Record<string, unknown> | null; terminal: boolean }>();
|
||||
const sequences = new Map<string, number>();
|
||||
let currentTurn: string | undefined;
|
||||
@@ -9584,6 +9589,7 @@ function createNativeTurnAccounting(provider: NativeExecutionInput["provider"])
|
||||
if (turns.size === 0) return { usage: null, turnId: undefined, complete: false };
|
||||
let inputTokens = 0, outputTokens = 0, cacheReadTokens = 0, cacheWriteTokens = 0;
|
||||
let cost = 0n, costKnown = false, costIncomplete = false, complete = true;
|
||||
let requestCount = 0, reportedRequestCount = 0;
|
||||
for (const [turnId, turn] of turns) {
|
||||
const normalized = normalizeNativeUsage(turn.usage);
|
||||
complete &&= turnId !== "unattributed" && turn.terminal && normalized !== undefined;
|
||||
@@ -9591,9 +9597,18 @@ function createNativeTurnAccounting(provider: NativeExecutionInput["provider"])
|
||||
outputTokens += normalized?.outputTokens ?? 0;
|
||||
cacheReadTokens += normalized?.cachedInputTokens ?? 0;
|
||||
cacheWriteTokens += normalized?.cacheWriteTokens ?? 0;
|
||||
const price = nativeUsageCostUsd(turn.usage, provider);
|
||||
const price = nativeUsageCostUsd(turn.usage, provider, selectedBiller);
|
||||
const billing = provider.kind === "acpx" && provider.agent === "hermes"
|
||||
? readProviderUsageBilling(turn.usage?.billing) : null;
|
||||
if (price === undefined) costIncomplete = true;
|
||||
else { cost += usdToUnits(price); costKnown = true; }
|
||||
else {
|
||||
cost += usdToUnits(billing?.amountUsdExact ?? price); costKnown = true;
|
||||
if (billing) {
|
||||
costIncomplete ||= !billing.complete;
|
||||
requestCount += billing.requestCount;
|
||||
reportedRequestCount += billing.reportedRequestCount;
|
||||
}
|
||||
}
|
||||
}
|
||||
// A free empty attempt contributes no price evidence for later paid work.
|
||||
// Keep an incomplete zero subtotal absent so complete aggregate tokens can
|
||||
@@ -9602,8 +9617,14 @@ function createNativeTurnAccounting(provider: NativeExecutionInput["provider"])
|
||||
const runDelta = { inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens,
|
||||
...(retainCost ? { providerCostUsd: Number(cost) / 1_000_000_000 } : {}) };
|
||||
const current = currentTurn ? turns.get(currentTurn) : undefined;
|
||||
const costExact = `${cost / 1_000_000_000n}.${String(cost % 1_000_000_000n).padStart(9, "0")}`;
|
||||
return { usage: { runDelta, accountingCostIncomplete: costIncomplete,
|
||||
...(retainCost ? { accountingCostUsdExact: `${cost / 1_000_000_000n}.${String(cost % 1_000_000_000n).padStart(9, "0")}` } : {}) },
|
||||
...(retainCost && provider.kind === "acpx" && provider.agent === "hermes" ? { billing: {
|
||||
schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: !costIncomplete, requestCount, reportedRequestCount,
|
||||
amountUsd: Number(costExact), amountUsdExact: costExact,
|
||||
} } : {}),
|
||||
...(retainCost ? { accountingCostUsdExact: costExact } : {}) },
|
||||
turnId: current && normalizeNativeUsage(current.usage) ? currentTurn : undefined, complete };
|
||||
};
|
||||
return {
|
||||
@@ -9628,7 +9649,8 @@ function createNativeTurnAccounting(provider: NativeExecutionInput["provider"])
|
||||
if (payload.kind === "usage") {
|
||||
const usage = record(payload.usage);
|
||||
turn.usage = Object.hasOwn(usage, "runDelta")
|
||||
? { runDelta: record(usage.runDelta), runDeltaComplete: usage.runDeltaComplete } : null;
|
||||
? { runDelta: record(usage.runDelta), runDeltaComplete: usage.runDeltaComplete,
|
||||
...(usage.billing === undefined ? {} : { billing: usage.billing }) } : null;
|
||||
}
|
||||
if (terminal) turn.terminal = true;
|
||||
turns.set(key, turn);
|
||||
@@ -9646,11 +9668,20 @@ function createNativeTurnAccounting(provider: NativeExecutionInput["provider"])
|
||||
export function nativeUsageCostUsd(
|
||||
usage: Record<string, unknown> | null,
|
||||
provider?: NativeExecutionInput["provider"],
|
||||
selectedBiller?: string | null,
|
||||
) {
|
||||
// The ACP normalization contract fills absent per-turn cost with zero and
|
||||
// reports actual cost cumulatively. Until it carries an authoritative run
|
||||
// delta with provenance, neither value is a candidate's billed USD receipt.
|
||||
if (provider?.kind === "acpx" && ["cursor", "copilot", "pi", "hermes"].includes(provider.agent)) return undefined;
|
||||
if (provider?.kind === "acpx" && provider.agent === "hermes") {
|
||||
const billing = readProviderUsageBilling(usage?.billing);
|
||||
if (!usage || !billing || selectedBiller !== billing.biller
|
||||
|| nativeUsageMeasurement(usage).providerCostUsd !== billing.amountUsd) return undefined;
|
||||
// Explicit provider-reported zero is authoritative only for a complete
|
||||
// receipt. Positive observed subtotals survive an incomplete request set.
|
||||
return billing.complete || billing.amountUsd > 0 ? billing.amountUsd : undefined;
|
||||
}
|
||||
if (provider?.kind === "acpx" && ["cursor", "copilot", "pi"].includes(provider.agent)) return undefined;
|
||||
if (!usage) return undefined;
|
||||
const measurement = nativeUsageMeasurement(usage);
|
||||
const hasRunDelta = Object.hasOwn(usage, "runDelta") || Object.hasOwn(record(usage.usage), "runDelta");
|
||||
@@ -9668,6 +9699,12 @@ export function nativeUsageCostUsd(
|
||||
return numericUsageField(cost, ["amount", "total"]);
|
||||
}
|
||||
|
||||
function nativeHermesPriceEvidence(usage: Record<string, unknown> | null, provider: NativeExecutionInput["provider"], selectedBiller?: string | null) {
|
||||
if (provider.kind !== "acpx" || provider.agent !== "hermes" || nativeUsageCostUsd(usage, provider, selectedBiller) === undefined) return undefined;
|
||||
return { pricingProvenance: { source: "provider_reported" as const, version: "hermes-openrouter-wire/v1",
|
||||
evidence: "Selected-account OpenRouter response usage.cost; incomplete attempts remain unpriced." } };
|
||||
}
|
||||
|
||||
/** Billing identity follows the selected runtime and resolved credentials.
|
||||
* Unknown authentication remains unknown; a provider name does not prove billing mode. */
|
||||
export function resolveNativeBilling(provider: NativeExecutionInput["provider"], env: NodeJS.ProcessEnv = {}, identity?: Pick<AdapterExecutionResult, "provider" | "biller" | "billingType">): Pick<AdapterExecutionResult, "provider" | "biller" | "billingType"> {
|
||||
|
||||
+1
@@ -130,6 +130,7 @@ export const parseNativeRuntimeContext = runner.parseNativeRuntimeContext;
|
||||
export const parsePaperclipQuestionSet = runner.parsePaperclipQuestionSet;
|
||||
export const parsePaperclipQuestionResponse =
|
||||
runner.parsePaperclipQuestionResponse;
|
||||
export const readProviderUsageBilling = runner.readProviderUsageBilling;
|
||||
export const resolveQualifiedAcpxProfile = runner.resolveQualifiedAcpxProfile;
|
||||
export const QUALIFIED_ACPX_PROFILES = runner.QUALIFIED_ACPX_PROFILES;
|
||||
export const QUALIFIED_ACPX_VERSION = runner.QUALIFIED_ACPX_VERSION;
|
||||
|
||||
@@ -303,6 +303,10 @@ definition includes the account/runner harness source digest. Its API-only scope
|
||||
does not substitute for subscription or lifecycle qualification.
|
||||
The task permits one attempt. Both public 200-cent budgets must match the fixture
|
||||
scope before task creation; unlimited or foreign budget records fail admission.
|
||||
OpenRouter also requires a settled provider-reported price with an exact USD
|
||||
amount and healthy company/agent budget state before fixture teardown. The
|
||||
oracle reads scoped public run, company and agent records; it never derives
|
||||
billing or health from the model's answer or the cleanup pause.
|
||||
|
||||
## Persistent agent files
|
||||
|
||||
|
||||
@@ -1542,6 +1542,11 @@ the ordinary completion, screenshot, token/cost and cleanup evidence.
|
||||
Each cell admits one attempt and configures 200-cent company and agent budgets.
|
||||
Public budget readback must pass before task creation. Unpriced model usage
|
||||
remains unknown; this configured limit is not an exact billing receipt.
|
||||
The OpenRouter cell additionally waits for its public run's reported wire cost
|
||||
to settle and checks that the company remains active and the agent idle with
|
||||
no pause reason before cleanup. Its separate settlement snapshot rejects missing
|
||||
price evidence, estimates and an unpriced budget pause even when the answer is
|
||||
correct. Other providers retain completion-only coverage.
|
||||
Before creating the task, public account readback must identify the selected
|
||||
connected personal account, its company and owner, and the authenticated caller.
|
||||
Both run attribution fields must match that independently recorded owner.
|
||||
|
||||
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
|
||||
import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, gradeHermesApiConnection } from "./hermes-api-connections.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection } from "./hermes-api-connections.js";
|
||||
|
||||
describe("Hermes managed API connection qualification", () => {
|
||||
const suite = runnerSuites.find(s => s.id === "hermes-api-connections")!;
|
||||
@@ -41,6 +41,39 @@ describe("Hermes managed API connection qualification", () => {
|
||||
it("accepts independently observed account/model metadata", () => {
|
||||
expect(gradeHermesApiConnection(valid).every(check => check.passed)).toBe(true);
|
||||
});
|
||||
it.each(["valid", "reported-zero", "company-scope", "agent-scope", "run-scope", "task-scope", "paused-agent", "paused-company", "pause-reason", "budget-changed",
|
||||
"pending", "missing-settlement", "missing-price", "estimated", "partial", "wrong-biller", "wrong-provenance", "mismatched-exact", "unknown-tokens"])("calibrates public OpenRouter settlement: %s", async fault => {
|
||||
const company: Record<string, unknown> = { id: "company", status: "active", pauseReason: null, budgetMonthlyCents: 200 };
|
||||
const agent: Record<string, unknown> = { id: "agent", companyId: "company", status: "idle", pauseReason: null, budgetMonthlyCents: 200 };
|
||||
const usage: Record<string, unknown> = { biller: "openrouter", billingType: "metered_api", costStatus: "reported", costUsd: 0.0042, costUsdExact: "0.004200000",
|
||||
inputTokens: 40, outputTokens: 10, accountingReceiptReady: true, pricingProvenance: { source: "provider_reported", version: "hermes-openrouter-wire/v1" } };
|
||||
const run: Record<string, unknown> = { id: "run", companyId: "company", agentId: "agent", issueId: "task", status: "succeeded", usageJson: usage,
|
||||
costAccountingPending: false, costAccountedAt: "2026-10-08T03:00:00Z" };
|
||||
if (fault === "reported-zero") { usage.costUsd = 0; usage.costUsdExact = "0.000000000"; }
|
||||
if (fault === "company-scope") company.id = "foreign";
|
||||
if (fault === "agent-scope") agent.companyId = "foreign";
|
||||
if (fault === "run-scope") run.agentId = "foreign";
|
||||
if (fault === "task-scope") run.issueId = "foreign";
|
||||
if (fault === "paused-agent") agent.status = "paused";
|
||||
if (fault === "paused-company") company.status = "paused";
|
||||
if (fault === "pause-reason") agent.pauseReason = "budget_unpriced";
|
||||
if (fault === "budget-changed") agent.budgetMonthlyCents = 0;
|
||||
if (fault === "pending") run.costAccountingPending = true;
|
||||
if (fault === "missing-settlement") delete run.costAccountedAt;
|
||||
if (fault === "missing-price") { usage.costUsd = null; usage.costUsdExact = null; }
|
||||
if (fault === "estimated") usage.costStatus = "estimated";
|
||||
if (fault === "partial") usage.costStatus = "unpriced";
|
||||
if (fault === "wrong-biller") usage.biller = "openai";
|
||||
if (fault === "wrong-provenance") usage.pricingProvenance = { source: "model_prices" };
|
||||
if (fault === "mismatched-exact") usage.costUsdExact = "0.040000000";
|
||||
if (fault === "unknown-tokens") delete usage.inputTokens;
|
||||
const paths: string[] = [];
|
||||
const receipt = await captureHermesOpenRouterSettlement({ companyId: "company", agentId: "agent", issueId: "task", runId: "run", api: {
|
||||
async get<T>(url: string) { paths.push(url); return (url === "/api/companies/company" ? company : url === "/api/agents/agent" ? agent : run) as T; },
|
||||
} });
|
||||
expect(new Set(paths)).toEqual(new Set(["/api/companies/company", "/api/agents/agent", "/api/heartbeat-runs/run"]));
|
||||
expect(receipt.checks.every(check => check.passed)).toBe(["valid", "reported-zero"].includes(fault));
|
||||
});
|
||||
it.each(["valid", "missing", "duplicate", "company", "provider", "method", "ownership", "status", "owner", "caller"])("establishes the expected user from public account readback: %s", async fault => {
|
||||
const account = { id: "account", companyId: "company", provider: "xai", method: "api_key", ownership: "personal", ownerUserId: "user", status: "connected" };
|
||||
if (fault === "company") account.companyId = "foreign";
|
||||
|
||||
@@ -53,6 +53,37 @@ export async function captureHermesApiBudgets(input: {
|
||||
] };
|
||||
}
|
||||
|
||||
/** Observe settled billing and budget health before fixture cleanup pauses the agent. */
|
||||
export async function captureHermesOpenRouterSettlement(input: {
|
||||
api: { get<T>(path: string): Promise<T> }; companyId: string; agentId: string; issueId: string; runId: string;
|
||||
}) {
|
||||
const [company, agent, run] = await Promise.all([
|
||||
input.api.get<Record<string, unknown>>(`/api/companies/${input.companyId}`),
|
||||
input.api.get<Record<string, unknown>>(`/api/agents/${input.agentId}`),
|
||||
input.api.get<Record<string, unknown>>(`/api/heartbeat-runs/${input.runId}`),
|
||||
]);
|
||||
const usage = record(run.usageJson), provenance = record(usage.pricingProvenance);
|
||||
const cost = usage.costUsd, exact = usage.costUsdExact;
|
||||
return { observation: {
|
||||
company: { id: company.id, status: company.status, pauseReason: company.pauseReason, budgetMonthlyCents: company.budgetMonthlyCents },
|
||||
agent: { id: agent.id, companyId: agent.companyId, status: agent.status, pauseReason: agent.pauseReason, budgetMonthlyCents: agent.budgetMonthlyCents },
|
||||
run: { id: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.issueId, status: run.status,
|
||||
costAccountingPending: run.costAccountingPending, costAccountedAt: run.costAccountedAt,
|
||||
usage: Object.fromEntries(["provider", "biller", "billingType", "costStatus", "costUsd", "costUsdExact", "inputTokens", "outputTokens", "accountingReceiptReady", "pricingProvenance"].map(key => [key, usage[key]])) },
|
||||
}, checks: [
|
||||
{ id: "billing-observation-scope", passed: company.id === input.companyId && agent.id === input.agentId && agent.companyId === input.companyId
|
||||
&& run.id === input.runId && run.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId },
|
||||
{ id: "settled-openrouter-reported-cost", passed: run.status === "succeeded" && run.costAccountingPending === false && present(run.costAccountedAt)
|
||||
&& usage.accountingReceiptReady === true && usage.biller === "openrouter" && usage.billingType === "metered_api" && usage.costStatus === "reported"
|
||||
&& provenance.source === "provider_reported" && provenance.version === "hermes-openrouter-wire/v1"
|
||||
&& typeof cost === "number" && Number.isFinite(cost) && cost >= 0 && typeof exact === "string" && /^(0|[1-9][0-9]{0,6})\.[0-9]{9}$/.test(exact) && Number(exact) === cost
|
||||
&& typeof usage.inputTokens === "number" && Number.isSafeInteger(usage.inputTokens) && usage.inputTokens >= 0
|
||||
&& typeof usage.outputTokens === "number" && Number.isSafeInteger(usage.outputTokens) && usage.outputTokens >= 0 && usage.inputTokens + usage.outputTokens > 0 },
|
||||
{ id: "budget-health-after-settlement", passed: company.status === "active" && company.pauseReason === null && agent.status === "idle" && agent.pauseReason === null
|
||||
&& company.budgetMonthlyCents === HERMES_API_CONNECTION_BUDGET_CENTS && agent.budgetMonthlyCents === HERMES_API_CONNECTION_BUDGET_CENTS },
|
||||
] };
|
||||
}
|
||||
|
||||
/** Grade public run/account/model metadata; a model's completion claim cannot supply it. */
|
||||
export function gradeHermesApiConnection(input: {
|
||||
companyId: string; agentId: string; issueId: string; connectionId: string; provider: string; model: string; expectedResponsibleUserId: string;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { runPlanTaskFlow } from "./plan-task-flow.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, gradeHermesApiConnection, isHermesConnectionSuite } from "./hermes-api-connections.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite } from "./hermes-api-connections.js";
|
||||
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
|
||||
import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js";
|
||||
import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js";
|
||||
@@ -2895,6 +2895,23 @@ for (const execution of executions) {
|
||||
{ id: "account-owner-preserved", passed: after.expectedResponsibleUserId === hermesApiAccountOwner.expectedResponsibleUserId });
|
||||
await writeSanitizedJson(snapshotsDir, "hermes-api-owner-after-execution.json", after, secrets);
|
||||
}
|
||||
if (fixtures.aiConnection.binding.provider === "openrouter") {
|
||||
const settlementScope = { companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id,
|
||||
runId: selectedRuns[0]?.id ?? "missing" };
|
||||
let receipt: Awaited<ReturnType<typeof captureHermesOpenRouterSettlement>> | undefined;
|
||||
try {
|
||||
await expect.poll(async () => {
|
||||
receipt = await captureHermesOpenRouterSettlement({ api, ...settlementScope });
|
||||
return receipt.checks.every(check => check.passed);
|
||||
}, { timeout: 30_000, message: "OpenRouter billing must settle without pausing the agent before cleanup" }).toBe(true);
|
||||
} finally {
|
||||
if (receipt) {
|
||||
await writeSanitizedJson(snapshotsDir, "hermes-openrouter-settlement.json", receipt, secrets);
|
||||
matcherResults.push(...receipt.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesBilling.${check.id}`, expected: true },
|
||||
passed: check.passed, detail: "Public reported cost and company/agent budget health must pass before teardown." })));
|
||||
}
|
||||
}
|
||||
}
|
||||
matcherResults.push(...checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesConnection.${check.id}`, expected: true }, passed: check.passed, detail: "Public native run metadata must match the selected managed API account and exact model." })));
|
||||
await writeSanitizedJson(snapshotsDir, "hermes-api-connection.json", { checks }, secrets);
|
||||
expect(checks.every(check => check.passed), "Hermes managed account and native model attribution").toBe(true);
|
||||
|
||||
Reference in new issue
Block a user