Files
PaperClipAI/server/src/services/native-runtime/native-usage-accounting.ts
T
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00

235 lines
12 KiB
TypeScript

import { usdToUnits } from "@paperclipai/shared";
import type { NativeExecutionInput, PrpEvent } from "../../vendor/paperclip-runner/index.js";
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
function numericUsageField(
usage: Record<string, unknown> | null,
keys: string[],
): number | undefined {
if (!usage) return undefined;
for (const key of keys) {
const value = usage[key];
if (typeof value === "number" && Number.isFinite(value) && value >= 0)
return value;
}
return undefined;
}
function nativeUsageMeasurement(usage: Record<string, unknown>) {
const nestedUsage = record(usage.usage);
// An explicit partial delta must not fall through to a full session total.
if (Object.hasOwn(usage, "runDelta")) return record(usage.runDelta);
if (Object.hasOwn(nestedUsage, "runDelta")) return record(nestedUsage.runDelta);
const candidates = [
record(usage.runDelta),
record(nestedUsage.runDelta),
record(usage.total),
record(nestedUsage.total),
record(usage.cumulative),
record(nestedUsage.cumulative),
nestedUsage,
usage,
];
return (
candidates.find(
(candidate) =>
numericUsageField(candidate, [
"inputTokens",
"input",
"promptTokens",
"outputTokens",
"output",
"completionTokens",
]) !== undefined,
) ?? usage
);
}
export function nativeUsageBiller(provider: NativeExecutionInput["provider"]): string {
if (provider.kind === "cursor_cloud") return "cursor";
if (provider.kind === "acpx") {
if (provider.agent === "cursor") return "cursor";
if (provider.agent === "copilot") return "github";
if (provider.agent === "pi") return "openrouter";
if (provider.agent === "grok") return "xai";
}
return "openai";
}
/** Provenance note for a receipt whose run includes a provider turn lost on
* restore. That turn's usage can never be reported, so the receipt is final
* but its usage and price are a lower bound. */
export const NATIVE_LOST_TURN_USAGE_EVIDENCE = "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound";
export const NATIVE_TURN_LOST_ON_RESTORE = "provider_turn_lost_on_restore";
const LOST_ON_RESTORE = NATIVE_TURN_LOST_ON_RESTORE;
export function createNativeTurnAccounting(provider: NativeExecutionInput["provider"]) {
const turns = new Map<string, { usage: Record<string, unknown> | null; terminal: boolean; lost?: boolean;
/** Open when the provider session was restored after process loss. */
restored?: boolean }>();
const sequences = new Map<string, number>();
let currentTurn: string | undefined;
let providerRequestId: string | undefined;
const terminalTypes = ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"];
const snapshot = () => {
if (turns.size === 0) return { usage: null, turnId: undefined, complete: false };
let inputTokens = 0, outputTokens = 0, cacheReadTokens = 0, cacheWriteTokens = 0;
let cost = 0n, costKnown = false, costIncomplete = false, complete = true, usageLost = false;
for (const [turnId, turn] of turns) {
const normalized = normalizeNativeUsage(turn.usage);
// A turn the restored Runner reports as lost on restore can never report
// its usage. It does not hold the receipt open; it makes it a lower bound.
const lost = turn.lost === true && normalized === undefined;
usageLost ||= lost;
complete &&= turnId !== "unattributed" && turn.terminal && (normalized !== undefined || lost);
inputTokens += (normalized?.inputTokens ?? 0) - (normalized?.cacheWriteTokens ?? 0);
outputTokens += normalized?.outputTokens ?? 0;
cacheReadTokens += normalized?.cachedInputTokens ?? 0;
cacheWriteTokens += normalized?.cacheWriteTokens ?? 0;
const price = nativeUsageCostUsd(turn.usage, provider);
if (price === undefined) costIncomplete = true;
else { cost += usdToUnits(price); costKnown = true; }
}
// A free empty attempt contributes no price evidence for later paid work.
// Keep an incomplete zero subtotal absent so complete aggregate tokens can
// still receive an estimate. Preserve positive known spend and true zero totals.
const retainCost = costKnown && (!costIncomplete || cost > 0n);
const runDelta = { inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens,
...(retainCost ? { providerCostUsd: Number(cost) / 1_000_000_000 } : {}) };
const current = currentTurn ? turns.get(currentTurn) : undefined;
return { usage: { runDelta, accountingCostIncomplete: costIncomplete, ...(usageLost ? { accountingUsageLost: true } : {}),
...(retainCost ? { accountingCostUsdExact: `${cost / 1_000_000_000n}.${String(cost % 1_000_000_000n).padStart(9, "0")}` } : {}),
...(providerRequestId ? { providerRequestId } : {}) },
// A turn closed as lost still names the terminal turn the executor must match.
turnId: current && (normalizeNativeUsage(current.usage) || (current.lost && current.terminal)) ? currentTurn : undefined, complete };
};
return {
observe(event: PrpEvent) {
if (!event) return false;
const payload = record(event.payload);
const terminal = terminalTypes.includes(event.eventType);
const resumed = event.eventType === "session.resumed";
if (payload.kind !== "usage" && event.eventType !== "turn.started" && !terminal && !resumed) return false;
// Historical replay can revisit an earlier report after restoration.
// Re-persist its aggregate, but never rewind or count that report twice.
if (event.sourceInstanceId && Number.isSafeInteger(event.sourceSeq)) {
if (event.sourceSeq <= (sequences.get(event.sourceInstanceId) ?? -1)) return true;
sequences.set(event.sourceInstanceId, event.sourceSeq);
}
// A restore after process loss ends the provider process that would
// have sent an open turn's final report. Mark those turns; a terminal
// event for one without a complete report closes it as a lower bound,
// like a turn the Runner fails as lost on restore.
if (resumed) {
for (const turn of turns.values()) if (!turn.terminal) turn.restored = true;
return false;
}
const key = event.turnId ?? "unattributed";
const turn = turns.get(key) ?? { usage: null, terminal: false };
if (event.eventType === "turn.started") {
currentTurn = event.turnId;
turn.usage = null;
turn.terminal = false;
turn.lost = false;
turn.restored = false;
} else if (payload.kind === "usage" && (!currentTurn || !turns.has(key))) currentTurn = event.turnId;
if (payload.kind === "usage") {
const usage = record(payload.usage);
if (!providerRequestId && typeof usage.providerRequestId === "string" && usage.providerRequestId)
providerRequestId = usage.providerRequestId.slice(0, 250);
turn.usage = Object.hasOwn(usage, "runDelta")
? { runDelta: record(usage.runDelta), runDeltaComplete: usage.runDeltaComplete } : null;
}
if (terminal) turn.terminal = true;
if (event.eventType === "turn.failed" && record(payload.error).code === LOST_ON_RESTORE) turn.lost = true;
if (terminal && turn.restored) turn.lost = true;
turns.set(key, turn);
return true;
},
snapshot,
/** A restore matters only while a turn is open. */
hasOpenTurn() {
for (const turn of turns.values()) if (!turn.terminal) return true;
return false;
},
finish(turnId: string | null) {
const turn = turnId ? turns.get(turnId) : undefined;
if (turn) turn.terminal = true;
return snapshot();
},
};
}
export function nativeUsageCostUsd(
usage: Record<string, unknown> | null,
provider?: NativeExecutionInput["provider"],
) {
if (provider?.kind === "cursor_cloud") return undefined;
// The ACP normalization contract fills absent per-turn cost with zero and
// reports actual cost cumulatively. Until it carries an authoritative run
// delta with provenance, neither value is a candidate's billed USD receipt.
if (provider?.kind === "acpx" && ["cursor", "copilot", "pi"].includes(provider.agent)) return undefined;
if (!usage) return undefined;
const measurement = nativeUsageMeasurement(usage);
const hasRunDelta = Object.hasOwn(usage, "runDelta") || Object.hasOwn(record(usage.usage), "runDelta");
const direct = numericUsageField(measurement, ["cacheAdjustedCostUsd", "providerCostUsd", "costUsd"])
?? (hasRunDelta ? undefined : numericUsageField(usage, ["cacheAdjustedCostUsd", "providerCostUsd", "costUsd"]));
// PRP currently defaults an absent providerCostUsd to zero. That zero is
// not proof of a free request when the runtime observed token consumption.
if (direct !== undefined && direct > 0) return direct;
const normalized = normalizeNativeUsage(usage);
if (direct === 0 && normalized && Object.values(normalized).every(value => value === 0)) return 0;
const cost = record(hasRunDelta ? measurement.cost : usage.cost);
const currency =
typeof cost.currency === "string" ? cost.currency.toUpperCase() : "USD";
if (currency !== "USD") return undefined;
return numericUsageField(cost, ["amount", "total"]);
}
export function normalizeNativeUsage(usage: Record<string, unknown> | null, options: { inputIncludesCacheReads?: boolean } = {}) {
if (!usage || usage.runDeltaComplete === false || record(usage.usage).runDeltaComplete === false) return undefined;
const measurement = nativeUsageMeasurement(usage);
const cache = record(measurement.cache);
const count = (source: Record<string, unknown>, keys: string[]) => {
for (const key of keys) {
if (source[key] === undefined) continue;
const value = source[key];
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : NaN;
}
return undefined;
};
const input = count(measurement, ["inputTokens", "input", "promptTokens"]);
const outputTokens = count(measurement, ["outputTokens", "output", "completionTokens"]);
const cachedInputTokens = count(measurement, ["cachedInputTokens", "cacheReadInputTokens", "cacheReadTokens", "cachedReadTokens"]) ?? count(cache, ["read"]);
const cacheWriteTokens = count(measurement, ["cacheWriteTokens", "cacheCreationInputTokens", "cachedWriteTokens"]) ?? count(cache, ["write"]);
if (input === undefined || outputTokens === undefined
|| ![input, outputTokens, cachedInputTokens ?? 0, cacheWriteTokens ?? 0].every(Number.isSafeInteger)) return undefined;
const inputTokens = input - (options.inputIncludesCacheReads ? cachedInputTokens ?? 0 : 0)
+ (options.inputIncludesCacheReads ? 0 : cacheWriteTokens ?? 0);
if (!Number.isSafeInteger(inputTokens) || inputTokens < (cacheWriteTokens ?? 0)) return undefined;
return {
inputTokens, outputTokens,
...(cachedInputTokens === undefined ? {} : { cachedInputTokens }),
...(cacheWriteTokens === undefined ? {} : { cacheWriteTokens }),
};
}
export function usesNativeUsageDelivery(input: {
runnerd: boolean; remote: boolean; provider?: NativeExecutionInput["provider"] | null;
}) {
return input.runnerd && input.remote && input.provider !== undefined && input.provider !== null
&& ["codex", "opencode", "acpx", "claude_managed", "aws_agentcore"].includes(input.provider.kind);
}
export function isNativeUsageAccountingEvent(event: Pick<PrpEvent, "eventType">) {
return ["usage.reported", "turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(event.eventType);
}
/** Producer delivery for the run itself also needs the session restore: a
* turn open across it can close as a lower bound (see createNativeTurnAccounting).
* Recovery routing keeps isNativeUsageAccountingEvent: a source run's restore
* carries no usage and must not require a proven recovery source. */
export function isNativeUsageDeliveryEvent(event: Pick<PrpEvent, "eventType">) {
return isNativeUsageAccountingEvent(event) || event.eventType === "session.resumed";
}