mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents can use models through OpenRouter, and the Runner proxies those model calls > - OpenRouter routes one model across many hosts, and these hosts use different quantizations (fp4, mxfp4, fp8) > - Chat evaluations with identical prompts behave very differently between runs, but the run log does not record which host served a call > - The run usage shows `provider: "deepseek"` and `biller: "openrouter"`, and `providerRequestId` is always `null` > - This pull request records the serving host and the OpenRouter generation ID for each model call in the run log > - The benefit is that we can compare evaluation cohorts by host, and the cost event names a real OpenRouter generation ## Linked Issues or Issue Description **Subsystem affected** Runner OpenCode provider proxy, runnerd usage normalization, and native run usage accounting. **Problem or motivation** OpenRouter's default routing sends a request to one of many hosts. Each host can use a different quantization. Chat evaluation results change a lot between runs with identical prompts. Paperclip does not record which host served a call, so we cannot compare results by host. `usageJson.providerRequestId` is always `null` for OpenCode runs. **Proposed solution** Every OpenRouter completion response has a top-level `provider` field and an `id` (the generation ID). Each streamed chunk has them too. The Runner proxy reads them from the first chunk after it forwards the bytes. The Runner attaches them to the usage of that call. The values go to the run log, and the run's first generation ID goes to `usageJson.providerRequestId`. **Alternatives considered** A new PRP event type such as `provider.upstream.served` would need a new event enum value. An older control plane rejects an unknown event type. A `harness.diagnostic` event shows in the task chat as a diagnostic. The usage payload already flows to every consumer, and older consumers ignore extra fields. Thus this change uses the usage payload. **Additional context** Run-log change only (AGENTS.md section 7). The data stays in `heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch `packages/shared/src/telemetry/` or OpenTelemetry. ## What Changed - New `createUpstreamServedTap` (`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`). It is a pass-through stream. It pushes each chunk downstream first, then it reads `provider` and `id` from the first SSE `data:` event (or from a JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies and unknown content types. Any parse problem or recorder error stops the scan and leaves the response unchanged. - `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The OpenCode session keeps the most recent served call. Each `item.completed` usage payload and `session.usage()` now carry `upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint does not include these fields, so the number of usage events does not change. - The OpenCode app-server proxy adds the two fields to `thread/tokenUsage/updated`. Rust `usage.reported` now fills its existing `providerRequestId` field and adds an optional `upstreamProvider`. The TypeScript runnerd transport keeps both fields when it rehydrates the usage notification. `usage.schema.json` documents `upstreamProvider` (the protocol manifest and schema bundle are regenerated). - Server native turn accounting keeps the first generation ID the run reports. The usage checkpoint and the native adapter result carry it as `providerRequestId`, so `usageJson.providerRequestId` and the final cost event name it. - `doc/run-log-events.md` documents the fields in a new "OpenRouter serving host" section. ## Verification - Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's stream through unchanged and records the host that served it on the turn's usage". The test sends a streamed fixture with a comment line, a host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks that the bytes are identical and that the turn's usage names `Novita` and the generation ID. The test fails without the proxy wiring. - `upstream-served.test.ts`: streamed fixture, chunk split inside a multibyte character, non-streamed JSON, malformed chunks, a truncated JSON body, an encoded body, the 64 KiB limit, and a recorder that throws. - `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts` (rehydration), Rust `provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`, and server `native-usage-accounting.test.ts` (first generation ID stays stable across turns). - Local: OpenCode driver, OpenCode proxy, and runnerd transport suites (31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes (one timing test failed once under load and passed alone). Runner and server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks pass. - Latency: a local benchmark proxied a 400-chunk SSE stream 200 times with the tap and 200 times without it. The median time to first byte was 292 µs with the tap and 259 µs without it (+33 µs, which is the extra stream stage). The total stream time changed from 491.4 ms to 492.5 ms. The tap spends about 5.5 µs of CPU to inspect the first chunk, after it has forwarded that chunk. OpenRouter time to first token is hundreds of milliseconds. - Paid evidence: one single-case run of `chat-features.runner-opencode.local.skills.fork-read-only` on this branch. The result is in a comment on this pull request. ## Risks - Low risk. All new fields are optional and additive. An older control plane ignores the extra usage fields. `usage.schema.json` is not bound by the event schema, and no validator enforces it on `usage.reported`. - Saved state from the previous release: this change adds no persisted run, session, checkpoint, or launch state. A saved session resumes unchanged. A run that is in flight during a deploy reports usage without the new fields until its Runner is new. `usageJson.providerRequestId` then stays `null`, as before. - `providerRequestId` on the run and its cost event is the run's first generation ID, not each call's ID. The value is stable, so a late usage receipt still matches the posted charge. Per-call IDs and hosts are in the run log. - If OpenCode starts a new model call before it reports the usage of the previous call, a usage event can name the newer call. OpenCode reports a message's usage when its step ends, so this is not expected. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
235 lines
12 KiB
TypeScript
235 lines
12 KiB
TypeScript
import { usdToUnits } from "@paperclipai/shared";
|
|
import type { NativeExecutionInput, PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
|
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
|
|
function numericUsageField(
|
|
usage: Record<string, unknown> | null,
|
|
keys: string[],
|
|
): number | undefined {
|
|
if (!usage) return undefined;
|
|
for (const key of keys) {
|
|
const value = usage[key];
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0)
|
|
return value;
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
function nativeUsageMeasurement(usage: Record<string, unknown>) {
|
|
const nestedUsage = record(usage.usage);
|
|
// An explicit partial delta must not fall through to a full session total.
|
|
if (Object.hasOwn(usage, "runDelta")) return record(usage.runDelta);
|
|
if (Object.hasOwn(nestedUsage, "runDelta")) return record(nestedUsage.runDelta);
|
|
const candidates = [
|
|
record(usage.runDelta),
|
|
record(nestedUsage.runDelta),
|
|
record(usage.total),
|
|
record(nestedUsage.total),
|
|
record(usage.cumulative),
|
|
record(nestedUsage.cumulative),
|
|
nestedUsage,
|
|
usage,
|
|
];
|
|
return (
|
|
candidates.find(
|
|
(candidate) =>
|
|
numericUsageField(candidate, [
|
|
"inputTokens",
|
|
"input",
|
|
"promptTokens",
|
|
"outputTokens",
|
|
"output",
|
|
"completionTokens",
|
|
]) !== undefined,
|
|
) ?? usage
|
|
);
|
|
}
|
|
|
|
export function nativeUsageBiller(provider: NativeExecutionInput["provider"]): string {
|
|
if (provider.kind === "cursor_cloud") return "cursor";
|
|
if (provider.kind === "acpx") {
|
|
if (provider.agent === "cursor") return "cursor";
|
|
if (provider.agent === "copilot") return "github";
|
|
if (provider.agent === "pi") return "openrouter";
|
|
if (provider.agent === "grok") return "xai";
|
|
}
|
|
return "openai";
|
|
}
|
|
|
|
/** Provenance note for a receipt whose run includes a provider turn lost on
|
|
* restore. That turn's usage can never be reported, so the receipt is final
|
|
* but its usage and price are a lower bound. */
|
|
export const NATIVE_LOST_TURN_USAGE_EVIDENCE = "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound";
|
|
export const NATIVE_TURN_LOST_ON_RESTORE = "provider_turn_lost_on_restore";
|
|
const LOST_ON_RESTORE = NATIVE_TURN_LOST_ON_RESTORE;
|
|
|
|
export function createNativeTurnAccounting(provider: NativeExecutionInput["provider"]) {
|
|
const turns = new Map<string, { usage: Record<string, unknown> | null; terminal: boolean; lost?: boolean;
|
|
/** Open when the provider session was restored after process loss. */
|
|
restored?: boolean }>();
|
|
const sequences = new Map<string, number>();
|
|
let currentTurn: string | undefined;
|
|
let providerRequestId: string | undefined;
|
|
const terminalTypes = ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"];
|
|
const snapshot = () => {
|
|
if (turns.size === 0) return { usage: null, turnId: undefined, complete: false };
|
|
let inputTokens = 0, outputTokens = 0, cacheReadTokens = 0, cacheWriteTokens = 0;
|
|
let cost = 0n, costKnown = false, costIncomplete = false, complete = true, usageLost = false;
|
|
for (const [turnId, turn] of turns) {
|
|
const normalized = normalizeNativeUsage(turn.usage);
|
|
// A turn the restored Runner reports as lost on restore can never report
|
|
// its usage. It does not hold the receipt open; it makes it a lower bound.
|
|
const lost = turn.lost === true && normalized === undefined;
|
|
usageLost ||= lost;
|
|
complete &&= turnId !== "unattributed" && turn.terminal && (normalized !== undefined || lost);
|
|
inputTokens += (normalized?.inputTokens ?? 0) - (normalized?.cacheWriteTokens ?? 0);
|
|
outputTokens += normalized?.outputTokens ?? 0;
|
|
cacheReadTokens += normalized?.cachedInputTokens ?? 0;
|
|
cacheWriteTokens += normalized?.cacheWriteTokens ?? 0;
|
|
const price = nativeUsageCostUsd(turn.usage, provider);
|
|
if (price === undefined) costIncomplete = true;
|
|
else { cost += usdToUnits(price); costKnown = true; }
|
|
}
|
|
// A free empty attempt contributes no price evidence for later paid work.
|
|
// Keep an incomplete zero subtotal absent so complete aggregate tokens can
|
|
// still receive an estimate. Preserve positive known spend and true zero totals.
|
|
const retainCost = costKnown && (!costIncomplete || cost > 0n);
|
|
const runDelta = { inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens,
|
|
...(retainCost ? { providerCostUsd: Number(cost) / 1_000_000_000 } : {}) };
|
|
const current = currentTurn ? turns.get(currentTurn) : undefined;
|
|
return { usage: { runDelta, accountingCostIncomplete: costIncomplete, ...(usageLost ? { accountingUsageLost: true } : {}),
|
|
...(retainCost ? { accountingCostUsdExact: `${cost / 1_000_000_000n}.${String(cost % 1_000_000_000n).padStart(9, "0")}` } : {}),
|
|
...(providerRequestId ? { providerRequestId } : {}) },
|
|
// A turn closed as lost still names the terminal turn the executor must match.
|
|
turnId: current && (normalizeNativeUsage(current.usage) || (current.lost && current.terminal)) ? currentTurn : undefined, complete };
|
|
};
|
|
return {
|
|
observe(event: PrpEvent) {
|
|
if (!event) return false;
|
|
const payload = record(event.payload);
|
|
const terminal = terminalTypes.includes(event.eventType);
|
|
const resumed = event.eventType === "session.resumed";
|
|
if (payload.kind !== "usage" && event.eventType !== "turn.started" && !terminal && !resumed) return false;
|
|
// Historical replay can revisit an earlier report after restoration.
|
|
// Re-persist its aggregate, but never rewind or count that report twice.
|
|
if (event.sourceInstanceId && Number.isSafeInteger(event.sourceSeq)) {
|
|
if (event.sourceSeq <= (sequences.get(event.sourceInstanceId) ?? -1)) return true;
|
|
sequences.set(event.sourceInstanceId, event.sourceSeq);
|
|
}
|
|
// A restore after process loss ends the provider process that would
|
|
// have sent an open turn's final report. Mark those turns; a terminal
|
|
// event for one without a complete report closes it as a lower bound,
|
|
// like a turn the Runner fails as lost on restore.
|
|
if (resumed) {
|
|
for (const turn of turns.values()) if (!turn.terminal) turn.restored = true;
|
|
return false;
|
|
}
|
|
const key = event.turnId ?? "unattributed";
|
|
const turn = turns.get(key) ?? { usage: null, terminal: false };
|
|
if (event.eventType === "turn.started") {
|
|
currentTurn = event.turnId;
|
|
turn.usage = null;
|
|
turn.terminal = false;
|
|
turn.lost = false;
|
|
turn.restored = false;
|
|
} else if (payload.kind === "usage" && (!currentTurn || !turns.has(key))) currentTurn = event.turnId;
|
|
if (payload.kind === "usage") {
|
|
const usage = record(payload.usage);
|
|
if (!providerRequestId && typeof usage.providerRequestId === "string" && usage.providerRequestId)
|
|
providerRequestId = usage.providerRequestId.slice(0, 250);
|
|
turn.usage = Object.hasOwn(usage, "runDelta")
|
|
? { runDelta: record(usage.runDelta), runDeltaComplete: usage.runDeltaComplete } : null;
|
|
}
|
|
if (terminal) turn.terminal = true;
|
|
if (event.eventType === "turn.failed" && record(payload.error).code === LOST_ON_RESTORE) turn.lost = true;
|
|
if (terminal && turn.restored) turn.lost = true;
|
|
turns.set(key, turn);
|
|
return true;
|
|
},
|
|
snapshot,
|
|
/** A restore matters only while a turn is open. */
|
|
hasOpenTurn() {
|
|
for (const turn of turns.values()) if (!turn.terminal) return true;
|
|
return false;
|
|
},
|
|
finish(turnId: string | null) {
|
|
const turn = turnId ? turns.get(turnId) : undefined;
|
|
if (turn) turn.terminal = true;
|
|
return snapshot();
|
|
},
|
|
};
|
|
}
|
|
|
|
export function nativeUsageCostUsd(
|
|
usage: Record<string, unknown> | null,
|
|
provider?: NativeExecutionInput["provider"],
|
|
) {
|
|
if (provider?.kind === "cursor_cloud") return undefined;
|
|
// The ACP normalization contract fills absent per-turn cost with zero and
|
|
// reports actual cost cumulatively. Until it carries an authoritative run
|
|
// delta with provenance, neither value is a candidate's billed USD receipt.
|
|
if (provider?.kind === "acpx" && ["cursor", "copilot", "pi"].includes(provider.agent)) return undefined;
|
|
if (!usage) return undefined;
|
|
const measurement = nativeUsageMeasurement(usage);
|
|
const hasRunDelta = Object.hasOwn(usage, "runDelta") || Object.hasOwn(record(usage.usage), "runDelta");
|
|
const direct = numericUsageField(measurement, ["cacheAdjustedCostUsd", "providerCostUsd", "costUsd"])
|
|
?? (hasRunDelta ? undefined : numericUsageField(usage, ["cacheAdjustedCostUsd", "providerCostUsd", "costUsd"]));
|
|
// PRP currently defaults an absent providerCostUsd to zero. That zero is
|
|
// not proof of a free request when the runtime observed token consumption.
|
|
if (direct !== undefined && direct > 0) return direct;
|
|
const normalized = normalizeNativeUsage(usage);
|
|
if (direct === 0 && normalized && Object.values(normalized).every(value => value === 0)) return 0;
|
|
const cost = record(hasRunDelta ? measurement.cost : usage.cost);
|
|
const currency =
|
|
typeof cost.currency === "string" ? cost.currency.toUpperCase() : "USD";
|
|
if (currency !== "USD") return undefined;
|
|
return numericUsageField(cost, ["amount", "total"]);
|
|
}
|
|
|
|
export function normalizeNativeUsage(usage: Record<string, unknown> | null, options: { inputIncludesCacheReads?: boolean } = {}) {
|
|
if (!usage || usage.runDeltaComplete === false || record(usage.usage).runDeltaComplete === false) return undefined;
|
|
const measurement = nativeUsageMeasurement(usage);
|
|
const cache = record(measurement.cache);
|
|
const count = (source: Record<string, unknown>, keys: string[]) => {
|
|
for (const key of keys) {
|
|
if (source[key] === undefined) continue;
|
|
const value = source[key];
|
|
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : NaN;
|
|
}
|
|
return undefined;
|
|
};
|
|
const input = count(measurement, ["inputTokens", "input", "promptTokens"]);
|
|
const outputTokens = count(measurement, ["outputTokens", "output", "completionTokens"]);
|
|
const cachedInputTokens = count(measurement, ["cachedInputTokens", "cacheReadInputTokens", "cacheReadTokens", "cachedReadTokens"]) ?? count(cache, ["read"]);
|
|
const cacheWriteTokens = count(measurement, ["cacheWriteTokens", "cacheCreationInputTokens", "cachedWriteTokens"]) ?? count(cache, ["write"]);
|
|
if (input === undefined || outputTokens === undefined
|
|
|| ![input, outputTokens, cachedInputTokens ?? 0, cacheWriteTokens ?? 0].every(Number.isSafeInteger)) return undefined;
|
|
const inputTokens = input - (options.inputIncludesCacheReads ? cachedInputTokens ?? 0 : 0)
|
|
+ (options.inputIncludesCacheReads ? 0 : cacheWriteTokens ?? 0);
|
|
if (!Number.isSafeInteger(inputTokens) || inputTokens < (cacheWriteTokens ?? 0)) return undefined;
|
|
return {
|
|
inputTokens, outputTokens,
|
|
...(cachedInputTokens === undefined ? {} : { cachedInputTokens }),
|
|
...(cacheWriteTokens === undefined ? {} : { cacheWriteTokens }),
|
|
};
|
|
}
|
|
|
|
export function usesNativeUsageDelivery(input: {
|
|
runnerd: boolean; remote: boolean; provider?: NativeExecutionInput["provider"] | null;
|
|
}) {
|
|
return input.runnerd && input.remote && input.provider !== undefined && input.provider !== null
|
|
&& ["codex", "opencode", "acpx", "claude_managed", "aws_agentcore"].includes(input.provider.kind);
|
|
}
|
|
|
|
export function isNativeUsageAccountingEvent(event: Pick<PrpEvent, "eventType">) {
|
|
return ["usage.reported", "turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(event.eventType);
|
|
}
|
|
|
|
/** Producer delivery for the run itself also needs the session restore: a
|
|
* turn open across it can close as a lower bound (see createNativeTurnAccounting).
|
|
* Recovery routing keeps isNativeUsageAccountingEvent: a source run's restore
|
|
* carries no usage and must not require a proven recovery source. */
|
|
export function isNativeUsageDeliveryEvent(event: Pick<PrpEvent, "eventType">) {
|
|
return isNativeUsageAccountingEvent(event) || event.eventType === "session.resumed";
|
|
}
|