fix(evals): preserve unknown Codex provider cost (#17158)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Direct Runner evals report usage and costs for each attempt.
> - Codex App Server supplies token counts without a provider USD
receipt.
> - The legacy Runner ledger fills an absent cost with zero.
> - The eval result exposed that ledger value as provider-reported cost.
> - This fix keeps provider cost unknown and preserves the estimate and
budget stop.

## Linked Issues or Issue Description

Refs #17009 and #17154. This is a separate direct Runner accounting fix
based on master.

## What Changed

- Codex direct eval usage now reports null provider USD cost with
unavailable provenance. Token counts and the pinned price estimate
remain available.
- Preserve the original ledger amount as optional
`legacyLedgerCostNanodollars`. Keep the original snapshot receipts
unchanged.
- Apply the existing budget ceiling to the retained ledger amount as
well as the token estimate. Correcting a label does not relax admission
or settlement checks.
- Use the unique operator UUID for Muse test company prefixes,
eliminating collisions from the previous 24-bit random suffix.
- Document the cost distinction in the Runner/Evals integration
contract. Other provider behavior and historical artifacts remain
unchanged.

## Verification

- Reproduced both zero and nonzero legacy amounts being labelled as
provider USD cost before the fix.
- The current eval-session contract and entrypoint tests pass (39
tests). The new regression uses the bounded usage consumer and actual
campaign cost projection. It checks unknown provider cost, the separate
estimate, unchanged raw evidence and the legacy ledger budget stop.
- All 15 Runner report, campaign and metrics Node tests pass. Four
adjacent CLI entrypoint/runtime tests also pass.
- Runner TypeScript validation passes with its standard dependency-build
prerequisite; `git diff --check` passes. Both Runner TypeScript
configurations also passed during the original accounting verification.
- Hosted CI run
[39960367674](https://github.com/paperclipai/paperclip/actions/runs/39960367674)
passes on `7d84f138044bac58cdced6baa2d0ec3d63767bb8`, including
typecheck, build, server tests and browser checks. Greptile and
Paperclip Review both report 5/5 on that commit with no actionable
findings.
- The previous CI failure was a PostgreSQL unique-index collision on
`companies_issue_prefix_idx` during Muse fixture setup. Local Muse
verification could not reach test execution because embedded PostgreSQL
timed out during macOS startup (initially reporting a missing zstd
library). The corrected fixture passed in hosted Linux CI.
- Verification used no model calls. Full repository checks were
performed by hosted CI; local verification was scoped to the changed
contracts and fixture.

## Risks

Consumers that assumed Codex always had a numeric provider cost must
handle null. The contract already permits null for other profiles, and
the report consumer retains missing-cost coverage. The legacy amount
remains available as unverified evidence and still triggers the existing
budget stop. Neither the estimate nor that ledger amount proves an
invoice charge. This change does not rewrite old artifacts or qualify
other provider routes.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and code execution.
The exact deployment variant and context-window size are not exposed by
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/paperclip-runner/docs/evals-integration.md

@@ -63,6 +63,21 @@ Evals owns conversion of this bundle into Evalbook ledger, scores, and reports.
 The App package does not implement an Evalbook environment loader, importer,
 grid, comparison, or report renderer.
 
+## Direct eval cost provenance
+
+Codex App Server supplies token usage, not a provider USD receipt. Its direct
+`eval-session` result therefore sets `providerReportedCostNanodollars` to null
+and `providerReportedCostProvenance` to `unavailable`. The pinned token-price
+estimate remains in `estimatedCostNanodollars` with `costCoverage: estimated`.
+A missing receipt must not appear as a free request.
+
+`legacyLedgerCostNanodollars` retains the unverified ledger amount for Codex.
+This amount and the token estimate still participate in the existing budget
+stop. The ledger amount is not provider billing, and consumers must not add it
+to the estimate or treat its zero placeholder as proof of zero spend. Raw
+usage receipts remain in the snapshot. Older artifacts remain their original
+measurements; this change does not reconcile invoices or reprice history.
+
 ## Deterministic conformance
 
 `runHarnessDriverConformance` with `DeterministicHarnessDriver` covers, without

packages/paperclip-runner/src/cli/eval-session-contract.test.ts

@@ -5,6 +5,7 @@ import { join } from "node:path";
 
 import { describe, expect, it, vi } from "vitest";
 import { createCapabilityFixtureState } from "../mock-core/capability-control-plane-types.js";
+import { sumAttemptCosts } from "../../scripts/runner-protocol-eval-metrics.mjs";
 
 import { parseNativeRuntimeContext } from "../contracts/runtime-context.js";
 import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js";
@@ -451,6 +452,39 @@ describe("eval-session usage", () => {
     });
   });
 
+  it.each([0, 6_000_000])("does not label Codex legacy ledger cost %d as a provider USD receipt", (legacyCost) => {
+    const snapshot = {
+      config: { provider: "codex" },
+      usageLedger: [{
+        receiptId: "codex-receipt", providerCalls: 1, providerRequests: 1,
+        inputTokens: 1_000, outputTokens: 100, cachedInputTokens: 400,
+        reasoningTokens: 50, costNanodollars: legacyCost,
+      }],
+    } as unknown as CapabilityLiveSessionSnapshot;
+    const usage = boundedEvalSessionUsage(parseEvalSessionRequest(request()), {
+      turnId: "codex-turn", status: "completed", assistantText: "Completed", snapshot,
+    });
+    expect(usage).toMatchObject({
+      agentTurns: 1, providerRequests: 1, inputTokens: 1_000, outputTokens: 100,
+      providerReportedCostNanodollars: null,
+      providerReportedCostProvenance: "unavailable",
+      estimatedCostNanodollars: 6_200_000,
+      legacyLedgerCostNanodollars: legacyCost,
+      costCoverage: "estimated",
+    });
+    expect(sumAttemptCosts([usage])).toMatchObject({
+      providerReported: { nanodollars: null, recordedAttempts: 0 },
+      estimated: { nanodollars: 6_200_000, recordedAttempts: 1 },
+    });
+    expect(snapshot.usageLedger[0]?.costNanodollars).toBe(legacyCost);
+    const limitedRequest = parseEvalSessionRequest(request());
+    limitedRequest.limits.maxEstimatedCostNanodollars = 10_000_000;
+    const costlySnapshot = { ...snapshot, usageLedger: [{ ...snapshot.usageLedger[0]!, costNanodollars: 10_000_001 }] };
+    expect(() => boundedEvalSessionUsage(limitedRequest, {
+      turnId: "codex-turn", status: "completed", assistantText: "Completed", snapshot: costlySnapshot,
+    })).toThrow("legacy ledger cost limit exceeded");
+  });
+
   it("fails closed when a completed turn has no usage receipt", () => {
     expect(() => evalSessionUsage(
       "gpt-5.6-sol",

packages/paperclip-runner/src/cli/eval-session-contract.ts

@@ -94,6 +94,7 @@ export interface EvalSessionUsage {
   cachedInputTokens: number;
   reasoningTokens: number;
   providerReportedCostNanodollars: number | null;
+  legacyLedgerCostNanodollars?: number;
   providerReportedCostProvenance: "legacy_runner_usage_ledger" | "unavailable";
   estimatedCostNanodollars: number | null;
   pricingVersion: EstimatedModelCost["pricingVersion"] | null;
@@ -401,6 +402,8 @@ export function evalSessionUsage(
   }
   const candidate = snapshot.config?.provider === "acpx"
     && ["pi", "cursor", "copilot"].includes(snapshot.config.acpxAgent ?? "");
+  const codex = snapshot.config?.provider === "codex";
+  const providerCostUnavailable = candidate || codex;
   let estimate: EstimatedModelCost | null;
   try {
     estimate = estimateModelCostNanodollars(model, totals);
@@ -413,14 +416,12 @@ export function evalSessionUsage(
   }
   return {
     ...totals,
-    // Candidate ACP adapters do not supply authenticated USD receipts. The
-    // shared legacy ledger fills absent cost with zero, so its numeric value
-    // cannot establish an invoice amount for these profiles.
-    providerReportedCostNanodollars: candidate ? null : totals.providerReportedCostNanodollars,
-    providerReportedCostProvenance: candidate ? "unavailable" : "legacy_runner_usage_ledger",
+    ...(codex ? { legacyLedgerCostNanodollars: totals.providerReportedCostNanodollars } : {}),
+    providerReportedCostNanodollars: providerCostUnavailable ? null : totals.providerReportedCostNanodollars,
+    providerReportedCostProvenance: providerCostUnavailable ? "unavailable" : "legacy_runner_usage_ledger",
     estimatedCostNanodollars: estimate?.estimatedCostNanodollars ?? null,
     pricingVersion: estimate?.pricingVersion ?? null,
     ratesUsdPerMillionTokens: estimate?.ratesUsdPerMillionTokens ?? null,
-    costCoverage: candidate ? (estimate === null ? "unpriced" : "estimated") : "legacy_ledger_and_estimate",
+    costCoverage: providerCostUnavailable ? (estimate === null ? "unpriced" : "estimated") : "legacy_ledger_and_estimate",
   };
 }

packages/paperclip-runner/src/cli/eval-session.ts

@@ -340,6 +340,9 @@ export function boundedEvalSessionUsage(
   ) {
     throw new EvalSessionBudgetError("provider-reported cost limit exceeded");
   }
+  if (usage?.legacyLedgerCostNanodollars !== undefined && usage.legacyLedgerCostNanodollars > request.limits.maxEstimatedCostNanodollars) {
+    throw new EvalSessionBudgetError("legacy ledger cost limit exceeded");
+  }
   if (unavailable || usage === null || (usage.estimatedCostNanodollars === null && usage.providerReportedCostNanodollars === null)) {
     throw new EvalSessionBudgetError("budget cost coverage is unavailable for one or more completed turns", true);
   }

server/src/__tests__/muse-control.test.ts

@@ -55,7 +55,7 @@ describe("personal Muse control plane",()=>{
   afterAll(async()=>{vi.unstubAllEnvs();await museReceiver(db).stop();if(externalDatabaseUrl)await closeRegisteredClients(externalDatabaseUrl);await temporary?.cleanup();});
   async function fixture(ready=true) {
     const operatorId=randomUUID();await db.insert(authUsers).values({id:operatorId,name:"Muse authorizer",email:operatorId+"@example.test",createdAt:new Date(),updatedAt:new Date()});
-    const [company]=await db.insert(companies).values({name:"Muse control",issuePrefix:"MU"+randomBytes(3).toString("hex")}).returning();
+    const [company]=await db.insert(companies).values({name:"Muse control",issuePrefix:"MU"+operatorId.replaceAll("-","")}).returning();
     const [agent]=await db.insert(agents).values({companyId:company.id,name:"Muse",role:"engineer",status:"active",adapterType:"paperclip_runner",adapterConfig:{provider:"muse",allowUnmeteredProvider:true,lifecycle:"per_turn"},permissions:{canCreateTasks:true}}).returning();
     await db.insert(companyMemberships).values([{companyId:company.id,principalType:"user",principalId:operatorId,membershipRole:"owner",status:"active"},{companyId:company.id,principalType:"agent",principalId:agent.id,membershipRole:"member",status:"active"}]);
     const ticket=randomBytes(32).toString("base64url");const [binding]=await db.insert(bindings).values({companyId:company.id,agentId:agent.id,operatorId,ticketHash:museCredentialHash(ticket),ticketExpiresAt:new Date(Date.now()+600000)}).returning();