From b6e690044a821faab52d5466db36c4330c754a2d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 23 Oct 2026 13:57:57 -0700 Subject: [PATCH] feat(evals): allow reviewed bounded provisional accounting (#17365) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns include model, setup, storage and observability costs. > - Some billing windows close after the tests and reports finish. > - Requiring settled receipts for retained storage can block a good campaign for weeks. > - A reservation alone does not prove that a pending charge has an upper bound. > - This change permits provisional completion only with reviewed evidence for the exact cost configuration. > - Reports preserve unknown amounts and holds until later reconciliation. ## Linked Issues or Issue Description Refs: #17360, #17354 **What existing behavior does this improve?** The Product E2E scheduled accounting gate and its retained cost reports. **Current behavior** Every pending receipt fails finalization. A completed test can wait through the retained-storage billing window. The journal has dollar reservations but no explicit qualification of their bounds. **Proposed behavior** Permit provisional completion when all pending charges have reviewed full-lifecycle bounds, no requests remain in flight, no receipts conflict, and known costs plus held amounts fit the campaign limit. Keep all existing holds and receipt-finality rules. Missing evidence still fails. **Reason and benefit** An operator can distinguish delayed but bounded billing from unbounded or missing costs. The campaign can finish without claiming that its invoice is settled. **Breaking changes** The journal accepts optional bound metadata and an invalidation record. New reports add an optional bounded-pending count. Current readers retain old journals and reports without granting them bounds. Older code must not read new bound-bearing journals. The existing daily/weekly enable gates and budget amounts stay unchanged. **Additional context** This implements the operator-approved provisional-cost policy for the existing eval harness. It does not change product budgets or the agent-eval roadmap. A targeted GitHub search found no duplicate open PR. ## What Changed - Accounting owns a narrow evidence contract: a reviewed audit hash, exact configuration hash and full-lifecycle scope. - Gateway reservations retain that evidence before dispatch. Changed limits or rates fail before dispatch. Unexpected routing, model identity or excessive usage invalidates a bound durably. - Operating allowances can carry separately reviewed bounds for each cost component, including its full storage lifetime. No existing allowance becomes qualified by default. - Reports and Braintrust experiment metadata distinguish provisional accounting, complete receipts, blocked accounting and unavailable evidence. Public output omits private audit hashes. - Scheduled finalization allows qualified provisional costs and explicitly rejects spending overruns, including overruns with complete receipts. - Add regressions for legacy data, restarts, late reconciliation, conflicts, malformed evidence, routing uncertainty and exceeded limits. Document the evidence and migration boundaries. ## Verification - Full offline eval-support suite: 4,062 Vitest tests passed, two existing tests skipped; 137 Node tests passed. - `pnpm test:e2e:runner:typecheck`: passed. - `git diff --check`: passed. - Read the retained hosted qualification journal with the new policy. All six legacy pending charges remain unqualified. Known cost `$0.421245200`, held `$19.327680000`, and committed `$19.748925200` remain unchanged. The journal's SHA-256 is unchanged. The original failed workflow remains failed. - No paid models, network provider calls, receipt imports or reservation releases were used for this verification. - Rebased onto master `4d90634f74`; 142 targeted tests and typecheck passed. - Review reproduced a lost OpenAI input-total limit check when cache details were missing (1 failed, 47 passed). Fix `5fef9ba7c149951a51eae468614a4b89f0b3d6db` preserves unpriced provider usage bounds. Then 178 focused tests, full offline support and typecheck passed. - Current-head hosted CI passed: https://github.com/paperclipai/paperclip/actions/runs/40020746585 (attempt 2). Typecheck, build, all general/Runner/serialized tests and browser gates are green. - CI attempt 1 hit a PostgreSQL deadlock in unchanged chat-test cleanup; its skipped flag restoration caused a second assertion failure. One failed-jobs-only retry passed. Original failure evidence is preserved in attempt 1. No paid eval campaign was rerun. - Greptile reviewed exact head `5fef9ba7c149951a51eae468614a4b89f0b3d6db` at 5/5; its one finding is fixed and resolved. The optional Paperclip review is paused and supplied no assessment. ## Risks - Evidence hashes bind an audit to a configuration; they do not prove the audit is true. Operators must review real provider limits, all price tiers, full resource lifetimes and enforceable infrastructure limits before configuring bounds. - Provisional completion is not invoice settlement. Later billing can reveal overruns, which must be reconciled and inspected. The ledger cannot stop external billing. - Existing pilot journals, configured allowances and price cards have no new bound approval. This PR alone does not qualify or enable recurring paid work. - The gateway cannot qualify a canceled Braintrust request without route confirmation. It preserves that unknown hold. - No new dependency, secret, repository variable, workflow, schedule or cloud resource changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/OPERATIONS-COVERAGE.md | 5 +- tests/runner-e2e/OPERATIONS.md | 81 ++++++++++- tests/runner-e2e/braintrust-experiment.ts | 2 + tests/runner-e2e/cost-bound-evidence.ts | 45 ++++++ tests/runner-e2e/cost-ledger.ts | 31 +++-- tests/runner-e2e/eval-gateway.test.ts | 46 ++++++ tests/runner-e2e/eval-gateway.ts | 23 ++- tests/runner-e2e/operating-overhead.ts | 14 +- tests/runner-e2e/operations-report.ts | 18 ++- tests/runner-e2e/operations.ts | 2 +- tests/runner-e2e/provider-usage.test.ts | 15 ++ tests/runner-e2e/provider-usage.ts | 21 ++- .../runner-e2e/provisional-accounting.test.ts | 131 ++++++++++++++++++ tests/runner-e2e/scheduled-finalization.ts | 8 +- 14 files changed, 419 insertions(+), 23 deletions(-) create mode 100644 tests/runner-e2e/cost-bound-evidence.ts create mode 100644 tests/runner-e2e/provisional-accounting.test.ts diff --git a/tests/runner-e2e/OPERATIONS-COVERAGE.md b/tests/runner-e2e/OPERATIONS-COVERAGE.md index da4e2efa6ec..4c6ba62aa58 100644 --- a/tests/runner-e2e/OPERATIONS-COVERAGE.md +++ b/tests/runner-e2e/OPERATIONS-COVERAGE.md @@ -133,7 +133,10 @@ the existing $249/month fee while including new usage and capacity charges. Scheduled admission now reserves the declared CI, storage, observability and subscription allowances inside the same $20/$100 journal limit. Missing or stale allowances stop scheduling before the AWS test runner starts. Missing receipts -keep their reservations held and accounting incomplete. This is a local admission +keep their reservations held. Explicit reviewed full-lifecycle bounds may allow +provisional completion while billing remains pending; missing bounds, conflicts +and overspending still fail. Legacy reservations gain no qualification from the +policy change. This is a local admission control, not a provider billing cap. Actual resource bounds, cost collection and reviewed allowances still need qualification before recurrence is enabled; see [the allowance contract](OPERATIONS.md#recurring-campaigns). diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 043097ac7fa..09fa4cd8ee3 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -173,6 +173,78 @@ Private request receipts retain transport IDs, response IDs, usage and cache TTL route/cache/failover evidence, and final accounting status without request bodies or credentials. Raw provider responses remain only in the normal private evidence. +## Provisional completion and later reconciliation + +Recurring campaigns may finish with provisional accounting when every pending +charge has an explicit reviewed bound, all requests have stopped, no receipts +conflict, and known costs plus retained holds fit the campaign cap. Product, +cleanup, feedback, trace and ingestion requirements still apply. Reports and +Braintrust experiment metadata label accounting as `provisional`, +`receipts_complete`, `blocked`, or `unavailable`. Receipt completeness is separate +from invoice reconciliation. The original failed campaign remains failed; a +later report or policy does not change its original result. + +The optional `provisionalBound` on a manifest model or operating component is: + +```json +{ + "version": 1, + "scope": "full_lifecycle", + "evidenceSha256": "SHA256_OF_RETAINED_REVIEWED_AUDIT", + "configurationSha256": "SHA256_OF_EXACT_REVIEWED_CONFIGURATION" +} +``` + +These placeholders are invalid until replaced with actual lowercase SHA-256 +values. Use `costBoundConfigurationSha256` from `cost-bound-evidence.ts` for +stable key-sorted JSON hashing. The model configuration contains `provider`, +`mode` (the manifest route), `price`, `maximumInputTokens`, +`maximumOutputTokens`, and `reservation` (the nine-decimal value from +`evalRequestReservation`). An operating configuration contains allowance +`reference`, `runner`, `verifiedAt`, `expiresAt`, and a `component` object with +its `component`, `maxUsd`, and `reference`. Changing any of those values +invalidates the approval hash. Evidence metadata itself is excluded from this +configuration hash. + +An operator must retain and review the audit before configuring its hash. +Hashes bind evidence and configuration; they do not prove the audit is true. +A model audit must establish the provider's full context ceiling, all billable +token prices and tiers, standard inference, and the route's billing behavior. +The gateway enforces output limits and supported request shapes. An operating +audit must cover setup, failed attempts, build time, publication, transfer, +observability and the full retained-storage lifecycle, including uncertainty in +resource deletion and invoice arrival. It must name the enforceable resource +limits and rates that keep the whole component below `maxUsd`. A current-day +usage estimate or an arbitrary contingency amount is insufficient. A zero bound +also needs evidence. Keep raw audits private; public reports expose only counts +and the existing journal checkpoint. + +The ledger retains bound evidence before model dispatch. Unexpected or missing +Braintrust route confirmation, failover, a returned model mismatch, or observed +usage above a declared limit disqualifies that request's bound. Provider usage +normalization retains a separate input/output lower bound even when missing or +contradictory cache counters prevent pricing. Malformed reported counts also +disqualify a bound. In-flight +requests, conflicts, pending charges above their bound, and campaign overruns +still block completion. Gateway cancellation before route confirmation remains +unqualified even if it retained an admission reservation. Later receipts remain +the way to resolve it. + +Provisional completion never releases a reservation or manufactures a zero +charge. A missing receipt stays unknown and holds its full bound, less any +known partial amount, until a valid final receipt supersedes it. The existing +receipt importer still requires complete billing/resource windows before it +marks a receipt final. Later bills can reveal an overrun; this local journal +cannot stop external billing. Reconcile the same retained journal after its +billing windows close, regenerate its report, and inspect any overruns before +admitting more work. Existing subscription costs remain separate from the +incremental daily/weekly budgets. + +Old journals and reports remain readable. They gain no bound evidence by +upgrade or by this policy; unresolved legacy charges still block. Existing +configured allowances and price cards are not automatically qualified by the +new optional field. Recurrence remains subject to its existing enable gates. + ## Manifest A manifest contains no credentials. This structural example uses fixture prices; @@ -420,8 +492,9 @@ lifecycles. They do not occupy model concurrency, but they count toward the finite charge-record limit. Resuming cannot change or drop them. Until matched cost observations arrive, the report shows them as unknown and keeps their allowance held; even a zero allowance does not claim a known zero charge. -The existing finalizer reports incomplete accounting while these receipts are -missing. Do not dismiss that finding to enable recurrence. +The finalizer permits provisional completion only when each pending charge has +explicit reviewed bound evidence as described below. Missing or unqualified +bounds remain incomplete accounting; a reservation alone does not qualify. Reconcile a component through its existing request ID with provenance-bound `infrastructure` or `invoice` observations. Partial observations keep the @@ -587,8 +660,8 @@ explicit, unbounded qualification path and does not inherit these allowances. After execution, deterministic review dismisses only exact synthetic feedback bodies corroborated by all feedback-persistence graders. Novel feedback remains -`needs_investigation`; no extra model call is made. Missing accounting, unresolved -receipts, feedback collection/submission failures, novel feedback, export failure +`needs_investigation`; no extra model call is made. Missing accounting, unbounded +pending charges, conflicts, spending overruns, feedback collection/submission failures, novel feedback, export failure and incomplete experiment import or indexing fail the job with bounded diagnostic codes. Successful unchanged runs produce artifacts without posting messages. GitHub's workflow failure notifications remain the operator's notification channel. diff --git a/tests/runner-e2e/braintrust-experiment.ts b/tests/runner-e2e/braintrust-experiment.ts index 4b2859e17d8..6db75341ede 100644 --- a/tests/runner-e2e/braintrust-experiment.ts +++ b/tests/runner-e2e/braintrust-experiment.ts @@ -1,3 +1,4 @@ +import { evalAccountingStatus } from "./operations-report.js"; import { createHash } from "node:crypto"; import { buildCampaignAccounting } from "./attempt-accounting.js"; import { exportBraintrustAttemptParent, type EvalBraintrustConfig } from "./braintrust.js"; @@ -98,6 +99,7 @@ export function buildBraintrustExperiment(projectId: string, campaign: RunnerE2E missing_attempts: accounting.missingAttempts, request_ledger: operations ? { journal_hash: operations.journalHash, known_usd: operations.knownUsd, additional_held_usd: operations.heldUsd, committed_usd: operations.committedUsd, + accounting_status: evalAccountingStatus(operations), bounded_unresolved_requests: operations.boundedUnresolvedRequests ?? 0, requests: operations.requests, unresolved_requests: operations.unresolvedRequests, breakdown: operations.breakdown } : null, feedback_review: campaign.feedbackReview ?? null, accounting_note: "Ledger amounts are separate observations, not additive trace metrics. Held reservations are not charges. Requests outside the gateway and infrastructure require separate accounting." }; diff --git a/tests/runner-e2e/cost-bound-evidence.ts b/tests/runner-e2e/cost-bound-evidence.ts new file mode 100644 index 00000000000..6d0bca53560 --- /dev/null +++ b/tests/runner-e2e/cost-bound-evidence.ts @@ -0,0 +1,45 @@ +import { createHash } from "node:crypto"; + +export interface EvalCostBoundEvidence { + version: 1; + scope: "full_lifecycle"; + evidenceSha256: string; + configurationSha256: string; +} +export interface EvalReservationBound { + basis: "model_limits" | "operating_allowance"; + evidence: EvalCostBoundEvidence; + calculationSha256: string; +} +const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value); + +export function validateCostBoundEvidence(value: EvalCostBoundEvidence): void { + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).some(key => !["version", "scope", "evidenceSha256", "configurationSha256"].includes(key)) + || value.version !== 1 || value.scope !== "full_lifecycle" || !hash(value.evidenceSha256) || !hash(value.configurationSha256)) { + throw new Error("Invalid reviewed cost bound evidence"); + } +} + +export function costBoundConfigurationSha256(calculation: unknown): string { + return createHash("sha256").update(JSON.stringify(calculation, (_key, value) => + value && typeof value === "object" && !Array.isArray(value) + ? Object.fromEntries(Object.keys(value).sort().map(key => [key, value[key]])) : value)).digest("hex"); +} + +export function reservationBound(basis: EvalReservationBound["basis"], evidence: EvalCostBoundEvidence, calculation: unknown): EvalReservationBound { + validateCostBoundEvidence(evidence); + const calculationSha256 = costBoundConfigurationSha256(calculation); + if (calculationSha256 !== evidence.configurationSha256) throw new Error("Reviewed cost bound configuration changed"); + return { basis, evidence: structuredClone(evidence), calculationSha256 }; +} + +export function validateReservationBound(value: EvalReservationBound, infrastructure: boolean): void { + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).some(key => !["basis", "evidence", "calculationSha256"].includes(key)) + || value.basis !== (infrastructure ? "operating_allowance" : "model_limits") || !hash(value.calculationSha256)) { + throw new Error("Invalid reservation bound"); + } + validateCostBoundEvidence(value.evidence); + if (value.calculationSha256 !== value.evidence.configurationSha256) throw new Error("Reviewed cost bound configuration changed"); +} diff --git a/tests/runner-e2e/cost-ledger.ts b/tests/runner-e2e/cost-ledger.ts index c132a6257fa..daa95eba7b3 100644 --- a/tests/runner-e2e/cost-ledger.ts +++ b/tests/runner-e2e/cost-ledger.ts @@ -1,9 +1,10 @@ +import { validateReservationBound, type EvalReservationBound } from "./cost-bound-evidence.js"; import { createHash } from "node:crypto"; import { constants } from "node:fs"; import { mkdir, open, readFile, rm, type FileHandle } from "node:fs/promises"; import path from "node:path"; import { usdToUnits } from "../../packages/shared/src/money.js"; -import { validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; +import { operatingReservationBound, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; export interface EvalSpendLimits { campaignId: string; @@ -46,11 +47,12 @@ type Event = | { type: "campaign"; limits: EvalSpendLimits } | ({ type: "attempt_policy" } & EvalLegacyAttemptAudit) | { type: "attempt"; executionId: string; attemptId: string } - | { type: "reserve"; requestId: string; identity: EvalChargeIdentity; maxUsd: string } + | { type: "reserve"; requestId: string; identity: EvalChargeIdentity; maxUsd: string; bound?: EvalReservationBound } + | { type: "invalidate_bound"; requestId: string } | { type: "observe"; observation: EvalCostObservation } | { type: "finish"; requestId: string; outcome: "succeeded" | "failed" | "cancelled" }; interface Line { seq: number; at: string; previous: string; event: Event; hash: string } -interface RequestState { identity: EvalChargeIdentity; reserved: bigint; observations: EvalCostObservation[]; finished: boolean } +interface RequestState { identity: EvalChargeIdentity; reserved: bigint; observations: EvalCostObservation[]; finished: boolean; bound?: EvalReservationBound; boundInvalidated?: boolean } const usd = (units: bigint) => `${units / 1_000_000_000n}.${String(units % 1_000_000_000n).padStart(9, "0")}`; export const validEvalLedgerId = (value: unknown): value is string => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:/-]{0,511}$/.test(value); const validId = validEvalLedgerId; @@ -89,6 +91,7 @@ export class EvalSpendState { for (const component of this.limits.overhead.components) this.requests.set(`overhead:${component.component}`, { identity: { executionId: "campaign-overhead", attemptId: component.component, phase: "infrastructure" }, reserved: amount(component.maxUsd), observations: [], finished: true, + ...(component.provisionalBound ? { bound: operatingReservationBound(this.limits.overhead, component) } : {}), }); } } @@ -105,7 +108,7 @@ export class EvalSpendState { resolved: finalized.length > 0 && amounts.size === 1 }; } snapshot() { - let known = 0n, held = 0n, committed = 0n, inflight = 0, unknown = 0, conflicts = 0; + let known = 0n, held = 0n, committed = 0n, inflight = 0, unknown = 0, conflicts = 0, boundedUnresolved = 0; const breakdown: Record<"reported" | "estimated" | "allocated" | "reconciled", bigint> = { reported: 0n, estimated: 0n, allocated: 0n, reconciled: 0n }; const requests = [...this.requests].map(([requestId, request]) => { const receipt = this.authoritative(request); @@ -113,6 +116,9 @@ export class EvalSpendState { if (receipt.conflict) conflicts++; if (!receipt.resolved) unknown++; const cost = receipt.cost ?? 0n; + const pendingBounded = !receipt.resolved && !receipt.conflict && request.finished && !!request.bound + && !request.boundInvalidated && cost <= request.reserved; + if (pendingBounded) boundedUnresolved++; known += cost; if (receipt.classification !== "unknown") breakdown[receipt.classification] += cost; const commitment = !request.finished || !receipt.resolved @@ -120,10 +126,10 @@ export class EvalSpendState { committed += commitment; held += commitment - cost; return { requestId, ...request.identity, reservedUsd: usd(request.reserved), costUsd: receipt.cost === null ? null : usd(cost), - classification: receipt.classification, conflict: receipt.conflict, receiptComplete: receipt.resolved, finished: request.finished }; + classification: receipt.classification, conflict: receipt.conflict, receiptComplete: receipt.resolved, finished: request.finished, pendingBounded }; }); return { campaignId: this.limits.campaignId, limitUsd: usd(amount(this.limits.maxUsd)), knownUsd: usd(known), heldUsd: usd(held), committedUsd: usd(committed), - overBudget: committed > amount(this.limits.maxUsd), unknownRequests: unknown, conflictingRequests: conflicts, inflight, + overBudget: committed > amount(this.limits.maxUsd), unknownRequests: unknown, boundedUnresolvedRequests: boundedUnresolved, conflictingRequests: conflicts, inflight, breakdown: Object.fromEntries(Object.entries(breakdown).map(([key, value]) => [key, usd(value)])), requests, admittedAttempts: [...this.attempts].map(([attemptId, executionId]) => ({ executionId, attemptId })) }; } @@ -156,6 +162,8 @@ export class EvalSpendState { if (now >= Date.parse(this.limits.deadlineAt)) throw new EvalAdmissionError("Eval campaign deadline reached"); if ([...this.attempts.values()].filter(id => id === event.executionId).length >= this.limits.maxAttemptsPerExecution) throw new EvalAdmissionError("Eval campaign attempt cap reached"); } else if (event.type === "reserve") { + fields(event, ["type", "requestId", "identity", "maxUsd", "bound"]); + if (event.bound !== undefined) validateReservationBound(event.bound, event.identity.phase === "infrastructure"); fields(event.identity, ["executionId", "attemptId", "phase", "runId"]); if (!validId(event.requestId) || !validId(event.identity.executionId) || !validId(event.identity.attemptId) || event.identity.runId !== undefined && !validId(event.identity.runId) @@ -189,6 +197,9 @@ export class EvalSpendState { const request = this.providerRequests.get(`${o.provider}:${o.providerRequestId}`); if (request && request !== o.requestId) throw new Error("Provider request belongs to another reservation; reconcile identities before import"); } + } else if (event.type === "invalidate_bound") { + fields(event, ["type", "requestId"]); + if (!this.requests.get(event.requestId)?.bound) throw new Error("Cannot invalidate an absent reservation bound"); } else if (event.type === "finish") { if (!this.requests.has(event.requestId) || !["succeeded", "failed", "cancelled"].includes(event.outcome)) throw new Error("Invalid request completion"); } else throw new Error("Unknown ledger event"); @@ -198,14 +209,15 @@ export class EvalSpendState { this.limits.maxAttemptsPerExecution = event.maxAttemptsPerExecution; for (const attempt of event.priorAttempts) this.attempts.set(attempt.attemptId, attempt.executionId); } else if (event.type === "attempt") this.attempts.set(event.attemptId, event.executionId); - else if (event.type === "reserve") this.requests.set(event.requestId, { identity: event.identity, reserved: amount(event.maxUsd), observations: [], finished: false }); + else if (event.type === "reserve") this.requests.set(event.requestId, { identity: event.identity, reserved: amount(event.maxUsd), observations: [], finished: false, ...(event.bound ? { bound: event.bound } : {}) }); else if (event.type === "observe") { const o = event.observation; if (this.observations.has(o.observationId)) return; this.observations.set(o.observationId, o); this.requests.get(o.requestId)!.observations.push(o); if (o.providerRequestId) this.providerRequests.set(`${o.provider}:${o.providerRequestId}`, o.requestId); - } else if (event.type === "finish") this.requests.get(event.requestId)!.finished = true; + } else if (event.type === "invalidate_bound") this.requests.get(event.requestId)!.boundInvalidated = true; + else if (event.type === "finish") this.requests.get(event.requestId)!.finished = true; } } @@ -273,7 +285,8 @@ export async function openEvalCostLedger(file: string, limits: EvalSpendLimits) return { adoptLegacyAttemptPolicy: (audit: EvalLegacyAttemptAudit) => append({ ...structuredClone(audit), type: "attempt_policy" }), admitAttempt: (executionId: string, attemptId: string) => append({ type: "attempt", executionId, attemptId }), - reserve: (requestId: string, identity: EvalChargeIdentity, maxUsd: string) => append({ type: "reserve", requestId, identity: structuredClone(identity), maxUsd }), + reserve: (requestId: string, identity: EvalChargeIdentity, maxUsd: string, bound?: EvalReservationBound) => append({ type: "reserve", requestId, identity: structuredClone(identity), maxUsd, ...(bound ? { bound: structuredClone(bound) } : {}) }), + invalidateBound: (requestId: string) => append({ type: "invalidate_bound", requestId }), observe: (observation: EvalCostObservation) => append({ type: "observe", observation: structuredClone(observation) }), finish: (requestId: string, outcome: "succeeded" | "failed" | "cancelled") => append({ type: "finish", requestId, outcome }), snapshot: () => state.snapshot(), diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index ba5266e72f3..6e38d356bea 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -1,3 +1,4 @@ +import { costBoundConfigurationSha256 } from "./cost-bound-evidence.js"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -355,3 +356,48 @@ it("refuses missing or unreviewed attribution checkpoints before request dispatc await writeFile(phasePath, JSON.stringify({ version: 1, ...identity, providerResponse: "private" })); await expect(readEvalAttemptPhase(phasePath, identity)).rejects.toThrow("Invalid eval phase"); }); + +const provisionalBound = { version: 1 as const, scope: "full_lifecycle" as const, evidenceSha256: "a".repeat(64), + configurationSha256: costBoundConfigurationSha256({ provider: route.provider, mode: route.mode, price: route.price, + maximumInputTokens: route.maximumInputTokens, maximumOutputTokens: route.maximumOutputTokens, reservation: evalRequestReservation(route) }) }; +it("retains a reviewed model bound through a partial response and durable reload without releasing the hold", async () => { + const f = await fixture(); + const receipt = await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => { + expect((await readEvalCostLedger(f.file)).state.requests.values().next().value?.bound?.evidence).toEqual(provisionalBound); + return Response.json({ object: "response", status: "in_progress", model: "fixture" }, + { headers: { "x-bt-used-endpoint": route.endpointName! } }); + } }); + expect(receipt).toMatchObject({ costUsd: null, final: false }); + expect((await readEvalCostLedger(f.file)).state.snapshot()).toMatchObject({ boundedUnresolvedRequests: 1, heldUsd: "0.003000000" }); +}); +it.each(["no-headers", "route", "failover", "model", "input-limit", "output-limit"])("disqualifies a model bound on %s evidence", async reason => { + const f = await fixture(); + await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => { + if (reason === "no-headers") throw new Error("No response headers"); + return Response.json({ object: "response", status: "in_progress", model: reason === "model" ? "other" : "fixture", + usage: { input_tokens: reason === "input-limit" ? 1001 : 10, input_tokens_details: { cache_write_tokens: 0, cached_tokens: 0 }, + output_tokens: reason === "output-limit" ? 101 : 0 } }, + { headers: { "x-bt-used-endpoint": reason === "route" ? "OTHER" : route.endpointName!, + ...(reason === "failover" ? { "x-bt-failover-to": "OTHER" } : {}) } }); + } }); + const replay = (await readEvalCostLedger(f.file)).state; + expect(replay.snapshot()).toMatchObject({ boundedUnresolvedRequests: 0, unknownRequests: 1, inflight: 0 }); + expect([...replay.requests.values()][0]!.boundInvalidated).toBe(true); +}); + +it("refuses a changed model limit or rate with a stale reviewed bound before dispatch", async () => { + const f = await fixture(), transport = vi.fn(); + for (const changed of [{ ...route, maximumInputTokens: 500 }, { ...route, price: { ...route.price, inputUsdPerMillion: "20" } }]) { + await expect(forwardEvalRequest({ ...f, route: { ...changed, provisionalBound }, transport })).rejects.toThrow("configuration changed"); + } + expect(transport).not.toHaveBeenCalled(); + expect(f.ledger.snapshot().requests).toHaveLength(0); +}); + +it("disqualifies excessive reported input even when missing cache details prevent pricing", async () => { + const f = await fixture(); + await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => Response.json({ + object: "response", status: "completed", model: "fixture", usage: { input_tokens: 1001, output_tokens: 1 }, + }, { headers: { "x-bt-used-endpoint": route.endpointName! } }) }); + expect((await readEvalCostLedger(f.file)).state.snapshot()).toMatchObject({ boundedUnresolvedRequests: 0, unknownRequests: 1, heldUsd: "0.003000000" }); +}); diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index 26af76ff106..330d87859f0 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -1,3 +1,4 @@ +import { reservationBound, validateCostBoundEvidence, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; import { randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; import { once } from "node:events"; @@ -17,6 +18,7 @@ export interface EvalGatewayRoute { maximumInputTokens: number; maximumOutputTokens: number; timeoutMs: number; + provisionalBound?: EvalCostBoundEvidence; } export interface EvalGatewayReceipt { requestId: string; status?: number; route: "direct" | "braintrust"; @@ -32,6 +34,7 @@ const row = (value: unknown): Record => value && typeof value = class EvalRequestError extends Error {} export function evalRequestReservation(route: EvalGatewayRoute): string { + if (route.provisionalBound !== undefined) validateCostBoundEvidence(route.provisionalBound); if (![route.maximumInputTokens, route.maximumOutputTokens, route.timeoutMs].every(n => Number.isSafeInteger(n) && n > 0) || route.timeoutMs > 300_000 || route.provider !== route.price.provider || !validEvalLedgerId(route.price.version) || !validEvalLedgerId(route.price.model)) throw new Error("Invalid eval gateway limits or price identity"); const parse = (rate: string) => { @@ -50,7 +53,16 @@ export function evalRequestReservation(route: EvalGatewayRoute): string { const output = parse(tier.outputUsdPerMillion); if (output > maximumOutputRate) maximumOutputRate = output; } const numerator = BigInt(route.maximumInputTokens) * maximumInputRate + BigInt(route.maximumOutputTokens) * maximumOutputRate; - return money((numerator + 999_999n) / 1_000_000n); + const reservation = money((numerator + 999_999n) / 1_000_000n); + modelReservationBound(route, reservation); + return reservation; +} + +function modelReservationBound(route: EvalGatewayRoute, reservation: string) { + return route.provisionalBound ? reservationBound("model_limits", route.provisionalBound, { + provider: route.provider, mode: route.mode, price: route.price, maximumInputTokens: route.maximumInputTokens, + maximumOutputTokens: route.maximumOutputTokens, reservation, + }) : undefined; } function boundedPayload(value: unknown, route: EvalGatewayRoute) { @@ -115,7 +127,8 @@ export async function forwardEvalRequest(input: { const receipt: EvalGatewayReceipt = { requestId, route: route.mode, outcome: "failed", usageComplete: false, final: false, costUsd: null, ...(route.mode === "braintrust" ? { braintrustParent: braintrustGatewayRequestParent(route.projectId!, requestId, ledger.snapshot().campaignId, input.identity) } : {}) }; const usage = new ProviderUsageAccumulator(route.provider, body); - await ledger.reserve(requestId, input.identity, reservation); + const bound = modelReservationBound(route, reservation); + await ledger.reserve(requestId, input.identity, reservation, bound); const abort = new AbortController(); const signal = AbortSignal.any([abort.signal, AbortSignal.timeout(Math.max(1, Math.min(route.timeoutMs, input.deadlineAt - Date.now()))), ...(input.signal ? [input.signal] : [])]); let response: Response | undefined; @@ -184,6 +197,12 @@ export async function forwardEvalRequest(input: { delete tokens.reasoning; receipt.errorCode ??= "invalid_usage"; } + const observedInput = observed.tokenBounds?.input ?? 0; + if (bound && (observed.tokenBounds?.valid === false || receipt.errorCode === "invalid_usage" || observed.model !== undefined && observed.model !== route.price.model + || !Number.isSafeInteger(observedInput) || observedInput > route.maximumInputTokens + || (observed.tokenBounds?.output ?? 0) > route.maximumOutputTokens + || route.mode === "braintrust" && (!response || receipt.endpoint !== route.endpointName + || receipt.cache === "HIT" || receipt.failoverFrom || receipt.failoverTo))) await ledger.invalidateBound(requestId); await input.persist(receipt); await ledger.observe({ observationId: `${requestId}:usage`, requestId, source: route.mode === "braintrust" ? "gateway" : "provider", provider: route.provider, diff --git a/tests/runner-e2e/operating-overhead.ts b/tests/runner-e2e/operating-overhead.ts index 4f53cd7517e..43eb5de338e 100644 --- a/tests/runner-e2e/operating-overhead.ts +++ b/tests/runner-e2e/operating-overhead.ts @@ -1,3 +1,4 @@ +import { reservationBound, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; import { usdToUnits } from "../../packages/shared/src/money.js"; export const EVAL_OVERHEAD_COMPONENTS = ["ci_compute", "artifact_storage", "observability", "shared_subscription"] as const; @@ -7,7 +8,7 @@ export interface EvalOperatingOverhead { runner: "github-hosted" | "aws-fleet"; verifiedAt: string; expiresAt: string; - components: Array<{ component: typeof EVAL_OVERHEAD_COMPONENTS[number]; maxUsd: string; reference: string }>; + components: Array<{ component: typeof EVAL_OVERHEAD_COMPONENTS[number]; maxUsd: string; reference: string; provisionalBound?: EvalCostBoundEvidence }>; } const object = (value: unknown, allowed: string[]) => { @@ -16,6 +17,14 @@ const object = (value: unknown, allowed: string[]) => { }; const reference = (value: unknown) => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:/-]{0,511}$/.test(value); +export function operatingReservationBound(allowance: EvalOperatingOverhead, component: EvalOperatingOverhead["components"][number]) { + if (component.provisionalBound === undefined) return undefined; + return reservationBound("operating_allowance", component.provisionalBound, { + reference: allowance.reference, runner: allowance.runner, verifiedAt: allowance.verifiedAt, expiresAt: allowance.expiresAt, + component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference }, + }); +} + export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint { object(value, ["version", "reference", "runner", "verifiedAt", "expiresAt", "components"]); if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner) @@ -29,7 +38,8 @@ export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd const seen = new Set(); let reserved = 0n; for (const component of value.components) { - object(component, ["component", "maxUsd", "reference"]); + object(component, ["component", "maxUsd", "reference", "provisionalBound"]); + if (component.provisionalBound !== undefined) operatingReservationBound(value, component); if (!EVAL_OVERHEAD_COMPONENTS.includes(component.component) || seen.has(component.component) || !reference(component.reference) || typeof component.maxUsd !== "string" || !/^(?:0|[1-9][0-9]*)(?:\.[0-9]{1,9})?$/.test(component.maxUsd)) throw new Error("Invalid operating overhead component"); diff --git a/tests/runner-e2e/operations-report.ts b/tests/runner-e2e/operations-report.ts index 97d8a3df776..b751d24bed6 100644 --- a/tests/runner-e2e/operations-report.ts +++ b/tests/runner-e2e/operations-report.ts @@ -17,6 +17,7 @@ interface EvalOperationsAvailableReport { committedUsd: string; requests: number; unresolvedRequests: number; + boundedUnresolvedRequests?: number; conflictingRequests: number; inflight: number; breakdown: { reported: string; estimated: string; allocated: string; reconciled: string }; @@ -38,11 +39,15 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void } const dollars = (value: unknown): bigint => typeof value === "string" && /^(?:0|[1-9][0-9]*)\.[0-9]{9}$/.test(value) ? usdToUnits(value) : fail(); const count = (value: unknown) => { if (!Number.isSafeInteger(value) || Number(value) < 0) fail(); }; - fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); + fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); if (report.version !== 1 || report.scope !== "shared_gateway_ledger" || !validEvalLedgerId(report.ledgerCampaignId) || !/^[a-f0-9]{64}$/.test(report.journalHash) || !isPublicReportTimestamp(report.deadlineAt) || !Array.isArray(report.attempts)) fail(); [report.records, report.requests, report.unresolvedRequests, report.conflictingRequests, report.inflight].forEach(count); if (report.records < 1 || report.unresolvedRequests > report.requests || report.conflictingRequests > report.unresolvedRequests || report.inflight > report.requests) fail(); + if (report.boundedUnresolvedRequests !== undefined) { + count(report.boundedUnresolvedRequests); + if (report.boundedUnresolvedRequests > report.unresolvedRequests - report.conflictingRequests) fail(); + } dollars(report.limitUsd); const known = dollars(report.knownUsd), held = dollars(report.heldUsd); if (known + held !== dollars(report.committedUsd)) fail(); @@ -84,18 +89,27 @@ async function readAvailableEvalOperationsReport(file: string): Promise usdToUnits(report.limitUsd) + || report.unresolvedRequests !== (report.boundedUnresolvedRequests ?? 0)) return "blocked"; + return report.unresolvedRequests ? "provisional" : "receipts_complete"; +} + export function operationsReportText(report: EvalOperationsReport): string[] { if (report.status === "unavailable") return [ "Request ledger accounting unavailable: the journal could not be read or validated. Request amounts, counts, and held reservations are unknown.", "Retained test results are preserved. Recover and validate the original journal before resuming paid work; do not reset the allowance or treat missing amounts as zero.", ]; return [ + `Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`, `Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`, `Charges: $${report.breakdown.reconciled} reconciled; $${report.breakdown.reported} reported; $${report.breakdown.estimated} estimated; $${report.breakdown.allocated} allocated.`, `${report.requests} recorded requests or overhead reservations; ${report.unresolvedRequests} unresolved; ${report.conflictingRequests} conflicting; ${report.inflight} in flight. Deadline ${report.deadlineAt}.`, diff --git a/tests/runner-e2e/operations.ts b/tests/runner-e2e/operations.ts index 2ab81eb2676..62507e05533 100644 --- a/tests/runner-e2e/operations.ts +++ b/tests/runner-e2e/operations.ts @@ -35,7 +35,7 @@ export function validateEvalOperationsManifest(value: EvalOperationsManifest, ex if (value.route === "braintrust" && !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value.projectId ?? "")) throw new Error("Braintrust gateway requires an explicit project UUID"); const models = new Map(); for (const model of value.models) { - fields(model, ["provider", "price", "maximumInputTokens", "maximumOutputTokens", "timeoutMs"]); + fields(model, ["provider", "price", "maximumInputTokens", "maximumOutputTokens", "timeoutMs", "provisionalBound"]); const rateFields = ["inputUsdPerMillion", "cachedReadUsdPerMillion", "outputUsdPerMillion", "cacheWrite5mUsdPerMillion", "cacheWrite1hUsdPerMillion", "cacheWrite30mUsdPerMillion", "cacheWrite12hUsdPerMillion"]; fields(model.price, ["provider", "model", "version", "sourceUrl", "longContext", ...rateFields]); if (model.price.longContext) { fields(model.price.longContext, ["aboveInputTokens", "rates"]); fields(model.price.longContext.rates, rateFields); } diff --git a/tests/runner-e2e/provider-usage.test.ts b/tests/runner-e2e/provider-usage.test.ts index b651c9e8a89..6549e15684e 100644 --- a/tests/runner-e2e/provider-usage.test.ts +++ b/tests/runner-e2e/provider-usage.test.ts @@ -111,3 +111,18 @@ it("prefers provider TTL splits and never repairs malformed or partial provider expect(estimateProviderCost(receipt, policyPrice)).toBeNull(); } }); + +it.each([ + ["openai", { input_tokens: 1001, output_tokens: 1 }, 1001], + ["openai", { input_tokens_details: { cached_tokens: 1001 } }, 1001], + ["anthropic", { input_tokens: 1, cache_read_input_tokens: 2, cache_creation_input_tokens: 1000, + cache_creation: { ephemeral_5m_input_tokens: 999, ephemeral_1h_input_tokens: 0 } }, 1003], + ["anthropic", { input_tokens: 1, cache_creation: { ephemeral_5m_input_tokens: 900, ephemeral_1h_input_tokens: 100 } }, 1001], +] as const)("retains %s input bounds independently of incomplete or contradictory price counters", (provider, usage, input) => { + const receipt = normalizeProviderUsage(provider, { usage }); + expect(receipt).toMatchObject({ complete: false, tokenBounds: { input, valid: true } }); + expect(estimateProviderCost(receipt, { ...price, provider })).toBeNull(); +}); +it.each([null, -1, 1.5, "1001", Number.MAX_SAFE_INTEGER + 1])("marks malformed input evidence %j ineligible for a bound", input_tokens => { + expect(normalizeProviderUsage("openai", { usage: { input_tokens } }).tokenBounds?.valid).toBe(false); +}); diff --git a/tests/runner-e2e/provider-usage.ts b/tests/runner-e2e/provider-usage.ts index bb3453a761d..4b5b702f832 100644 --- a/tests/runner-e2e/provider-usage.ts +++ b/tests/runner-e2e/provider-usage.ts @@ -9,6 +9,7 @@ export interface ProviderUsageReceipt { provider: "openai" | "anthropic"; tokens: Tokens; complete: boolean; + tokenBounds?: { input: number; output: number; valid: boolean }; providerResponseId?: string; model?: string; cacheWrite5m?: number; @@ -38,6 +39,24 @@ function openAiRequestCachePolicy(payload: unknown): { model: string; ttl: "30m" return { model: body.model, ttl }; } +function tokenBounds(provider: "openai" | "anthropic", usage: Row) { + const input = row(usage.input_tokens_details), output = row(usage.output_tokens_details); + const writes = provider === "openai" ? row(input.cache_write_tokens_details) : row(usage.cache_creation); + const rawWrite = provider === "openai" ? input.cache_write_tokens : usage.cache_creation_input_tokens; + const rawRead = provider === "openai" ? input.cached_tokens : usage.cache_read_input_tokens; + const ttls = provider === "openai" ? [writes.ttl_30m_tokens, writes.ttl_12h_tokens] + : [writes.ephemeral_5m_input_tokens, writes.ephemeral_1h_input_tokens]; + const inputs = [usage.input_tokens, rawRead, rawWrite, ...ttls]; + const outputs = [usage.output_tokens, ...(provider === "openai" ? [output.reasoning_tokens] : [])]; + const value = (raw: unknown) => count(raw) ?? 0; + const cached = value(rawRead) + Math.max(value(rawWrite), value(ttls[0]) + value(ttls[1])); + const inputBound = provider === "openai" ? Math.max(value(usage.input_tokens), cached) : value(usage.input_tokens) + cached; + const outputBound = Math.max(...outputs.map(value)); + return { input: inputBound, output: outputBound, + valid: [...inputs, ...outputs].every(raw => raw === undefined || count(raw) !== undefined) + && Number.isSafeInteger(inputBound) && Number.isSafeInteger(outputBound) }; +} + export function normalizeProviderUsage(provider: "openai" | "anthropic", payload: unknown): ProviderUsageReceipt { const body = row(payload), usage = row(body.usage); const output = count(usage.output_tokens); @@ -65,7 +84,7 @@ export function normalizeProviderUsage(provider: "openai" | "anthropic", payload } const complete = tokens.input !== undefined && tokens.cachedRead !== undefined && tokens.cachedWrite !== undefined && output !== undefined && !(tokens.reasoning !== undefined && tokens.reasoning > output); - return { provider, tokens: Object.fromEntries(Object.entries(tokens).filter(([, value]) => value !== undefined)), complete, + return { provider, tokenBounds: tokenBounds(provider, usage), tokens: Object.fromEntries(Object.entries(tokens).filter(([, value]) => value !== undefined)), complete, ...(typeof body.id === "string" ? { providerResponseId: body.id } : {}), ...(typeof body.model === "string" ? { model: body.model } : {}), ...(cacheWrite5m !== undefined ? { cacheWrite5m } : {}), ...(cacheWrite1h !== undefined ? { cacheWrite1h } : {}), diff --git a/tests/runner-e2e/provisional-accounting.test.ts b/tests/runner-e2e/provisional-accounting.test.ts new file mode 100644 index 00000000000..fde5e01603d --- /dev/null +++ b/tests/runner-e2e/provisional-accounting.test.ts @@ -0,0 +1,131 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, expect, it } from "vitest"; +import { openEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js"; +import { costBoundConfigurationSha256, reservationBound, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; +import { EVAL_OVERHEAD_COMPONENTS } from "./operating-overhead.js"; +import { evalAccountingStatus, readEvalOperationsReport, renderEvalOperationsReport, validateEvalOperationsReport } from "./operations-report.js"; +import { scheduledCampaignFindings } from "./scheduled-finalization.js"; +import type { RunnerE2ECampaign } from "./types.js"; + +const directories: string[] = []; +afterEach(async () => { for (const dir of directories.splice(0)) await rm(dir, { recursive: true, force: true }); }); +const evidence: EvalCostBoundEvidence = { version: 1, scope: "full_lifecycle", evidenceSha256: "a".repeat(64), configurationSha256: costBoundConfigurationSha256({ fixture: true }) }; +const identity = { executionId: "cell", attemptId: "one", phase: "agent" as const }; +const bound = reservationBound("model_limits", evidence, { fixture: true }); +async function fixture(overhead = false) { + const dir = await mkdtemp(path.join(tmpdir(), "provisional-accounting-")); directories.push(dir); + const file = path.join(dir, "ledger.jsonl"); + const limits: EvalSpendLimits = { campaignId: "fixture", maxUsd: "20", maxRequests: 20, maxConcurrent: 2, + deadlineAt: "2099-01-01T00:00:00Z", maxAttemptsPerExecution: 1, + ...(overhead ? { overhead: { version: 1 as const, reference: "fixture-audit", runner: "github-hosted" as const, + verifiedAt: new Date(Date.now() - 60_000).toISOString(), expiresAt: new Date(Date.now() + 86_400_000).toISOString(), + components: EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: component === "shared_subscription" ? "0" : "1", + reference: `fixture-${component}`, provisionalBound: evidence })) } } : {}) }; + if (limits.overhead) for (const component of limits.overhead.components) { + component.provisionalBound = { ...evidence, configurationSha256: costBoundConfigurationSha256({ + reference: limits.overhead.reference, runner: limits.overhead.runner, verifiedAt: limits.overhead.verifiedAt, expiresAt: limits.overhead.expiresAt, + component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference }, + }) }; + } + const ledger = await openEvalCostLedger(file, limits); + await ledger.admitAttempt(identity.executionId, identity.attemptId); + return { ledger, file, limits }; +} +function accountingFindings(operations: Awaited>) { + return scheduledCampaignFindings({ results: [], expected: [], operations } as unknown as RunnerE2ECampaign) + .filter(finding => finding.startsWith("accounting_")); +} + +it("allows reviewed pending operating/model costs without changing holds or finality, including after restart", async () => { + const { ledger, file, limits } = await fixture(true); + await ledger.reserve("request", identity, "2", bound); + await ledger.observe({ observationId: "partial", requestId: "request", source: "provider", provider: "fixture", currency: "USD", + route: "direct", classification: "estimated", pricingVersion: "fixture", costUsd: "0.1", final: false }); + await ledger.finish("request", "cancelled"); + await ledger.close(); + const original = await readFile(file, "utf8"); + const report = await readEvalOperationsReport(file); + expect(report).toMatchObject({ unresolvedRequests: 5, boundedUnresolvedRequests: 5, knownUsd: "0.100000000", heldUsd: "4.900000000", committedUsd: "5.000000000" }); + expect(evalAccountingStatus(report)).toBe("provisional"); + expect(accountingFindings(report)).toEqual([]); + const rendered = renderEvalOperationsReport(report); + expect(rendered).toContain("Accounting status: provisional"); + expect(rendered).toContain("requires later reconciliation"); + expect(rendered).not.toContain(evidence.evidenceSha256); + const resumed = await openEvalCostLedger(file, limits); + try { expect(resumed.snapshot()).toMatchObject({ boundedUnresolvedRequests: 5, heldUsd: "4.900000000" }); } + finally { await resumed.close(); } + expect(await readFile(file, "utf8")).toBe(original); +}); + +it("keeps legacy reservations and legacy reports unqualified instead of inferring a bound from dollars", async () => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2"); + await ledger.finish("request", "cancelled"); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(accountingFindings(report)).toEqual(["accounting_incomplete"]); + if (report.status === "unavailable") throw new Error("Missing fixture report"); + delete report.boundedUnresolvedRequests; + validateEvalOperationsReport(report); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(() => validateEvalOperationsReport({ ...report, boundedUnresolvedRequests: 2 })).toThrow(); + } finally { await ledger.close(); } +}); + +it.each(["inflight", "invalidated", "partial-over-bound", "conflict", "over-budget"])("blocks %s even with reviewed bound evidence", async reason => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2", bound); + if (reason === "invalidated") await ledger.invalidateBound("request"); + if (reason !== "inflight") await ledger.finish("request", "cancelled"); + if (["partial-over-bound", "conflict", "over-budget"].includes(reason)) { + await ledger.observe({ observationId: "first", requestId: "request", source: "provider", provider: "fixture", currency: "USD", route: "direct", + classification: "reported", costUsd: reason === "over-budget" ? "21" : "3", final: reason !== "partial-over-bound" }); + } + if (reason === "conflict") await ledger.observe({ observationId: "second", requestId: "request", source: "provider", provider: "fixture", + currency: "USD", route: "direct", classification: "reported", costUsd: "4", final: true }); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(accountingFindings(report)).toContain("accounting_incomplete"); + if (reason === "over-budget") expect(accountingFindings(report)).toContain("accounting_over_budget"); + expect(report).toMatchObject({ boundedUnresolvedRequests: 0 }); + } finally { await ledger.close(); } +}); + +it("lets a later complete receipt settle the original reservation without treating the provisional hold as a charge", async () => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2", bound); + await ledger.finish("request", "cancelled"); + expect(evalAccountingStatus(await readEvalOperationsReport(file))).toBe("provisional"); + await ledger.observe({ observationId: "invoice", requestId: "request", source: "invoice", provider: "fixture", currency: "USD", + route: "direct", classification: "reconciled", costUsd: "0.2", final: true }); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("receipts_complete"); + expect(report).toMatchObject({ boundedUnresolvedRequests: 0, knownUsd: "0.200000000", heldUsd: "0.000000000" }); + } finally { await ledger.close(); } +}); + +it("rejects malformed or mismatched evidence before creating a reservation", async () => { + const { ledger } = await fixture(); + try { + for (const invalid of [ { ...bound, calculationSha256: "missing" }, { ...bound, basis: "operating_allowance" }, + { ...bound, evidence: { ...evidence, scope: "current_day" } }, { ...bound, evidence: { ...evidence, evidenceSha256: "bad" } }, + { ...bound, evidence: { ...evidence, privateField: "not-allowed" } } ]) { + await expect(ledger.reserve("request", identity, "2", invalid as never)).rejects.toThrow(); + } + expect(ledger.snapshot().requests).toHaveLength(0); + } finally { await ledger.close(); } +}); + +it("does not accept null as reviewed operating-bound evidence", async () => { + const { ledger, limits } = await fixture(true); + await ledger.close(); + limits.overhead!.components[0]!.provisionalBound = null as never; + const directory = await mkdtemp(path.join(tmpdir(), "provisional-malformed-")); directories.push(directory); + await expect(openEvalCostLedger(path.join(directory, "ledger.jsonl"), limits)).rejects.toThrow("Invalid reviewed cost bound evidence"); +}); diff --git a/tests/runner-e2e/scheduled-finalization.ts b/tests/runner-e2e/scheduled-finalization.ts index b9653b17723..83dfe9bbdcd 100644 --- a/tests/runner-e2e/scheduled-finalization.ts +++ b/tests/runner-e2e/scheduled-finalization.ts @@ -1,3 +1,5 @@ +import { usdToUnits } from "../../packages/shared/src/money.js"; +import { evalAccountingStatus } from "./operations-report.js"; import { readFile, writeFile } from "node:fs/promises"; import path from "node:path"; import { pathToFileURL } from "node:url"; @@ -44,7 +46,10 @@ export function scheduledCampaignFindings(campaign: RunnerE2ECampaign): string[] if (campaign.accounting?.attempts.some(row => row.cleanup !== "passed")) findings.push("cleanup_failure"); const operations = campaign.operations; if (!operations || operations.status === "unavailable") findings.push("accounting_unavailable"); - else if (operations.unresolvedRequests || operations.conflictingRequests || operations.inflight) findings.push("accounting_incomplete"); + else { + if (evalAccountingStatus(operations) === "blocked") findings.push("accounting_incomplete"); + if (usdToUnits(operations.committedUsd) > usdToUnits(operations.limitUsd)) findings.push("accounting_over_budget"); + } const feedback = summarizeCampaignFeedback(campaign.accounting?.attempts ?? campaign.results); if (feedback.incomplete || feedback.failedSubmissions || feedback.unknownSubmissions) findings.push("feedback_collection_incomplete"); if (!campaign.feedbackReview || campaign.feedbackReview.decisions.some(row => ["actionable", "needs_investigation"].includes(row.disposition))) findings.push("feedback_needs_review"); @@ -107,6 +112,7 @@ export async function finalizeScheduledCampaign(env: NodeJS.ProcessEnv) { if (indexing !== "verified") findings.push("trace_or_experiment_indexing_incomplete"); await writeFile(path.join(directory, "findings.json"), `${JSON.stringify({ version: 1, findings, nativeIndexVerification: indexing, invoiceReconciliation: "pending", + accountingStatus: campaign.operations ? evalAccountingStatus(campaign.operations) : "unavailable", feedbackReviewMethod: "Only exact synthetic fixture bodies with all corroborating graders are automatically dismissed; all other feedback requires review" }, null, 2)}\n`, { mode: 0o600 }); for (const finding of findings) console.error(`::error title=Eval operations::${finding}`); return findings;