diff --git a/tests/runner-e2e/OPERATIONS-COVERAGE.md b/tests/runner-e2e/OPERATIONS-COVERAGE.md index da4e2efa6ec..4c6ba62aa58 100644 --- a/tests/runner-e2e/OPERATIONS-COVERAGE.md +++ b/tests/runner-e2e/OPERATIONS-COVERAGE.md @@ -133,7 +133,10 @@ the existing $249/month fee while including new usage and capacity charges. Scheduled admission now reserves the declared CI, storage, observability and subscription allowances inside the same $20/$100 journal limit. Missing or stale allowances stop scheduling before the AWS test runner starts. Missing receipts -keep their reservations held and accounting incomplete. This is a local admission +keep their reservations held. Explicit reviewed full-lifecycle bounds may allow +provisional completion while billing remains pending; missing bounds, conflicts +and overspending still fail. Legacy reservations gain no qualification from the +policy change. This is a local admission control, not a provider billing cap. Actual resource bounds, cost collection and reviewed allowances still need qualification before recurrence is enabled; see [the allowance contract](OPERATIONS.md#recurring-campaigns). diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 043097ac7fa..09fa4cd8ee3 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -173,6 +173,78 @@ Private request receipts retain transport IDs, response IDs, usage and cache TTL route/cache/failover evidence, and final accounting status without request bodies or credentials. Raw provider responses remain only in the normal private evidence. +## Provisional completion and later reconciliation + +Recurring campaigns may finish with provisional accounting when every pending +charge has an explicit reviewed bound, all requests have stopped, no receipts +conflict, and known costs plus retained holds fit the campaign cap. Product, +cleanup, feedback, trace and ingestion requirements still apply. Reports and +Braintrust experiment metadata label accounting as `provisional`, +`receipts_complete`, `blocked`, or `unavailable`. Receipt completeness is separate +from invoice reconciliation. The original failed campaign remains failed; a +later report or policy does not change its original result. + +The optional `provisionalBound` on a manifest model or operating component is: + +```json +{ + "version": 1, + "scope": "full_lifecycle", + "evidenceSha256": "SHA256_OF_RETAINED_REVIEWED_AUDIT", + "configurationSha256": "SHA256_OF_EXACT_REVIEWED_CONFIGURATION" +} +``` + +These placeholders are invalid until replaced with actual lowercase SHA-256 +values. Use `costBoundConfigurationSha256` from `cost-bound-evidence.ts` for +stable key-sorted JSON hashing. The model configuration contains `provider`, +`mode` (the manifest route), `price`, `maximumInputTokens`, +`maximumOutputTokens`, and `reservation` (the nine-decimal value from +`evalRequestReservation`). An operating configuration contains allowance +`reference`, `runner`, `verifiedAt`, `expiresAt`, and a `component` object with +its `component`, `maxUsd`, and `reference`. Changing any of those values +invalidates the approval hash. Evidence metadata itself is excluded from this +configuration hash. + +An operator must retain and review the audit before configuring its hash. +Hashes bind evidence and configuration; they do not prove the audit is true. +A model audit must establish the provider's full context ceiling, all billable +token prices and tiers, standard inference, and the route's billing behavior. +The gateway enforces output limits and supported request shapes. An operating +audit must cover setup, failed attempts, build time, publication, transfer, +observability and the full retained-storage lifecycle, including uncertainty in +resource deletion and invoice arrival. It must name the enforceable resource +limits and rates that keep the whole component below `maxUsd`. A current-day +usage estimate or an arbitrary contingency amount is insufficient. A zero bound +also needs evidence. Keep raw audits private; public reports expose only counts +and the existing journal checkpoint. + +The ledger retains bound evidence before model dispatch. Unexpected or missing +Braintrust route confirmation, failover, a returned model mismatch, or observed +usage above a declared limit disqualifies that request's bound. Provider usage +normalization retains a separate input/output lower bound even when missing or +contradictory cache counters prevent pricing. Malformed reported counts also +disqualify a bound. In-flight +requests, conflicts, pending charges above their bound, and campaign overruns +still block completion. Gateway cancellation before route confirmation remains +unqualified even if it retained an admission reservation. Later receipts remain +the way to resolve it. + +Provisional completion never releases a reservation or manufactures a zero +charge. A missing receipt stays unknown and holds its full bound, less any +known partial amount, until a valid final receipt supersedes it. The existing +receipt importer still requires complete billing/resource windows before it +marks a receipt final. Later bills can reveal an overrun; this local journal +cannot stop external billing. Reconcile the same retained journal after its +billing windows close, regenerate its report, and inspect any overruns before +admitting more work. Existing subscription costs remain separate from the +incremental daily/weekly budgets. + +Old journals and reports remain readable. They gain no bound evidence by +upgrade or by this policy; unresolved legacy charges still block. Existing +configured allowances and price cards are not automatically qualified by the +new optional field. Recurrence remains subject to its existing enable gates. + ## Manifest A manifest contains no credentials. This structural example uses fixture prices; @@ -420,8 +492,9 @@ lifecycles. They do not occupy model concurrency, but they count toward the finite charge-record limit. Resuming cannot change or drop them. Until matched cost observations arrive, the report shows them as unknown and keeps their allowance held; even a zero allowance does not claim a known zero charge. -The existing finalizer reports incomplete accounting while these receipts are -missing. Do not dismiss that finding to enable recurrence. +The finalizer permits provisional completion only when each pending charge has +explicit reviewed bound evidence as described below. Missing or unqualified +bounds remain incomplete accounting; a reservation alone does not qualify. Reconcile a component through its existing request ID with provenance-bound `infrastructure` or `invoice` observations. Partial observations keep the @@ -587,8 +660,8 @@ explicit, unbounded qualification path and does not inherit these allowances. After execution, deterministic review dismisses only exact synthetic feedback bodies corroborated by all feedback-persistence graders. Novel feedback remains -`needs_investigation`; no extra model call is made. Missing accounting, unresolved -receipts, feedback collection/submission failures, novel feedback, export failure +`needs_investigation`; no extra model call is made. Missing accounting, unbounded +pending charges, conflicts, spending overruns, feedback collection/submission failures, novel feedback, export failure and incomplete experiment import or indexing fail the job with bounded diagnostic codes. Successful unchanged runs produce artifacts without posting messages. GitHub's workflow failure notifications remain the operator's notification channel. diff --git a/tests/runner-e2e/braintrust-experiment.ts b/tests/runner-e2e/braintrust-experiment.ts index 4b2859e17d8..6db75341ede 100644 --- a/tests/runner-e2e/braintrust-experiment.ts +++ b/tests/runner-e2e/braintrust-experiment.ts @@ -1,3 +1,4 @@ +import { evalAccountingStatus } from "./operations-report.js"; import { createHash } from "node:crypto"; import { buildCampaignAccounting } from "./attempt-accounting.js"; import { exportBraintrustAttemptParent, type EvalBraintrustConfig } from "./braintrust.js"; @@ -98,6 +99,7 @@ export function buildBraintrustExperiment(projectId: string, campaign: RunnerE2E missing_attempts: accounting.missingAttempts, request_ledger: operations ? { journal_hash: operations.journalHash, known_usd: operations.knownUsd, additional_held_usd: operations.heldUsd, committed_usd: operations.committedUsd, + accounting_status: evalAccountingStatus(operations), bounded_unresolved_requests: operations.boundedUnresolvedRequests ?? 0, requests: operations.requests, unresolved_requests: operations.unresolvedRequests, breakdown: operations.breakdown } : null, feedback_review: campaign.feedbackReview ?? null, accounting_note: "Ledger amounts are separate observations, not additive trace metrics. Held reservations are not charges. Requests outside the gateway and infrastructure require separate accounting." }; diff --git a/tests/runner-e2e/cost-bound-evidence.ts b/tests/runner-e2e/cost-bound-evidence.ts new file mode 100644 index 00000000000..6d0bca53560 --- /dev/null +++ b/tests/runner-e2e/cost-bound-evidence.ts @@ -0,0 +1,45 @@ +import { createHash } from "node:crypto"; + +export interface EvalCostBoundEvidence { + version: 1; + scope: "full_lifecycle"; + evidenceSha256: string; + configurationSha256: string; +} +export interface EvalReservationBound { + basis: "model_limits" | "operating_allowance"; + evidence: EvalCostBoundEvidence; + calculationSha256: string; +} +const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value); + +export function validateCostBoundEvidence(value: EvalCostBoundEvidence): void { + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).some(key => !["version", "scope", "evidenceSha256", "configurationSha256"].includes(key)) + || value.version !== 1 || value.scope !== "full_lifecycle" || !hash(value.evidenceSha256) || !hash(value.configurationSha256)) { + throw new Error("Invalid reviewed cost bound evidence"); + } +} + +export function costBoundConfigurationSha256(calculation: unknown): string { + return createHash("sha256").update(JSON.stringify(calculation, (_key, value) => + value && typeof value === "object" && !Array.isArray(value) + ? Object.fromEntries(Object.keys(value).sort().map(key => [key, value[key]])) : value)).digest("hex"); +} + +export function reservationBound(basis: EvalReservationBound["basis"], evidence: EvalCostBoundEvidence, calculation: unknown): EvalReservationBound { + validateCostBoundEvidence(evidence); + const calculationSha256 = costBoundConfigurationSha256(calculation); + if (calculationSha256 !== evidence.configurationSha256) throw new Error("Reviewed cost bound configuration changed"); + return { basis, evidence: structuredClone(evidence), calculationSha256 }; +} + +export function validateReservationBound(value: EvalReservationBound, infrastructure: boolean): void { + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).some(key => !["basis", "evidence", "calculationSha256"].includes(key)) + || value.basis !== (infrastructure ? "operating_allowance" : "model_limits") || !hash(value.calculationSha256)) { + throw new Error("Invalid reservation bound"); + } + validateCostBoundEvidence(value.evidence); + if (value.calculationSha256 !== value.evidence.configurationSha256) throw new Error("Reviewed cost bound configuration changed"); +} diff --git a/tests/runner-e2e/cost-ledger.ts b/tests/runner-e2e/cost-ledger.ts index c132a6257fa..daa95eba7b3 100644 --- a/tests/runner-e2e/cost-ledger.ts +++ b/tests/runner-e2e/cost-ledger.ts @@ -1,9 +1,10 @@ +import { validateReservationBound, type EvalReservationBound } from "./cost-bound-evidence.js"; import { createHash } from "node:crypto"; import { constants } from "node:fs"; import { mkdir, open, readFile, rm, type FileHandle } from "node:fs/promises"; import path from "node:path"; import { usdToUnits } from "../../packages/shared/src/money.js"; -import { validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; +import { operatingReservationBound, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; export interface EvalSpendLimits { campaignId: string; @@ -46,11 +47,12 @@ type Event = | { type: "campaign"; limits: EvalSpendLimits } | ({ type: "attempt_policy" } & EvalLegacyAttemptAudit) | { type: "attempt"; executionId: string; attemptId: string } - | { type: "reserve"; requestId: string; identity: EvalChargeIdentity; maxUsd: string } + | { type: "reserve"; requestId: string; identity: EvalChargeIdentity; maxUsd: string; bound?: EvalReservationBound } + | { type: "invalidate_bound"; requestId: string } | { type: "observe"; observation: EvalCostObservation } | { type: "finish"; requestId: string; outcome: "succeeded" | "failed" | "cancelled" }; interface Line { seq: number; at: string; previous: string; event: Event; hash: string } -interface RequestState { identity: EvalChargeIdentity; reserved: bigint; observations: EvalCostObservation[]; finished: boolean } +interface RequestState { identity: EvalChargeIdentity; reserved: bigint; observations: EvalCostObservation[]; finished: boolean; bound?: EvalReservationBound; boundInvalidated?: boolean } const usd = (units: bigint) => `${units / 1_000_000_000n}.${String(units % 1_000_000_000n).padStart(9, "0")}`; export const validEvalLedgerId = (value: unknown): value is string => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:/-]{0,511}$/.test(value); const validId = validEvalLedgerId; @@ -89,6 +91,7 @@ export class EvalSpendState { for (const component of this.limits.overhead.components) this.requests.set(`overhead:${component.component}`, { identity: { executionId: "campaign-overhead", attemptId: component.component, phase: "infrastructure" }, reserved: amount(component.maxUsd), observations: [], finished: true, + ...(component.provisionalBound ? { bound: operatingReservationBound(this.limits.overhead, component) } : {}), }); } } @@ -105,7 +108,7 @@ export class EvalSpendState { resolved: finalized.length > 0 && amounts.size === 1 }; } snapshot() { - let known = 0n, held = 0n, committed = 0n, inflight = 0, unknown = 0, conflicts = 0; + let known = 0n, held = 0n, committed = 0n, inflight = 0, unknown = 0, conflicts = 0, boundedUnresolved = 0; const breakdown: Record<"reported" | "estimated" | "allocated" | "reconciled", bigint> = { reported: 0n, estimated: 0n, allocated: 0n, reconciled: 0n }; const requests = [...this.requests].map(([requestId, request]) => { const receipt = this.authoritative(request); @@ -113,6 +116,9 @@ export class EvalSpendState { if (receipt.conflict) conflicts++; if (!receipt.resolved) unknown++; const cost = receipt.cost ?? 0n; + const pendingBounded = !receipt.resolved && !receipt.conflict && request.finished && !!request.bound + && !request.boundInvalidated && cost <= request.reserved; + if (pendingBounded) boundedUnresolved++; known += cost; if (receipt.classification !== "unknown") breakdown[receipt.classification] += cost; const commitment = !request.finished || !receipt.resolved @@ -120,10 +126,10 @@ export class EvalSpendState { committed += commitment; held += commitment - cost; return { requestId, ...request.identity, reservedUsd: usd(request.reserved), costUsd: receipt.cost === null ? null : usd(cost), - classification: receipt.classification, conflict: receipt.conflict, receiptComplete: receipt.resolved, finished: request.finished }; + classification: receipt.classification, conflict: receipt.conflict, receiptComplete: receipt.resolved, finished: request.finished, pendingBounded }; }); return { campaignId: this.limits.campaignId, limitUsd: usd(amount(this.limits.maxUsd)), knownUsd: usd(known), heldUsd: usd(held), committedUsd: usd(committed), - overBudget: committed > amount(this.limits.maxUsd), unknownRequests: unknown, conflictingRequests: conflicts, inflight, + overBudget: committed > amount(this.limits.maxUsd), unknownRequests: unknown, boundedUnresolvedRequests: boundedUnresolved, conflictingRequests: conflicts, inflight, breakdown: Object.fromEntries(Object.entries(breakdown).map(([key, value]) => [key, usd(value)])), requests, admittedAttempts: [...this.attempts].map(([attemptId, executionId]) => ({ executionId, attemptId })) }; } @@ -156,6 +162,8 @@ export class EvalSpendState { if (now >= Date.parse(this.limits.deadlineAt)) throw new EvalAdmissionError("Eval campaign deadline reached"); if ([...this.attempts.values()].filter(id => id === event.executionId).length >= this.limits.maxAttemptsPerExecution) throw new EvalAdmissionError("Eval campaign attempt cap reached"); } else if (event.type === "reserve") { + fields(event, ["type", "requestId", "identity", "maxUsd", "bound"]); + if (event.bound !== undefined) validateReservationBound(event.bound, event.identity.phase === "infrastructure"); fields(event.identity, ["executionId", "attemptId", "phase", "runId"]); if (!validId(event.requestId) || !validId(event.identity.executionId) || !validId(event.identity.attemptId) || event.identity.runId !== undefined && !validId(event.identity.runId) @@ -189,6 +197,9 @@ export class EvalSpendState { const request = this.providerRequests.get(`${o.provider}:${o.providerRequestId}`); if (request && request !== o.requestId) throw new Error("Provider request belongs to another reservation; reconcile identities before import"); } + } else if (event.type === "invalidate_bound") { + fields(event, ["type", "requestId"]); + if (!this.requests.get(event.requestId)?.bound) throw new Error("Cannot invalidate an absent reservation bound"); } else if (event.type === "finish") { if (!this.requests.has(event.requestId) || !["succeeded", "failed", "cancelled"].includes(event.outcome)) throw new Error("Invalid request completion"); } else throw new Error("Unknown ledger event"); @@ -198,14 +209,15 @@ export class EvalSpendState { this.limits.maxAttemptsPerExecution = event.maxAttemptsPerExecution; for (const attempt of event.priorAttempts) this.attempts.set(attempt.attemptId, attempt.executionId); } else if (event.type === "attempt") this.attempts.set(event.attemptId, event.executionId); - else if (event.type === "reserve") this.requests.set(event.requestId, { identity: event.identity, reserved: amount(event.maxUsd), observations: [], finished: false }); + else if (event.type === "reserve") this.requests.set(event.requestId, { identity: event.identity, reserved: amount(event.maxUsd), observations: [], finished: false, ...(event.bound ? { bound: event.bound } : {}) }); else if (event.type === "observe") { const o = event.observation; if (this.observations.has(o.observationId)) return; this.observations.set(o.observationId, o); this.requests.get(o.requestId)!.observations.push(o); if (o.providerRequestId) this.providerRequests.set(`${o.provider}:${o.providerRequestId}`, o.requestId); - } else if (event.type === "finish") this.requests.get(event.requestId)!.finished = true; + } else if (event.type === "invalidate_bound") this.requests.get(event.requestId)!.boundInvalidated = true; + else if (event.type === "finish") this.requests.get(event.requestId)!.finished = true; } } @@ -273,7 +285,8 @@ export async function openEvalCostLedger(file: string, limits: EvalSpendLimits) return { adoptLegacyAttemptPolicy: (audit: EvalLegacyAttemptAudit) => append({ ...structuredClone(audit), type: "attempt_policy" }), admitAttempt: (executionId: string, attemptId: string) => append({ type: "attempt", executionId, attemptId }), - reserve: (requestId: string, identity: EvalChargeIdentity, maxUsd: string) => append({ type: "reserve", requestId, identity: structuredClone(identity), maxUsd }), + reserve: (requestId: string, identity: EvalChargeIdentity, maxUsd: string, bound?: EvalReservationBound) => append({ type: "reserve", requestId, identity: structuredClone(identity), maxUsd, ...(bound ? { bound: structuredClone(bound) } : {}) }), + invalidateBound: (requestId: string) => append({ type: "invalidate_bound", requestId }), observe: (observation: EvalCostObservation) => append({ type: "observe", observation: structuredClone(observation) }), finish: (requestId: string, outcome: "succeeded" | "failed" | "cancelled") => append({ type: "finish", requestId, outcome }), snapshot: () => state.snapshot(), diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index ba5266e72f3..6e38d356bea 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -1,3 +1,4 @@ +import { costBoundConfigurationSha256 } from "./cost-bound-evidence.js"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -355,3 +356,48 @@ it("refuses missing or unreviewed attribution checkpoints before request dispatc await writeFile(phasePath, JSON.stringify({ version: 1, ...identity, providerResponse: "private" })); await expect(readEvalAttemptPhase(phasePath, identity)).rejects.toThrow("Invalid eval phase"); }); + +const provisionalBound = { version: 1 as const, scope: "full_lifecycle" as const, evidenceSha256: "a".repeat(64), + configurationSha256: costBoundConfigurationSha256({ provider: route.provider, mode: route.mode, price: route.price, + maximumInputTokens: route.maximumInputTokens, maximumOutputTokens: route.maximumOutputTokens, reservation: evalRequestReservation(route) }) }; +it("retains a reviewed model bound through a partial response and durable reload without releasing the hold", async () => { + const f = await fixture(); + const receipt = await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => { + expect((await readEvalCostLedger(f.file)).state.requests.values().next().value?.bound?.evidence).toEqual(provisionalBound); + return Response.json({ object: "response", status: "in_progress", model: "fixture" }, + { headers: { "x-bt-used-endpoint": route.endpointName! } }); + } }); + expect(receipt).toMatchObject({ costUsd: null, final: false }); + expect((await readEvalCostLedger(f.file)).state.snapshot()).toMatchObject({ boundedUnresolvedRequests: 1, heldUsd: "0.003000000" }); +}); +it.each(["no-headers", "route", "failover", "model", "input-limit", "output-limit"])("disqualifies a model bound on %s evidence", async reason => { + const f = await fixture(); + await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => { + if (reason === "no-headers") throw new Error("No response headers"); + return Response.json({ object: "response", status: "in_progress", model: reason === "model" ? "other" : "fixture", + usage: { input_tokens: reason === "input-limit" ? 1001 : 10, input_tokens_details: { cache_write_tokens: 0, cached_tokens: 0 }, + output_tokens: reason === "output-limit" ? 101 : 0 } }, + { headers: { "x-bt-used-endpoint": reason === "route" ? "OTHER" : route.endpointName!, + ...(reason === "failover" ? { "x-bt-failover-to": "OTHER" } : {}) } }); + } }); + const replay = (await readEvalCostLedger(f.file)).state; + expect(replay.snapshot()).toMatchObject({ boundedUnresolvedRequests: 0, unknownRequests: 1, inflight: 0 }); + expect([...replay.requests.values()][0]!.boundInvalidated).toBe(true); +}); + +it("refuses a changed model limit or rate with a stale reviewed bound before dispatch", async () => { + const f = await fixture(), transport = vi.fn(); + for (const changed of [{ ...route, maximumInputTokens: 500 }, { ...route, price: { ...route.price, inputUsdPerMillion: "20" } }]) { + await expect(forwardEvalRequest({ ...f, route: { ...changed, provisionalBound }, transport })).rejects.toThrow("configuration changed"); + } + expect(transport).not.toHaveBeenCalled(); + expect(f.ledger.snapshot().requests).toHaveLength(0); +}); + +it("disqualifies excessive reported input even when missing cache details prevent pricing", async () => { + const f = await fixture(); + await forwardEvalRequest({ ...f, route: { ...route, provisionalBound }, transport: async () => Response.json({ + object: "response", status: "completed", model: "fixture", usage: { input_tokens: 1001, output_tokens: 1 }, + }, { headers: { "x-bt-used-endpoint": route.endpointName! } }) }); + expect((await readEvalCostLedger(f.file)).state.snapshot()).toMatchObject({ boundedUnresolvedRequests: 0, unknownRequests: 1, heldUsd: "0.003000000" }); +}); diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index 26af76ff106..330d87859f0 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -1,3 +1,4 @@ +import { reservationBound, validateCostBoundEvidence, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; import { randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; import { once } from "node:events"; @@ -17,6 +18,7 @@ export interface EvalGatewayRoute { maximumInputTokens: number; maximumOutputTokens: number; timeoutMs: number; + provisionalBound?: EvalCostBoundEvidence; } export interface EvalGatewayReceipt { requestId: string; status?: number; route: "direct" | "braintrust"; @@ -32,6 +34,7 @@ const row = (value: unknown): Record => value && typeof value = class EvalRequestError extends Error {} export function evalRequestReservation(route: EvalGatewayRoute): string { + if (route.provisionalBound !== undefined) validateCostBoundEvidence(route.provisionalBound); if (![route.maximumInputTokens, route.maximumOutputTokens, route.timeoutMs].every(n => Number.isSafeInteger(n) && n > 0) || route.timeoutMs > 300_000 || route.provider !== route.price.provider || !validEvalLedgerId(route.price.version) || !validEvalLedgerId(route.price.model)) throw new Error("Invalid eval gateway limits or price identity"); const parse = (rate: string) => { @@ -50,7 +53,16 @@ export function evalRequestReservation(route: EvalGatewayRoute): string { const output = parse(tier.outputUsdPerMillion); if (output > maximumOutputRate) maximumOutputRate = output; } const numerator = BigInt(route.maximumInputTokens) * maximumInputRate + BigInt(route.maximumOutputTokens) * maximumOutputRate; - return money((numerator + 999_999n) / 1_000_000n); + const reservation = money((numerator + 999_999n) / 1_000_000n); + modelReservationBound(route, reservation); + return reservation; +} + +function modelReservationBound(route: EvalGatewayRoute, reservation: string) { + return route.provisionalBound ? reservationBound("model_limits", route.provisionalBound, { + provider: route.provider, mode: route.mode, price: route.price, maximumInputTokens: route.maximumInputTokens, + maximumOutputTokens: route.maximumOutputTokens, reservation, + }) : undefined; } function boundedPayload(value: unknown, route: EvalGatewayRoute) { @@ -115,7 +127,8 @@ export async function forwardEvalRequest(input: { const receipt: EvalGatewayReceipt = { requestId, route: route.mode, outcome: "failed", usageComplete: false, final: false, costUsd: null, ...(route.mode === "braintrust" ? { braintrustParent: braintrustGatewayRequestParent(route.projectId!, requestId, ledger.snapshot().campaignId, input.identity) } : {}) }; const usage = new ProviderUsageAccumulator(route.provider, body); - await ledger.reserve(requestId, input.identity, reservation); + const bound = modelReservationBound(route, reservation); + await ledger.reserve(requestId, input.identity, reservation, bound); const abort = new AbortController(); const signal = AbortSignal.any([abort.signal, AbortSignal.timeout(Math.max(1, Math.min(route.timeoutMs, input.deadlineAt - Date.now()))), ...(input.signal ? [input.signal] : [])]); let response: Response | undefined; @@ -184,6 +197,12 @@ export async function forwardEvalRequest(input: { delete tokens.reasoning; receipt.errorCode ??= "invalid_usage"; } + const observedInput = observed.tokenBounds?.input ?? 0; + if (bound && (observed.tokenBounds?.valid === false || receipt.errorCode === "invalid_usage" || observed.model !== undefined && observed.model !== route.price.model + || !Number.isSafeInteger(observedInput) || observedInput > route.maximumInputTokens + || (observed.tokenBounds?.output ?? 0) > route.maximumOutputTokens + || route.mode === "braintrust" && (!response || receipt.endpoint !== route.endpointName + || receipt.cache === "HIT" || receipt.failoverFrom || receipt.failoverTo))) await ledger.invalidateBound(requestId); await input.persist(receipt); await ledger.observe({ observationId: `${requestId}:usage`, requestId, source: route.mode === "braintrust" ? "gateway" : "provider", provider: route.provider, diff --git a/tests/runner-e2e/operating-overhead.ts b/tests/runner-e2e/operating-overhead.ts index 4f53cd7517e..43eb5de338e 100644 --- a/tests/runner-e2e/operating-overhead.ts +++ b/tests/runner-e2e/operating-overhead.ts @@ -1,3 +1,4 @@ +import { reservationBound, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; import { usdToUnits } from "../../packages/shared/src/money.js"; export const EVAL_OVERHEAD_COMPONENTS = ["ci_compute", "artifact_storage", "observability", "shared_subscription"] as const; @@ -7,7 +8,7 @@ export interface EvalOperatingOverhead { runner: "github-hosted" | "aws-fleet"; verifiedAt: string; expiresAt: string; - components: Array<{ component: typeof EVAL_OVERHEAD_COMPONENTS[number]; maxUsd: string; reference: string }>; + components: Array<{ component: typeof EVAL_OVERHEAD_COMPONENTS[number]; maxUsd: string; reference: string; provisionalBound?: EvalCostBoundEvidence }>; } const object = (value: unknown, allowed: string[]) => { @@ -16,6 +17,14 @@ const object = (value: unknown, allowed: string[]) => { }; const reference = (value: unknown) => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9_.:/-]{0,511}$/.test(value); +export function operatingReservationBound(allowance: EvalOperatingOverhead, component: EvalOperatingOverhead["components"][number]) { + if (component.provisionalBound === undefined) return undefined; + return reservationBound("operating_allowance", component.provisionalBound, { + reference: allowance.reference, runner: allowance.runner, verifiedAt: allowance.verifiedAt, expiresAt: allowance.expiresAt, + component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference }, + }); +} + export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint { object(value, ["version", "reference", "runner", "verifiedAt", "expiresAt", "components"]); if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner) @@ -29,7 +38,8 @@ export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd const seen = new Set(); let reserved = 0n; for (const component of value.components) { - object(component, ["component", "maxUsd", "reference"]); + object(component, ["component", "maxUsd", "reference", "provisionalBound"]); + if (component.provisionalBound !== undefined) operatingReservationBound(value, component); if (!EVAL_OVERHEAD_COMPONENTS.includes(component.component) || seen.has(component.component) || !reference(component.reference) || typeof component.maxUsd !== "string" || !/^(?:0|[1-9][0-9]*)(?:\.[0-9]{1,9})?$/.test(component.maxUsd)) throw new Error("Invalid operating overhead component"); diff --git a/tests/runner-e2e/operations-report.ts b/tests/runner-e2e/operations-report.ts index 97d8a3df776..b751d24bed6 100644 --- a/tests/runner-e2e/operations-report.ts +++ b/tests/runner-e2e/operations-report.ts @@ -17,6 +17,7 @@ interface EvalOperationsAvailableReport { committedUsd: string; requests: number; unresolvedRequests: number; + boundedUnresolvedRequests?: number; conflictingRequests: number; inflight: number; breakdown: { reported: string; estimated: string; allocated: string; reconciled: string }; @@ -38,11 +39,15 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void } const dollars = (value: unknown): bigint => typeof value === "string" && /^(?:0|[1-9][0-9]*)\.[0-9]{9}$/.test(value) ? usdToUnits(value) : fail(); const count = (value: unknown) => { if (!Number.isSafeInteger(value) || Number(value) < 0) fail(); }; - fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); + fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); if (report.version !== 1 || report.scope !== "shared_gateway_ledger" || !validEvalLedgerId(report.ledgerCampaignId) || !/^[a-f0-9]{64}$/.test(report.journalHash) || !isPublicReportTimestamp(report.deadlineAt) || !Array.isArray(report.attempts)) fail(); [report.records, report.requests, report.unresolvedRequests, report.conflictingRequests, report.inflight].forEach(count); if (report.records < 1 || report.unresolvedRequests > report.requests || report.conflictingRequests > report.unresolvedRequests || report.inflight > report.requests) fail(); + if (report.boundedUnresolvedRequests !== undefined) { + count(report.boundedUnresolvedRequests); + if (report.boundedUnresolvedRequests > report.unresolvedRequests - report.conflictingRequests) fail(); + } dollars(report.limitUsd); const known = dollars(report.knownUsd), held = dollars(report.heldUsd); if (known + held !== dollars(report.committedUsd)) fail(); @@ -84,18 +89,27 @@ async function readAvailableEvalOperationsReport(file: string): Promise usdToUnits(report.limitUsd) + || report.unresolvedRequests !== (report.boundedUnresolvedRequests ?? 0)) return "blocked"; + return report.unresolvedRequests ? "provisional" : "receipts_complete"; +} + export function operationsReportText(report: EvalOperationsReport): string[] { if (report.status === "unavailable") return [ "Request ledger accounting unavailable: the journal could not be read or validated. Request amounts, counts, and held reservations are unknown.", "Retained test results are preserved. Recover and validate the original journal before resuming paid work; do not reset the allowance or treat missing amounts as zero.", ]; return [ + `Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`, `Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`, `Charges: $${report.breakdown.reconciled} reconciled; $${report.breakdown.reported} reported; $${report.breakdown.estimated} estimated; $${report.breakdown.allocated} allocated.`, `${report.requests} recorded requests or overhead reservations; ${report.unresolvedRequests} unresolved; ${report.conflictingRequests} conflicting; ${report.inflight} in flight. Deadline ${report.deadlineAt}.`, diff --git a/tests/runner-e2e/operations.ts b/tests/runner-e2e/operations.ts index 2ab81eb2676..62507e05533 100644 --- a/tests/runner-e2e/operations.ts +++ b/tests/runner-e2e/operations.ts @@ -35,7 +35,7 @@ export function validateEvalOperationsManifest(value: EvalOperationsManifest, ex if (value.route === "braintrust" && !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value.projectId ?? "")) throw new Error("Braintrust gateway requires an explicit project UUID"); const models = new Map(); for (const model of value.models) { - fields(model, ["provider", "price", "maximumInputTokens", "maximumOutputTokens", "timeoutMs"]); + fields(model, ["provider", "price", "maximumInputTokens", "maximumOutputTokens", "timeoutMs", "provisionalBound"]); const rateFields = ["inputUsdPerMillion", "cachedReadUsdPerMillion", "outputUsdPerMillion", "cacheWrite5mUsdPerMillion", "cacheWrite1hUsdPerMillion", "cacheWrite30mUsdPerMillion", "cacheWrite12hUsdPerMillion"]; fields(model.price, ["provider", "model", "version", "sourceUrl", "longContext", ...rateFields]); if (model.price.longContext) { fields(model.price.longContext, ["aboveInputTokens", "rates"]); fields(model.price.longContext.rates, rateFields); } diff --git a/tests/runner-e2e/provider-usage.test.ts b/tests/runner-e2e/provider-usage.test.ts index b651c9e8a89..6549e15684e 100644 --- a/tests/runner-e2e/provider-usage.test.ts +++ b/tests/runner-e2e/provider-usage.test.ts @@ -111,3 +111,18 @@ it("prefers provider TTL splits and never repairs malformed or partial provider expect(estimateProviderCost(receipt, policyPrice)).toBeNull(); } }); + +it.each([ + ["openai", { input_tokens: 1001, output_tokens: 1 }, 1001], + ["openai", { input_tokens_details: { cached_tokens: 1001 } }, 1001], + ["anthropic", { input_tokens: 1, cache_read_input_tokens: 2, cache_creation_input_tokens: 1000, + cache_creation: { ephemeral_5m_input_tokens: 999, ephemeral_1h_input_tokens: 0 } }, 1003], + ["anthropic", { input_tokens: 1, cache_creation: { ephemeral_5m_input_tokens: 900, ephemeral_1h_input_tokens: 100 } }, 1001], +] as const)("retains %s input bounds independently of incomplete or contradictory price counters", (provider, usage, input) => { + const receipt = normalizeProviderUsage(provider, { usage }); + expect(receipt).toMatchObject({ complete: false, tokenBounds: { input, valid: true } }); + expect(estimateProviderCost(receipt, { ...price, provider })).toBeNull(); +}); +it.each([null, -1, 1.5, "1001", Number.MAX_SAFE_INTEGER + 1])("marks malformed input evidence %j ineligible for a bound", input_tokens => { + expect(normalizeProviderUsage("openai", { usage: { input_tokens } }).tokenBounds?.valid).toBe(false); +}); diff --git a/tests/runner-e2e/provider-usage.ts b/tests/runner-e2e/provider-usage.ts index bb3453a761d..4b5b702f832 100644 --- a/tests/runner-e2e/provider-usage.ts +++ b/tests/runner-e2e/provider-usage.ts @@ -9,6 +9,7 @@ export interface ProviderUsageReceipt { provider: "openai" | "anthropic"; tokens: Tokens; complete: boolean; + tokenBounds?: { input: number; output: number; valid: boolean }; providerResponseId?: string; model?: string; cacheWrite5m?: number; @@ -38,6 +39,24 @@ function openAiRequestCachePolicy(payload: unknown): { model: string; ttl: "30m" return { model: body.model, ttl }; } +function tokenBounds(provider: "openai" | "anthropic", usage: Row) { + const input = row(usage.input_tokens_details), output = row(usage.output_tokens_details); + const writes = provider === "openai" ? row(input.cache_write_tokens_details) : row(usage.cache_creation); + const rawWrite = provider === "openai" ? input.cache_write_tokens : usage.cache_creation_input_tokens; + const rawRead = provider === "openai" ? input.cached_tokens : usage.cache_read_input_tokens; + const ttls = provider === "openai" ? [writes.ttl_30m_tokens, writes.ttl_12h_tokens] + : [writes.ephemeral_5m_input_tokens, writes.ephemeral_1h_input_tokens]; + const inputs = [usage.input_tokens, rawRead, rawWrite, ...ttls]; + const outputs = [usage.output_tokens, ...(provider === "openai" ? [output.reasoning_tokens] : [])]; + const value = (raw: unknown) => count(raw) ?? 0; + const cached = value(rawRead) + Math.max(value(rawWrite), value(ttls[0]) + value(ttls[1])); + const inputBound = provider === "openai" ? Math.max(value(usage.input_tokens), cached) : value(usage.input_tokens) + cached; + const outputBound = Math.max(...outputs.map(value)); + return { input: inputBound, output: outputBound, + valid: [...inputs, ...outputs].every(raw => raw === undefined || count(raw) !== undefined) + && Number.isSafeInteger(inputBound) && Number.isSafeInteger(outputBound) }; +} + export function normalizeProviderUsage(provider: "openai" | "anthropic", payload: unknown): ProviderUsageReceipt { const body = row(payload), usage = row(body.usage); const output = count(usage.output_tokens); @@ -65,7 +84,7 @@ export function normalizeProviderUsage(provider: "openai" | "anthropic", payload } const complete = tokens.input !== undefined && tokens.cachedRead !== undefined && tokens.cachedWrite !== undefined && output !== undefined && !(tokens.reasoning !== undefined && tokens.reasoning > output); - return { provider, tokens: Object.fromEntries(Object.entries(tokens).filter(([, value]) => value !== undefined)), complete, + return { provider, tokenBounds: tokenBounds(provider, usage), tokens: Object.fromEntries(Object.entries(tokens).filter(([, value]) => value !== undefined)), complete, ...(typeof body.id === "string" ? { providerResponseId: body.id } : {}), ...(typeof body.model === "string" ? { model: body.model } : {}), ...(cacheWrite5m !== undefined ? { cacheWrite5m } : {}), ...(cacheWrite1h !== undefined ? { cacheWrite1h } : {}), diff --git a/tests/runner-e2e/provisional-accounting.test.ts b/tests/runner-e2e/provisional-accounting.test.ts new file mode 100644 index 00000000000..fde5e01603d --- /dev/null +++ b/tests/runner-e2e/provisional-accounting.test.ts @@ -0,0 +1,131 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, expect, it } from "vitest"; +import { openEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js"; +import { costBoundConfigurationSha256, reservationBound, type EvalCostBoundEvidence } from "./cost-bound-evidence.js"; +import { EVAL_OVERHEAD_COMPONENTS } from "./operating-overhead.js"; +import { evalAccountingStatus, readEvalOperationsReport, renderEvalOperationsReport, validateEvalOperationsReport } from "./operations-report.js"; +import { scheduledCampaignFindings } from "./scheduled-finalization.js"; +import type { RunnerE2ECampaign } from "./types.js"; + +const directories: string[] = []; +afterEach(async () => { for (const dir of directories.splice(0)) await rm(dir, { recursive: true, force: true }); }); +const evidence: EvalCostBoundEvidence = { version: 1, scope: "full_lifecycle", evidenceSha256: "a".repeat(64), configurationSha256: costBoundConfigurationSha256({ fixture: true }) }; +const identity = { executionId: "cell", attemptId: "one", phase: "agent" as const }; +const bound = reservationBound("model_limits", evidence, { fixture: true }); +async function fixture(overhead = false) { + const dir = await mkdtemp(path.join(tmpdir(), "provisional-accounting-")); directories.push(dir); + const file = path.join(dir, "ledger.jsonl"); + const limits: EvalSpendLimits = { campaignId: "fixture", maxUsd: "20", maxRequests: 20, maxConcurrent: 2, + deadlineAt: "2099-01-01T00:00:00Z", maxAttemptsPerExecution: 1, + ...(overhead ? { overhead: { version: 1 as const, reference: "fixture-audit", runner: "github-hosted" as const, + verifiedAt: new Date(Date.now() - 60_000).toISOString(), expiresAt: new Date(Date.now() + 86_400_000).toISOString(), + components: EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: component === "shared_subscription" ? "0" : "1", + reference: `fixture-${component}`, provisionalBound: evidence })) } } : {}) }; + if (limits.overhead) for (const component of limits.overhead.components) { + component.provisionalBound = { ...evidence, configurationSha256: costBoundConfigurationSha256({ + reference: limits.overhead.reference, runner: limits.overhead.runner, verifiedAt: limits.overhead.verifiedAt, expiresAt: limits.overhead.expiresAt, + component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference }, + }) }; + } + const ledger = await openEvalCostLedger(file, limits); + await ledger.admitAttempt(identity.executionId, identity.attemptId); + return { ledger, file, limits }; +} +function accountingFindings(operations: Awaited>) { + return scheduledCampaignFindings({ results: [], expected: [], operations } as unknown as RunnerE2ECampaign) + .filter(finding => finding.startsWith("accounting_")); +} + +it("allows reviewed pending operating/model costs without changing holds or finality, including after restart", async () => { + const { ledger, file, limits } = await fixture(true); + await ledger.reserve("request", identity, "2", bound); + await ledger.observe({ observationId: "partial", requestId: "request", source: "provider", provider: "fixture", currency: "USD", + route: "direct", classification: "estimated", pricingVersion: "fixture", costUsd: "0.1", final: false }); + await ledger.finish("request", "cancelled"); + await ledger.close(); + const original = await readFile(file, "utf8"); + const report = await readEvalOperationsReport(file); + expect(report).toMatchObject({ unresolvedRequests: 5, boundedUnresolvedRequests: 5, knownUsd: "0.100000000", heldUsd: "4.900000000", committedUsd: "5.000000000" }); + expect(evalAccountingStatus(report)).toBe("provisional"); + expect(accountingFindings(report)).toEqual([]); + const rendered = renderEvalOperationsReport(report); + expect(rendered).toContain("Accounting status: provisional"); + expect(rendered).toContain("requires later reconciliation"); + expect(rendered).not.toContain(evidence.evidenceSha256); + const resumed = await openEvalCostLedger(file, limits); + try { expect(resumed.snapshot()).toMatchObject({ boundedUnresolvedRequests: 5, heldUsd: "4.900000000" }); } + finally { await resumed.close(); } + expect(await readFile(file, "utf8")).toBe(original); +}); + +it("keeps legacy reservations and legacy reports unqualified instead of inferring a bound from dollars", async () => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2"); + await ledger.finish("request", "cancelled"); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(accountingFindings(report)).toEqual(["accounting_incomplete"]); + if (report.status === "unavailable") throw new Error("Missing fixture report"); + delete report.boundedUnresolvedRequests; + validateEvalOperationsReport(report); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(() => validateEvalOperationsReport({ ...report, boundedUnresolvedRequests: 2 })).toThrow(); + } finally { await ledger.close(); } +}); + +it.each(["inflight", "invalidated", "partial-over-bound", "conflict", "over-budget"])("blocks %s even with reviewed bound evidence", async reason => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2", bound); + if (reason === "invalidated") await ledger.invalidateBound("request"); + if (reason !== "inflight") await ledger.finish("request", "cancelled"); + if (["partial-over-bound", "conflict", "over-budget"].includes(reason)) { + await ledger.observe({ observationId: "first", requestId: "request", source: "provider", provider: "fixture", currency: "USD", route: "direct", + classification: "reported", costUsd: reason === "over-budget" ? "21" : "3", final: reason !== "partial-over-bound" }); + } + if (reason === "conflict") await ledger.observe({ observationId: "second", requestId: "request", source: "provider", provider: "fixture", + currency: "USD", route: "direct", classification: "reported", costUsd: "4", final: true }); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(accountingFindings(report)).toContain("accounting_incomplete"); + if (reason === "over-budget") expect(accountingFindings(report)).toContain("accounting_over_budget"); + expect(report).toMatchObject({ boundedUnresolvedRequests: 0 }); + } finally { await ledger.close(); } +}); + +it("lets a later complete receipt settle the original reservation without treating the provisional hold as a charge", async () => { + const { ledger, file } = await fixture(); + try { + await ledger.reserve("request", identity, "2", bound); + await ledger.finish("request", "cancelled"); + expect(evalAccountingStatus(await readEvalOperationsReport(file))).toBe("provisional"); + await ledger.observe({ observationId: "invoice", requestId: "request", source: "invoice", provider: "fixture", currency: "USD", + route: "direct", classification: "reconciled", costUsd: "0.2", final: true }); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("receipts_complete"); + expect(report).toMatchObject({ boundedUnresolvedRequests: 0, knownUsd: "0.200000000", heldUsd: "0.000000000" }); + } finally { await ledger.close(); } +}); + +it("rejects malformed or mismatched evidence before creating a reservation", async () => { + const { ledger } = await fixture(); + try { + for (const invalid of [ { ...bound, calculationSha256: "missing" }, { ...bound, basis: "operating_allowance" }, + { ...bound, evidence: { ...evidence, scope: "current_day" } }, { ...bound, evidence: { ...evidence, evidenceSha256: "bad" } }, + { ...bound, evidence: { ...evidence, privateField: "not-allowed" } } ]) { + await expect(ledger.reserve("request", identity, "2", invalid as never)).rejects.toThrow(); + } + expect(ledger.snapshot().requests).toHaveLength(0); + } finally { await ledger.close(); } +}); + +it("does not accept null as reviewed operating-bound evidence", async () => { + const { ledger, limits } = await fixture(true); + await ledger.close(); + limits.overhead!.components[0]!.provisionalBound = null as never; + const directory = await mkdtemp(path.join(tmpdir(), "provisional-malformed-")); directories.push(directory); + await expect(openEvalCostLedger(path.join(directory, "ledger.jsonl"), limits)).rejects.toThrow("Invalid reviewed cost bound evidence"); +}); diff --git a/tests/runner-e2e/scheduled-finalization.ts b/tests/runner-e2e/scheduled-finalization.ts index b9653b17723..83dfe9bbdcd 100644 --- a/tests/runner-e2e/scheduled-finalization.ts +++ b/tests/runner-e2e/scheduled-finalization.ts @@ -1,3 +1,5 @@ +import { usdToUnits } from "../../packages/shared/src/money.js"; +import { evalAccountingStatus } from "./operations-report.js"; import { readFile, writeFile } from "node:fs/promises"; import path from "node:path"; import { pathToFileURL } from "node:url"; @@ -44,7 +46,10 @@ export function scheduledCampaignFindings(campaign: RunnerE2ECampaign): string[] if (campaign.accounting?.attempts.some(row => row.cleanup !== "passed")) findings.push("cleanup_failure"); const operations = campaign.operations; if (!operations || operations.status === "unavailable") findings.push("accounting_unavailable"); - else if (operations.unresolvedRequests || operations.conflictingRequests || operations.inflight) findings.push("accounting_incomplete"); + else { + if (evalAccountingStatus(operations) === "blocked") findings.push("accounting_incomplete"); + if (usdToUnits(operations.committedUsd) > usdToUnits(operations.limitUsd)) findings.push("accounting_over_budget"); + } const feedback = summarizeCampaignFeedback(campaign.accounting?.attempts ?? campaign.results); if (feedback.incomplete || feedback.failedSubmissions || feedback.unknownSubmissions) findings.push("feedback_collection_incomplete"); if (!campaign.feedbackReview || campaign.feedbackReview.decisions.some(row => ["actionable", "needs_investigation"].includes(row.disposition))) findings.push("feedback_needs_review"); @@ -107,6 +112,7 @@ export async function finalizeScheduledCampaign(env: NodeJS.ProcessEnv) { if (indexing !== "verified") findings.push("trace_or_experiment_indexing_incomplete"); await writeFile(path.join(directory, "findings.json"), `${JSON.stringify({ version: 1, findings, nativeIndexVerification: indexing, invoiceReconciliation: "pending", + accountingStatus: campaign.operations ? evalAccountingStatus(campaign.operations) : "unavailable", feedbackReviewMethod: "Only exact synthetic fixture bodies with all corroborating graders are automatically dismissed; all other feedback requires review" }, null, 2)}\n`, { mode: 0o600 }); for (const finding of findings) console.error(`::error title=Eval operations::${finding}`); return findings;