fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929)

## Thinking Path

> - Paperclip must account for paid eval work.
> - The maintained Runner workflow distributes one immutable controller
manifest.
> - Its cost gate currently requires vendor proofs for every operating
charge.
> - Model-spend qualification must also work under an explicit policy
that keeps operating charges separate.
> - This PR applies the shared model-readiness policy to the hosted
controller.
> - It retains every operating hold and still rejects missing model
receipts and incomplete workers.

## Linked Issues or Issue Description

Refs #17928, #17889.

**What happened?**
Runner qualification requires full-lifecycle vendor operating proofs
even when the operator selects model-request readiness. That blocks the
model-token qualification path.

**Expected behavior**
An explicit `overhead.accountingReadiness: "model_requests"` policy
permits absent operating cost proofs. Supplied proofs still validate.
Complete final model receipts are required.

**Steps to reproduce**
Prepare a two-worker hosted manifest with the explicit policy and no
vendor cost proofs. Preserve missing-worker allocations, then complete
both workers and inspect the private archive summary. A second test
returns a provider refusal without terminal usage and requires failure
with the full hold.

## What Changed

- Apply the shared explicit policy to hosted capacity and observability
cost-proof admission.
- Preserve pricing windows, model bounds, immutable
source/request/runtime identities, trace quotas and every allocation.
- Reject zero or unresolved model requests in this policy, even if a
request has a provisional bound.
- Retain operating amounts and pending reconciliation through campaign
and private archive summaries.
- Document the separate model-readiness contract. Preserve strict
behavior when the field is absent.

## Verification

- Regression first failed at the original vendor-bound gate.
- All 80 controlled-eval Node checks pass, including source and copied
portable runtime checks, archived partial/missing workers, credential
exclusion and model receipt failure.
- Product readiness prerequisite #17928 has its own eval unit/typecheck
and full CI checks.
- No paid workflow, credential change or recurring activation is part of
this source PR.

## Risks

This opt-in policy does not prove a full-lifecycle operating-cost
ceiling. The report retains operating holds and pending reconciliation.
Default policies retain the existing vendor-proof gate. Source and
worker identity checks remain required. A model refusal without final
usage now fails readiness even if its reservation has a reviewed bound.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/paperclip-runner/docs/runner-protocol-live-evals.md

@@ -930,3 +930,25 @@ portable runtime isolation, private archive replay, and complete/partial/empty
 campaign accounting. They do not establish current vendor prices, future
 retention charges, live Braintrust attribution or invoice settlement. Keep
 paid recurrence disabled until these independent gates are qualified.
+
+
+### Token-spend qualification with separate operating reconciliation
+
+A policy may explicitly set `overhead.accountingReadiness: "model_requests"`
+when model-token qualification is the operator's priority. In that mode an
+absent campaign capacity proof and absent observability cost proof do not block
+admission. Any supplied proofs must still validate. Current allowance/pricing
+windows, trace quotas, immutable source/request/runtime bindings, complete
+selection, per-cell allocations, request limits, no retries and the shared
+campaign admission cap remain required. Model reservation proofs remain required.
+
+The final report retains the once-only operating allowance and labels operating
+reconciliation pending. Missing, failed or canceled model evidence still holds
+the full cell allocation and fails completion. This policy does not certify a
+vendor operating-cost bound. Omit the field to retain full-lifecycle proof
+requirements. Existing manifests and held amounts are not rewritten.
+
+For this policy version, `operatingReconciliation` is always `pending`; the
+immutable controller summary has no operating invoice settlement step. Mixed
+histories retain that warning when any campaign uses the policy. A shared
+`accountingReadiness` label appears only when every campaign uses it.

packages/paperclip-runner/scripts/controlled-evals/archive.mjs

@@ -170,6 +170,9 @@ export async function summarizeControlledEvidence(bundles, options) {
 function history(campaigns) {
   const sum = field => usd(campaigns.reduce((total, campaign) => total + units(campaign[field]), 0n));
   return { schema: 'paperclip.runner-eval.control-history/v1', costStatus: campaigns.some(c => c.costStatus === 'incomplete') ? 'incomplete' : 'provisional',
+    ...(campaigns.length && campaigns.every(c => c.accountingReadiness === 'model_requests')
+      ? { accountingReadiness: 'model_requests' } : {}),
+    ...(campaigns.some(c => c.operatingReconciliation === 'pending') ? { operatingReconciliation: 'pending' } : {}),
     knownUsd: sum('knownUsd'), heldUsd: sum('heldUsd'), committedUsd: sum('committedUsd'), operatingHeldUsd: sum('operatingHeldUsd'),
     incompleteAttempts: campaigns.reduce((total, c) => total + c.incompleteAttempts, 0), overBudget: campaigns.some(c => c.overBudget), campaigns };
 }

packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs

@@ -787,3 +787,42 @@ test('retained failure codes distinguish native startup, child exit and invalid
   const artifact = await fixture(t, { missingRoute: true, requests: 0 });
   assert.equal((await artifact.run()).report.failure, 'artifact_invalid');
 });
+
+
+test('model readiness rejects a missing terminal model receipt even when its reservation has a reviewed bound', async t => {
+  const f = await fixture(t, { requests: 1 });
+  f.manifest.overhead.accountingReadiness = 'model_requests';
+  for (const component of f.manifest.overhead.components) delete component.provisionalBound;
+  await f.save();
+  const result = await f.run(0, async () => Response.json({ error: 'synthetic refusal' }, { status: 429 }));
+  assert.equal(result.report.failure, 'request_accounting_unbounded');
+  const summary = await summarizeControlledCampaign(f.manifest);
+  assert.equal(summary.incompleteAttempts, 1);
+  assert.equal(summary.heldUsd, '0.014000000');
+  assert.equal(summary.operatingReconciliation, 'pending');
+});
+
+
+test('model readiness rejects zero model requests and keeps the full allocation', async t => {
+  const f = await fixture(t, { requests: 0 });
+  f.manifest.overhead.accountingReadiness = 'model_requests';
+  for (const component of f.manifest.overhead.components) delete component.provisionalBound;
+  await f.save();
+  const result = await f.run();
+  assert.equal(result.report.failure, 'request_accounting_unbounded');
+  assert.equal(result.report.accounting.requests.length, 0);
+  const summary = await summarizeControlledCampaign(f.manifest);
+  assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.heldUsd, '0.014000000');
+});
+
+test('mixed-policy private histories keep the pending operating reconciliation warning', async t => {
+  const model = await fixture(t, { requests: 1 }), strict = await fixture(t, { requests: 1 });
+  model.manifest.campaignId = 'model-readiness-history';
+  model.manifest.overhead.accountingReadiness = 'model_requests';
+  for (const component of model.manifest.overhead.components) delete component.provisionalBound;
+  await model.save(); await model.run(); await strict.run();
+  const bundles = await Promise.all([model, strict].map(f => packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: f.environment })));
+  const history = await summarizeControlledEvidence(bundles);
+  assert.equal(history.incompleteAttempts, 0); assert.equal(history.operatingHeldUsd, '0.008000000');
+  assert.equal(history.accountingReadiness, undefined); assert.equal(history.operatingReconciliation, 'pending');
+});

packages/paperclip-runner/scripts/controlled-evals/hosted.mjs

@@ -23,7 +23,9 @@ export function prepareHostedManifest({ catalog, catalogSha256, prepared, policy
       || !Number.isSafeInteger(policy.maxCells) || policy.maxCells < 1 || policy.maxCells > 512
       || !Array.isArray(policy.allocations) || !policy.allocations.length) throw new Error('A bounded hosted policy with tracing is required');
   if (!/^[a-f0-9]{64}$/.test(policy.controllerLockSha256 ?? '')) throw new Error('Invalid hosted controller lock binding');
-  costRuntime.reservationBound('operating_allowance', policy.capacityBound, hostedPolicyConfiguration(policy));
+  if (policy.overhead?.accountingReadiness !== 'model_requests' || policy.capacityBound !== undefined) {
+    costRuntime.reservationBound('operating_allowance', policy.capacityBound, hostedPolicyConfiguration(policy));
+  }
   fields(policy.observability, ['projectId', 'capture', 'nativeLimits']);
   const runId = environment.GITHUB_RUN_ID, paths = hostedPaths(runId);
   if (prepared.schema !== 'paperclip.runner-eval.prepared-requests/v1' || prepared.githubRunId !== runId || prepared.githubRunAttempt !== 1
@@ -65,7 +67,7 @@ export function prepareHostedManifest({ catalog, catalogSha256, prepared, policy
     models: policy.models, overhead: policy.overhead, observability: { ...policy.observability },
     sourceRevision: prepared.sourceRevision, definitionsRevision: prepared.definitionsRevision, ...artifacts,
     outputDirectory: paths.evidence, hosted: { repository: environment.GITHUB_REPOSITORY, runId, runAttempt: 1, workflowRevision: environment.GITHUB_SHA, controllerLockSha256: policy.controllerLockSha256, policySha256: sha256(canonical(policy)) }, attempts };
-  manifest.observability.provisionalBound = { ...policy.capacityBound, configurationSha256: sha256(canonical(observabilityBoundConfiguration(manifest))) };
+  if (policy.capacityBound !== undefined) manifest.observability.provisionalBound = { ...policy.capacityBound, configurationSha256: sha256(canonical(observabilityBoundConfiguration(manifest))) };
   const validated = validateManifest(manifest, now);
   assertHostedAdmission(manifest, attempts[0], { ...environment, PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: attempts[0].executionId });
   return validated;

packages/paperclip-runner/scripts/controlled-evals/hosted.test.mjs

@@ -119,3 +119,52 @@ test('hosted CLI reports safe admission reasons without echoing private paths or
   assert.match(result.stderr, /A required hosted input file is missing/);
   assert.doesNotMatch(result.stderr + result.stdout, /SENSITIVE_SENTINEL/);
 });
+
+
+test('model-readiness policy retains operating holds without requiring vendor cost proofs', async t => {
+  const c = await campaign(t);
+  c.input.policy.overhead.accountingReadiness = 'model_requests';
+  for (const component of c.input.policy.overhead.components) delete component.provisionalBound;
+  delete c.input.policy.capacityBound;
+  const result = c.build();
+  assert.equal(result.manifest.overhead.accountingReadiness, 'model_requests');
+  assert.equal(result.manifest.observability.provisionalBound, undefined);
+  Object.assign(c.f.manifest, result.manifest); await c.f.save();
+  const missing = await summarizeControlledManifest(result.manifest, []);
+  assert.equal(missing.incompleteAttempts, 2);
+  assert.equal(missing.operatingHeldUsd, '0.004000000');
+  Object.assign(c.f.environment, c.environment, { PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: 'execution-0' });
+  assert.equal((await c.f.run(0, () => success())).code, 0);
+  const first = await packControlledAttempt({ manifest: result.manifest, attemptId: 'attempt-0', environment: c.f.environment });
+  Object.assign(c.f.environment, { PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: 'execution-1' });
+  assert.equal((await c.f.run(1, () => success())).code, 0);
+  const second = await packControlledAttempt({ manifest: result.manifest, attemptId: 'attempt-1', environment: c.f.environment });
+  const complete = await summarizeControlledManifest(result.manifest, [first, second]);
+  assert.equal(complete.incompleteAttempts, 0);
+  assert.equal(complete.operatingHeldUsd, '0.004000000');
+  assert.equal(complete.accountingReadiness, 'model_requests');
+  assert.equal(complete.operatingReconciliation, 'pending');
+  delete c.input.policy.overhead.accountingReadiness;
+  assert.throws(() => c.build(), /cost bound|operating bound/);
+});
+
+
+test('model readiness still validates supplied operating and capacity proofs', async t => {
+  const c = await campaign(t), policy = c.input.policy;
+  policy.overhead.accountingReadiness = 'model_requests';
+  for (const component of policy.overhead.components) {
+    component.provisionalBound.configurationSha256 = sha256(canonical({ accountingReadiness: 'model_requests',
+      reference: policy.overhead.reference, runner: policy.overhead.runner, verifiedAt: policy.overhead.verifiedAt,
+      expiresAt: policy.overhead.expiresAt, component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference } }));
+  }
+  policy.capacityBound.configurationSha256 = sha256(canonical(hostedPolicyConfiguration(policy)));
+  assert.equal(c.build().manifest.overhead.accountingReadiness, 'model_requests');
+  const mismatch = structuredClone(c.input); mismatch.policy.capacityBound.evidenceSha256 = '0'.repeat(64);
+  assert.throws(() => prepareHostedManifest(mismatch), /Observability capacity requires its operating bound evidence/);
+  const stale = structuredClone(c.input); stale.policy.capacityBound.configurationSha256 = '0'.repeat(64);
+  assert.throws(() => prepareHostedManifest(stale), /Reviewed cost bound configuration changed/);
+  const absent = structuredClone(c.input);
+  for (const component of absent.policy.overhead.components) delete component.provisionalBound;
+  absent.policy.capacityBound.configurationSha256 = sha256(canonical(hostedPolicyConfiguration(absent.policy)));
+  assert.throws(() => prepareHostedManifest(absent), /Observability capacity requires its operating bound evidence/);
+});

packages/paperclip-runner/scripts/controlled-evals/manifest.mjs

@@ -49,7 +49,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } =
   validateHostedIdentity(value);
   const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd);
   if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline
-      || value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds');
+      || value.overhead.accountingReadiness !== 'model_requests' && value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds');
   if (!Array.isArray(value.models) || !value.models.length || value.models.length > 16
       || !Array.isArray(value.attempts) || !value.attempts.length || value.attempts.length > 1000) throw new Error('Invalid controlled campaign selection');
   const prices = new Map();
@@ -141,7 +141,13 @@ function validateObservability(manifest) {
       || Object.keys(policy.capture).length !== 4 || Object.values(policy.capture).some(value => typeof value !== 'boolean')
       || canonical(Object.keys(policy.nativeLimits ?? {}).sort()) !== canonical(Object.keys(costRuntime.EVAL_OTLP_LIMITS).sort())
       || Object.entries(costRuntime.EVAL_OTLP_LIMITS).some(([key, max]) => !Number.isSafeInteger(policy.nativeLimits[key]) || policy.nativeLimits[key] < 1 || policy.nativeLimits[key] > max)) throw new Error('Invalid controller observability allocation');
+  if (manifest.overhead.accountingReadiness === 'model_requests' && policy.provisionalBound === undefined) return;
   const operating = manifest.overhead.components.find(item => item.component === 'observability');
   if (!operating || policy.provisionalBound?.evidenceSha256 !== operating.provisionalBound?.evidenceSha256) throw new Error('Observability capacity requires its operating bound evidence');
   costRuntime.reservationBound('operating_allowance', policy.provisionalBound, observabilityBoundConfiguration(manifest));
 }
+
+
+export function modelReadinessDefect(manifest, snapshot) {
+  return manifest.overhead.accountingReadiness === 'model_requests' && (!snapshot.requests.length || snapshot.unknownRequests > 0);
+}

packages/paperclip-runner/scripts/controlled-evals/portable.test.mjs

@@ -92,7 +92,7 @@ test('copied controller runs without source dependencies, retains native evidenc
   const hosted = await exec(process.execPath, ['--test', '--test-reporter=tap', join(source, 'hosted.test.mjs')], {
     cwd: root, env: { ...environment, PAPERCLIP_CONTROLLER_TEST_RUNTIME: join(directory, 'controlled-evals') }, timeout: 30000, maxBuffer: 1_048_576,
   });
-  assert.match(hosted.stdout, /# pass 4\b/);
+  assert.match(hosted.stdout, /# pass 6\b/);
 
 
   const cli = await exec(process.execPath, [join(directory, 'controlled-evals/archive.mjs'), 'report-attempts', join(root, 'empty-attempts')], {

packages/paperclip-runner/scripts/controlled-evals/report.mjs

@@ -1,6 +1,6 @@
 import { isEntrypoint } from './entrypoint.mjs';
 import { join } from 'node:path';
-import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
+import { modelReadinessDefect, canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
 import { inspectControlledObservability } from './observability.mjs';
 import { attemptDirectory } from './run.mjs';
 import { limitFor, readRegularFile } from './evidence-files.mjs';
@@ -79,7 +79,8 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {}
             campaignId: manifest.campaignId, executionId: allocation.executionId, attemptId: allocation.attemptId, descriptorSha256: sha256(descriptorBytes) })
           || artifact.attemptId !== allocation.attemptId || artifact.requestedModel !== allocation.model
           || snapshot.inflight || snapshot.conflictingRequests || snapshot.overBudget
-          || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) throw new Error('Incomplete controller evidence');
+          || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests
+          || modelReadinessDefect(manifest, snapshot)) throw new Error('Incomplete controller evidence');
       verified = true;
     } catch {}
     const cost = snapshot ? units(snapshot.knownUsd) : 0n;
@@ -92,6 +93,7 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {}
       unknownRequests: snapshot?.unknownRequests ?? null, boundedUnresolvedRequests: snapshot?.boundedUnresolvedRequests ?? null });
   }
   return { schema: 'paperclip.runner-eval.control-campaign/v1', campaignId: manifest.campaignId, manifestSha256,
+    ...(manifest.overhead.accountingReadiness ? { accountingReadiness: manifest.overhead.accountingReadiness, operatingReconciliation: 'pending' } : {}),
     limitUsd: manifest.maxUsd, allocatedUsd, operatingHeldUsd: overheadUsd, knownUsd: usd(known), heldUsd: usd(committed - known),
     committedUsd: usd(committed), overBudget: committed > units(manifest.maxUsd), incompleteAttempts: attempts.filter(attempt => attempt.status === 'incomplete').length,
     costStatus: attempts.some(attempt => attempt.status === 'incomplete') || committed > units(manifest.maxUsd) ? 'incomplete' : 'provisional', attempts };

packages/paperclip-runner/scripts/controlled-evals/run.mjs

@@ -1,3 +1,4 @@
+import { modelReadinessDefect } from './manifest.mjs';
 import { isEntrypoint } from './entrypoint.mjs';
 import { spawn } from 'node:child_process';
 import { once } from 'node:events';
@@ -185,7 +186,8 @@ registerHooks({ resolve(specifier, context, nextResolve) {
       catch { cleanupVerified = false; failure ??= 'executable_cleanup_unverified'; }
     }
     const snapshot = ledger.snapshot();
-    if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded';
+    if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests
+        || modelReadinessDefect(manifest, snapshot)) failure ??= 'request_accounting_unbounded';
     const receiptHashes = {};
     try {
       for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readRegularFile(join(root, 'receipts', `${request.requestId}.json`), limitFor(`receipts/${request.requestId}.json`)));