Files
PaperClipAI/packages/paperclip-runner/scripts/controlled-evals
DottaandPaperclip 35820cf398 fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929)
## Thinking Path

> - Paperclip must account for paid eval work.
> - The maintained Runner workflow distributes one immutable controller
manifest.
> - Its cost gate currently requires vendor proofs for every operating
charge.
> - Model-spend qualification must also work under an explicit policy
that keeps operating charges separate.
> - This PR applies the shared model-readiness policy to the hosted
controller.
> - It retains every operating hold and still rejects missing model
receipts and incomplete workers.

## Linked Issues or Issue Description

Refs #17928, #17889.

**What happened?**
Runner qualification requires full-lifecycle vendor operating proofs
even when the operator selects model-request readiness. That blocks the
model-token qualification path.

**Expected behavior**
An explicit `overhead.accountingReadiness: "model_requests"` policy
permits absent operating cost proofs. Supplied proofs still validate.
Complete final model receipts are required.

**Steps to reproduce**
Prepare a two-worker hosted manifest with the explicit policy and no
vendor cost proofs. Preserve missing-worker allocations, then complete
both workers and inspect the private archive summary. A second test
returns a provider refusal without terminal usage and requires failure
with the full hold.

## What Changed

- Apply the shared explicit policy to hosted capacity and observability
cost-proof admission.
- Preserve pricing windows, model bounds, immutable
source/request/runtime identities, trace quotas and every allocation.
- Reject zero or unresolved model requests in this policy, even if a
request has a provisional bound.
- Retain operating amounts and pending reconciliation through campaign
and private archive summaries.
- Document the separate model-readiness contract. Preserve strict
behavior when the field is absent.

## Verification

- Regression first failed at the original vendor-bound gate.
- All 80 controlled-eval Node checks pass, including source and copied
portable runtime checks, archived partial/missing workers, credential
exclusion and model receipt failure.
- Product readiness prerequisite #17928 has its own eval unit/typecheck
and full CI checks.
- No paid workflow, credential change or recurring activation is part of
this source PR.

## Risks

This opt-in policy does not prove a full-lifecycle operating-cost
ceiling. The report retains operating holds and pending reconciliation.
Default policies retain the existing vendor-proof gate. Source and
worker identity checks remain required. A model refusal without final
usage now fails readiness even if its reservation has a reviewed bound.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 15:41:07 -05:00
..