Files
PaperClipAI/packages/paperclip-runner/scripts/controlled-evals
DottaandPaperclip 6b9e5f3a80 fix(evals): enforce qualified ACPX profile admission before paid turns (#17964)
## Thinking Path

> - Paperclip coordinates agent work through Runner.
> - Controlled evals route model calls through a budget gateway.
> - The definitions pin each ACPX runtime identity.
> - A stale qualified profile currently reaches a paid turn before
scoring rejects it.
> - The Runner CLI already checks an exact expected profile before it
creates a provider.
> - This change forwards that existing admission argument through the
controller.
> - Profile drift can then stop before model spending.

## Linked Issues or Issue Description

Refs #17943 and #17929.

**What happened?**
The worker in run 40618022700 passed all four behavior checks and
retained three final request receipts. Its grader then rejected the old
Claude SDK pin. The controlled wrapper could not pass the existing
`--expected-acpx-profile` argument to the built CLI.

**What did you expect to happen?**
The built CLI must reject a stale expected profile before provider
creation.

**Steps to reproduce**
Run a controlled ACPX cell with a definition that pins a different SDK
from the built Runner. The old wrapper accepts only request and output
arguments. It cannot enforce the profile admission supplied by updated
definitions.

**Paperclip version**
The observed run used e4b30163bb.

**Deployment mode**
The controlled GitHub Actions Runner protocol workflow.

## What Changed

- Forward bounded expected-profile JSON through the controlled wrapper
to the same built CLI.
- Reject invalid objects, oversized arguments, and profiles on non-ACPX
requests before opening evidence.
- Exercise a real child boundary and the actual qualified Claude CLI
admission gate.
- Document the definition/controller compatibility requirement.

## Verification

- Controller suite: 77 tests passed. The new regressions fail against
the original wrapper.
- Hosted and portable controller suites: 8 tests passed.
- Actual eval-session CLI contract suite: 39 tests passed. A stale
qualified Claude SDK never creates a provider service.
- Runner package typecheck passed.
- The non-ACPX rejection guard also passed its focused regression after
review.
- No live provider request was made for this change.
- Full repository checks and current-head reviews run on this PR.

## Risks

- Updated definitions require this wrapper before they can pass the new
argument. Merge the controller first.
- Profile identity remains owned by the built Runner CLI. The controller
only forwards its existing argument.
- Existing callers that omit the argument retain their current behavior.
- Optional review suggestions to duplicate the expected-profile hash
into controller evidence and add a full built-CLI/controller integration
case are deferred. The pinned definition revision retains the expected
profile, and both controller entrypoints plus the actual CLI admission
boundary are covered. A compatibility error fails closed without a paid
fallback.

## Model Used

OpenAI Codex, GPT-6 family, with tool execution and extended reasoning.
The exact serving model ID and context window are not exposed in this
session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:59:20 -05:00
..