The live matrix injected `idempotencyKey` into every case's dictated JSON
input, but only mutating operations declare that key -- read operations close
their schema with `additionalProperties: false`. `search_tasks` therefore
rejected the input the prompt told the model to send.
Nothing caught it. A rejected call still emits a `tool_call` entry, so the
observed-vs-expected call comparison matched, and a rejected read leaves the
control plane `unchanged` exactly like a successful one, so the semantic
outcome matched by coincidence. The lone signal was an authorization mismatch,
and the hard-invariant gate only fires on escalation (denied -> allowed), never
on the reverse. se-get-issue-01 scored 1.0 on all five dimensions while its
call was failing.
Fixed on three levels:
- Root cause: derive the retry key from the catalog descriptor so the dictated
input always satisfies the operation's own schema.
- Harness gate: fail any live case whose typed result came back `ok: false`,
so a denial can never pass on matching calls and state alone.
- Scorer: a call the case declared allowed but that was denied now fails
`semantic_outcome` -- a rejected call cannot have produced its effect.
Trajectory stays green, since the model did choose the right tool.
Negative control: reintroducing the injection now aborts the matrix with
`deniedCalls: ["search_tasks"]` instead of reporting a silent pass.
Live matrix re-run 16/16 with zero authorization mismatches; the fault matrix
is unchanged (8 green / 8 red, mean 0.691), confirming the scorer change only
catches genuine denials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduce src/eval: the two pieces the whole-stack runner eval slice was
missing on top of the existing conformance corpus and real-Codex matrix.
- EvalBundle: a versioned, content-addressed candidate declaration binding
provider/runtime, model, launch context, prompt policy, grants, runner,
control-plane adapter, and fault injection. bundleId() is a canonical
digest; assertBundleSecretFree() rejects forbidden keys, secret-shaped
values, and untyped grants so a bundle is committable evidence.
- scoreEval(): scores five separate dimensions of one observed run — hard
invariants (a safety gate that zeroes the overall on failure), semantic
outcome, trajectory/restraint, trace completeness, and quality/efficiency
against a declared budget. Pure and deterministic; observations carry only
safe identifiers. Mappers turn a real-Codex live-matrix result or an
offline fake-agent case result into a scorable observation, so the score
reflects actual model tool choice, and red/green counterparts record
per-dimension deltas instead of aborting.
- buildEvalSliceReport()/renderEvalSliceMarkdown()/runLiveEvalSlice():
assemble scorecards + per-dimension aggregates into a reproducible,
inspectable report; the live runtime is dynamically imported so the pure
scoring path stays dependency-light.
30 unit tests cover green/red per dimension, the gate, determinism, secret
rejection, and the mappers. Adds docs/capability-eval-slice.md and a
test:eval-slice script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
