mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 02:44:50 +01:00
DottaandClaude Opus 4.8
76de2c16f2
Introduce src/eval: the two pieces the whole-stack runner eval slice was missing on top of the existing conformance corpus and real-Codex matrix. - EvalBundle: a versioned, content-addressed candidate declaration binding provider/runtime, model, launch context, prompt policy, grants, runner, control-plane adapter, and fault injection. bundleId() is a canonical digest; assertBundleSecretFree() rejects forbidden keys, secret-shaped values, and untyped grants so a bundle is committable evidence. - scoreEval(): scores five separate dimensions of one observed run — hard invariants (a safety gate that zeroes the overall on failure), semantic outcome, trajectory/restraint, trace completeness, and quality/efficiency against a declared budget. Pure and deterministic; observations carry only safe identifiers. Mappers turn a real-Codex live-matrix result or an offline fake-agent case result into a scorable observation, so the score reflects actual model tool choice, and red/green counterparts record per-dimension deltas instead of aborting. - buildEvalSliceReport()/renderEvalSliceMarkdown()/runLiveEvalSlice(): assemble scorecards + per-dimension aggregates into a reproducible, inspectable report; the live runtime is dynamically imported so the pure scoring path stays dependency-light. 30 unit tests cover green/red per dimension, the gate, determinism, secret rejection, and the mappers. Adds docs/capability-eval-slice.md and a test:eval-slice script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>