mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 18:31:34 +01:00
DottaandClaude Opus 5
00897dc7c1
The live matrix injected `idempotencyKey` into every case's dictated JSON input, but only mutating operations declare that key -- read operations close their schema with `additionalProperties: false`. `search_tasks` therefore rejected the input the prompt told the model to send. Nothing caught it. A rejected call still emits a `tool_call` entry, so the observed-vs-expected call comparison matched, and a rejected read leaves the control plane `unchanged` exactly like a successful one, so the semantic outcome matched by coincidence. The lone signal was an authorization mismatch, and the hard-invariant gate only fires on escalation (denied -> allowed), never on the reverse. se-get-issue-01 scored 1.0 on all five dimensions while its call was failing. Fixed on three levels: - Root cause: derive the retry key from the catalog descriptor so the dictated input always satisfies the operation's own schema. - Harness gate: fail any live case whose typed result came back `ok: false`, so a denial can never pass on matching calls and state alone. - Scorer: a call the case declared allowed but that was denied now fails `semantic_outcome` -- a rejected call cannot have produced its effect. Trajectory stays green, since the model did choose the right tool. Negative control: reintroducing the injection now aborts the matrix with `deniedCalls: ["search_tasks"]` instead of reporting a silent pass. Live matrix re-run 16/16 with zero authorization mismatches; the fault matrix is unchanged (8 green / 8 red, mean 0.691), confirming the scorer change only catches genuine denials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>