DottaandClaude Opus 4.8
ab7b85878a
Remove trailing generation-wrapper markup (</content>, plus </invoke> in the tutorial) that had leaked at EOF into eleven committed Phase 7 reference, tutorial, evidence, and OKF-journal files. Sync the stated suite counts in the current-state docs (tutorial, scenario-explorer reference, verification-commands reference, and OKF journal) to the verified results: test:phase7 49, test:browser:phase7 25, docs:validate 73 files / 28 OKF concepts / 5 indexes. Drop a brittle wall-clock timing from the journal. The dated 7F verification evidence and log.md milestone entry keep their point-in-time 45/19 figures on purpose. Add a wrapper-markup guard to docs:validate (via scripts/lib/doc-wrappers.mjs) that fails with the exact file and line when generator envelope tags reappear, plus a node:test regression suite wired into the package test script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.9 KiB
Phase 7 Eval-Derived Conformance
Phase 7E turns the Paperclip Evals corpus into an executable, offline
conformance suite. The suite derives its cases from the checked-in Phase 7
traceability derivative (spec/phase-07/eval-traceability.yaml) — it does
not clone or read an external eval repository, and it starts only the
in-process mock control plane. It hard-fails unless the derivative declares
schema version 2, exactly 106 rows, exactly 16 groups, and unique case IDs.
Sources: src/conformance/phase7-eval-suite.ts and its test
src/conformance/phase7-eval-suite.test.ts; the reporter
scripts/run-phase7-eval-suite.mjs.
The corpus
- 106 cases across 16 groups. Per-group counts: hb 5, co 6, st 8, cm 6, se 4, su 4, bl 5, dp 3, ix 9, ap 6, ar 4, er 9, rf 22, mh 4, rs 3, wk 8.
- Each case declares an actor role, a task mode, an input scenario, and the expected outcome, all bound to a row in the capability contract.
Assertion classes
Every case belongs to one assertion class, and each class checks a different kind of invariant:
agent_tool_contract— a semantic tool call produces the expected typed effect and state change.authorization_policy— a capability is exposed, denied, or unlocked by a grant exactly as its disposition requires.control_plane_invariant— a control-plane-owned action happens without any agent tool, and no tool can perform it.combined_multi_hop— a sequence of operations across turns produces the expected cumulative state and respects forbidden-operation rules.restraint_no_call— the correct behavior is to make no call; the case passes only if the agent deliberately does nothing further.
Fake-agent matrix and bounded Codex sample
The suite executes each case with a deterministic fake agent whose plan is
fixed by a fixture seed, so repeat runs are byte-identical. Optional-tool rows
are run twice — once with the unlocking grants (must be allowed and must mutate
state) and once ungranted (must be absent or denied with no state change);
control-plane-owned rows remain absent in every configuration; and
restraint_no_call rows must produce an empty state diff.
The fake-agent surface is 14 always-agent tools plus 4 optional tools unlocked
by four seed grants (discovery:tasks:read, discovery:agents:read,
delegation:tasks:create, governance:approvals:request) — 18 operations.
The suite binds that surface through both the fake-agent and Codex bindings and
asserts the two operation lists are byte-identical (18/18).
A bounded Codex binding sample picks one representative case from nine
groups (hb, dp, bl, ap, ar, ix, mh, rs, wk) and checks that
checkout_task is absent from the Codex surface while every other sampled
operation is present. It is an offline parity check; no real Codex or network is
contacted, and the browser explorer holds no credential.
Running it
# Run the 106-case suite in-process (one vitest file drives all cases).
pnpm --filter @paperclipai/paperclip-runner test:phase7-evals
# Build the public surface and write the parity report with per-group counts,
# assertion classes, the fake-agent matrix, the bounded Codex sample, and the
# semantic-operation execution counts.
pnpm --filter @paperclipai/paperclip-runner report:phase7-evals
The reporter writes knowledge/evidence/phase-07/eval-parity-report.{json,md}.
Each failure carries its case ID, assertion class, semantic operation,
authorization decision, and final state diff. The report is generated on demand
and is not committed; delete it before running docs:validate (it carries no
OKF frontmatter). See the
verification commands reference.