Add versioned semantic-tool and budget stop receipts with replay parity, conformance fixtures, and normalized adapter evidence. Score eval trace completeness from PRP receipts while preserving the existing live evidence fallback.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The live matrix injected `idempotencyKey` into every case's dictated JSON
input, but only mutating operations declare that key -- read operations close
their schema with `additionalProperties: false`. `search_tasks` therefore
rejected the input the prompt told the model to send.
Nothing caught it. A rejected call still emits a `tool_call` entry, so the
observed-vs-expected call comparison matched, and a rejected read leaves the
control plane `unchanged` exactly like a successful one, so the semantic
outcome matched by coincidence. The lone signal was an authorization mismatch,
and the hard-invariant gate only fires on escalation (denied -> allowed), never
on the reverse. se-get-issue-01 scored 1.0 on all five dimensions while its
call was failing.
Fixed on three levels:
- Root cause: derive the retry key from the catalog descriptor so the dictated
input always satisfies the operation's own schema.
- Harness gate: fail any live case whose typed result came back `ok: false`,
so a denial can never pass on matching calls and state alone.
- Scorer: a call the case declared allowed but that was denied now fails
`semantic_outcome` -- a rejected call cannot have produced its effect.
Trajectory stays green, since the model did choose the right tool.
Negative control: reintroducing the injection now aborts the matrix with
`deniedCalls: ["search_tasks"]` instead of reporting a silent pass.
Live matrix re-run 16/16 with zero authorization mismatches; the fault matrix
is unchanged (8 green / 8 red, mean 0.691), confirming the scorer change only
catches genuine denials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduce src/eval: the two pieces the whole-stack runner eval slice was
missing on top of the existing conformance corpus and real-Codex matrix.
- EvalBundle: a versioned, content-addressed candidate declaration binding
provider/runtime, model, launch context, prompt policy, grants, runner,
control-plane adapter, and fault injection. bundleId() is a canonical
digest; assertBundleSecretFree() rejects forbidden keys, secret-shaped
values, and untyped grants so a bundle is committable evidence.
- scoreEval(): scores five separate dimensions of one observed run — hard
invariants (a safety gate that zeroes the overall on failure), semantic
outcome, trajectory/restraint, trace completeness, and quality/efficiency
against a declared budget. Pure and deterministic; observations carry only
safe identifiers. Mappers turn a real-Codex live-matrix result or an
offline fake-agent case result into a scorable observation, so the score
reflects actual model tool choice, and red/green counterparts record
per-dimension deltas instead of aborting.
- buildEvalSliceReport()/renderEvalSliceMarkdown()/runLiveEvalSlice():
assemble scorecards + per-dimension aggregates into a reproducible,
inspectable report; the live runtime is dynamically imported so the pure
scoring path stays dependency-light.
30 unit tests cover green/red per dimension, the gate, determinism, secret
rejection, and the mappers. Adds docs/capability-eval-slice.md and a
test:eval-slice script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PAP-17039. Both catalog modules now derive their operation set, placement,
required claims, and task-mode policy from a single hand-authored source,
src/catalog/canonical-operations.ts; neither declares an independent op list.
- New canonical-operations.ts is the single 41-op source of truth. tools/ and
semantic-tools/ project it and supply only per-surface presentation (title,
description, schema, and the scenario-only mock mapping + redaction). A
bidirectional parity check fails the build on any drift.
- Unify generic_api_request to one claim, test:generic_api_request (test-only,
never product coverage); update scenario grants/plan/tests.
- One canonical task-mode policy: the live runtime policy unioned with
skill_test. Resolves the ledger's 15 task-mode divergences to zero. The live
contract golden is byte-identical (allowedModes is not serialized); the live
runtime allowedModes gained skill_test for the 9 standard-only ops so the
eval harness can drive them.
- reconciliation.ts inverts to derive from canonical-operations and recompute
the drift ledger: zero claim/mode divergences; the sole surviving divergence
is the reviewed intentional input-schema projection (idempotency in-band for
the provider, out-of-band for the observable runtime).
Focused catalog, scenario, conformance, issue-thread, mock, and package-export
suites pass; golden verified byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The catalog doc claimed 36 tools / 22 optional while the catalog holds 37 / 23
(the 36-vs-37 drift the reconciliation plan flagged). Correct the prose and add
src/catalog/catalog-docs.test.ts, which recomputes the total/always/optional
counts and the operation-id membership from the catalog so a stale hand count
can no longer pass.
Refs PAP-17025.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The committed matrix was captured while fontconfig resolved the sans stack
to the DejaVu Sans fallback; ambient user-font churn then made Inter (the
stack's first choice) resolve, drifting all 24 shots at once. Re-recorded
under Chromium 151.0.7922.34 with Inter active, taught the recorder to
refuse to record or compare when Inter does not resolve distinctly, and
wrote the recording environment into the generated evidence index.
PAP-17001
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both findings from the exact-build security review (PAP-16983) against
`b73524ec2ac5`.
Sensitive information disclosure: every projected view carried raw
`provider_event` records and unredacted tool results, so one ordinary turn
handed an admitted browser provider protocol traffic, the echoed prompt,
provider thread/turn/item identifiers, model and token metadata, and complete
tool payloads. Redaction now happens where evidence is recorded, not where it
is rendered — `redactPhase7EvidenceData` is the only way an entry enters a live
session, so a raw payload never reaches the snapshot, the store, a frame, or a
log. `toPhase7PublicThreadView` then publishes an allowlisted DTO on every
response path, aliases provider-authored identifiers consistently, and scrubs
withheld provider identity. A streamed failure answers with a code and fixed
operator copy instead of the underlying message.
Cross-session BOLA: the only capability was minted once per gateway process, so
possession of a session id authorized reads and mutations of it. Each surface
now mints a per-browser capability, stores only its SHA-256 with the session,
and compares it in constant time on every session-scoped route. A valid id
without its capability is answered 404, exactly like an id that never existed.
`New chat`, a scenario POST, and reset rotate the capability after revoking the
caller's prior bindings.
Also normalises the stream frame cap to encoded UTF-8 bytes rather than UTF-16
code units, which the review flagged as low-severity hardening.
Each new test was observed failing against the reinstated rejected behaviour
(19 failed, 18 pre-existing passed) before passing against the fix.
Refs PAP-16991.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Phase 7 issue-thread live region re-armed "A request is waiting for
your answer." on every snapshot while a request was pending. On the live
path a submit re-renders with the same request still pending (state
`submitting`), so the pending copy overwrote "Your answer was recorded."
and then stayed put once the request settled — clean-room QA (PAP-16974)
heard the stale guidance during later turns.
The pending copy is now announced once per request, on arrival, and the
settle path replaces it (and only it) with the request's resolved state
label, so a request that goes away server-side corrects itself too.
Covers pending -> answered -> later turn, plus a server-side withdrawal,
in the issue-thread browser suite: 60/60 pass, `check:phase7:ui` still
reproduces all 24 screenshots byte-for-byte.
Adds a second primary path beside the preset scenario explorer. `#/chat`
opens a blank thread on a freshly minted mock tenant — one company, one
agent, one blank issue — and drives real Codex through real runnerd. The
scenario explorer and its deterministic deep links are unchanged.
The clean room is live-only by construction. The route pins `mode=live`
and drops `shot`/`at`, the package server has no scripted path for it and
refuses any non-live projection, and a failure to start runnerd or Codex
is reported with a retry rather than replaced by a fixture.
`createPhase7CleanRoomSessionInput` seeds the minimum a runner session
needs and fails closed if anything else is present, so "starts blank" is
enforced at the seam rather than assumed. The exposure profile is fixed
and inspectable while the calls stay conversation-driven; three grants
(`governance:approvals:decide`, `workspace:control`,
`test:generic_api_request`) stay withheld so a typed denial is reachable
in a chat nobody scripted.
`Reset` and `New chat` retire the previous session's authority, drop its
workspace, and mint new mock identities; the old session id answers 404
afterwards. The projection now publishes the real-API block as a
`network-guard-<sessionId>` control-plane record, so "no request reached
a real Paperclip API" is inspectable in the drawer on both surfaces.
Also fixes two things the clean room surfaced: the unanchored `sk-`
credential heuristic matched mock ids of the form `task-cleanroom-<token>`,
and a hash change left one committed frame reporting `settled` for the
surface just navigated away from.
The 24-image screenshot matrix is re-recorded: the surface rail carrying
the clean-room entry is a real, always-visible element, and it reproduces
byte-for-byte against Chromium 151.0.7922.34.
Verified: test:phase7 159/159; test:browser:issue-thread 58/58 with axe
clean; check:phase7:ui 24/24 byte-identical; docs:validate 93 files, 34
OKF concepts; smoke:phase7:cleanroom all 15 assertions true against real
Codex 0.132.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the final Phase 7 build 865d1a72 for clean-room QA and 7L review:
add the live process topology to architecture, an execution-modes reference
that names Real Codex / Real runnerd / Mock Paperclip and the fake-vs-live
eligibility rules, a canonical clean-start issue-thread tutorial (offline fake
mode plus optional real Codex), the authoritative Revision 3/4 evidence
manifest, the screenshot-index eligibility note, and the OKF journal entry.
Mark the Revision 2 preview (build da0d32d74a and its URL) ineligible for
final acceptance. Offline commands re-run on this tree; docs:validate green
(89 files, 33 OKF concepts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Preserve the existing explorer while wiring the accepted mock, semantic, live, and issue-thread surfaces together. Keep fixture-only command capabilities separate from model-visible grants and split browser-safe exports from Node transports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Revisions 12-14 verified on the re-captured chat-streaming shots (byte-identical on re-record). Revision 15 accepts the post-capture composer disclaimer and isolated-mock wording from the demo-hardening commits and refreshes the five captures they staled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
UX review for PAP-16920: §11a revisions 1-9 ruled on (all accepted; the
stage=streaming staging depth is the exception), reviewer revisions 10-11
accept the three-segment mobile shell and the home-rail hint card, and
revisions 12-14 record the fixes shot 5 is gated on: hold the activity
stream at the transcript's reveal ceiling, leave a model card visibly
mid-reveal, and give disabled session controls a visible reason.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a chat surface to the Phase 7 browser explorer where the board sends
prompts to a scenario and sees exactly how the mock Paperclip control plane
was used on every turn — the point of the phase is that the activity stays
visible rather than hidden behind the conversation (PAP-16916).
Runtime owns the turn model, so the UI derives nothing:
- `Phase7ChatSession` keeps one long-lived mock core across turns, so the
state a turn changes is still there for the next one. Turn 0 is the
control-plane seed (checkout, wake routing, exposure) that exists before
anyone can type.
- Timeline entries gain a `turn`; the artifact gains `turns[]` with per-turn
exposure, authorization records, state diff, reconciliation events, counts,
and a turn-scoped verdict. A new `user_message` kind joins the five
existing ones.
- `phase7TurnParity` judges a turn on what that turn did. A session-level
expectation about work nobody has asked for yet would read as a failure of
turn 1 rather than as work still to come.
- Recording machinery moves to `scenario-execution.ts`, shared with the 7F
single-shot runner so a scenario reads the same either way.
Two scenarios carry authored multi-turn scripts: `ap-mcp-gate-01` (allowed
turn, then a denied one) and `ix-checkbox-01` (open an interaction, then the
board answers it — a control-plane-owned action with no agent tool, which is
what makes the control plane schedule the continuation wake). Every other
corpus scenario falls back to its 7F plan as one scripted turn.
The surface implements the approved 7I interaction map: turn strips bridging
each turn to its evidence, the six-section activity stream, the mobile
Chat/Activity segments with a deny count chip, reset and scenario-switch
confirmation, and deterministic deep links including a staged streaming hold.
Deviations taken during implementation are recorded in §11a of the map rather
than left for a reviewer to find.
No ACPX implementation, no real control-plane integration, no credential in
the browser: Codex mode stays disabled with a named reason and scripted mode
drives the same in-page mock core offline.
Tests: 30 new runtime/component tests and 18 browser tests; 79 phase-7 unit
and 43 phase-7 browser tests pass. Ten §11 screenshots recorded by
`record:phase7i`, which fails the run on any horizontal page scroll.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remove trailing generation-wrapper markup (</content>, plus </invoke> in the
tutorial) that had leaked at EOF into eleven committed Phase 7 reference,
tutorial, evidence, and OKF-journal files.
Sync the stated suite counts in the current-state docs (tutorial,
scenario-explorer reference, verification-commands reference, and OKF journal)
to the verified results: test:phase7 49, test:browser:phase7 25,
docs:validate 73 files / 28 OKF concepts / 5 indexes. Drop a brittle wall-clock
timing from the journal. The dated 7F verification evidence and log.md milestone
entry keep their point-in-time 45/19 figures on purpose.
Add a wrapper-markup guard to docs:validate (via scripts/lib/doc-wrappers.mjs)
that fails with the exact file and line when generator envelope tags reappear,
plus a node:test regression suite wired into the package test script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 7G (PAP-16906): package-local reference pages for capability
disposition, mock ControlPlanePort, semantic tools, authorization and
exposure, eval-derived conformance, browser explorer, the future Phase 8
binding boundary, and the focused verification commands; a clean-start
tutorial that runs from the repo root with no Paperclip service or external
eval repo; an OKF evidence manifest cross-linking every focused command to
its observed result and the security/UX/QA gate issues (PAP-16902, PAP-16905,
PAP-16907); and the Phase 7 OKF journal entry. Additive updates to the docs
index, architecture boundary, and knowledge indexes. Keeps ACPX as Phase 8.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revisions 8-12: continuation evidence split across surfaces, empty
forbidden-operations corpus, keyboard deltas accepted as interim,
inline redaction-rule chips, and expanded-disclosure capture policy.
Recorded at the PAP-16905 acceptance gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements the standalone, package-local Phase 7 scenario explorer against the
approved interaction map (docs/design/phase-7-scenario-explorer-ux.md).
Scenario runtime (src/phase7, no DOM, no network):
- Scenario index over all 106 eval cases and 16 groups, with actor role, task
mode, fixture seed, and catalog claims declared in a profile table rather
than inferred from case prose. Claims that cannot unlock an operation in a
scenario's task mode or for its actor role are dropped, so no scenario shows
a grant that unlocks nothing.
- Deterministic fixture seeds and fake-agent plans: twelve authored plans for
the UX acceptance scenarios, derived plans for the rest. Derived plans only
invoke operations the run actually exposes, so any denial in a derived
transcript is a real policy result, not a scripting artefact.
- A run-artifact builder that executes each plan against the Phase 7C mock
control plane through the Phase 7D authorization engine and records the
ordered timeline (both channels), tool exposure, authorization records, an
immutable ten-domain state diff, and a parity verdict. Descriptor redaction
rules are applied at the artifact boundary so no raw secret can reach a
rendered record.
- Parity is judged in the runtime, never in a component. Phase 7E's per-case
result is carried through verbatim in its own labelled block and renders
"Not run" when no report is bundled.
Explorer (examples/phase7-explorer):
- Read-only surface over the frozen SDK: hash routing, scenario picker with
faceted filters, agent and control-plane transcript channels, and Context,
Authorization, State diff, and Parity inspector tabs. Tokens only; the
0.1.2 export surface is unchanged and the runtime is reached through the
deliberately package-local @paperclip-runner-local/phase7 alias.
- Checked-in inputs arrive as declared virtual modules, so the example
consumer keeps its boundary and fake mode does no network I/O.
Verification: 45 unit tests (index, plans, all-scenario runs and parity,
components, routes), 19 Playwright tests (route determinism, evidence routes,
credential boundary, keyboard paths, responsive), and the 24-image screenshot
acceptance set.
Fixes found while building: SDK badges overflowed at 390px inside disclosure
summaries; SDK badge capitalisation broke the label vocabulary; hash-change
deep links left the wrong mobile segment; screenshot capture leaked run state
between shots. All recorded as revisions in the interaction map.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Interaction map, screenshot acceptance matrix, and annotated mockup for
the Phase 7 browser scenario explorer (PAP-16899). Binds picker facets,
transcript channels, inspector tabs, deterministic routes, and the
twelve-shot acceptance set to the 7A capability contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route corpus operations through cancellation, reconciliation, attention, and rollout entrypoints, materialize effects through owning services, and document the live-versus-policy proof boundary.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Exercise all atomic-liveness failpoints, make native cancellation retry-safe, and align the allowlist and verification evidence with the final database-backed matrix.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
