Add versioned semantic-tool and budget stop receipts with replay parity, conformance fixtures, and normalized adapter evidence. Score eval trace completeness from PRP receipts while preserving the existing live evidence fallback.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The live matrix injected `idempotencyKey` into every case's dictated JSON
input, but only mutating operations declare that key -- read operations close
their schema with `additionalProperties: false`. `search_tasks` therefore
rejected the input the prompt told the model to send.
Nothing caught it. A rejected call still emits a `tool_call` entry, so the
observed-vs-expected call comparison matched, and a rejected read leaves the
control plane `unchanged` exactly like a successful one, so the semantic
outcome matched by coincidence. The lone signal was an authorization mismatch,
and the hard-invariant gate only fires on escalation (denied -> allowed), never
on the reverse. se-get-issue-01 scored 1.0 on all five dimensions while its
call was failing.
Fixed on three levels:
- Root cause: derive the retry key from the catalog descriptor so the dictated
input always satisfies the operation's own schema.
- Harness gate: fail any live case whose typed result came back `ok: false`,
so a denial can never pass on matching calls and state alone.
- Scorer: a call the case declared allowed but that was denied now fails
`semantic_outcome` -- a rejected call cannot have produced its effect.
Trajectory stays green, since the model did choose the right tool.
Negative control: reintroducing the injection now aborts the matrix with
`deniedCalls: ["search_tasks"]` instead of reporting a silent pass.
Live matrix re-run 16/16 with zero authorization mismatches; the fault matrix
is unchanged (8 green / 8 red, mean 0.691), confirming the scorer change only
catches genuine denials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduce src/eval: the two pieces the whole-stack runner eval slice was
missing on top of the existing conformance corpus and real-Codex matrix.
- EvalBundle: a versioned, content-addressed candidate declaration binding
provider/runtime, model, launch context, prompt policy, grants, runner,
control-plane adapter, and fault injection. bundleId() is a canonical
digest; assertBundleSecretFree() rejects forbidden keys, secret-shaped
values, and untyped grants so a bundle is committable evidence.
- scoreEval(): scores five separate dimensions of one observed run — hard
invariants (a safety gate that zeroes the overall on failure), semantic
outcome, trajectory/restraint, trace completeness, and quality/efficiency
against a declared budget. Pure and deterministic; observations carry only
safe identifiers. Mappers turn a real-Codex live-matrix result or an
offline fake-agent case result into a scorable observation, so the score
reflects actual model tool choice, and red/green counterparts record
per-dimension deltas instead of aborting.
- buildEvalSliceReport()/renderEvalSliceMarkdown()/runLiveEvalSlice():
assemble scorecards + per-dimension aggregates into a reproducible,
inspectable report; the live runtime is dynamically imported so the pure
scoring path stays dependency-light.
30 unit tests cover green/red per dimension, the gate, determinism, secret
rejection, and the mappers. Adds docs/capability-eval-slice.md and a
test:eval-slice script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish the PRP v1 expressiveness audit and its crosswalk as a
self-validating artifact. Adds the JSON Schema the crosswalk already
referenced (previously dangling) encoding the audit invariants: control-
plane-local and breaking-v2 facts carry no wire target, additive v1
changes must name a correction plus JSON fixtures, and no fact may be
justified by a permissive payload field. The Vitest gate now validates
the crosswalk against that schema in addition to the structural checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PAP-17039. Both catalog modules now derive their operation set, placement,
required claims, and task-mode policy from a single hand-authored source,
src/catalog/canonical-operations.ts; neither declares an independent op list.
- New canonical-operations.ts is the single 41-op source of truth. tools/ and
semantic-tools/ project it and supply only per-surface presentation (title,
description, schema, and the scenario-only mock mapping + redaction). A
bidirectional parity check fails the build on any drift.
- Unify generic_api_request to one claim, test:generic_api_request (test-only,
never product coverage); update scenario grants/plan/tests.
- One canonical task-mode policy: the live runtime policy unioned with
skill_test. Resolves the ledger's 15 task-mode divergences to zero. The live
contract golden is byte-identical (allowedModes is not serialized); the live
runtime allowedModes gained skill_test for the 9 standard-only ops so the
eval harness can drive them.
- reconciliation.ts inverts to derive from canonical-operations and recompute
the drift ledger: zero claim/mode divergences; the sole surviving divergence
is the reviewed intentional input-schema projection (idempotency in-band for
the provider, out-of-band for the observable runtime).
Focused catalog, scenario, conformance, issue-thread, mock, and package-export
suites pass; golden verified byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add spec/capability/catalog-reconciliation.md: the canonical authority decision,
the 37/28/24/13/4/41 inventory, real-binding-status classification, the divergence
ledger with dispositions, legacy MCP alias handling, and the deliberate decisions
the schema-migration follow-on and the surface-ledger/eval tasks must execute.
Also fix a fabricated legacy-alias id (paperclipAddComment) and drift-check that
every alias named by the authority is a real inventory row.
Refs PAP-17025.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The catalog doc claimed 36 tools / 22 optional while the catalog holds 37 / 23
(the 36-vs-37 drift the reconciliation plan flagged). Correct the prose and add
src/catalog/catalog-docs.test.ts, which recomputes the total/always/optional
counts and the operation-id membership from the catalog so a stale hand count
can no longer pass.
Refs PAP-17025.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce src/catalog/ as the single machine-readable authority over the
union (41) of the two hand-maintained semantic catalogs — src/tools/ (37,
scenario/eval rich descriptor) and src/semantic-tools/ (28, live runtime
engine). It names every operation's placement, claims, task modes, roles,
side-effect class, idempotency, redaction, mock mapping, real-binding status,
and PRP-evidence shape, and computes the full op-set + metadata divergence
ledger (one claim, 15 task-mode, and the input-schema-shape differences) each
with a recorded disposition. The drift test pins the reconciled relationship
so a new operation or an unreviewed metadata divergence fails the gate.
Refs PAP-17025.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`src/index.ts` re-exported `./standalone/phase6-demo.js`, but that module only
ever existed as an untracked file in the shared worktree. The commit therefore
built there and failed `build:typescript` with TS2307 in any clean checkout,
which blocked the Phase 7U deployment. Nothing tracked consumes the module, so
removing the export restores a buildable tree without touching the accepted
remediation semantics — the issue-thread bundle stays byte-identical at
index-Df3v8-4P.js, SHA-256 2e37dcb74d544ed103fdf6f22554de9e8747ab11067f7b3e8fb6c5811e38a528.
Guard the class of defect rather than the instance. `check:tracked-imports`
rejects any tracked source whose relative import resolves only against an
untracked file, and any `package.json` script whose entry point is untracked.
Ignored build output is excused via `git check-ignore`.
The same sweep found the second half of the trap: the repository-root
`.gitignore` swallows `packages/paperclip-runner/scripts/check-*.mjs` because
its `!scripts/check-*.mjs` negation is anchored to the root. Existing checks
survived only through `git add -f`. This package's `.gitignore` now re-includes
them so new checks cannot be silently dropped from a commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The committed matrix was captured while fontconfig resolved the sans stack
to the DejaVu Sans fallback; ambient user-font churn then made Inter (the
stack's first choice) resolve, drifting all 24 shots at once. Re-recorded
under Chromium 151.0.7922.34 with Inter active, taught the recorder to
refuse to record or compare when Inter does not resolve distinctly, and
wrote the recording environment into the generated evidence index.
PAP-17001
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both findings from the exact-build security review (PAP-16983) against
`b73524ec2ac5`.
Sensitive information disclosure: every projected view carried raw
`provider_event` records and unredacted tool results, so one ordinary turn
handed an admitted browser provider protocol traffic, the echoed prompt,
provider thread/turn/item identifiers, model and token metadata, and complete
tool payloads. Redaction now happens where evidence is recorded, not where it
is rendered — `redactPhase7EvidenceData` is the only way an entry enters a live
session, so a raw payload never reaches the snapshot, the store, a frame, or a
log. `toPhase7PublicThreadView` then publishes an allowlisted DTO on every
response path, aliases provider-authored identifiers consistently, and scrubs
withheld provider identity. A streamed failure answers with a code and fixed
operator copy instead of the underlying message.
Cross-session BOLA: the only capability was minted once per gateway process, so
possession of a session id authorized reads and mutations of it. Each surface
now mints a per-browser capability, stores only its SHA-256 with the session,
and compares it in constant time on every session-scoped route. A valid id
without its capability is answered 404, exactly like an id that never existed.
`New chat`, a scenario POST, and reset rotate the capability after revoking the
caller's prior bindings.
Also normalises the stream frame cap to encoded UTF-8 bytes rather than UTF-16
code units, which the review flagged as low-severity hardening.
Each new test was observed failing against the reinstated rejected behaviour
(19 failed, 18 pre-existing passed) before passing against the fix.
Refs PAP-16991.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The clean-room chat revealed a finished reply instead of writing one: the
live session accumulated agent deltas in its turn waiter, the package
session server awaited the whole turn and answered with one JSON body, and
the browser waited on `response.json()`. Nothing between the provider and
the page could show work in progress, so the board saw the answer appear
all at once.
A turn now streams:
- `Phase7LiveSession` publishes bounded, ordered live turn events and
mirrors the waiter's assembled text into the transcript entry the reply
will finally occupy, so one rendered card grows and the settled text is
still assembled in exactly one place.
- `src/phase7/turn-stream.ts` defines the NDJSON frame codec and reader.
The decoder refuses a replayed, reordered, oversized, late, or truncated
frame rather than applying a stale view.
- `POST /api/phase7/ui/message` flushes headers, then writes one frame per
event-loop tick of provider activity. Admission checks still answer with
a JSON body and their own status code, because they are decided before
the first frame. Client disconnect interrupts the provider turn.
- The browser client consumes frames and renders each projection in place;
the settled payload is applied last and remains the authority. Stop,
reset, and `New chat` abandon the stream, and frames from an abandoned
turn generation are dropped.
Measured against real Codex through real runnerd: headers at 265 ms, 62
growing assistant states from the first character to a 301-character reply
at 3.1 s, the last interim state identical to the settled one, and a single
assistant card in the settled thread.
The scenario explorer and every fixture, preset, and replay route are
unchanged and stay deterministic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Phase 7 issue-thread live region re-armed "A request is waiting for
your answer." on every snapshot while a request was pending. On the live
path a submit re-renders with the same request still pending (state
`submitting`), so the pending copy overwrote "Your answer was recorded."
and then stayed put once the request settled — clean-room QA (PAP-16974)
heard the stale guidance during later turns.
The pending copy is now announced once per request, on arrival, and the
settle path replaces it (and only it) with the request's resolved state
label, so a request that goes away server-side corrects itself too.
Covers pending -> answered -> later turn, plus a server-side withdrawal,
in the issue-thread browser suite: 60/60 pass, `check:phase7:ui` still
reproduces all 24 screenshots byte-for-byte.
Adds a second primary path beside the preset scenario explorer. `#/chat`
opens a blank thread on a freshly minted mock tenant — one company, one
agent, one blank issue — and drives real Codex through real runnerd. The
scenario explorer and its deterministic deep links are unchanged.
The clean room is live-only by construction. The route pins `mode=live`
and drops `shot`/`at`, the package server has no scripted path for it and
refuses any non-live projection, and a failure to start runnerd or Codex
is reported with a retry rather than replaced by a fixture.
`createPhase7CleanRoomSessionInput` seeds the minimum a runner session
needs and fails closed if anything else is present, so "starts blank" is
enforced at the seam rather than assumed. The exposure profile is fixed
and inspectable while the calls stay conversation-driven; three grants
(`governance:approvals:decide`, `workspace:control`,
`test:generic_api_request`) stay withheld so a typed denial is reachable
in a chat nobody scripted.
`Reset` and `New chat` retire the previous session's authority, drop its
workspace, and mint new mock identities; the old session id answers 404
afterwards. The projection now publishes the real-API block as a
`network-guard-<sessionId>` control-plane record, so "no request reached
a real Paperclip API" is inspectable in the drawer on both surfaces.
Also fixes two things the clean room surfaced: the unanchored `sk-`
credential heuristic matched mock ids of the form `task-cleanroom-<token>`,
and a hash change left one committed frame reporting `settled` for the
surface just navigated away from.
The 24-image screenshot matrix is re-recorded: the surface rail carrying
the clean-room entry is a real, always-visible element, and it reproduces
byte-for-byte against Chromium 151.0.7922.34.
Verified: test:phase7 159/159; test:browser:issue-thread 58/58 with axe
clean; check:phase7:ui 24/24 byte-identical; docs:validate 93 files, 34
OKF concepts; smoke:phase7:cleanroom all 15 assertions true against real
Codex 0.132.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the final Phase 7 build 865d1a72 for clean-room QA and 7L review:
add the live process topology to architecture, an execution-modes reference
that names Real Codex / Real runnerd / Mock Paperclip and the fake-vs-live
eligibility rules, a canonical clean-start issue-thread tutorial (offline fake
mode plus optional real Codex), the authoritative Revision 3/4 evidence
manifest, the screenshot-index eligibility note, and the OKF journal entry.
Mark the Revision 2 preview (build da0d32d74a and its URL) ineligible for
final acceptance. Offline commands re-run on this tree; docs:validate green
(89 files, 33 OKF concepts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Preserve the existing explorer while wiring the accepted mock, semantic, live, and issue-thread surfaces together. Keep fixture-only command capabilities separate from model-visible grants and split browser-safe exports from Node transports.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Revisions 12-14 verified on the re-captured chat-streaming shots (byte-identical on re-record). Revision 15 accepts the post-capture composer disclaimer and isolated-mock wording from the demo-hardening commits and refreshes the five captures they staled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hold staged replay on a partial model message, limit activity to revealed records, and visibly explain disabled controls. Re-record the two streaming acceptance shots.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
UX review for PAP-16920: §11a revisions 1-9 ruled on (all accepted; the
stage=streaming staging depth is the exception), reviewer revisions 10-11
accept the three-segment mobile shell and the home-rail hint card, and
revisions 12-14 record the fixes shot 5 is gated on: hold the activity
stream at the transcript's reveal ceiling, leave a model card visibly
mid-reveal, and give disabled session controls a visible reason.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a chat surface to the Phase 7 browser explorer where the board sends
prompts to a scenario and sees exactly how the mock Paperclip control plane
was used on every turn — the point of the phase is that the activity stays
visible rather than hidden behind the conversation (PAP-16916).
Runtime owns the turn model, so the UI derives nothing:
- `Phase7ChatSession` keeps one long-lived mock core across turns, so the
state a turn changes is still there for the next one. Turn 0 is the
control-plane seed (checkout, wake routing, exposure) that exists before
anyone can type.
- Timeline entries gain a `turn`; the artifact gains `turns[]` with per-turn
exposure, authorization records, state diff, reconciliation events, counts,
and a turn-scoped verdict. A new `user_message` kind joins the five
existing ones.
- `phase7TurnParity` judges a turn on what that turn did. A session-level
expectation about work nobody has asked for yet would read as a failure of
turn 1 rather than as work still to come.
- Recording machinery moves to `scenario-execution.ts`, shared with the 7F
single-shot runner so a scenario reads the same either way.
Two scenarios carry authored multi-turn scripts: `ap-mcp-gate-01` (allowed
turn, then a denied one) and `ix-checkbox-01` (open an interaction, then the
board answers it — a control-plane-owned action with no agent tool, which is
what makes the control plane schedule the continuation wake). Every other
corpus scenario falls back to its 7F plan as one scripted turn.
The surface implements the approved 7I interaction map: turn strips bridging
each turn to its evidence, the six-section activity stream, the mobile
Chat/Activity segments with a deny count chip, reset and scenario-switch
confirmation, and deterministic deep links including a staged streaming hold.
Deviations taken during implementation are recorded in §11a of the map rather
than left for a reviewer to find.
No ACPX implementation, no real control-plane integration, no credential in
the browser: Codex mode stays disabled with a named reason and scripted mode
drives the same in-page mock core offline.
Tests: 30 new runtime/component tests and 18 browser tests; 79 phase-7 unit
and 43 phase-7 browser tests pass. Ten §11 screenshots recorded by
`record:phase7i`, which fails the run on any horizontal page scroll.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remove trailing generation-wrapper markup (</content>, plus </invoke> in the
tutorial) that had leaked at EOF into eleven committed Phase 7 reference,
tutorial, evidence, and OKF-journal files.
Sync the stated suite counts in the current-state docs (tutorial,
scenario-explorer reference, verification-commands reference, and OKF journal)
to the verified results: test:phase7 49, test:browser:phase7 25,
docs:validate 73 files / 28 OKF concepts / 5 indexes. Drop a brittle wall-clock
timing from the journal. The dated 7F verification evidence and log.md milestone
entry keep their point-in-time 45/19 figures on purpose.
Add a wrapper-markup guard to docs:validate (via scripts/lib/doc-wrappers.mjs)
that fails with the exact file and line when generator envelope tags reappear,
plus a node:test regression suite wired into the package test script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
