0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-09 16:35:27 +02:00

Files

T

DottaandPaperclip 3db2e6bdd2 feat(mcp) [split 8/8]: add e2e coverage and operator docs (#9563 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Governed MCP access spans contracts, runtime enforcement, adapters,
UI surfaces, and operator verification
> - The parity reference PR #9534 is too large for effective automated
or human review
> - The feature therefore needs a linear stack whose individual diffs
stay below the 100-file review limit
> - This pull request is split 8/8 and focuses on end-to-end coverage,
operator docs, evals, and release notes
> - The benefit is a standalone, testable review boundary while
preserving byte-for-byte parity at the top of the stack

## Linked Issues or Issue Description

- Related parity reference: #9534
- Problem: The complete stack needs discoverable browser scenarios,
operator guidance, threat modeling, eval coverage, and a parity proof
before merge.
- Proposed solution: Adds MCP user-story and Smoke Lab e2e suites,
docs/evals/release notes, the skill update, and the root e2e driver
script registration.
- Alternatives considered: keeping #9534 as one 403-file review, or
rewriting the feature to manufacture seams; both were rejected in favor
of path extraction plus compile-driven boundary moves.
- Roadmap alignment: this advances the existing governed MCP/tool-access
work already represented by #9534; it does not introduce a separate
roadmap initiative.
- Stack position: base branch is `pap10341-split/07-ui-apps-activation`.
- Merge policy: merge bottom-up, in order, only after the complete
eight-PR stack has been reviewed and the top-of-stack parity gate
remains empty.
- Requested review: QA for flag audit and e2e/browser acceptance;
Greptile on every PR.

## What Changed

- Adds MCP user-story and Smoke Lab e2e suites, docs/evals/release
notes, the skill update, and the root e2e driver script registration.
- Keeps this PR below 100 changed files and independently typecheckable.
- Preserves the final tree from #9534 when combined with the other seven
stack levels.

## Verification

- `pnpm typecheck`
- `node --check scripts/e2e-mcp-user-stories.mjs`
- `pnpm exec playwright test --config tests/e2e/playwright.config.ts
--list` — 43 tests discovered
- `git diff pap10341-split/08-e2e-docs
6b40e3876d9297105d4ec306e47e46d351c86172` — empty (0 bytes)

## Risks

- Browser suites depend on runtime services and environment setup; this
PR validates discovery locally while QA owns full flag-on/flag-off
execution.
- Stack risk: merging out of order can expose incomplete layers;
mitigate by following the documented bottom-up merge policy.
- Parity risk: later edits to an intermediate branch can drift from
#9534; mitigate by re-running the empty top-of-stack diff before merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, exact model ID `gpt-5.4`; runtime-managed context
window; medium reasoning with repository, shell, Git, GitHub CLI, and
code-execution tools enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] Internal references are omitted except the execution-plan link
explicitly required for this coordinated split stack
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


## Stack Coordination

- Internal execution plan:
[PAP-13874](/PAP/issues/PAP-13874#document-plan)
- Parity reference: #9534
- Stack: #9556 → #9557 → #9558 → #9559 → #9560 → #9561 → #9562 → #9563
- Merge bottom-up only after full-stack review and an empty parity diff
at #9563.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-07-14 15:48:57 -05:00

1.7 KiB

Raw Permalink Blame History

MCP Gateway Promptfoo Run Summary

Date: 2026-06-16 Issue: PAP-11188

Suite

Config: evals/promptfoo/promptfooconfig.yaml
Cases: evals/promptfoo/tests/mcp-gateway.yaml
Provider used for local baseline: echo override
Filter: ^mcp_gateway\.

Command

npx promptfoo@0.103.3 eval -c evals/promptfoo/promptfooconfig.yaml --providers echo --filter-pattern '^mcp_gateway\.' --no-cache --no-progress-bar --no-write -o evals/promptfoo/mcp-gateway-results.json

Config validation:

npx promptfoo@latest validate -c evals/promptfoo/promptfooconfig.yaml

Result

Total cases: 12
Passed: 12
Failed: 0
Errors: 0

Coverage

allow / profile_allows_tool read-only success behavior
403 deny_default denied unsafe tool behavior
409 approval_required pending gateway approval behavior
409 action_not_approved rejected approval behavior
409 formal_approval_required formal board approval wait behavior
429 rate_limited rate-limit handling behavior
422 remote_http_missing_secret credential repair without secret leakage
401 session_revoked stale gateway-token handling
remote HTTP header forwarding and redacted credential audit evidence
exact named/on-demand gateway target selection
elicitation-required human input wait path
409 approved_tool_target_changed stale approval / target drift behavior

Residual Risk

No provider API keys were present in the heartbeat environment, so the live OpenRouter model matrix was not executed. The local baseline proves promptfoo wiring and deterministic assertions. Live model scoring should be run with OPENROUTER_API_KEY or equivalent before using this as a release gate.

1.7 KiB Raw Permalink Blame History