diff --git a/doc/connections/BRAINTRUST.md b/doc/connections/BRAINTRUST.md new file mode 100644 index 00000000000..839fa630fce --- /dev/null +++ b/doc/connections/BRAINTRUST.md @@ -0,0 +1,201 @@ +# Braintrust + +Updated: 2026-10-21. Status: US OAuth connected and exercised against an +authorized account; US API-key and both EU methods still require live account +qualification. The remaining qualification gaps are listed below. + +Braintrust is a P2 resource connection using Paperclip's existing remote MCP +transport, vault, catalog, access profiles, policy evaluation, run-scoped gateway, +and audit. Sign-in tokens are never reused as resource tokens; id.paperclip.ing +never stores resource tokens; no connections hub on the ID service. + +## Methods and setup + +The curated definition is authored in +[`scripts/app-definition-overrides/braintrust.json`](../../scripts/app-definition-overrides/braintrust.json) +and regenerated into the shared catalog. All methods use Streamable HTTP, S4 +risk, and the same read/write capability profile. + +| Method key | Account region | Authentication | MCP resource | +| --- | --- | --- | --- | +| `mcp-oauth` | US | OAuth; recommended default | `https://api.braintrust.dev/mcp` | +| `mcp-api-key` | US | API key | `https://api.braintrust.dev/mcp` | +| `mcp-eu-oauth` | EU | OAuth | `https://api-eu.braintrust.dev/mcp` | +| `mcp-eu-api-key` | EU | API key | `https://api-eu.braintrust.dev/mcp` | + +Open **Apps → Braintrust** or `/apps/connect?source=braintrust`, choose access, +then select the method for the account's region. OAuth registers a client +dynamically and requests only `mcp`. No client ID or secret is entered manually. +For an API key, create a separate key in Braintrust **Settings → API keys**, then +paste it into Paperclip's password field. Paperclip projects the write-only +credential as `Authorization: Bearer `. See the official +[MCP setup guide](https://www.braintrust.dev/docs/integrations/developer-tools/mcp). + +Braintrust OAuth and API keys inherit their account's permissions. Paperclip +cannot grant an issued key additional access. See Braintrust's +[authentication model](https://www.braintrust.dev/docs/admin/authentication). +Use the normal personal or shared credential identity and agent selection. +Personal invocation credentials remain owned by their user; shared credentials +use the company vault. Client-registration credentials are separate from +invocation credentials. Agents receive gateway access, never the provider key. + +There is no Paperclip project filter: project IDs in tool arguments do not +establish an access boundary. Resource authorization remains with Braintrust, +while Paperclip enforces company, grant, agent, task, and action policy. + +Self-hosted Braintrust uses **Connect your own MCP server** with the deployment's +MCP URL. See the [generic remote MCP guide](./GENERIC-REMOTE-MCP.md). This +connector does not configure Paperclip's Braintrust trace exporter. + +## Public protocol evidence + +Unauthenticated probes on 2026-10-21 returned an HTTP 401 challenge from both +regional MCP resources. Each challenge identified its own protected-resource +document, with the exact `/mcp` resource and same-origin authorization server. + +| Metadata or operation | US | EU | +| --- | --- | --- | +| Protected resource | `https://api.braintrust.dev/.well-known/oauth-protected-resource/mcp` | `https://api-eu.braintrust.dev/.well-known/oauth-protected-resource/mcp` | +| Authorization server | `https://api.braintrust.dev/.well-known/oauth-authorization-server` | `https://api-eu.braintrust.dev/.well-known/oauth-authorization-server` | +| Authorization | `https://api.braintrust.dev/oauth/authorize` | `https://api-eu.braintrust.dev/oauth/authorize` | +| Token | `https://api.braintrust.dev/oauth/token` | `https://api-eu.braintrust.dev/oauth/token` | +| Registration | `https://api.braintrust.dev/oauth/register` | `https://api-eu.braintrust.dev/oauth/register` | + +Both servers advertise `authorization_code` and `refresh_token`, scope `mcp`, +PKCE `S256` and `plain`, and token authentication `client_secret_post`, +`client_secret_basic`, and `none`. Paperclip uses PKCE S256 and declares only +`ownershipModes: ["dcr"]`. Neither metadata document advertised CIMD or a +revocation endpoint. No dynamic registration, consent, token exchange, or +authenticated tool discovery was performed by these public probes. + +## Actions and governance + +The Braintrust connector owns the reviewed tool-risk inventory. The connector +policy registry selects it and declares catalog review behavior; the shared +connection service applies those policies. Retrieval tools are reads, including SQL, schema and +experiment inspection, object lookup, trace inspection, chart previews, +documentation search, and skill loading. Loading a provider skill is a governed +read; it does not install a Paperclip skill. + +`edit_dataset_rows` is destructive because rows can be permanently deleted. +`set_automation_status` is destructive because activation can enable retention +or delivery automation. Other mutations and execution tools are writes, +including evaluations, evaluator/facet/preprocessor tests, dataset additions, +patterns, dashboards, alerts, online scoring, and Topics processing. Evaluations +and model-backed processing can incur provider costs. Unknown tools default to +write. Provider annotations may raise risk but cannot downgrade reviewed +mutations. The +[permission reviews](./tool-method-permission-reviews.json) record the scope and +provider-permission rationale for every method. + +Risk classification does not change the central action defaults: discovered +actions initially use Paperclip's current **Allowed** defaults. Operators can +set actions to **Ask first** or **Off** on **Permissions**. Existing decisions +survive reconnect and refresh. After initial discovery, new or changed tools +are quarantined for review; reconnect and refresh preserve that setting. + +## Recovery and removal + +**Save & exit**, cancellation, **Finish setup**, and reconnect use the existing +setup controller. Resume the saved connection and its exact regional method. +An invalid or expired key needs replacement; OAuth access failures need fresh +authorization. Do not switch regions to repair a credential error. An +insufficient-permission failure requires the owner to authorize an account or +key with the intended Braintrust access. + +Remove the connection in Paperclip to clear its local credentials and tool +access. Deactivate an API key in Braintrust separately to invalidate copies +outside Paperclip. API keys are displayed once; expired keys require replacement, +as described in the +[authentication documentation](https://www.braintrust.dev/docs/admin/authentication). +OAuth provider-side revocation is unqualified: the public metadata advertises +no revocation endpoint, so local removal must not be described as remote token +invalidation. + +## Qualification record + +Public metadata is protocol evidence, not proof of account authorization or +tool execution. An existing signed-in US account was authorized for a temporary +personal connection in an isolated source-checkout instance on 2026-10-21. + +| Method | Public regional metadata | Connect / authenticated discovery | Bounded read / disposable write | Real run gateway / refresh or reconnect | Local removal / provider revocation / secret inspection | +| --- | --- | --- | --- | --- | --- | +| `mcp-oauth` | pass, US | pass; 49 tools | pass; synthetic insert and readback | pass; process-agent runs and OAuth reconnect; token expiry/refresh not forced | removal and structural redaction pass; remote revocation unqualified | +| `mcp-api-key` | pass, US; does not validate a key | not run; no test key supplied | not run | not run | not run | +| `mcp-eu-oauth` | pass, EU | not run; available account uses US data plane | not run | not run | not run | +| `mcp-eu-api-key` | pass, EU; does not validate a key | not run; no EU account/key supplied | not run | not run | not run | + +The live US connection discovered 17 reads and 32 non-read actions. Provider +annotations raised several write classifications to destructive; unreviewed +names remained writes. `list_recent_objects` performed a bounded read. +`edit_dataset_rows` created one synthetic row in a new disposable dataset, and +`sql_query` read it back. Both ran through real process-agent heartbeats and +run-scoped gateway sessions. The qualification harness delivered the gateway +capability through a private temporary file; this does not qualify automatic +credential delivery for a native CLI harness. No provider credential was sent +to the process agent and no evaluation/model execution was invoked. + +Catalog refresh and OAuth reconnect retained the same connection, US method, +49 actions, and changed-tool quarantine configuration. A subsequent bounded +read succeeded. The provider returned `openid profile email offline_access` in +the token response scope despite the authorization request containing only +`mcp`; Paperclip retained that discrepancy as provider metadata. Access-token +expiry and refresh were tested with fixtures, not forced on this live grant. + +Cleanup deleted the synthetic row, verified an empty result, and removed the +new empty dataset in Braintrust after explicit approval. Local connection +removal revoked two vault secrets and one grant, removed the installation and +49 profile entries, and rejected a subsequent call with `tool_not_found`. +Qualification gateway sessions were revoked and their private files deleted. +The owned server and database were stopped and the disposable instance data +was removed. +Remote OAuth token invalidation remains unqualified because no revocation +endpoint is advertised. + +Connection/catalog API responses contained no raw credential fields. The +isolated server log had no bearer values, access/refresh-token values, or OAuth +callback codes under credential-shaped scans. This was a structural redaction +check, not an exact decrypted-token search. Non-submitted key canaries were +absent from browser storage during the separate UI walkthrough. Fixture tests +provide exact canary redaction assertions. The sanitized +[qualification record](./braintrust-qualification-2026-10-21.json) separates +live, browser, and simulated-provider evidence. + +Qualify each method independently in an owned isolated instance. Record the +revision, deployment, region, credential identity, discovered inventory, action +decision, and audit correlation. Use synthetic resources for writes; prove +Allowed, Ask-first, Off, quarantine, wrong-owner, and cross-company behavior +through the real run-scoped gateway. Verify cancellation/resume and reconnect +without duplicate connections or regional changes. Inspect API responses, +browser storage, logs, and stored secret references without copying tokens or +customer traces into evidence. Fixture-backed verification establishes +Paperclip behavior only; record its commands/results separately from live +provider proof, following the +[connector playbook](./CONNECTOR-PLAYBOOK.md#step-9-align-with-production-validation). + +### Deterministic verification + +On 2026-10-21, the Braintrust service fixture exercised all four methods with a +simulated provider and the actual encrypted vault, database, catalog, policy, +and run-scoped gateway. It covered OAuth DCR/PKCE and cancellation/resume, +personal/shared credential ownership, allowed reads and writes, approval and +Off decisions, schema-change quarantine, refresh/reconnect, wrong-owner and +cross-company denial, revocation, and redaction. See +[`braintrust-connection.test.ts`](../../server/src/__tests__/braintrust-connection.test.ts) +and the connector-owned +[`risk.ts`](../../server/src/services/connectors/braintrust/risk.ts). + +The catalog suite validates all four reviewed contracts. Apps setup tests cover +each region/authentication choice and retrying the saved EU OAuth connection. +Brand validation checks the official local artwork; provenance is recorded in +[`braintrust-brand-provenance-2026-10-21.json`](./braintrust-brand-provenance-2026-10-21.json). +These checks establish Paperclip behavior with fixtures, not live Braintrust +account compatibility. + +The isolated browser walkthrough passed 15 checks covering Apps and direct-link +entry, all four method choices and reloads, password-field validation, +cancellation, personal/shared access controls, and official light/dark artwork. +Its first pass exposed missing pre-auth caveats: warnings were present in the +definition but were not rendered by the setup form. The capability summary and +key helper now show account permissions, costs, permanent deletion, and +automation risks before authorization; the repeated walkthrough passed. diff --git a/doc/connections/README.md b/doc/connections/README.md index 4c8f13512d2..5a06e5a4856 100644 --- a/doc/connections/README.md +++ b/doc/connections/README.md @@ -13,7 +13,7 @@ Runtime authentication: [AI Connections](./AI-CONNECTIONS.md). Long-term memory: [Experimental memory connectors](./MEMORY.md). Provider notes: [Google Workspace](./GOOGLE-WORKSPACE.md), -[Gmail](./GMAIL.md), [Asana](./ASANA.md), [Sentry](./SENTRY.md), [PostHog](./POSTHOG.md), [Neon](./NEON.md), [Superagent](./SUPERAGENT.md), [Gauge](./GAUGE.md), [Ditto](./DITTO.md), +[Gmail](./GMAIL.md), [Asana](./ASANA.md), [Sentry](./SENTRY.md), [PostHog](./POSTHOG.md), [Braintrust](./BRAINTRUST.md), [Neon](./NEON.md), [Superagent](./SUPERAGENT.md), [Gauge](./GAUGE.md), [Ditto](./DITTO.md), [AgentMail](./AGENTMAIL.md), [Telem.AI](./TELEM.md), [iMessage Photon](./IMESSAGE-PHOTON.md), and [Enterpret](./ENTERPRET.md). Optional credential custody: [Vercel Connect](./VERCEL-CONNECT.md). diff --git a/doc/connections/braintrust-brand-provenance-2026-10-21.json b/doc/connections/braintrust-brand-provenance-2026-10-21.json new file mode 100644 index 00000000000..71d5f3bdc18 --- /dev/null +++ b/doc/connections/braintrust-brand-provenance-2026-10-21.json @@ -0,0 +1,17 @@ +{ + "retrievedAt": "2026-10-21", + "providers": [ + { + "slug": "braintrust", + "provider": "Braintrust", + "localAsset": "/brands/apps/braintrust.png", + "darkAsset": "/brands/apps/braintrust-dark.png", + "sourceUrl": "https://www.braintrust.dev/icon180.png?v=2", + "darkSourceUrl": "https://www.braintrust.dev/icon-dark180.png?v=2", + "linkedFrom": "https://www.braintrust.dev/", + "sha256": "aa4434ae1033d523f11d08761933ec317ef30c04ac543ec06ae7443d2fd38cb3", + "darkSha256": "6c64f3fd8b32bb84544cb438ea18aac4f461402d1fdb6960022f5d410bc0e1ca", + "note": "Official 180\u00d7180 transparent Braintrust website icons, retained without modification. Purple mark for light mode and white mark for dark mode; use the existing Paperclip gray icon frame." + } + ] +} diff --git a/doc/connections/braintrust-qualification-2026-10-21.json b/doc/connections/braintrust-qualification-2026-10-21.json new file mode 100644 index 00000000000..00462d06fe8 --- /dev/null +++ b/doc/connections/braintrust-qualification-2026-10-21.json @@ -0,0 +1,101 @@ +{ + "recordedAt": "2026-10-21", + "baseRevision": "16ea56fd3eb492e3636ebc60e0c9fd4deb328f14", + "worktreeChangesDuringProof": "Pre-auth capability summaries and key permission helper copy; generated catalog and setup assertions.", + "environment": { + "artifact": "source checkout", + "platform": "darwin arm64", + "node": "v26.4.0", + "instance": "isolated local_trusted with fresh embedded PostgreSQL", + "actor": "temporary personal board-owned grant", + "modelExecution": false + }, + "live": { + "method": "mcp-oauth", + "region": "US", + "endpoint": "https://api.braintrust.dev/mcp", + "requestedScopes": [ + "mcp" + ], + "providerReportedScope": "openid profile email offline_access", + "connect": "pass", + "discoveredTools": 49, + "readTools": 17, + "nonReadTools": 32, + "boundedRead": { + "tool": "list_recent_objects", + "limit": 1, + "result": "pass" + }, + "syntheticWrite": { + "tool": "edit_dataset_rows", + "rows": 1, + "newDisposableDataset": true, + "readbackTool": "sql_query", + "result": "pass" + }, + "runGateway": "Two successful process-agent heartbeats using actual run-scoped gateway capabilities. Capability delivered by private qualification harness file; provider credentials remained vaulted.", + "catalogRefresh": "pass; 49 tools retained", + "oauthReconnect": "pass; same connection, regional method and quarantine configuration; subsequent bounded read passed", + "cleanup": { + "syntheticRow": "deleted; empty result verified", + "disposableDataset": "deleted after explicit user approval; Braintrust dataset list empty", + "localConnection": "archived; 2 secrets and 1 grant revoked; installation and 49 profile entries removed", + "postRemovalCall": "denied with tool_not_found", + "gatewaySessions": "revoked; private capability files deleted", + "isolatedInstance": "Owned server/database stopped; disposable instance data removed; sanitized evidence retained." + }, + "redaction": { + "api": "No raw credential fields in connection and catalog responses", + "serverLog": "No credential-shaped bearer/access-token/refresh-token/callback-code values", + "limit": "No token was decrypted for exact-value scanning." + } + }, + "browser": { + "checksPassed": 15, + "providerCalls": false, + "coverage": [ + "Apps card", + "all four method choices", + "personal/shared control", + "pre-auth risk copy", + "password input and required validation", + "key canary absent from browser storage", + "cancel without connection creation", + "direct method link and reload", + "plain source default", + "light and dark artwork" + ], + "observedAndFixed": "Initial form did not render definition warnings; moved risk caveats into visible capability summary and key helper, then repeated walkthrough.", + "unrelatedObservation": "Assistant Connection card requests an unavailable /api/mcp/connections endpoint; Braintrust setup succeeds." + }, + "fixture": { + "braintrustServiceTestsPassed": 56, + "catalogTestsPassed": 36, + "coverage": [ + "all four methods", + "DCR and PKCE", + "personal/shared vault ownership", + "read/write execution", + "Off and Ask-first", + "changed-tool quarantine", + "refresh/reconnect", + "cancellation/resume", + "cross-company and wrong-owner denial", + "revocation and failed-setup cleanup", + "exact canary redaction" + ] + }, + "qualificationGaps": [ + "US API-key method: no test key supplied", + "EU OAuth and API-key methods: available account uses US data plane; no EU account/key supplied", + "Live token expiry/refresh not forced; reconnect passed", + "Provider-side OAuth revocation has no advertised endpoint", + "Native CLI harness automatic capability delivery not exercised" + ], + "harnessCorrections": [ + "Initial process harness used the wrong authentication header; corrected to x-paperclip-tool-gateway-token.", + "Initial write harness parsed normalized MCP content incorrectly; corrected before successful synthetic write." + ], + "delivery": "Live qualification is partial. Outstanding account methods, token lifecycle checks and native CLI capability delivery are listed above; successful CI does not qualify those provider behaviors." +} diff --git a/doc/connections/tool-method-permission-reviews.json b/doc/connections/tool-method-permission-reviews.json index e04dd4d36fa..f5a516146b9 100644 --- a/doc/connections/tool-method-permission-reviews.json +++ b/doc/connections/tool-method-permission-reviews.json @@ -2805,6 +2805,80 @@ "reviewedAt": "2026-10-20", "liveProof": "not-run", "evidenceLimit": "First-party implementation contract; canonical service availability and live OAuth qualification are deployment prerequisites. Uses the existing personal OAuth flow with default connection labels; automatic remote revocation and custom keyed refresh recovery are deferred." + }, + { + "app": "braintrust", + "method": "mcp-oauth", + "auth": "oauth", + "policy": "explicit", + "requestedScopes": [ + "mcp" + ], + "capability": "write", + "supportedActions": "Read logs, schemas, experiment summaries, datasets, projects, monitoring views, and documentation; create or update prompts, evaluators, datasets, patterns, charts, and automations; run evaluations and Topics jobs that may consume credits. Dataset row edits can permanently delete rows. Automation status can enable retention or delivery. Access remains bounded by the authenticated Braintrust account.", + "evidence": [ + "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "https://api.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "https://api.braintrust.dev/.well-known/oauth-authorization-server" + ], + "reviewedAt": "2026-10-21", + "liveProof": "2026-10-21 US OAuth: temporary personal connection authorized in an isolated instance; 49 tools discovered (17 reads, 32 non-reads). Real process-agent runs completed bounded retrieval and synthetic dataset insert/readback through run-scoped gateway sessions. Catalog refresh and OAuth reconnect retained the connection, region and quarantine; subsequent read passed. Synthetic row and dataset removed. Local removal revoked 2 secrets and 1 grant and denied further calls. Structural API/log redaction checked. Token expiry/refresh, provider-side revocation and native CLI capability delivery remain unqualified. See BRAINTRUST.md and braintrust-qualification-2026-10-21.json.", + "evidenceLimit": "Live proof covers a temporary personal US OAuth grant, bounded retrieval, one synthetic dataset insert/readback, real process-agent gateway calls, catalog refresh, OAuth reconnect, local removal and structural redaction. Live token expiry/refresh, remote OAuth revocation, native CLI automatic capability delivery and exact decrypted-token scanning remain unqualified. The other three methods need separate live accounts/keys." + }, + { + "app": "braintrust", + "method": "mcp-api-key", + "auth": "api_key", + "policy": "provider-key", + "requestedScopes": [], + "capability": "write", + "supportedActions": "Read logs, schemas, experiment summaries, datasets, projects, monitoring views, and documentation; create or update prompts, evaluators, datasets, patterns, charts, and automations; run evaluations and Topics jobs that may consume credits. Dataset row edits can permanently delete rows. Automation status can enable retention or delivery. Access remains bounded by the authenticated Braintrust account.", + "evidence": [ + "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "https://api.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "https://api.braintrust.dev/.well-known/oauth-authorization-server" + ], + "reviewedAt": "2026-10-21", + "liveProof": "not-run", + "evidenceLimit": "Public provider documentation and OAuth metadata only. Account-bound connect, discovery, read/write execution, run-scoped gateway, refresh/reconnect, removal, and secret handling remain to be qualified for this method.", + "keyPermissions": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + { + "app": "braintrust", + "method": "mcp-eu-oauth", + "auth": "oauth", + "policy": "explicit", + "requestedScopes": [ + "mcp" + ], + "capability": "write", + "supportedActions": "Read logs, schemas, experiment summaries, datasets, projects, monitoring views, and documentation; create or update prompts, evaluators, datasets, patterns, charts, and automations; run evaluations and Topics jobs that may consume credits. Dataset row edits can permanently delete rows. Automation status can enable retention or delivery. Access remains bounded by the authenticated Braintrust account.", + "evidence": [ + "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "https://api-eu.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "https://api-eu.braintrust.dev/.well-known/oauth-authorization-server" + ], + "reviewedAt": "2026-10-21", + "liveProof": "not-run", + "evidenceLimit": "Public provider documentation and OAuth metadata only. Account-bound connect, discovery, read/write execution, run-scoped gateway, refresh/reconnect, removal, and secret handling remain to be qualified for this method." + }, + { + "app": "braintrust", + "method": "mcp-eu-api-key", + "auth": "api_key", + "policy": "provider-key", + "requestedScopes": [], + "capability": "write", + "supportedActions": "Read logs, schemas, experiment summaries, datasets, projects, monitoring views, and documentation; create or update prompts, evaluators, datasets, patterns, charts, and automations; run evaluations and Topics jobs that may consume credits. Dataset row edits can permanently delete rows. Automation status can enable retention or delivery. Access remains bounded by the authenticated Braintrust account.", + "evidence": [ + "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "https://api-eu.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "https://api-eu.braintrust.dev/.well-known/oauth-authorization-server" + ], + "reviewedAt": "2026-10-21", + "liveProof": "not-run", + "evidenceLimit": "Public provider documentation and OAuth metadata only. Account-bound connect, discovery, read/write execution, run-scoped gateway, refresh/reconnect, removal, and secret handling remain to be qualified for this method.", + "keyPermissions": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." } ] } diff --git a/doc/plans/2026-08-26-self-serve-mcp-connections.md b/doc/plans/2026-08-26-self-serve-mcp-connections.md index 2581f8e6832..1f6fe1053f8 100644 --- a/doc/plans/2026-08-26-self-serve-mcp-connections.md +++ b/doc/plans/2026-08-26-self-serve-mcp-connections.md @@ -93,6 +93,34 @@ Store visibility is a separate release gate from having an implemented definitio | Vercel | Blocked | [x] | n/a | Reconsider when reviewed-client approval is removed or Paperclip is approved. | | Zomato | Blocked | [x] | n/a | Reconsider when third-party clients and unallowlisted redirect URIs are supported. | +## Braintrust addition — 2026-10-21 + +Braintrust adds four hosted Streamable HTTP methods: US `mcp-oauth` and +`mcp-api-key` at `https://api.braintrust.dev/mcp`, plus EU `mcp-eu-oauth` and +`mcp-eu-api-key` at `https://api-eu.braintrust.dev/mcp`. US OAuth is the +recommended default. OAuth declares DCR and requests only `mcp`; API keys use +the bearer header. All four methods are S4 and write-capable. They reuse +Access → Connect, personal/shared vault ownership, policy, and gateway paths. + +Both regions' unauthenticated challenge and metadata checks passed on +2026-10-21. They advertise DCR, PKCE S256, and refresh tokens, with no CIMD or +revocation endpoint. These public probes performed no registration or +authenticated discovery; separate US OAuth account proof is recorded below. +The reviewed risk inventory preserves retrieval tools as reads, treats dataset +row edits and automation-status changes as destructive, and treats other +mutations, billable execution, and unknown tools as writes. Central Allowed +defaults and operator Off/Ask-first decisions remain intact. Catalog drift is +quarantined after initial discovery and stays enabled across reconnect. + +US OAuth account proof covers connect, discovery, bounded read, a synthetic +dataset write/readback through real process-agent gateway calls, catalog +refresh, OAuth reconnect, local removal and structural secret inspection. The +synthetic data and disposable instance were removed. US API-key and both EU +methods remain **not run** against a real account. Live token expiry/refresh, +provider-side revocation and native CLI capability delivery remain unqualified. +Metadata, fixture and CI results do not qualify those provider behaviors. See [Braintrust](../connections/BRAINTRUST.md) for the per-method matrix, +provider evidence, permissions, and revocation limits. + ## Browser authorization redirect audit This is a local, credential-free handoff check performed through the real BOB catalog UI on 2026-08-26. A checked provider reached its own login, consent, domain-picker, or authorization page. It does **not** satisfy the account-bound live-proof column above, which still requires consent, tool discovery, a safe read, reconnect/refresh, revocation, and secret-leak inspection. diff --git a/maps/agent-access-map/app-permissions.md b/maps/agent-access-map/app-permissions.md index d40da4f5489..d3cacb64ece 100644 --- a/maps/agent-access-map/app-permissions.md +++ b/maps/agent-access-map/app-permissions.md @@ -71,6 +71,19 @@ own policy and access review. Gaps: An agent cannot authorize itself by editing the policy. Decide whether any explicitly delegated administration should be supported. +Braintrust's reviewed classifier treats retrieval as read, dataset row edits and +automation-status changes as destructive, and other mutations, billable +execution, and unknown tools as write. Provider hints cannot lower a reviewed +mutation's risk. The classifier grants no authority: shared Allowed defaults +and explicit Off/Ask-first policies still govern each agent's actual run-bound +call. See [Braintrust](../../doc/connections/BRAINTRUST.md#actions-and-governance). +Live US OAuth bounded reads and a synthetic dataset insert/readback passed +through actual process-agent runs and run-scoped gateway sessions; local removal +then denied further use. This does not establish live Off/Ask-first policy +behavior, which is covered with provider fixtures. US API-key and both EU +methods remain `not-run` against real accounts; board Test calls do not +establish agent execution evidence. + ## `catalog-change` Capability ID: `app-permissions/catalog-change`. @@ -97,6 +110,18 @@ own policy and access review. Gaps: Catalog refresh/review and quarantined-action enablement are management operations. +Braintrust enables changed-tool quarantine after initial discovery and preserves +it through reconnect and refresh. Agents cannot approve newly quarantined tools +or bypass a changed schema by reusing its old name. Verify the quarantined call +denial through a real run-scoped gateway separately from board catalog review; +the [provider qualification record](../../doc/connections/BRAINTRUST.md#qualification-record) +records live US OAuth catalog refresh and reconnect preserving the regional +method and quarantine setting. Live changed-tool denial remains outstanding; +fixtures cover that denial and policy preservation across refresh/reconnect. +US API-key and both EU methods have no account-bound qualification. Live token +expiry/refresh, remote OAuth revocation, and native CLI automatic capability +delivery also remain unqualified. + Daytona also checks its deployed native contract on the board's `GET /api/tool-connections/{connectionId}/catalog`. A changed saved catalog refreshes under the viewing human's credential context. Retired tools are diff --git a/maps/agent-access-map/connection-setup.md b/maps/agent-access-map/connection-setup.md index 0d3c322f2a8..c800966e8f8 100644 --- a/maps/agent-access-map/connection-setup.md +++ b/maps/agent-access-map/connection-setup.md @@ -114,6 +114,22 @@ remaining operations before qualifying the whole capability. Gaps: An initiation endpoint exists; it is not authorization to complete the person's credential flow. +Braintrust uses the existing setup/request path for US and EU OAuth and API-key +methods. OAuth requests only `mcp`; provider authorization follows the selected +account's permissions. Personal invocation credentials must belong to their +user, while shared identities use company credentials. Region and explicit +method survive resume and reconnect. Agents use assigned tools through the +run-scoped gateway and receive no Braintrust token. See +[Braintrust](../../doc/connections/BRAINTRUST.md) for method keys and the +qualification matrix. Separate live US OAuth proof passed browser authorization, +49-tool discovery, bounded read, synthetic dataset insert/readback through actual +process-agent runs and gateway sessions, catalog refresh, OAuth reconnect, local +removal, post-removal denial, and structural API/log redaction. US API-key and +both EU methods remain `not-run` against real accounts. Live token expiry/refresh, +remote OAuth revocation, and native CLI automatic capability delivery remain +unqualified. Public metadata probes performed no registration or account +authorization; fixtures do not establish live proof for the remaining methods. + Daytona native setup uses an organization API key in the selected personal or company vault grant. It does not borrow sandbox-provider credentials or CLI login tokens. Agents use assigned provider tools through the normal run-bound diff --git a/maps/feature-map/app-permissions.md b/maps/feature-map/app-permissions.md index f68a93a3fe2..025b6660d8e 100644 --- a/maps/feature-map/app-permissions.md +++ b/maps/feature-map/app-permissions.md @@ -59,6 +59,19 @@ qualified by this coverage. Manual: Grant one agent access, leave another excluded, and exercise off, ask, and allowed states for a harmless action. Reload between changes. Verify actual denied calls stay denied and ask-first creates a decision rather than executing early. +For Braintrust, distinguish reviewed action risk from effective policy: S4 +methods retain the shared Allowed defaults. Explicit Off and Ask-first choices +must survive refresh and reconnect. Exercise the read, write, and destructive +classifications with a fixture, including billable tests classified as writes +and dataset row edits/automation activation classified as destructive. After +initial discovery, newly added or changed tools remain quarantined until +reviewed. Use synthetic data for provider calls; see +[Braintrust](../../doc/connections/BRAINTRUST.md#actions-and-governance). Live US +OAuth proof covers bounded reads, a synthetic dataset insert/readback through +actual process-agent gateway calls, and denial after local removal. Off, +Ask-first, and changed-tool denial use fixture evidence; the other three methods +still need live account qualification. + ### `test-and-maintain` Automated: [app detail tests](../../ui/src/pages/apps/AppDetail.test.tsx) cover test-dialog/UI contracts. Real provider execution and revocation require the selected provider’s live check. diff --git a/maps/feature-map/connection-setup.md b/maps/feature-map/connection-setup.md index 3efb0d3713b..679a090e769 100644 --- a/maps/feature-map/connection-setup.md +++ b/maps/feature-map/connection-setup.md @@ -123,6 +123,20 @@ checks the catalog, password field, access controls, and optional organization pin. It does not submit a key or qualify a real provider account. See [Daytona](../../doc/connections/DAYTONA.md) for API-key authority and OAuth limits. +For Braintrust, verify US OAuth (default), US API key, EU OAuth, and EU API key +from both Apps and `/apps/connect?source=braintrust`. Inspect the exact regional +endpoint and method after Save & exit, cancellation/resume, and reconnect; +these operations must retain the saved connection and its access choices. +Verify the key stays out of browser draft storage and public responses, then +perform a bounded read and disposable write through an actual eligible run. +The [Braintrust qualification matrix](../../doc/connections/BRAINTRUST.md#qualification-record) +records completed US OAuth connect/discovery, bounded read, synthetic write +through actual process-agent gateway calls, catalog refresh, reconnect, local +removal, and structural redaction. US API-key and both EU methods still need +account-bound proof. Live token expiry/refresh, remote OAuth revocation, and +native CLI automatic capability delivery remain unqualified; public metadata +and fixtures do not establish those results. + ### `mcp-link` Automated: AppsConnect's generic MCP cases cover probe-driven auth, private or diff --git a/packages/shared/src/app-definitions.generated.ts b/packages/shared/src/app-definitions.generated.ts index 677f8e57953..77f51bb2b8b 100644 --- a/packages/shared/src/app-definitions.generated.ts +++ b/packages/shared/src/app-definitions.generated.ts @@ -82,28 +82,29 @@ import a80 from "./app-definitions/gsc-wizard.json" with { type: "json" }; import a81 from "./app-definitions/clickup.json" with { type: "json" }; import a82 from "./app-definitions/klaviyo.json" with { type: "json" }; import a83 from "./app-definitions/ditto.json" with { type: "json" }; -import a84 from "./app-definitions/google-workspace.json" with { type: "json" }; -import a85 from "./app-definitions/gmail.json" with { type: "json" }; -import a86 from "./app-definitions/google-drive.json" with { type: "json" }; -import a87 from "./app-definitions/google-docs.json" with { type: "json" }; -import a88 from "./app-definitions/google-sheets.json" with { type: "json" }; -import a89 from "./app-definitions/google-slides.json" with { type: "json" }; -import a90 from "./app-definitions/google-calendar.json" with { type: "json" }; -import a91 from "./app-definitions/google-chat.json" with { type: "json" }; -import a92 from "./app-definitions/google-people.json" with { type: "json" }; -import a93 from "./app-definitions/google-workspace-search.json" with { type: "json" }; -import a94 from "./app-definitions/openai.json" with { type: "json" }; -import a95 from "./app-definitions/openrouter.json" with { type: "json" }; -import a96 from "./app-definitions/xai.json" with { type: "json" }; -import a97 from "./app-definitions/google.json" with { type: "json" }; -import a98 from "./app-definitions/bedrock.json" with { type: "json" }; -import a99 from "./app-definitions/responses-api.json" with { type: "json" }; -import a100 from "./app-definitions/messages-api.json" with { type: "json" }; -import a101 from "./app-definitions/chat-completions-api.json" with { type: "json" }; -import a102 from "./app-definitions/local.json" with { type: "json" }; -import a103 from "./app-definitions/daytona.json" with { type: "json" }; -import a104 from "./app-definitions/monday.json" with { type: "json" }; -import a105 from "./app-definitions/paperclip-notebook.json" with { type: "json" }; -import a106 from "./app-definitions/speko.json" with { type: "json" }; +import a84 from "./app-definitions/braintrust.json" with { type: "json" }; +import a85 from "./app-definitions/google-workspace.json" with { type: "json" }; +import a86 from "./app-definitions/gmail.json" with { type: "json" }; +import a87 from "./app-definitions/google-drive.json" with { type: "json" }; +import a88 from "./app-definitions/google-docs.json" with { type: "json" }; +import a89 from "./app-definitions/google-sheets.json" with { type: "json" }; +import a90 from "./app-definitions/google-slides.json" with { type: "json" }; +import a91 from "./app-definitions/google-calendar.json" with { type: "json" }; +import a92 from "./app-definitions/google-chat.json" with { type: "json" }; +import a93 from "./app-definitions/google-people.json" with { type: "json" }; +import a94 from "./app-definitions/google-workspace-search.json" with { type: "json" }; +import a95 from "./app-definitions/openai.json" with { type: "json" }; +import a96 from "./app-definitions/openrouter.json" with { type: "json" }; +import a97 from "./app-definitions/xai.json" with { type: "json" }; +import a98 from "./app-definitions/google.json" with { type: "json" }; +import a99 from "./app-definitions/bedrock.json" with { type: "json" }; +import a100 from "./app-definitions/responses-api.json" with { type: "json" }; +import a101 from "./app-definitions/messages-api.json" with { type: "json" }; +import a102 from "./app-definitions/chat-completions-api.json" with { type: "json" }; +import a103 from "./app-definitions/local.json" with { type: "json" }; +import a104 from "./app-definitions/daytona.json" with { type: "json" }; +import a105 from "./app-definitions/monday.json" with { type: "json" }; +import a106 from "./app-definitions/paperclip-notebook.json" with { type: "json" }; +import a107 from "./app-definitions/speko.json" with { type: "json" }; import type { AppDefinition } from "./types/app-definition.js"; -export const APP_DEFINITIONS=[a0,a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11,a12,a13,a14,a15,a16,a17,a18,a19,a20,a21,a22,a23,a24,a25,a26,a27,a28,a29,a30,a31,a32,a33,a34,a35,a36,a37,a38,a39,a40,a41,a42,a43,a44,a45,a46,a47,a48,a49,a50,a51,a52,a53,a54,a55,a56,a57,a58,a59,a60,a61,a62,a63,a64,a65,a66,a67,a68,a69,a70,a71,a72,a73,a74,a75,a76,a77,a78,a79,a80,a81,a82,a83,a84,a85,a86,a87,a88,a89,a90,a91,a92,a93,a94,a95,a96,a97,a98,a99,a100,a101,a102,a103,a104,a105,a106] as AppDefinition[]; +export const APP_DEFINITIONS=[a0,a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11,a12,a13,a14,a15,a16,a17,a18,a19,a20,a21,a22,a23,a24,a25,a26,a27,a28,a29,a30,a31,a32,a33,a34,a35,a36,a37,a38,a39,a40,a41,a42,a43,a44,a45,a46,a47,a48,a49,a50,a51,a52,a53,a54,a55,a56,a57,a58,a59,a60,a61,a62,a63,a64,a65,a66,a67,a68,a69,a70,a71,a72,a73,a74,a75,a76,a77,a78,a79,a80,a81,a82,a83,a84,a85,a86,a87,a88,a89,a90,a91,a92,a93,a94,a95,a96,a97,a98,a99,a100,a101,a102,a103,a104,a105,a106,a107] as AppDefinition[]; diff --git a/packages/shared/src/app-definitions.test.ts b/packages/shared/src/app-definitions.test.ts index b647d66d3b3..3b1febbbde9 100644 --- a/packages/shared/src/app-definitions.test.ts +++ b/packages/shared/src/app-definitions.test.ts @@ -284,7 +284,7 @@ describe("AppDefinition catalog", () => { "google-workspace-search", ]), ); - expect(SELF_SERVE_MCP_CANDIDATES).toHaveLength(64); + expect(SELF_SERVE_MCP_CANDIDATES).toHaveLength(65); expect(BLOCKED_MCP_PROVIDERS.map((entry) => entry.slug)).toEqual([ "g2", "vercel", @@ -450,15 +450,15 @@ describe("AppDefinition catalog", () => { expect(channel("slack")?.guidanceMd).toContain("reactions"); expect(channel("slack")?.guidanceMd).toContain("direct messages"); }); - it("keeps a complete, unique, dated evidence ledger for all 68 researched MCP providers", () => { + it("keeps a complete, unique, dated evidence ledger for all 69 researched MCP providers", () => { // Ledger-wide date reflects the last full re-verification (2026-08-26); // later provider additions carry their own research evidence, but // bumping the shared date would overstate freshness for the other providers. expect(SELF_SERVE_MCP_RESEARCH.verifiedAt).toBe("2026-08-26"); - expect(SELF_SERVE_MCP_RESEARCH.entries).toHaveLength(68); + expect(SELF_SERVE_MCP_RESEARCH.entries).toHaveLength(69); expect( new Set(SELF_SERVE_MCP_RESEARCH.entries.map((entry) => entry.slug)), - ).toHaveProperty("size", 68); + ).toHaveProperty("size", 69); for (const entry of SELF_SERVE_MCP_RESEARCH.entries) { expect(new URL(entry.docsUrl).protocol).toBe("https:"); expect(new URL(entry.serverUrl).protocol).toBe("https:"); @@ -886,7 +886,7 @@ describe("AppDefinition catalog", () => { "ticktick", "xero", ]); - expect(APP_STORE_DEFINITIONS).toHaveLength(84); + expect(APP_STORE_DEFINITIONS).toHaveLength(85); const connectableSlugs = new Set( CONNECTABLE_APP_DEFINITIONS.map((entry) => entry.slug), ); @@ -1096,6 +1096,71 @@ describe("AppDefinition catalog", () => { expect(method.guidanceMd).toContain("optional advanced controls"); } }); + it("connects Braintrust US and EU with discovered OAuth or a vaulted bearer key", () => { + const braintrust = getConnectableAppDefinition("braintrust")!; + expect(getAppStoreDefinition("braintrust")).toBe(braintrust); + expect(braintrust).toMatchObject({ + name: "Braintrust", + categories: ["developer", "data"], + urlPatterns: ["https://api.braintrust.dev/*", "https://api-eu.braintrust.dev/*"], + docsUrl: "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + redirectConstraints: "https-or-loopback-http", + branding: { + logoUrl: "/brands/apps/braintrust.png", + darkLogoUrl: "/brands/apps/braintrust-dark.png", + }, + }); + expect(braintrust.methods.map((method) => method.key)).toEqual([ + "mcp-oauth", "mcp-api-key", "mcp-eu-oauth", "mcp-eu-api-key", + ]); + expect(getAvailableConnectionMethod(braintrust)?.key).toBe("mcp-oauth"); + for (const method of braintrust.methods) { + const region = method.key.startsWith("mcp-eu-") ? "EU" : "US"; + const host = region === "EU" ? "api-eu.braintrust.dev" : "api.braintrust.dev"; + const serverUrl = `https://${host}/mcp`; + expect(getAvailableConnectionMethod(braintrust, method.key)).toBe(method); + expect(resolveConnectionMethodServerUrl(method, {})).toBe(serverUrl); + expect(getAppDefinitionForUrl(serverUrl)?.slug).toBe("braintrust"); + expect(method).toMatchObject({ + transport: "mcp_remote", riskTier: "S4", + capabilityProfile: { key: "write", label: "Read and write" }, + defaults: { serverUrl }, + }); + expect(method.label).toContain(region); + expect(method.tenantFields).toBeUndefined(); + expect(method.requiredResourceFilters).toBeUndefined(); + expect(method.grantKinds).toBeUndefined(); + expect(method.credentialSources).toBeUndefined(); + expect(recommendedDefaultsForApp(braintrust, method.key)).toEqual({ + access: "all_agents", askFirstRiskLevels: [], + }); + expect(method.warnings?.join(" ")).toContain("permanently delete"); + expect(method.warnings?.join(" ")).toContain("credits"); + if (method.auth === "oauth") { + expect(method.ownershipModes).toEqual(["dcr"]); + expect(method.defaults).toEqual({ + serverUrl, + discoveryUrl: `https://${host}/.well-known/oauth-protected-resource/mcp`, + scopesHint: ["mcp"], + }); + expect(method.credentialFields).toBeUndefined(); + } else { + expect(method.auth).toBe("api_key"); + expect(method.ownershipModes).toEqual(["customer"]); + expect(method.defaults).toEqual({ serverUrl }); + expect(method.keyPlacement).toEqual({ + location: "header", name: "Authorization", prefix: "Bearer ", + }); + expect(method.credentialFields).toEqual([ + expect.objectContaining({ + key: "authorization", type: "password", required: true, secret: true, + helperMd: expect.stringContaining("reads and writes"), + }), + ]); + } + } + expect(getAppDefinitionForUrl("https://api.braintrust.dev.example.com/mcp")).toBeNull(); + }); it("connects Neon's hosted server with optional project pinning and read-only mode", () => { const neon = APP_DEFINITIONS.find((app) => app.slug === "neon")!; expect(neon).toMatchObject({ diff --git a/packages/shared/src/app-definitions/braintrust.json b/packages/shared/src/app-definitions/braintrust.json new file mode 100644 index 00000000000..c7b52e8588b --- /dev/null +++ b/packages/shared/src/app-definitions/braintrust.json @@ -0,0 +1,177 @@ +{ + "schemaVersion": 1, + "slug": "braintrust", + "name": "Braintrust", + "description": "Explore AI traces, manage evaluation datasets and scorers, and run Braintrust evaluations.", + "categories": [ + "developer", + "data" + ], + "featured": false, + "branding": { + "logoUrl": "/brands/apps/braintrust.png", + "darkLogoUrl": "/brands/apps/braintrust-dark.png" + }, + "urlPatterns": [ + "https://api.braintrust.dev/*", + "https://api-eu.braintrust.dev/*" + ], + "docsUrl": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "redirectConstraints": "https-or-loopback-http", + "methods": [ + { + "key": "mcp-oauth", + "label": "Sign in with Braintrust (US)", + "transport": "mcp_remote", + "auth": "oauth", + "ownershipModes": [ + "dcr" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use browser sign-in for a Braintrust organization hosted in the US.", + "defaults": { + "serverUrl": "https://api.braintrust.dev/mcp", + "discoveryUrl": "https://api.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "scopesHint": [ + "mcp" + ] + }, + "guidanceMd": "Choose the US endpoint for the organization’s data plane region. Sign in with the Braintrust account whose data agents should access. No developer app registration is needed. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4" + }, + { + "key": "mcp-api-key", + "label": "Use a Braintrust API key (US)", + "transport": "mcp_remote", + "auth": "api_key", + "ownershipModes": [ + "customer" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use an API key for a Braintrust organization hosted in the US.", + "defaults": { + "serverUrl": "https://api.braintrust.dev/mcp" + }, + "guidanceMd": "Choose the US endpoint for the organization’s data plane region. Create a key in Braintrust Settings → API keys. Paste the key for the account whose data agents should access. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "keys": "https://www.braintrust.dev/app/~/configuration/org/api-keys" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4", + "credentialFields": [ + { + "key": "authorization", + "label": "Braintrust API key", + "type": "password", + "required": true, + "placeholder": "Paste your Braintrust API key", + "secret": true, + "helperMd": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + } + ], + "keyPlacement": { + "location": "header", + "name": "Authorization", + "prefix": "Bearer " + } + }, + { + "key": "mcp-eu-oauth", + "label": "Sign in with Braintrust (EU)", + "transport": "mcp_remote", + "auth": "oauth", + "ownershipModes": [ + "dcr" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use browser sign-in for a Braintrust organization hosted in the EU.", + "defaults": { + "serverUrl": "https://api-eu.braintrust.dev/mcp", + "discoveryUrl": "https://api-eu.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "scopesHint": [ + "mcp" + ] + }, + "guidanceMd": "Choose the EU endpoint for the organization’s data plane region. Sign in with the Braintrust account whose data agents should access. No developer app registration is needed. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4" + }, + { + "key": "mcp-eu-api-key", + "label": "Use a Braintrust API key (EU)", + "transport": "mcp_remote", + "auth": "api_key", + "ownershipModes": [ + "customer" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use an API key for a Braintrust organization hosted in the EU.", + "defaults": { + "serverUrl": "https://api-eu.braintrust.dev/mcp" + }, + "guidanceMd": "Choose the EU endpoint for the organization’s data plane region. Create a key in Braintrust Settings → API keys. Paste the key for the account whose data agents should access. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "keys": "https://www.braintrust.dev/app/~/configuration/org/api-keys" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4", + "credentialFields": [ + { + "key": "authorization", + "label": "Braintrust API key", + "type": "password", + "required": true, + "placeholder": "Paste your Braintrust API key", + "secret": true, + "helperMd": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + } + ], + "keyPlacement": { + "location": "header", + "name": "Authorization", + "prefix": "Bearer " + } + } + ] +} diff --git a/packages/shared/src/self-serve-mcp-research.json b/packages/shared/src/self-serve-mcp-research.json index 1f2e624aad3..e9f6f621627 100644 --- a/packages/shared/src/self-serve-mcp-research.json +++ b/packages/shared/src/self-serve-mcp-research.json @@ -69,6 +69,7 @@ { "slug": "klaviyo", "name": "Klaviyo", "wave": 2, "status": "self_serve", "docsUrl": "https://developers.klaviyo.com/en/docs/connect_to_the_klaviyo_mcp_server", "serverUrl": "https://mcp.klaviyo.com/mcp?toolsets=campaigns:read,campaigns:write,metrics:read,segments:read", "authMode": "dcr", "prerequisite": "Hosted MCP requires a Klaviyo Owner, Admin, or Manager role. The server's toolsets parameter selects campaign read/write, metrics read, and segment read catalog families; exact authenticated actions still need review.", "riskTier": "S4", "verifiedAt": "2026-10-09", "browserOnboarding": "pending-client-approval", "liveRead": "not-run", "blocker": "Initial client authorization awaits user approval; account and scopes not yet enumerated" }, { "slug": "monday", "name": "monday.com", "wave": 2, "status": "blocked", "docsUrl": "https://developer.monday.com/api-reference/docs/integrate-with-monday-mcp", "serverUrl": "https://mcp.monday.com/mcp", "authMode": "provider_approval", "prerequisite": "Provider DCR rejects the Paperclip HTTPS callback with HTTP 400: redirect_uris are not allowed. Contact monday.com to allowlist the callback before OAuth setup; account-admin approval may also be needed.", "riskTier": "S4", "verifiedAt": "2026-10-09", "browserOnboarding": "blocked-callback-allowlist", "liveRead": "not-run", "blocker": "Provider callback allowlisting required" }, { "slug": "context7", "name": "Context7", "wave": 2, "status": "self_serve", "docsUrl": "https://context7.com/docs/api-guide", "serverUrl": "https://mcp.context7.com/mcp", "authMode": "public", "prerequisite": "No account or API key is required for anonymous remote MCP use; the official docs state anonymous requests have low rate limits. An optional API key raises limits and supports private repositories, but this curated method does not request or store one.", "riskTier": "S1", "verifiedAt": "2026-10-09", "browserOnboarding": "completed-public-no-auth", "liveRead": "passed: resolve-library-id and query-docs returned React public documentation; repeat query passed after reload with 0 agents selected", "liveWrite": "not-applicable", "blocker": null }, - { "slug": "ditto", "name": "Ditto", "wave": 4, "status": "self_serve", "docsUrl": "https://developer.dittowords.com/guides/connect-the-mcp", "serverUrl": "https://api.dittowords.com/v2/mcp", "authMode": "dcr_or_api_key", "prerequisite": "A Ditto account in the workspace whose text and style guides agents should use. API keys require developer integrations on the Ditto plan.", "riskTier": "S4" } + { "slug": "ditto", "name": "Ditto", "wave": 4, "status": "self_serve", "docsUrl": "https://developer.dittowords.com/guides/connect-the-mcp", "serverUrl": "https://api.dittowords.com/v2/mcp", "authMode": "dcr_or_api_key", "prerequisite": "A Ditto account in the workspace whose text and style guides agents should use. API keys require developer integrations on the Ditto plan.", "riskTier": "S4" }, + { "slug": "braintrust", "name": "Braintrust", "wave": 4, "status": "self_serve", "docsUrl": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", "serverUrl": "https://api.braintrust.dev/mcp", "authMode": "dcr_or_api_key", "prerequisite": "A Braintrust account or API key. Select US or EU to match the organization’s data plane region. Account permissions bound access; evaluations and Topics processing may consume credits, dataset edits can permanently delete rows, and automation status can enable retention or delivery.", "riskTier": "S4", "verifiedAt": "2026-10-21", "browserOnboarding": "completed-oauth: US mcp-oauth authorized a temporary personal connection in an isolated instance; authenticated discovery returned 49 tools", "liveRead": "passed: US mcp-oauth bounded list_recent_objects and synthetic dataset sql_query readback through actual process-agent runs and run-scoped gateway sessions", "liveWrite": "passed: US mcp-oauth edit_dataset_rows inserted one synthetic row into a new disposable dataset through an actual process-agent gateway call; row and dataset removed", "blocker": "US API-key and both EU methods remain untested against real accounts. Live token expiry/refresh, provider-side OAuth revocation, and native CLI automatic capability delivery remain unqualified.", "evidenceLimit": "Public regional metadata probes confirmed endpoints, Streamable HTTP, DCR, PKCE S256, refresh-token support, and scope mcp without registration or account authorization. Separate US OAuth account proof passed connect, discovery, bounded read, synthetic write/readback, real process-agent gateway calls, catalog refresh, OAuth reconnect, local removal, post-removal denial, and structural API/log redaction. Token expiry/refresh was not forced; structural redaction did not decrypt tokens for exact-value scanning. The provider reported identity scopes despite the mcp request. See doc/connections/BRAINTRUST.md and braintrust-qualification-2026-10-21.json for method-specific proof and limits; metadata and fixtures do not qualify the remaining methods or lifecycle gaps." } ] } diff --git a/scripts/app-definition-overrides/braintrust.json b/scripts/app-definition-overrides/braintrust.json new file mode 100644 index 00000000000..c7b52e8588b --- /dev/null +++ b/scripts/app-definition-overrides/braintrust.json @@ -0,0 +1,177 @@ +{ + "schemaVersion": 1, + "slug": "braintrust", + "name": "Braintrust", + "description": "Explore AI traces, manage evaluation datasets and scorers, and run Braintrust evaluations.", + "categories": [ + "developer", + "data" + ], + "featured": false, + "branding": { + "logoUrl": "/brands/apps/braintrust.png", + "darkLogoUrl": "/brands/apps/braintrust-dark.png" + }, + "urlPatterns": [ + "https://api.braintrust.dev/*", + "https://api-eu.braintrust.dev/*" + ], + "docsUrl": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "redirectConstraints": "https-or-loopback-http", + "methods": [ + { + "key": "mcp-oauth", + "label": "Sign in with Braintrust (US)", + "transport": "mcp_remote", + "auth": "oauth", + "ownershipModes": [ + "dcr" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use browser sign-in for a Braintrust organization hosted in the US.", + "defaults": { + "serverUrl": "https://api.braintrust.dev/mcp", + "discoveryUrl": "https://api.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "scopesHint": [ + "mcp" + ] + }, + "guidanceMd": "Choose the US endpoint for the organization’s data plane region. Sign in with the Braintrust account whose data agents should access. No developer app registration is needed. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4" + }, + { + "key": "mcp-api-key", + "label": "Use a Braintrust API key (US)", + "transport": "mcp_remote", + "auth": "api_key", + "ownershipModes": [ + "customer" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use an API key for a Braintrust organization hosted in the US.", + "defaults": { + "serverUrl": "https://api.braintrust.dev/mcp" + }, + "guidanceMd": "Choose the US endpoint for the organization’s data plane region. Create a key in Braintrust Settings → API keys. Paste the key for the account whose data agents should access. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "keys": "https://www.braintrust.dev/app/~/configuration/org/api-keys" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4", + "credentialFields": [ + { + "key": "authorization", + "label": "Braintrust API key", + "type": "password", + "required": true, + "placeholder": "Paste your Braintrust API key", + "secret": true, + "helperMd": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + } + ], + "keyPlacement": { + "location": "header", + "name": "Authorization", + "prefix": "Bearer " + } + }, + { + "key": "mcp-eu-oauth", + "label": "Sign in with Braintrust (EU)", + "transport": "mcp_remote", + "auth": "oauth", + "ownershipModes": [ + "dcr" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use browser sign-in for a Braintrust organization hosted in the EU.", + "defaults": { + "serverUrl": "https://api-eu.braintrust.dev/mcp", + "discoveryUrl": "https://api-eu.braintrust.dev/.well-known/oauth-protected-resource/mcp", + "scopesHint": [ + "mcp" + ] + }, + "guidanceMd": "Choose the EU endpoint for the organization’s data plane region. Sign in with the Braintrust account whose data agents should access. No developer app registration is needed. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4" + }, + { + "key": "mcp-eu-api-key", + "label": "Use a Braintrust API key (EU)", + "transport": "mcp_remote", + "auth": "api_key", + "ownershipModes": [ + "customer" + ], + "capabilityProfile": { + "key": "write", + "label": "Read and write", + "description": "Inspect Braintrust data and run evaluations. Access follows your Braintrust account permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + }, + "whenToUse": "Use an API key for a Braintrust organization hosted in the EU.", + "defaults": { + "serverUrl": "https://api-eu.braintrust.dev/mcp" + }, + "guidanceMd": "Choose the EU endpoint for the organization’s data plane region. Create a key in Braintrust Settings → API keys. Paste the key for the account whose data agents should access. Access follows that account’s Braintrust permissions. Review reads, writes, and destructive actions on Permissions after connecting.", + "consoleLinks": { + "docs": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", + "keys": "https://www.braintrust.dev/app/~/configuration/org/api-keys" + }, + "warnings": [ + "Access follows the signed-in Braintrust account or API key. Paperclip action policies do not narrow the provider credential or enforce a separate project boundary.", + "Evaluations, evaluator tests, and Topics processing can consume Braintrust or model credits.", + "Dataset row edits can permanently delete data. Changing automation status can enable retention or delivery automations; review these actions on Permissions." + ], + "riskTier": "S4", + "credentialFields": [ + { + "key": "authorization", + "label": "Braintrust API key", + "type": "password", + "required": true, + "placeholder": "Paste your Braintrust API key", + "secret": true, + "helperMd": "Create a Braintrust API key in Settings → API keys for the account and organization whose data agents should access. The key must authorize the reads and writes you intend to allow; Paperclip cannot increase or narrow its provider permissions. Evaluations and Topics processing can incur costs. Dataset edits can permanently delete rows; automation activation can enable retention or delivery." + } + ], + "keyPlacement": { + "location": "header", + "name": "Authorization", + "prefix": "Bearer " + } + } + ] +} diff --git a/server/src/__tests__/braintrust-connection.test.ts b/server/src/__tests__/braintrust-connection.test.ts new file mode 100644 index 00000000000..0f021200e2b --- /dev/null +++ b/server/src/__tests__/braintrust-connection.test.ts @@ -0,0 +1,252 @@ +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { + agents, approvals, companies, companyMemberships, companySecrets, connectionGrants, + createDb, heartbeatRuns, issues, toolAccessAuditEvents, toolActionRequests, + toolApplications, toolCallEvents, toolConnections, toolOauthStates, +} from "@paperclipai/db"; +import { classifyRisk, toolAccessService, type McpToolDescriptor } from "../services/tool-access.js"; +import { createToolGatewayService } from "../services/tool-gateway.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const actor = { actorType: "user" as const, actorId: "braintrust-reviewer" }; +const redirectUri = "https://paperclip.fixture.test/api/tools/oauth/callback"; +const initialTools: McpToolDescriptor[] = [ + { name: "sql_query", inputSchema: { type: "object", properties: { query: { type: "string" } } }, annotations: { readOnlyHint: true } }, + // Deliberately incorrect provider hints must never downgrade reviewed risk. + { name: "run_eval", inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }, + { name: "edit_dataset_rows", inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }, + { name: "set_automation_status", inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }, +]; + +describe("Braintrust reviewed tool risks", () => { + it.each([ + "sql_query", "infer_schema", "summarize_experiment", "list_recent_objects", "resolve_object", + "generate_permalink", "lookup_users", "lookup_api_keys", "get_trace_work_items", "search_patterns", + "generate_monitor_chart", "list_monitoring_views", "get_monitoring_view", "list_automations", + "list_slack_channels", "get_project_settings", "search_docs", "load_braintrust_skill", + ])("allows only reviewed retrieval %s to be read risk", name => { + expect(classifyRisk({ name }, "braintrust")).toBe("read"); + expect(classifyRisk({ name, annotations: { readOnlyHint: false } }, "braintrust")).toBe("write"); + expect(classifyRisk({ name, annotations: { writeHint: true } }, "braintrust")).toBe("write"); + expect(classifyRisk({ name, annotations: { destructiveHint: true } }, "braintrust")).toBe("destructive"); + }); + + it.each([ + "update_trace_work_report", "new_pattern", "update_pattern", "create_preprocessor", "test_preprocessor_on_trace", + "create_facet", "test_facet_on_trace", "enable_topics_automation", "set_topics_automation", "rewind_topics_automation", + "create_monitoring_view", "update_monitoring_view", "create_log_alert", "create_environment_update_alert", + "create_threshold_alert", "create_scheduled_loop_job", "create_prompt", "create_evaluator", "create_human_review_score", + "test_evaluator", "update_online_scoring_rule", "run_eval", "add_to_dataset", "set_project_default_preprocessor", + "new_provider_tool", "sql_query_v2", "SQL_QUERY", + ])("keeps %s at write risk even when the provider advertises read-only", name => { + expect(classifyRisk({ name, annotations: { readOnlyHint: true } }, "braintrust")).toBe("write"); + expect(classifyRisk({ name, annotations: { destructive: true } }, "braintrust")).toBe("destructive"); + }); + + it.each(["edit_dataset_rows", "set_automation_status"])("keeps %s destructive regardless of hints", name => { + expect(classifyRisk({ name, annotations: { readOnlyHint: true, writeHint: true } }, "braintrust")).toBe("destructive"); + }); +}); + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("Braintrust connection lifecycle and governed gateway", () => { + let db: ReturnType; + let temp: Awaited>; + beforeAll(async () => { + temp = await startEmbeddedPostgresTestDatabase("paperclip-braintrust-"); + db = createDb(temp.connectionString); + }, 30000); + afterAll(async () => { await temp?.cleanup(); }); + + async function fixture(options: { methodKey?: string; personal?: boolean; deferOAuth?: boolean } = {}) { + const methodKey = options.methodKey ?? "mcp-api-key"; + const oauth = methodKey.endsWith("oauth"); + const origin = methodKey.includes("-eu-") ? "https://api-eu.braintrust.dev" : "https://api.braintrust.dev"; + const endpoint = `${origin}/mcp`; + const accessToken = `braintrust-fixture-${randomUUID()}`; + const refreshToken = `braintrust-refresh-${randomUUID()}`; + const [company] = await db.insert(companies).values({ name: "Braintrust fixture", issuePrefix: `BT${randomUUID().slice(0, 6)}` }).returning(); + await db.insert(companyMemberships).values({ companyId: company.id, principalType: "user", principalId: actor.actorId, status: "active", membershipRole: "admin" }); + let tools = [...initialTools]; + let authorized = !oauth; + const request = vi.fn(async (url: string, init: RequestInit) => { + // A region-specific token must never be sent to another region. + expect(new URL(url).origin).toBe(origin); + const headers = new Headers(init.headers); + const body = String(init.body ?? ""); + if (url === endpoint && init.method === "POST") { + if (!authorized || headers.get("authorization") !== `Bearer ${accessToken}`) { + return new Response(null, { status: 401, headers: { "www-authenticate": `Bearer resource_metadata="${origin}/.well-known/oauth-protected-resource/mcp"` } }); + } + const rpc = JSON.parse(body); + if (rpc.method === "initialize") return Response.json({ jsonrpc: "2.0", id: rpc.id, + result: { protocolVersion: "2025-06-18", capabilities: { tools: {} }, serverInfo: { name: "Braintrust fixture", version: "1" } } }); + if (rpc.method === "notifications/initialized") return new Response(null, { status: 202 }); + if (rpc.method === "tools/list") return Response.json({ jsonrpc: "2.0", id: rpc.id, result: { tools } }); + if (rpc.method === "tools/call") return Response.json({ jsonrpc: "2.0", id: rpc.id, + result: { content: [{ type: "text", text: `Executed ${rpc.params.name}` }], structuredContent: { operation: rpc.params.name, reflectedCredential: accessToken } } }); + } + if (url.includes("oauth-protected-resource")) return Response.json({ resource: endpoint, authorization_servers: [origin], scopes_supported: ["mcp"] }); + if (url === `${origin}/.well-known/oauth-authorization-server`) return Response.json({ + issuer: origin, authorization_endpoint: `${origin}/oauth/authorize`, token_endpoint: `${origin}/oauth/token`, + registration_endpoint: `${origin}/oauth/register`, code_challenge_methods_supported: ["S256"], + token_endpoint_auth_methods_supported: ["none"], scopes_supported: ["mcp"], + }); + if (url === `${origin}/oauth/register`) return Response.json({ ...JSON.parse(body), client_id: "braintrust-fixture-client" }); + if (url === `${origin}/oauth/token`) { + authorized = true; + return Response.json({ access_token: accessToken, refresh_token: refreshToken, expires_in: 3600, token_type: "Bearer", scope: "mcp" }); + } + return new Response(null, { status: 404 }); + }); + const service = toolAccessService(db, { remoteHttpRequest: request, remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 }] }); + const connectInput = { galleryKey: "braintrust", connectionMethodKey: methodKey, + ...(options.personal ? { grantKind: "user" as const } : {}), + ...(!oauth ? { credentialValues: { "credentials.authorization": accessToken } } : {}) }; + const connect = () => service.connectGalleryApp(company.id, connectInput, actor); + const signIn = async (connectionId: string) => { + const start = await service.startOAuth(company.id, connectionId, { redirectUri, actor }); + const authorization = new URL(start.authorizationUrl); + expect(start.registrationSource).toBe("dcr"); + expect(authorization.origin).toBe(origin); + expect(authorization.searchParams.get("scope")).toBe("mcp"); + expect(authorization.searchParams.get("code_challenge_method")).toBe("S256"); + expect(authorization.searchParams.get("resource")).toBe(endpoint); + return service.completeOAuthCallback({ state: authorization.searchParams.get("state")!, code: "fixture-code", iss: origin, redirectUri, actor }); + }; + let connected = await connect(); + if (oauth && options.deferOAuth) return { company, service, request, connect, accessToken, endpoint, origin, signIn, connected, + agent: undefined, gateway: undefined, session: undefined, call: undefined, run: undefined, issue: undefined, setTools: undefined }; + if (oauth && !options.deferOAuth) connected = await signIn(connected.connectionId); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Braintrust analyst", role: "engineer", adapterType: "process" }).returning(); + await service.finishGalleryAppConnection(company.id, connected.connectionId, { enabledCatalogEntryIds: connected.catalog.map(entry => entry.id), askFirstCatalogEntryIds: [], access: { agentIds: [agent.id] } }, actor); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Braintrust fixture work", assigneeAgentId: agent.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, + scopeKind: "issue", invocationSource: "on_demand", status: "running", responsibleUserId: actor.actorId, contextSnapshot: { issueId: issue.id } }).returning(); + const gateway = createToolGatewayService(db, { remoteHttpRequest: request, toolActionSigningSecret: "braintrust-fixture-signing-key" }); + const session = await gateway.createSession({ companyId: company.id, agentId: agent.id, runId: run.id, issueId: issue.id }); + const listed = await gateway.listToolsForSession(session.token); + const call = (name: string, parameters = {}) => { + const tool = listed.find(entry => entry.upstreamToolName === name); + expect(tool, name).toBeTruthy(); + return gateway.executeTool({ sessionToken: session.token, tool: tool!.name, parameters, idempotencyKey: randomUUID() }); + }; + return { company, service, request, connect, accessToken, endpoint, origin, signIn, + connected, agent, gateway, session, call, run, issue, setTools: (next: McpToolDescriptor[]) => { tools = next; } }; + } + + it.each([ + ["mcp-api-key", false], ["mcp-eu-api-key", true], ["mcp-oauth", true], ["mcp-eu-oauth", false], + ])("connects %s with the selected credential ownership and run-scoped reads and writes", async (methodKey, personal) => { + const f = await fixture({ methodKey: String(methodKey), personal: Boolean(personal) }); + const connected = f.connected!; + expect(connected.connection.config).toMatchObject({ url: f.endpoint, sourceTemplateKey: "braintrust", connectionMethodKey: methodKey, quarantineNewEntries: true }); + expect(connected.catalog.every(entry => entry.status === "active")).toBe(true); + expect(connected.suggestedDefaults.askFirstRiskLevels).toEqual([]); + const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.connectionId, connected.connectionId)); + expect(grant.kind).toBe(personal ? "user" : "organization"); + for (const ref of grant.credentialSecretRefs) { + const [secret] = await db.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId)); + expect(secret.scope).toBe(personal ? "user" : "company"); + expect(secret.ownerUserId).toBe(personal ? actor.actorId : null); + } + expect(JSON.stringify(await f.call!("sql_query", { query: "SELECT id FROM project_logs LIMIT 1" }))).toContain("Executed sql_query"); + const write = JSON.stringify(await f.call!("run_eval")); + expect(write).toContain("Executed run_eval"); + expect(write).not.toContain(f.accessToken); + const persisted = { connection: await f.service.getConnection(connected.connectionId), + audit: await db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.companyId, f.company.id)), + calls: await db.select().from(toolCallEvents).where(eq(toolCallEvents.companyId, f.company.id)) }; + expect(JSON.stringify(persisted)).not.toContain(f.accessToken); + }); + + it("keeps EU OAuth cancellation resumable with the same region and method", async () => { + const f = await fixture({ methodKey: "mcp-eu-oauth", deferOAuth: true }); + const connected = f.connected!; + const start = await f.service.startOAuth(f.company.id, connected.connectionId, { redirectUri, actor }); + await expect(f.service.completeOAuthCallback({ state: new URL(start.authorizationUrl).searchParams.get("state")!, error: "access_denied", redirectUri, actor })).rejects.toMatchObject({ status: 400 }); + expect(await db.select().from(toolOauthStates).where(eq(toolOauthStates.connectionId, connected.connectionId))).toHaveLength(0); + const resumed = await f.service.connectGalleryApp(f.company.id, { galleryKey: "braintrust", connectionMethodKey: "mcp-eu-oauth", resumeConnectionId: connected.connectionId }, actor); + expect(resumed.connectionId).toBe(connected.connectionId); + expect(resumed.connection.config).toMatchObject({ url: f.endpoint, connectionMethodKey: "mcp-eu-oauth" }); + await f.signIn(connected.connectionId); + expect((await f.service.listCatalog(connected.connectionId)).every(entry => entry.status === "active")).toBe(true); + }); + + it("cleans up vaulted credentials and the new identity after failed discovery", async () => { + const [company] = await db.insert(companies).values({ name: "Braintrust failed setup", issuePrefix: `BF${randomUUID().slice(0, 6)}` }).returning(); + await db.insert(companyMemberships).values({ companyId: company.id, principalType: "user", principalId: actor.actorId, status: "active", membershipRole: "admin" }); + const service = toolAccessService(db, { + remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 }], + remoteHttpRequest: async () => new Response("provider-private-detail", { status: 503 }), + }); + await expect(service.connectGalleryApp(company.id, { galleryKey: "braintrust", connectionMethodKey: "mcp-api-key", grantKind: "user", + credentialValues: { "credentials.authorization": "braintrust-failed-setup-key" } }, actor)).rejects.toMatchObject({ status: 502 }); + for (const table of [toolConnections, toolApplications, connectionGrants, companySecrets]) { + expect(await db.select().from(table).where(eq(table.companyId, company.id))).toHaveLength(0); + } + }); + + it.each(["mcp-api-key", "mcp-eu-oauth"])("preserves policy and quarantines added or changed tools on %s refresh and reconnect", async methodKey => { + const f = await fixture({ methodKey }); + const connected = f.connected!; + const offName = "set_automation_status", askName = "edit_dataset_rows"; + await f.service.finishGalleryAppConnection(f.company.id, connected.connectionId, { + enabledCatalogEntryIds: connected.catalog.filter(entry => entry.toolName !== offName).map(entry => entry.id), + askFirstCatalogEntryIds: connected.catalog.filter(entry => entry.toolName === askName).map(entry => entry.id), access: { agentIds: [f.agent!.id] }, + }, actor); + f.request.mockClear(); + await expect(f.call!(offName)).rejects.toThrow(); + await expect(f.call!(askName)).rejects.toMatchObject({ reasonCode: "approval_required" }); + expect(f.request).not.toHaveBeenCalled(); + const [pending] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.companyId, f.company.id)); + if (pending.approvalId) await db.update(approvals).set({ status: "approved", decidedByUserId: actor.actorId, decidedAt: new Date() }).where(eq(approvals.id, pending.approvalId)); + await f.gateway!.approveActionRequest({ companyId: f.company.id, actionRequestId: pending.id, actor: { userId: actor.actorId } }); + expect(f.request.mock.calls.filter(([, init]) => String(init.body).includes('"method":"tools/call"'))).toHaveLength(1); + f.setTools!([...initialTools.map(tool => tool.name === "sql_query" ? { + ...tool, inputSchema: { type: "object", properties: { query: { type: "string" }, project_id: { type: "string" } } }, + } : tool), { name: "provider_added_tool", annotations: { readOnlyHint: true } }]); + const refreshed = await f.service.refreshCatalog(connected.connectionId, actor, { enableAllByDefault: true }); + expect(refreshed.catalog.find(entry => entry.toolName === "sql_query")?.status).toBe("quarantined"); + expect(refreshed.catalog.find(entry => entry.toolName === "provider_added_tool")).toMatchObject({ riskLevel: "write", status: "quarantined" }); + if (methodKey.endsWith("oauth")) await f.signIn(connected.connectionId); + else await f.service.reconnectGalleryApp(connected.connectionId, f.company.id, { credentialValues: { "credentials.authorization": f.accessToken } }, actor); + const afterReconnect = await f.service.refreshCatalog(connected.connectionId, actor, { enableAllByDefault: true }); + expect(afterReconnect.connection.config.quarantineNewEntries).toBe(true); + expect(afterReconnect.connection.transportConfig.quarantineNewEntries).toBe(true); + expect(afterReconnect.catalog.filter(entry => ["sql_query", "provider_added_tool"].includes(entry.toolName)).every(entry => entry.status === "quarantined")).toBe(true); + f.request.mockClear(); + await expect(f.call!("sql_query")).rejects.toThrow(); + await expect(f.call!(offName)).rejects.toThrow(); + await expect(f.call!(askName, { dataset_id: "new-dataset-after-reconnect" })).rejects.toMatchObject({ reasonCode: "approval_required" }); + expect(f.request).not.toHaveBeenCalled(); + }); + + it("refreshes the EU OAuth grant at its issuer and enforces ownership, company and revocation boundaries", async () => { + const f = await fixture({ methodKey: "mcp-eu-oauth", personal: true }); + const connected = f.connected!; + const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.connectionId, connected.connectionId)); + f.request.mockClear(); + await f.service.refreshOAuthGrantCredentials({ companyId: f.company.id, connectionId: connected.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(f.request.mock.calls.some(([url, init]) => url === `${f.origin}/oauth/token` && new URLSearchParams(String(init.body)).get("grant_type") === "refresh_token")).toBe(true); + expect(JSON.stringify(await f.call!("sql_query"))).not.toContain(f.accessToken); + await expect(f.service.getConnection(connected.connectionId, randomUUID())).rejects.toMatchObject({ status: 404 }); + await expect(f.gateway!.createSession({ companyId: randomUUID(), agentId: f.agent!.id, runId: f.run!.id })).rejects.toThrow(); + await expect(f.service.reconnectGalleryApp(connected.connectionId, f.company.id, { credentialValues: { "credentials.authorization": "other-user-value" } }, { actorType: "user", actorId: "another-user" })).rejects.toThrow(); + // Existing gateway sessions must reject a secret that no longer belongs to + // the selected personal identity; they cannot fall back to company secrets. + const accessRef = grant.credentialSecretRefs.find(ref => ref.configPath === "oauth.access_token")!; + await db.update(companySecrets).set({ ownerUserId: "another-user" }).where(eq(companySecrets.id, accessRef.secretId)); + f.request.mockClear(); + await expect(f.call!("sql_query")).rejects.toMatchObject({ reasonCode: "grant_credential_invalid" }); + expect(f.request).not.toHaveBeenCalled(); + await db.update(companySecrets).set({ ownerUserId: actor.actorId }).where(eq(companySecrets.id, accessRef.secretId)); + await f.service.revokeConnectionGrant(connected.connectionId, grant.id, actor); + await expect(f.call!("sql_query")).rejects.toThrow(); + expect(f.request).not.toHaveBeenCalled(); + await f.service.archiveConnection(connected.connectionId, f.company.id, actor); + expect(await db.select().from(companySecrets).where(and(eq(companySecrets.companyId, f.company.id), eq(companySecrets.ownerUserId, actor.actorId)))).toHaveLength(0); + }); +}); diff --git a/server/src/__tests__/connector-tool-policy.test.ts b/server/src/__tests__/connector-tool-policy.test.ts new file mode 100644 index 00000000000..9c094e3d27a --- /dev/null +++ b/server/src/__tests__/connector-tool-policy.test.ts @@ -0,0 +1,119 @@ +import { randomUUID } from "node:crypto"; +import { companies, createDb } from "@paperclipai/db"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { connectorToolPolicy } from "../services/connectors/tool-policy.js"; +import { DAYTONA_API_URL, DAYTONA_METHOD } from "../services/daytona.js"; +import { classifyRisk, toolAccessService, type McpToolDescriptor } from "../services/tool-access.js"; +import { RAILWAY_MCP_URL } from "../services/railway.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +describe("reviewed connector risk dispatch", () => { + it("keeps destructive provider hints above every reviewed classifier", () => { + for (const [source, name] of [ + ["railway", "list-projects"], ["daytona", "daytona_api_list_sandboxes"], + ["enterpret", "run_graph_query"], ["braintrust", "sql_query"], + ]) { + for (const annotations of [{ destructiveHint: true }, { destructive: true }]) { + expect(classifyRisk({ name, annotations }, source)).toBe("destructive"); + } + } + }); + + it("retains each provider's reviewed annotation precedence", () => { + expect(classifyRisk({ name: "listProjects" }, "railway")).toBe("read"); + expect(classifyRisk({ name: "listProjects", annotations: { writeHint: true } }, "railway")).toBe("write"); + expect(classifyRisk({ name: "redeploy", annotations: { readOnlyHint: true } }, "railway")).toBe("destructive"); + // The native catalog owns this risk; a write hint does not replace it. + expect(classifyRisk({ name: "daytona_api_list_sandboxes", annotations: { writeHint: true } }, "daytona")).toBe("read"); + expect(classifyRisk({ name: "unknown", annotations: { readOnlyHint: true } }, "daytona")).toBe("destructive"); + expect(classifyRisk({ name: "runGraphQuery", annotations: { readOnlyHint: true } }, "enterpret")).toBe("write"); + expect(classifyRisk({ name: "get_organization_details", annotations: { readOnlyHint: true } }, "enterpret")).toBe("read"); + expect(classifyRisk({ name: "edit_dataset_rows", annotations: { readOnlyHint: false } }, "braintrust")).toBe("destructive"); + expect(classifyRisk({ name: "sql_query", annotations: { writeHint: true } }, "braintrust")).toBe("write"); + expect(classifyRisk({ name: "new_tool", annotations: { readOnlyHint: true } }, "braintrust")).toBe("write"); + }); + + it("retains generic risk behavior when the connector has no reviewed override", () => { + expect(classifyRisk({ name: "get_object" }, "custom")).toBe("read"); + expect(classifyRisk({ name: "new_tool", annotations: { readOnlyHint: true } }, "custom")).toBe("read"); + expect(connectorToolPolicy("custom").classifyRisk).toBeUndefined(); + }); +}); + +describe("reviewed connector catalog policies", () => { + it.each(["railway", "enterpret", "daytona", "braintrust"])("declares the initial drift policy for %s", source => { + expect(connectorToolPolicy(source).quarantineNewEntries).toBe(true); + }); + + it.each([undefined, null, "custom", "notion"])("keeps existing initial defaults for %s", source => { + expect(connectorToolPolicy(source).quarantineNewEntries).toBe(false); + }); + + it.each(["https://mcp.railway.com", "https://mcp.railway.com/"])("recognizes the exact Railway endpoint for a custom connection: %s", url => { + const policy = connectorToolPolicy("custom", { transport: "mcp_remote", config: { url } }); + expect(policy.sourceTemplateKey).toBe("railway"); + expect(policy.quarantineNewEntries).toBe(true); + expect(classifyRisk({ name: "redeploy", annotations: { readOnlyHint: true } }, policy.sourceTemplateKey)).toBe("destructive"); + }); + + it.each(["https://mcp.railway.com.evil.test", "https://mcp.railway.com/custom", "https://other.test"])("does not extend Railway's catalog policy to %s", url => { + expect(connectorToolPolicy("custom", { transport: "mcp_remote", config: { url } }).quarantineNewEntries).toBe(false); + expect(connectorToolPolicy("railway", { transport: "mcp_remote", config: { url } }).quarantineNewEntries).toBe(false); + }); + + it("preserves native Daytona review only for the reviewed REST method and endpoint", () => { + const config = { sourceTemplateKey: "daytona", connectionMethodKey: DAYTONA_METHOD, url: DAYTONA_API_URL }; + expect(connectorToolPolicy("daytona", { transport: "rest_api", config }).quarantineNewEntries).toBe(true); + for (const connection of [ + { transport: "mcp_remote", config }, + { transport: "rest_api", config: { ...config, connectionMethodKey: "mcp-oauth" } }, + { transport: "rest_api", config: { ...config, url: "https://other.test" } }, + ]) expect(connectorToolPolicy("daytona", connection).quarantineNewEntries).toBe(false); + }); + + it.each(["enterpret", "braintrust"])("preserves %s review when reconnect resets the saved flag", sourceTemplateKey => { + const policy = connectorToolPolicy(sourceTemplateKey, { + transport: "mcp_remote", config: { sourceTemplateKey, quarantineNewEntries: false }, + }); + expect(policy.quarantineNewEntries).toBe(true); + }); +}); + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("custom connection catalog policy", () => { + let db: ReturnType; + let temp: Awaited>; + beforeAll(async () => { + temp = await startEmbeddedPostgresTestDatabase("paperclip-connector-policy-"); + db = createDb(temp.connectionString); + }, 30000); + afterAll(async () => { await temp?.cleanup(); }); + + it("applies reviewed endpoint risks and keeps changed tools quarantined without enabling a native bridge", async () => { + const [company] = await db.insert(companies).values({ name: "Custom policy fixture", issuePrefix: `CP${randomUUID().slice(0, 6)}` }).returning(); + let tools: McpToolDescriptor[] = [{ name: "redeploy", annotations: { readOnlyHint: true } }]; + const request = vi.fn(async (url: string, init: RequestInit) => { + expect(url.replace(/\/$/, "")).toBe(RAILWAY_MCP_URL); + const rpc = JSON.parse(String(init.body)); + if (rpc.method === "initialize") return Response.json({ jsonrpc: "2.0", id: rpc.id, + result: { protocolVersion: "2025-06-18", capabilities: { tools: {} }, serverInfo: { name: "Policy fixture", version: "1" } } }); + if (rpc.method === "notifications/initialized") return new Response(null, { status: 202 }); + return Response.json({ jsonrpc: "2.0", id: rpc.id, result: { tools } }); + }); + const service = toolAccessService(db, { remoteHttpRequest: request, + remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 }] }); + const actor = { actorType: "user" as const, actorId: "policy-reviewer" }; + const connection = await service.createConnection(company.id, { + name: "Custom reviewed endpoint", transport: "mcp_remote", enabled: true, status: "active", + config: { sourceTemplateKey: "custom", url: RAILWAY_MCP_URL, quarantineNewEntries: true }, + }, actor); + const initial = await service.refreshCatalog(connection.id, actor, { enableAllByDefault: true }); + expect(initial.catalog).toEqual([expect.objectContaining({ toolName: "redeploy", riskLevel: "destructive", status: "active" })]); + tools = [...tools, { name: "provider_added_tool", annotations: { readOnlyHint: true } }]; + const refreshed = await service.refreshCatalog(connection.id, actor, { enableAllByDefault: true }); + expect(refreshed.catalog.find(tool => tool.toolName === "provider_added_tool")).toMatchObject({ riskLevel: "write", status: "quarantined" }); + expect(refreshed.connection.config.quarantineNewEntries).toBe(true); + expect(refreshed.connection.config.sourceTemplateKey).toBe("custom"); + expect(request).toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/connectors/braintrust/risk.ts b/server/src/services/connectors/braintrust/risk.ts new file mode 100644 index 00000000000..ac354c8dc8f --- /dev/null +++ b/server/src/services/connectors/braintrust/risk.ts @@ -0,0 +1,45 @@ +import type { ToolRiskLevel } from "@paperclipai/shared"; + +// Reviewed against Braintrust's hosted MCP tool reference on 2026-10-21: +// https://www.braintrust.dev/docs/integrations/developer-tools/mcp#tools +// Exact names make additions and renames require review. In particular, test +// and evaluation tools can execute code or consume model credits without +// changing a source trace, so readOnlyHint cannot make them reads. +const READ_TOOLS = new Set([ + "sql_query", + "infer_schema", + "summarize_experiment", + "list_recent_objects", + "resolve_object", + "generate_permalink", + "lookup_users", + "lookup_api_keys", + "get_trace_work_items", + "search_patterns", + "generate_monitor_chart", + "list_monitoring_views", + "get_monitoring_view", + "list_automations", + "list_slack_channels", + "get_project_settings", + "search_docs", + "load_braintrust_skill", +]); + +const DESTRUCTIVE_TOOLS = new Set([ + // One action supports insertion, replacement, and permanent deletion. + "edit_dataset_rows", + // Activating an existing automation can apply retention or deliver data. + "set_automation_status", +]); + +export function braintrustRisk(tool: { + name: string; + annotations?: Record; +}): ToolRiskLevel { + const annotations = tool.annotations ?? {}; + if (annotations.destructiveHint === true || annotations.destructive === true + || DESTRUCTIVE_TOOLS.has(tool.name)) return "destructive"; + if (annotations.readOnlyHint === false || annotations.writeHint === true) return "write"; + return READ_TOOLS.has(tool.name) ? "read" : "write"; +} diff --git a/server/src/services/connectors/enterpret/risk.ts b/server/src/services/connectors/enterpret/risk.ts new file mode 100644 index 00000000000..8989ba0da74 --- /dev/null +++ b/server/src/services/connectors/enterpret/risk.ts @@ -0,0 +1,9 @@ +import type { ToolRiskLevel } from "@paperclipai/shared"; + +export function enterpretRisk(tool: { name: string }): ToolRiskLevel | null { + const name = tool.name.replace(/([a-z0-9])([A-Z])/g, "$1-$2") + .toLowerCase().replace(/[:._-]+/g, "-"); + // This tool advertises readOnlyHint, but Cypher can mutate the graph and + // live qualification has not established a read-only execution boundary. + return name === "run-graph-query" ? "write" : null; +} diff --git a/server/src/services/connectors/tool-policy.ts b/server/src/services/connectors/tool-policy.ts new file mode 100644 index 00000000000..6b5cfe63d93 --- /dev/null +++ b/server/src/services/connectors/tool-policy.ts @@ -0,0 +1,64 @@ +import type { ToolRiskLevel } from "@paperclipai/shared"; +import { daytonaRisk, isDaytonaConnection } from "../daytona.js"; +import { classifyRailwayToolRisk, isRailwayEndpoint } from "../railway.js"; +import { braintrustRisk } from "./braintrust/risk.js"; +import { enterpretRisk } from "./enterpret/risk.js"; + +interface ToolDescriptor { + name: string; + annotations?: Record; +} + +interface ConnectionIdentity { + transport: string; + config: Record; +} + +interface ReviewedToolPolicy { + sourceTemplateKey: string; + classifyRisk?: (tool: ToolDescriptor) => ToolRiskLevel | null; + quarantineNewEntries: boolean; + matchesConnection?: (connection: ConnectionIdentity) => boolean; + matchesEndpoint?: (connection: ConnectionIdentity) => boolean; +} + +// Provider selection belongs here; authorization, catalog transitions, and +// operator-selected action policies remain owned by the shared tool service. +const reviewedPolicies: ReviewedToolPolicy[] = [ + { + sourceTemplateKey: "railway", + quarantineNewEntries: true, + classifyRisk: classifyRailwayToolRisk, + matchesConnection: connection => isRailwayEndpoint(connection.config.url), + // A custom connection to the exact reviewed endpoint gets the same risk + // review and drift handling, without enabling the native API bridge. + matchesEndpoint: connection => isRailwayEndpoint(connection.config.url), + }, + { + sourceTemplateKey: "daytona", + quarantineNewEntries: true, + classifyRisk: tool => daytonaRisk(tool.name), + matchesConnection: isDaytonaConnection, + }, + { sourceTemplateKey: "enterpret", quarantineNewEntries: true, classifyRisk: enterpretRisk }, + { sourceTemplateKey: "braintrust", quarantineNewEntries: true, classifyRisk: braintrustRisk }, +]; + +const bySource = new Map(reviewedPolicies.map(policy => [policy.sourceTemplateKey, policy])); + +export function connectorToolPolicy(sourceTemplateKey?: string | null, connection?: ConnectionIdentity): { + sourceTemplateKey: string | null; + classifyRisk?: (tool: ToolDescriptor) => ToolRiskLevel | null; + quarantineNewEntries: boolean; +} { + const policy = (connection && reviewedPolicies.find(candidate => candidate.matchesEndpoint?.(connection))) + || bySource.get(sourceTemplateKey ?? ""); + return { + sourceTemplateKey: policy?.sourceTemplateKey ?? sourceTemplateKey ?? null, + classifyRisk: policy?.classifyRisk, + // Initial setup declares this default before discovery. Subsequent + // refresh/reconnect preserves it only for the reviewed connection identity. + quarantineNewEntries: Boolean(policy?.quarantineNewEntries + && (!connection || !policy.matchesConnection || policy.matchesConnection(connection))), + }; +} diff --git a/server/src/services/railway.ts b/server/src/services/railway.ts index 05052f127b8..15fcf6d6790 100644 --- a/server/src/services/railway.ts +++ b/server/src/services/railway.ts @@ -106,6 +106,13 @@ export function railwayRisk(name: string): "read" | "write" | "destructive" { return "write"; } +export function classifyRailwayToolRisk(tool: { name: string; annotations?: Record }): "read" | "write" | "destructive" { + const risk = railwayRisk(tool.name); + const annotations = tool.annotations ?? {}; + return risk === "read" && (annotations.readOnlyHint === false || annotations.writeHint === true) + ? "write" : risk; +} + export class RailwayError extends Error { constructor(readonly code: string, message: string, readonly status = 502) { super(message); this.name = "RailwayError"; } } diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 8dbc93a84d5..1f5a02b08c4 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -2,6 +2,7 @@ import { getAppDefinitionForApplicationKey, isConnectorFeatureEnabled } from "@p import { stableHash, descriptorHash } from "./tool-catalog-hash.js"; import { assertProfileEntryInputs } from "./tool-profile-entry-validation.js"; import { DAYTONA_TOOLS, daytonaClient, daytonaRisk, isDaytonaConnection, isDaytonaToolName } from "./daytona.js"; +import { connectorToolPolicy } from "./connectors/tool-policy.js"; import { chatCredentialMutationLease } from "./chat-credential-mutation-lease.js"; import { notifyChatEndpointWork } from "./chat-work-notifications.js"; import { removeSlackRegistration } from "./chat-slack-registration-cleanup.js"; @@ -249,7 +250,7 @@ import { connectionCredentialConfigPath as credentialRefConfigPath, connectionGr import { agentmailApi } from "./agentmail-api.js"; import type { ConfigureRailwaySsh, RailwaySshSetup } from "@paperclipai/shared"; import { generateRailwaySshKey, RAILWAY_SSH_SECRET_PATH, validateRailwayKnownHosts } from "./railway-ssh.js"; -import { createRailwayClient, discoverRailwayWorkspace, isRailwayConnection, isRailwayEndpoint, isRailwayToolBlocked, normalizeRailwayToolName, RAILWAY_TOOLS, RAILWAY_TOOL_PREFIX, railwayRisk, RailwayError } from "./railway.js"; +import { createRailwayClient, discoverRailwayWorkspace, isRailwayConnection, isRailwayEndpoint, isRailwayToolBlocked, normalizeRailwayToolName, RAILWAY_TOOLS, RAILWAY_TOOL_PREFIX, RailwayError } from "./railway.js"; import { toolAccessPolicyService } from "./tool-access-policy.js"; import { readSignedToolArgumentsPayload, @@ -2401,6 +2402,8 @@ export function classifyRisk( const annotations = tool.annotations ?? {}; if (annotations.destructiveHint === true || annotations.destructive === true) return "destructive"; + const reviewedRisk = connectorToolPolicy(sourceTemplateKey).classifyRisk?.(tool); + if (reviewedRisk) return reviewedRisk; const normalizedToolName = normalizedProviderToolName(tool.name); if (isMemoryConnectorId(sourceTemplateKey)) { if (verbMatches(tool.name, "delete|remove|destroy|forget|prune|reset")) return "destructive"; @@ -2420,21 +2423,11 @@ export function classifyRisk( if (sourceTemplateKey === "browser-use-cloud") { return BROWSER_USE_TOOLS.find(t => t.name === tool.name)?.annotations.readOnlyHint ? "read" : "destructive"; } - if (sourceTemplateKey === "daytona") return daytonaRisk(tool.name); - if (sourceTemplateKey === "railway") { - const reviewed = railwayRisk(normalizedToolName); - return reviewed === "read" && (annotations.readOnlyHint === false || annotations.writeHint === true) ? "write" : reviewed; - } // Fireflies sharing, moving, and access revocation are mutations even when // a provider omits annotations or mistakenly advertises a read hint. if (sourceTemplateKey === "fireflies" && [ "fireflies-share-meeting", "fireflies-revoke-meeting-access", "fireflies-move-meeting", ].includes(normalizedToolName)) return "write"; - // Enterpret's run_graph_query self-reports readOnlyHint: true, but Cypher is - // not a read-only language and live validation never established the tool as - // safe. Treat it as write so Ask-first defaults can restrict it. - if (sourceTemplateKey === "enterpret" && normalizedToolName === "run-graph-query") - return "write"; // Gmail archive removes a system label. Keep it behind the archive method's // destructive Ask-first default even when the preview provider omits hints. if (sourceTemplateKey === "gmail" && normalizedToolName === "unlabel-thread") @@ -7964,9 +7957,8 @@ export function toolAccessService( isGoogleWorkspaceConnectorProfileId(googleProfileValue) ? googleProfileValue : null; - const preserveReviewedCatalog = - (isRailwayEndpoint(connection.config.url) || sourceTemplateKey === "enterpret" || isDaytonaConnection(connection)) && - existingRows.length > 0; + const connectorPolicy = connectorToolPolicy(sourceTemplateKey, connection); + const preserveReviewedCatalog = connectorPolicy.quarantineNewEntries && existingRows.length > 0; const quarantineOnRefresh = (!refreshOptions.enableAllByDefault || preserveReviewedCatalog) && shouldQuarantineNewEntries(connection) && @@ -7976,7 +7968,7 @@ export function toolAccessService( refreshOptions.quarantineManagedOAuthDraft === true); const safeDefault = asRecord(connection.config).safeDefault === true; for (const descriptor of descriptors) { - const riskLevel = classifyRisk(descriptor, isRailwayEndpoint(connection.config.url) ? "railway" : sourceTemplateKey); + const riskLevel = classifyRisk(descriptor, connectorPolicy.sourceTemplateKey); const hash = descriptorHash(descriptor, riskLevel); const schemaHash = stableHash(descriptor.inputSchema ?? {}); const existing = existingByName.get(descriptor.name); @@ -8043,8 +8035,7 @@ export function toolAccessService( const catalogByName = new Map(catalogRows.map(row => [row.toolName, row])); const updatedEntries = descriptors.map(descriptor => toCatalogEntry(catalogByName.get(descriptor.name)!)); - const preserveQuarantine = - isRailwayEndpoint(connection.config.url) || sourceTemplateKey === "enterpret" || isDaytonaConnection(connection); + const preserveQuarantine = connectorPolicy.quarantineNewEntries; const normalizedConfig = preserveQuarantine ? { ...connection.config, quarantineNewEntries: true } : refreshOptions.enableAllByDefault @@ -12979,7 +12970,7 @@ export function toolAccessService( // Grant-backed setup keeps the full discovered catalog selectable; // the wizard projects the app's action defaults into policies at // finish time instead of using catalog quarantine as access state. - quarantineNewEntries: ["railway", "enterpret", "daytona"].includes(galleryEntry.slug), + quarantineNewEntries: connectorToolPolicy(galleryEntry.slug).quarantineNewEntries, ...(remoteMcpConnector ? { mcpSessionRequired: true, mcpAuthMode: input.authMode ?? "auto", diff --git a/ui/public/brands/apps/braintrust-dark.png b/ui/public/brands/apps/braintrust-dark.png new file mode 100644 index 00000000000..e2accd1e3fe Binary files /dev/null and b/ui/public/brands/apps/braintrust-dark.png differ diff --git a/ui/public/brands/apps/braintrust.png b/ui/public/brands/apps/braintrust.png new file mode 100644 index 00000000000..ca6c2d1c643 Binary files /dev/null and b/ui/public/brands/apps/braintrust.png differ diff --git a/ui/public/brands/apps/manifest.json b/ui/public/brands/apps/manifest.json index a6c9e5c8e88..e77518449c9 100644 --- a/ui/public/brands/apps/manifest.json +++ b/ui/public/brands/apps/manifest.json @@ -56,6 +56,13 @@ "catalogVisible": true, "localAsset": "/brands/apps/box.svg" }, + { + "slug": "braintrust", + "provider": "Braintrust", + "catalogVisible": true, + "localAsset": "/brands/apps/braintrust.png", + "darkAsset": "/brands/apps/braintrust-dark.png" + }, { "slug": "brex", "provider": "Brex", diff --git a/ui/src/pages/apps/AppsConnect.test.tsx b/ui/src/pages/apps/AppsConnect.test.tsx index 71b107f22b7..96d3fe7c6b4 100644 --- a/ui/src/pages/apps/AppsConnect.test.tsx +++ b/ui/src/pages/apps/AppsConnect.test.tsx @@ -67,6 +67,7 @@ const NEON = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "neon")!; const SUPERAGENT = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "superagent")!; const GAUGE = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "gauge")!; const DITTO = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "ditto")!; +const BRAINTRUST = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "braintrust")!; const POSTMAN = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "postman")!; const SHOPIFY = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "shopify")!; const GOOGLE_SHEETS = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "google-sheets")!; @@ -2152,6 +2153,65 @@ describe("AppsConnect — Connect with a link (M4 frame)", () => { expect(container.textContent).not.toContain("Pick the app you want your agents to use."); }); + it.each([ + "mcp-oauth", "mcp-api-key", "mcp-eu-oauth", "mcp-eu-api-key", + ])("submits the selected Braintrust region and authentication (%s)", async (methodKey) => { + mockSearch.value = "source=braintrust"; + listGalleryMock.mockResolvedValueOnce({ apps: [BRAINTRUST] }); + const method = BRAINTRUST.methods.find((candidate) => candidate.key === methodKey)!; + await render(); + + expect(buttonByText("Continue to sign in")?.disabled).toBe(false); + expect(container.textContent).toContain("permissions"); + expect(container.textContent).toContain("cost"); + expect(container.textContent).toContain("permanently delete"); + expect(container.querySelector('input[type="password"]')).toBeNull(); + await openAccessAdvanced(); + expect(radioContaining(BRAINTRUST.methods[0]!.label!)?.getAttribute("aria-checked")).toBe("true"); + await act(async () => { radioContaining(method.label!)!.click(); }); + await flushReact(); + + expect(container.textContent).toContain("permissions"); + expect(container.textContent).toContain("cost"); + expect(container.textContent).toContain("permanently delete"); + if (method.auth === "api_key") { + const input = container.querySelector('input[type="password"]')!; + expect(input).toBeTruthy(); + expect(buttonByText("Connect")?.disabled).toBe(true); + await act(async () => setInputValue(input, "braintrust-ui-canary")); + await flushReact(); + } + await act(async () => { buttonByText(method.auth === "oauth" ? "Continue to sign in" : "Connect")!.click(); }); + await flushReact(); + + expect(connectAppMock).toHaveBeenCalledWith("company-1", expect.objectContaining({ + galleryKey: "braintrust", + connectionMethodKey: methodKey, + ...(method.auth === "api_key" ? { credentialValues: { "credentials.authorization": "braintrust-ui-canary" } } : {}), + })); + expect(JSON.stringify({ ...window.sessionStorage, ...window.localStorage })).not.toContain("braintrust-ui-canary"); + }); + + it("resumes the selected Braintrust EU OAuth draft after cancellation", async () => { + const connectionId = "22222222-2222-4222-8222-222222222222"; + mockSearch.value = `source=braintrust&resume=${connectionId}&oauth=denied`; + listGalleryMock.mockResolvedValueOnce({ apps: [BRAINTRUST] }); + listApplicationsMock.mockResolvedValueOnce({ applications: [{ + id: "braintrust-app", status: "draft", metadata: { sourceTemplateKey: "braintrust" }, + }] }); + listConnectionsMock.mockResolvedValueOnce({ connections: [{ + id: connectionId, applicationId: "braintrust-app", authKind: "oauth", credentialPolicy: "per_user", status: "draft", + config: { sourceTemplateKey: "braintrust", connectionMethodKey: "mcp-eu-oauth", url: "https://api-eu.braintrust.dev/mcp" }, transportConfig: {}, + }] }); + startOAuthMock.mockResolvedValueOnce(oauthStartResponse(connectionId, "braintrust", "https://api-eu.braintrust.dev/oauth/authorize")); + await render(); + await act(async () => { buttonByText("Try again")!.click(); }); + await flushReact(); + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).toHaveBeenCalledWith(connectionId, { asCurrentUser: true }); + expect(navigateTopLevelMock).toHaveBeenCalledWith("https://api-eu.braintrust.dev/oauth/authorize"); + }); + it("connects Superagent with only an organization API key", async () => { mockParams.appKey = "superagent"; listGalleryMock.mockResolvedValueOnce({ apps: [SUPERAGENT] });