Files
PaperClipAI/tests/fixtures/connection-review-provider.ts
DottaandPaperclip aa8fc86331 feat(connections): prefer native apps and ask users to choose external providers (#13941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections give agents governed access to external services.
> - Native connections should remain the first choice for a supported
app.
> - Other apps may be available through an external MCP provider.
> - The user must know which external provider handles the connection
and choose it before setup.
> - This pull request adds ranked alternatives and server-authored
instructions to connection search.
> - Agents can follow the returned instructions while Paperclip
validates saved choices and access.

## Linked Issues or Issue Description

Related: #13879, which fixed inline MCP provider setup. This PR adds
discovery and provider selection on top of that work.

**Subsystem affected**

Cross-cutting: shared connection contracts, server search and intent
services, native runtime, CLI, inline setup UI, and evals.

**Problem or motivation**

An agent cannot offer a clear external-provider choice when Paperclip
has no native connection for an app. Adding provider-specific branches
to the core prompt would make those instructions harder to maintain.

**Proposed solution**

Prefer a native connection. Otherwise return verified alternatives in
Composio, Arcade, Executor, Zapier order. Include an external-service
disclosure, a question with None, and the next instruction in the search
result. Validate the saved human choice before creating a selected
fallback setup card. Reuse existing provider accounts and verify
underlying app access separately.

**Alternatives considered**

Do not silently choose a provider. Do not claim that broad execution
tools prove support for every app. Reuse existing questions and
connection intents rather than add another connection model.

**Roadmap alignment**

Extends the existing MCP Tool Gateway & Apps and Agent evals & feedback
capabilities. The MCP aggregators experiment remains the gate. No
duplicate provider-routing PR was found in the public search.

## What Changed

- Add a dated support index and authorized cached-tool evidence for
external routes.
- Return provider questions and next-step instructions from
`connections_search`.
- Preserve pending choices and declines across continuation. Validate
company, task, agent, human, app, and current route eligibility.
- Carry the selected app into new setup and account reuse, validate
explicit provider requests against persisted human messages, and
distinguish provider readiness from app authorization.
- Sync native, MCP, REST, and CLI contracts. Keep core agent
instructions provider-neutral.
- Add production-component Storybooks, focused database tests, and three
real-agent browser eval cases.
- Record the plan, observed failures, fixes, passing evidence, and
acceptance limits.

## Verification

- Latest head `586f0e6cd`: 54 checks passed, 2 skipped; Greptile 5/5 and
all review threads resolved.

- After rebasing on master `18dac1e1e`: 64 focused shared, validator,
route-contract, and database tests passed; server typecheck passed.
- Embedded-browser test drive on the rebased head: native Jira card,
HubSpot external-provider question, Arcade account reuse, one actual MCP
read against a local synthetic fixture, reload persistence, and None
preventing further calls. A real OpenAI-backed agent performed discovery
and continuation.
- UX observation: the agent initially combined mutually exclusive
request fields; the server rejected it and the agent recovered without
changing access. This extra retry remains visible in the transcript.
- After rebase: 23 focused eval grader/catalog tests, affected
TypeScript checks, token gates, production UI build, and Storybook build
passed. Full local tests are intentionally excluded at the maintainer's
request.
- Before rebase: four browser/real-agent attempts passed: native Jira,
None, and reuse of the second provider on two Codex profiles.
- Browser evals used an isolated deterministic MCP fixture through the
real Paperclip gateway. They do not prove production compatibility with
all four providers.
- Review `Apps / Connections / Provider choice` in Storybook. Choose
Arcade, continue through Access, and verify the app name,
external-service disclosure, and URL configuration.
- The detailed verification report is
`doc/connections/2026-09-23-aggregator-routing-verification.md`.

## Risks

- The public support index is finite and can age. Account capability and
app authorization still require verification after selection.
- Existing installed-tool permissions remain in effect. Provider choice
is not a new execution permission boundary. An early Mini attempt
skipped search; clearer provider-neutral instructions made the targeted
rerun pass. This is not a measured reliability rate.
- Explicit requests skip provider confirmation only when a clear
persisted human message or saved provider choice supports them. Other
phrasing falls back to confirmation; the agent query alone is not
consent. These routes do not add tool permissions.
- No database migration or legacy Composio broker is added.
Real-provider acceptance remains separate from fixture proof.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser tools. The exact deployment variant and
context-window size are not exposed in this session. Product evals
separately used the repository's primary Codex and Codex Mini profiles;
those agents supplied test behavior, not independent provider
compatibility proof.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-24 16:36:49 -05:00

88 lines
2.9 KiB
TypeScript

import { createServer, type Server } from "node:http";
import { listenOnFetchAllowedPort } from "../e2e/fetch-allowed-port.js";
export async function startReviewProvider(
successText = "Pages: Roadmap, Meeting notes",
bearerToken?: string,
tool?: { name: string; title: string; description: string },
) {
const captures: Array<{ method: string; toolName: string | null; authorized?: boolean }> = [];
const server: Server = createServer(async (req, res) => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(chunk as Buffer);
const payload = JSON.parse(
Buffer.concat(chunks).toString("utf8") || "{}",
) as {
id?: string | number;
method?: string;
params?: { name?: string; arguments?: { query?: string } };
};
const authorized = !bearerToken || req.headers.authorization === `Bearer ${bearerToken}`;
captures.push({
method: String(payload.method ?? "<unknown>"),
toolName: payload.params?.name ?? null,
...(bearerToken ? { authorized } : {}),
});
if (!authorized) {
res.writeHead(401, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Service credential required" }));
return;
}
res.writeHead(200, { "Content-Type": "application/json" });
if (payload.method === "tools/list") {
res.end(
JSON.stringify({
jsonrpc: "2.0",
id: payload.id ?? null,
result: {
tools: [
{
name: tool?.name ?? "notion:list_pages",
title: tool?.title ?? "List fixture pages",
description: tool?.description ?? "Reads deterministic pages from the fake Notion provider.",
annotations: { readOnlyHint: true },
inputSchema: {
type: "object",
properties: { query: { type: "string" } },
additionalProperties: false,
},
},
],
},
}),
);
return;
}
if (payload.method === "tools/call") {
res.end(
JSON.stringify({
jsonrpc: "2.0",
id: payload.id ?? null,
result: {
isError: payload.params?.arguments?.query === "fail",
content: [
{
type: "text",
text:
payload.params?.arguments?.query === "fail"
? "Fixture provider unavailable"
: successText,
},
],
},
}),
);
return;
}
res.end(
JSON.stringify({ jsonrpc: "2.0", id: payload.id ?? null, result: {} }),
);
});
const port = await listenOnFetchAllowedPort(server);
return {
url: `http://127.0.0.1:${port}/`,
captures,
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
};
}