Files
PaperClipAI/packages/shared/src/connection-routing.test.ts
T
DottaandPaperclip aa8fc86331 feat(connections): prefer native apps and ask users to choose external providers (#13941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections give agents governed access to external services.
> - Native connections should remain the first choice for a supported
app.
> - Other apps may be available through an external MCP provider.
> - The user must know which external provider handles the connection
and choose it before setup.
> - This pull request adds ranked alternatives and server-authored
instructions to connection search.
> - Agents can follow the returned instructions while Paperclip
validates saved choices and access.

## Linked Issues or Issue Description

Related: #13879, which fixed inline MCP provider setup. This PR adds
discovery and provider selection on top of that work.

**Subsystem affected**

Cross-cutting: shared connection contracts, server search and intent
services, native runtime, CLI, inline setup UI, and evals.

**Problem or motivation**

An agent cannot offer a clear external-provider choice when Paperclip
has no native connection for an app. Adding provider-specific branches
to the core prompt would make those instructions harder to maintain.

**Proposed solution**

Prefer a native connection. Otherwise return verified alternatives in
Composio, Arcade, Executor, Zapier order. Include an external-service
disclosure, a question with None, and the next instruction in the search
result. Validate the saved human choice before creating a selected
fallback setup card. Reuse existing provider accounts and verify
underlying app access separately.

**Alternatives considered**

Do not silently choose a provider. Do not claim that broad execution
tools prove support for every app. Reuse existing questions and
connection intents rather than add another connection model.

**Roadmap alignment**

Extends the existing MCP Tool Gateway & Apps and Agent evals & feedback
capabilities. The MCP aggregators experiment remains the gate. No
duplicate provider-routing PR was found in the public search.

## What Changed

- Add a dated support index and authorized cached-tool evidence for
external routes.
- Return provider questions and next-step instructions from
`connections_search`.
- Preserve pending choices and declines across continuation. Validate
company, task, agent, human, app, and current route eligibility.
- Carry the selected app into new setup and account reuse, validate
explicit provider requests against persisted human messages, and
distinguish provider readiness from app authorization.
- Sync native, MCP, REST, and CLI contracts. Keep core agent
instructions provider-neutral.
- Add production-component Storybooks, focused database tests, and three
real-agent browser eval cases.
- Record the plan, observed failures, fixes, passing evidence, and
acceptance limits.

## Verification

- Latest head `586f0e6cd`: 54 checks passed, 2 skipped; Greptile 5/5 and
all review threads resolved.

- After rebasing on master `18dac1e1e`: 64 focused shared, validator,
route-contract, and database tests passed; server typecheck passed.
- Embedded-browser test drive on the rebased head: native Jira card,
HubSpot external-provider question, Arcade account reuse, one actual MCP
read against a local synthetic fixture, reload persistence, and None
preventing further calls. A real OpenAI-backed agent performed discovery
and continuation.
- UX observation: the agent initially combined mutually exclusive
request fields; the server rejected it and the agent recovered without
changing access. This extra retry remains visible in the transcript.
- After rebase: 23 focused eval grader/catalog tests, affected
TypeScript checks, token gates, production UI build, and Storybook build
passed. Full local tests are intentionally excluded at the maintainer's
request.
- Before rebase: four browser/real-agent attempts passed: native Jira,
None, and reuse of the second provider on two Codex profiles.
- Browser evals used an isolated deterministic MCP fixture through the
real Paperclip gateway. They do not prove production compatibility with
all four providers.
- Review `Apps / Connections / Provider choice` in Storybook. Choose
Arcade, continue through Access, and verify the app name,
external-service disclosure, and URL configuration.
- The detailed verification report is
`doc/connections/2026-09-23-aggregator-routing-verification.md`.

## Risks

- The public support index is finite and can age. Account capability and
app authorization still require verification after selection.
- Existing installed-tool permissions remain in effect. Provider choice
is not a new execution permission boundary. An early Mini attempt
skipped search; clearer provider-neutral instructions made the targeted
rerun pass. This is not a measured reliability rate.
- Explicit requests skip provider confirmation only when a clear
persisted human message or saved provider choice supports them. Other
phrasing falls back to confirmation; the agent query alone is not
consent. These routes do not add tool permissions.
- No database migration or legacy Composio broker is added.
Real-provider acceptance remains separate from fixture proof.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser tools. The exact deployment variant and
context-window size are not exposed in this session. Product evals
separately used the repository's primary Codex and Codex Mini profiles;
those agents supplied test behavior, not independent provider
compatibility proof.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-24 16:36:49 -05:00

100 lines
3.5 KiB
TypeScript

import { describe, expect, it } from "vitest";
import {
AGGREGATOR_PRIORITY,
AGGREGATOR_SUPPORT_INDEX,
explicitAggregatorQuery,
findAggregatorService,
parseAggregatorRoute,
aggregatorProviderQuestion,
aggregatorContinuationInstruction,
} from "./connection-routing.js";
import type { ConnectionSearchResultItem } from "./types/connection-intent.js";
describe("aggregator routing contract", () => {
it("ranks providers and resolves aliases without fuzzy service claims", () => {
expect(AGGREGATOR_PRIORITY).toEqual([
"composio",
"arcade",
"executor",
"zapier",
]);
expect(findAggregatorService(" Hub Spot ")?.slug).toBe("hubspot");
expect(findAggregatorService("my unknown crm")).toBeUndefined();
});
it("rejects forged or unbounded route identifiers", () => {
expect(parseAggregatorRoute("via:arcade:hubspot")).toEqual({
provider: "arcade",
targetService: "hubspot",
});
for (const value of [
"via:other:hubspot",
"via:composio:https://evil",
"via:composio:../secret",
"via:executor:",
])
expect(parseAggregatorRoute(value)).toBeNull();
});
it("always discloses external handling and offers None even with one provider", () => {
const results = [
{
service: "via:composio:hubspot",
aggregator: { provider: "composio" },
reason: "Verify app authorization",
},
] as ConnectionSearchResultItem[];
const question = aggregatorProviderQuestion("hubspot", "HubSpot", results);
expect(question.prompt).toContain("external service");
expect(question.prompt).toContain("handle the connection and requests");
expect(question.options.map((option) => option.id)).toEqual([
"via:composio:hubspot",
"none",
]);
expect(question.selectionMode).toBe("single");
expect(question.allowOther).toBe(false);
});
it.each(AGGREGATOR_PRIORITY)(
"does not claim underlying app readiness for %s",
(provider) => {
expect(aggregatorContinuationInstruction(provider, "HubSpot")).toContain(
"HubSpot access is not yet verified",
);
expect(aggregatorContinuationInstruction(provider, "HubSpot")).toContain(
"ask before switching providers",
);
},
);
});
it("recognizes a service in capability searches without guessing ambiguous or partial names", () => {
expect(findAggregatorService("HubSpot recent contacts")?.slug).toBe(
"hubspot",
);
expect(
findAggregatorService("connect atlassian jira and read issues")?.slug,
).toBe("jira");
expect(
findAggregatorService("HubSpot and Salesforce contacts"),
).toBeUndefined();
expect(findAggregatorService("notionally similar")).toBeUndefined();
});
it("keeps dates per app/provider claim and parses only an unambiguous explicit provider", () => {
for (const app of AGGREGATOR_SUPPORT_INDEX)
for (const date of Object.values(app.providers))
expect(date).toMatch(/^\d{4}-\d{2}-\d{2}$/);
expect(explicitAggregatorQuery("HubSpot through Arcade")).toEqual({
provider: "arcade",
serviceQuery: "HubSpot",
});
expect(
explicitAggregatorQuery("HubSpot via Arcade or via Composio"),
).toBeNull();
expect(explicitAggregatorQuery("arcade games")).toBeNull();
expect(explicitAggregatorQuery("HubSpot via Arcade or Composio")).toBeNull();
expect(explicitAggregatorQuery("Jira via Arcade, not Zapier")).toEqual({
provider: "arcade",
serviceQuery: "Jira",
});
expect(explicitAggregatorQuery("Jira via Arcade, not Arcade")).toBeNull();
});