mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-18 14:06:27 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connections give agents governed access to external services. > - Native connections should remain the first choice for a supported app. > - Other apps may be available through an external MCP provider. > - The user must know which external provider handles the connection and choose it before setup. > - This pull request adds ranked alternatives and server-authored instructions to connection search. > - Agents can follow the returned instructions while Paperclip validates saved choices and access. ## Linked Issues or Issue Description Related: #13879, which fixed inline MCP provider setup. This PR adds discovery and provider selection on top of that work. **Subsystem affected** Cross-cutting: shared connection contracts, server search and intent services, native runtime, CLI, inline setup UI, and evals. **Problem or motivation** An agent cannot offer a clear external-provider choice when Paperclip has no native connection for an app. Adding provider-specific branches to the core prompt would make those instructions harder to maintain. **Proposed solution** Prefer a native connection. Otherwise return verified alternatives in Composio, Arcade, Executor, Zapier order. Include an external-service disclosure, a question with None, and the next instruction in the search result. Validate the saved human choice before creating a selected fallback setup card. Reuse existing provider accounts and verify underlying app access separately. **Alternatives considered** Do not silently choose a provider. Do not claim that broad execution tools prove support for every app. Reuse existing questions and connection intents rather than add another connection model. **Roadmap alignment** Extends the existing MCP Tool Gateway & Apps and Agent evals & feedback capabilities. The MCP aggregators experiment remains the gate. No duplicate provider-routing PR was found in the public search. ## What Changed - Add a dated support index and authorized cached-tool evidence for external routes. - Return provider questions and next-step instructions from `connections_search`. - Preserve pending choices and declines across continuation. Validate company, task, agent, human, app, and current route eligibility. - Carry the selected app into new setup and account reuse, validate explicit provider requests against persisted human messages, and distinguish provider readiness from app authorization. - Sync native, MCP, REST, and CLI contracts. Keep core agent instructions provider-neutral. - Add production-component Storybooks, focused database tests, and three real-agent browser eval cases. - Record the plan, observed failures, fixes, passing evidence, and acceptance limits. ## Verification - Latest head `586f0e6cd`: 54 checks passed, 2 skipped; Greptile 5/5 and all review threads resolved. - After rebasing on master `18dac1e1e`: 64 focused shared, validator, route-contract, and database tests passed; server typecheck passed. - Embedded-browser test drive on the rebased head: native Jira card, HubSpot external-provider question, Arcade account reuse, one actual MCP read against a local synthetic fixture, reload persistence, and None preventing further calls. A real OpenAI-backed agent performed discovery and continuation. - UX observation: the agent initially combined mutually exclusive request fields; the server rejected it and the agent recovered without changing access. This extra retry remains visible in the transcript. - After rebase: 23 focused eval grader/catalog tests, affected TypeScript checks, token gates, production UI build, and Storybook build passed. Full local tests are intentionally excluded at the maintainer's request. - Before rebase: four browser/real-agent attempts passed: native Jira, None, and reuse of the second provider on two Codex profiles. - Browser evals used an isolated deterministic MCP fixture through the real Paperclip gateway. They do not prove production compatibility with all four providers. - Review `Apps / Connections / Provider choice` in Storybook. Choose Arcade, continue through Access, and verify the app name, external-service disclosure, and URL configuration. - The detailed verification report is `doc/connections/2026-09-23-aggregator-routing-verification.md`. ## Risks - The public support index is finite and can age. Account capability and app authorization still require verification after selection. - Existing installed-tool permissions remain in effect. Provider choice is not a new execution permission boundary. An early Mini attempt skipped search; clearer provider-neutral instructions made the targeted rerun pass. This is not a measured reliability rate. - Explicit requests skip provider confirmation only when a clear persisted human message or saved provider choice supports them. Other phrasing falls back to confirmation; the agent query alone is not consent. These routes do not add tool permissions. - No database migration or legacy Composio broker is added. Real-provider acceptance remains separate from fixture proof. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The exact deployment variant and context-window size are not exposed in this session. Product evals separately used the repository's primary Codex and Codex Mini profiles; those agents supplied test behavior, not independent provider compatibility proof. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
100 lines
3.5 KiB
TypeScript
100 lines
3.5 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import {
|
|
AGGREGATOR_PRIORITY,
|
|
AGGREGATOR_SUPPORT_INDEX,
|
|
explicitAggregatorQuery,
|
|
findAggregatorService,
|
|
parseAggregatorRoute,
|
|
aggregatorProviderQuestion,
|
|
aggregatorContinuationInstruction,
|
|
} from "./connection-routing.js";
|
|
import type { ConnectionSearchResultItem } from "./types/connection-intent.js";
|
|
|
|
describe("aggregator routing contract", () => {
|
|
it("ranks providers and resolves aliases without fuzzy service claims", () => {
|
|
expect(AGGREGATOR_PRIORITY).toEqual([
|
|
"composio",
|
|
"arcade",
|
|
"executor",
|
|
"zapier",
|
|
]);
|
|
expect(findAggregatorService(" Hub Spot ")?.slug).toBe("hubspot");
|
|
expect(findAggregatorService("my unknown crm")).toBeUndefined();
|
|
});
|
|
it("rejects forged or unbounded route identifiers", () => {
|
|
expect(parseAggregatorRoute("via:arcade:hubspot")).toEqual({
|
|
provider: "arcade",
|
|
targetService: "hubspot",
|
|
});
|
|
for (const value of [
|
|
"via:other:hubspot",
|
|
"via:composio:https://evil",
|
|
"via:composio:../secret",
|
|
"via:executor:",
|
|
])
|
|
expect(parseAggregatorRoute(value)).toBeNull();
|
|
});
|
|
it("always discloses external handling and offers None even with one provider", () => {
|
|
const results = [
|
|
{
|
|
service: "via:composio:hubspot",
|
|
aggregator: { provider: "composio" },
|
|
reason: "Verify app authorization",
|
|
},
|
|
] as ConnectionSearchResultItem[];
|
|
const question = aggregatorProviderQuestion("hubspot", "HubSpot", results);
|
|
expect(question.prompt).toContain("external service");
|
|
expect(question.prompt).toContain("handle the connection and requests");
|
|
expect(question.options.map((option) => option.id)).toEqual([
|
|
"via:composio:hubspot",
|
|
"none",
|
|
]);
|
|
expect(question.selectionMode).toBe("single");
|
|
expect(question.allowOther).toBe(false);
|
|
});
|
|
it.each(AGGREGATOR_PRIORITY)(
|
|
"does not claim underlying app readiness for %s",
|
|
(provider) => {
|
|
expect(aggregatorContinuationInstruction(provider, "HubSpot")).toContain(
|
|
"HubSpot access is not yet verified",
|
|
);
|
|
expect(aggregatorContinuationInstruction(provider, "HubSpot")).toContain(
|
|
"ask before switching providers",
|
|
);
|
|
},
|
|
);
|
|
});
|
|
|
|
it("recognizes a service in capability searches without guessing ambiguous or partial names", () => {
|
|
expect(findAggregatorService("HubSpot recent contacts")?.slug).toBe(
|
|
"hubspot",
|
|
);
|
|
expect(
|
|
findAggregatorService("connect atlassian jira and read issues")?.slug,
|
|
).toBe("jira");
|
|
expect(
|
|
findAggregatorService("HubSpot and Salesforce contacts"),
|
|
).toBeUndefined();
|
|
expect(findAggregatorService("notionally similar")).toBeUndefined();
|
|
});
|
|
|
|
it("keeps dates per app/provider claim and parses only an unambiguous explicit provider", () => {
|
|
for (const app of AGGREGATOR_SUPPORT_INDEX)
|
|
for (const date of Object.values(app.providers))
|
|
expect(date).toMatch(/^\d{4}-\d{2}-\d{2}$/);
|
|
expect(explicitAggregatorQuery("HubSpot through Arcade")).toEqual({
|
|
provider: "arcade",
|
|
serviceQuery: "HubSpot",
|
|
});
|
|
expect(
|
|
explicitAggregatorQuery("HubSpot via Arcade or via Composio"),
|
|
).toBeNull();
|
|
expect(explicitAggregatorQuery("arcade games")).toBeNull();
|
|
expect(explicitAggregatorQuery("HubSpot via Arcade or Composio")).toBeNull();
|
|
expect(explicitAggregatorQuery("Jira via Arcade, not Zapier")).toEqual({
|
|
provider: "arcade",
|
|
serviceQuery: "Jira",
|
|
});
|
|
expect(explicitAggregatorQuery("Jira via Arcade, not Arcade")).toBeNull();
|
|
});
|