mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 07:55:47 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents governed access to external tools. > - Several provider MCP servers lacked a supported catalog entry or failed during setup. > - Real browser tests identified specific registration, session, and form defects. > - This pull request adds seven catalog entries and fixes the shared paths used by eleven verified providers. > - Users can connect these providers through the existing Apps flow and control each action. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Apps catalog and remote MCP setup, discovery, and action tester. **Subsystem affected** Shared app definitions, the server tool services, and the Apps UI. **Current behavior** Seven providers lack a catalog entry. Airtable can select an advertised client-metadata flow that fails. Calendly rejects the registration name. Tavily needs an initialized session before a call. Firecrawl exposes invalid defaults for optional nested form objects. HTTPS setup can display a callback that differs from the server callback. **Proposed behavior** Add Calendly, Exa, Firecrawl, GSC Wizard, Parallel Search, Tavily, and Windsor.ai. Preserve Airtable, Linear, Make, and PostHog. Use reviewed provider options through the shared MCP and OAuth paths. Display the callback supplied by the server. Omit untouched optional object inputs. **Reason and benefit** Eleven providers passed bounded reads through the real Paperclip browser action tester. This PR includes that verified set and its required shared fixes. Unfinished providers remain outside this change. AgentMail keeps its existing integration. **Breaking changes** No database migration. Existing OAuth credentials and action policies keep their ownership and access rules. Airtable's reviewed method re-registers a retained client-metadata binding through DCR. Tavily's reviewed method initializes sessions before dispatch. The existing customer, managed, and Vercel OAuth gateway paths now refresh on upstream 401 and return `oauth_refreshed_retry_required` (409), requiring an explicit caller retry. They do not replay the rejected call automatically; the next invocation initializes a fresh credential-scoped session when required. Related catalog work: #15545. This branch preserves current master catalog entries and does not add another Google Workspace integration. Open and closed provider PRs and public MCP issues were searched. No duplicate for this verified set was found. The change extends the shipped Connected Apps roadmap item. ## What Changed - Add seven catalog entries with official provider branding and reviewed permissions. - Update Airtable, Linear, and Make metadata and show Make in Apps. - Add authoritative provider source overrides that use the existing generation and review checks. - Add the reviewed Airtable DCR option and compatible Calendly registration names. - Initialize Tavily sessions before discovery and calls, including retained connections. Keep credential-scoped session caching and single call dispatch. - Use the server's callback URL in the OAuth setup UI. - Omit untouched optional object defaults; keep supplied empty strings, false, zero, and object values intact in the action tester, with strict required-child validation. - Require an explicit retry after OAuth refresh instead of replaying a tools/call with missing session headers. - Record all eleven successful reads, write policies, auth-method limits, and qualification caveats. Omit credentials and private account data. - Find chat connector cards through catalog search in browser tests, so pagination does not hide Slack or other later entries. ## Verification - Real browser qualification: eleven bounded reads passed. Catalog refresh and reload proof are recorded in `doc/connections/verified-mcp-qualification-2026-10-08.md`. - Provider writes and actual agent-adapter sessions were not run. Alternative documented auth methods and expiry refresh remain unqualified. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Focused shared/catalog tests: 41 passed. Source override tests: 3 passed. Catalog regeneration fixture suite: 9 passed. Latest form regression suites: 31 passed. Gateway suite: 38 passed. Other focused UI suites passed before these last fixes. - UI token gates and both token sync checks: passed. - The catalog regeneration fixture includes the authoritative provider overrides; its full suite passes. The focused OAuth socket case passed on isolated rerun. - Complete local UI suite: 7,866 passed. CLI suite: 511 passed, 6 skipped. Complete shared suite: 892 passed. - The unchanged AgentMail/ClickUp discovery fallback suite passes all 45 cases. An OpenAI login test passed on isolated rerun. - Local broad database coverage is limited by macOS PostgreSQL shared-memory exhaustion (`shmget: No space left on device`, not disk space). The broad run was interrupted after diagnosis; no host settings or other running services were changed. - CI found that the Slack browser test assumed its catalog card was on the first page. The test now uses catalog search; the Slack case passes locally. Adjacent GitHub and iMessage cases could not start locally because embedded PostgreSQL initialization failed before browser assertions. - Final commit [`febe4520e`](https://github.com/paperclipai/paperclip/commit/febe4520e13d4b3a4121eafc3d22540c2b11379d): all 54 checks passed, with none pending or failed. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37847182065) passed typecheck, build, all general and serialized test suites, all eight browser shards, runner checks, canary dry run, and the aggregate verification gate. - Greptile reviewed that exact final commit at 2026-10-08 21:33 UTC and returned 5/5 with no outstanding findings or unresolved threads. - PR UI preview against the existing local test server: Calendly catalog/setup observed; Firecrawl read passed after expanding More options with nested optional inputs untouched. This retest proves frontend behavior, not the revised OAuth backend against a live provider. <details> <summary>Browser evidence</summary>    </details> ## Risks - Provider registration and consent behavior can change. Airtable's DCR option is explicit and keeps the existing issuer, resource, redirect, and PKCE checks. - Tavily uses initialized sessions without automatic call retries. After a successful OAuth refresh following 401, callers now receive a retry-required result. A later explicit retry can still require reauthorization if the provider rejects the refreshed token. Provider writes are not live-qualified. - Optional object cleanup affects the shared action tester. Regression tests cover absent objects, required children, defaults, and populated values. - GSC Wizard's underlying Google data scopes and Google flow completion were not independently verified. Its account reported paid/trial metadata of unknown origin. No purchase was performed. - No database migration, new AgentMail integration, or change to existing action grants is included. ## Model Used OpenAI GPT-6 through Codex assisted with implementation, research, tool use, and code execution. The root backend model ID and context window are not exposed in this session. The cheaper subagents used OpenAI `gpt-6-luna`. No model context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass — targeted checks and complete UI/CLI/shared suites passed; the broad local database run was blocked by the macOS PostgreSQL startup limitation documented above. The full database/workspace suite passed in CI. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
261 lines
9.8 KiB
TypeScript
261 lines
9.8 KiB
TypeScript
import { checkMcpRemoteHeaderName, checkMcpRemoteHeaderValue, mcpRemoteHeaderRejectionMessage } from "@paperclipai/shared";
|
||
import type { GenericMcpAuthMode } from "@paperclipai/shared";
|
||
|
||
/**
|
||
* Logic behind the guided "Connect your own MCP server" flow (PAP-17087).
|
||
*
|
||
* Kept out of the component so the branch an operator lands on — and the
|
||
* corrective advice they get — is directly testable, and so the wizard never has
|
||
* to reason about protocol error codes inline.
|
||
*/
|
||
|
||
export type { GenericMcpAuthMode };
|
||
|
||
export interface CustomHeaderRow {
|
||
/** Stable key so React can track rows across add/remove without reordering values. */
|
||
id: string;
|
||
name: string;
|
||
value: string;
|
||
}
|
||
|
||
let customHeaderRowSeq = 0;
|
||
|
||
/** A blank custom-header row. The id is local only — it never reaches the API. */
|
||
export function newCustomHeaderRow(): CustomHeaderRow {
|
||
customHeaderRowSeq += 1;
|
||
return { id: `header-${customHeaderRowSeq}`, name: "", value: "" };
|
||
}
|
||
|
||
/** The endpoint host, shown prominently so the operator always sees who they are trusting. */
|
||
export function endpointHost(url: string): string | null {
|
||
try {
|
||
return new URL(url).host || null;
|
||
} catch {
|
||
return null;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* The callback URI shown to operators must exactly match the URI the API sends.
|
||
*
|
||
* Xero and a few other OAuth providers reject numeric loopback hosts even though
|
||
* they point at the same machine. The API therefore uses `localhost` for local
|
||
* HTTP OAuth, so the setup form must advertise that same canonical spelling.
|
||
*/
|
||
export function oauthCallbackUrlForBrowser(
|
||
origin: string = window.location.origin,
|
||
serverCallbackUrl?: string,
|
||
): string {
|
||
if (serverCallbackUrl) return serverCallbackUrl;
|
||
const callbackUrl = new URL("/api/tools/oauth/callback", origin);
|
||
const hostname = callbackUrl.hostname.toLowerCase().replace(/^\[|\]$/g, "");
|
||
if (
|
||
callbackUrl.protocol === "http:"
|
||
&& (hostname === "127.0.0.1" || hostname === "::1")
|
||
) {
|
||
callbackUrl.hostname = "localhost";
|
||
}
|
||
return callbackUrl.toString();
|
||
}
|
||
|
||
/**
|
||
* A useful, non-secret default label for an arbitrary endpoint.
|
||
*
|
||
* Keep the port and path: one host commonly serves several MCP endpoints, and
|
||
* collapsing all of them to the hostname makes the second connection fail the
|
||
* company-wide application-name constraint. Query strings are deliberately
|
||
* excluded because remote MCP URLs can carry credentials there.
|
||
*/
|
||
export function defaultGenericMcpName(url: string): string | null {
|
||
try {
|
||
const parsed = new URL(url);
|
||
const host = parsed.host.replace(/^www\./, "");
|
||
const path = parsed.pathname === "/" ? "" : parsed.pathname.replace(/\/+$/, "");
|
||
return `${host}${path}`.slice(0, 160) || null;
|
||
} catch {
|
||
return null;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Where the operator has to look to fix a failed connect attempt. The wizard uses
|
||
* this to decide whether to reopen the URL field or the credential section, rather
|
||
* than leaving them to guess which of the two was wrong.
|
||
*/
|
||
export type GenericConnectFocus = "url" | "credentials" | "deployment" | "none";
|
||
|
||
export interface GenericConnectGuidance {
|
||
title: string;
|
||
body: string;
|
||
focus: GenericConnectFocus;
|
||
}
|
||
|
||
/**
|
||
* Turn an API failure into the specific corrective action the plan asks for:
|
||
* invalid, unsafe, unreachable, or "needs a credential we could not discover".
|
||
*
|
||
* Anything unrecognised falls through to the server's own message, which is
|
||
* already UI-safe (upstream bodies and secrets are stripped server-side).
|
||
*/
|
||
export function genericConnectGuidance(
|
||
code: string | null | undefined,
|
||
message: string | null | undefined,
|
||
): GenericConnectGuidance {
|
||
const fallback = message?.trim() || "Paperclip couldn't connect to that address. Check it and try again.";
|
||
switch (code) {
|
||
case "mcp_remote_url_missing":
|
||
case "mcp_remote_url_invalid":
|
||
return {
|
||
title: "That doesn't look like a server address",
|
||
body: "Paste the full address, starting with https:// — for example https://mcp.example.com/mcp.",
|
||
focus: "url",
|
||
};
|
||
case "remote_http_private_endpoint":
|
||
return {
|
||
title: "That address is inside a private network",
|
||
body: "This Paperclip is reachable from the internet, so it won't call addresses on your local network. Use the server's public address instead.",
|
||
focus: "url",
|
||
};
|
||
case "remote_http_dns_failed":
|
||
return {
|
||
title: "We couldn't find that host",
|
||
body: "The address didn't resolve. Check the spelling, or confirm the server is published on the internet.",
|
||
focus: "url",
|
||
};
|
||
case "mcp_header_rejected":
|
||
return {
|
||
title: "Paperclip can't send that header",
|
||
body: fallback,
|
||
focus: "credentials",
|
||
};
|
||
case "tool_access_name_conflict":
|
||
return {
|
||
title: "Paperclip couldn’t name this connection",
|
||
body: "Try connecting again.",
|
||
focus: "none",
|
||
};
|
||
case "oauth_challenge":
|
||
return {
|
||
title: "This server wants a credential",
|
||
body: "It asked us to authenticate but didn't offer a sign-in Paperclip can complete on its own. Add the key or headers its docs list under Advanced authentication.",
|
||
focus: "credentials",
|
||
};
|
||
case "oauth_manual_client_required":
|
||
case "oauth_manual_client_rebinding_required":
|
||
return {
|
||
title: "This server needs sign-in details you create yourself",
|
||
body: "Register Paperclip in the provider's settings, then add the client ID and secret it gives you under Advanced authentication.",
|
||
focus: "credentials",
|
||
};
|
||
case "oauth_redirect_origin_unsupported":
|
||
case "oauth_redirect_uri_invalid":
|
||
return {
|
||
title: "This Paperclip needs a public HTTPS address first",
|
||
body: "Sign-in sends the operator back to Paperclip, so this instance has to be reachable over HTTPS. Ask your Paperclip admin to configure it.",
|
||
focus: "deployment",
|
||
};
|
||
case "runtime_error":
|
||
return {
|
||
title: "We couldn't reach that server",
|
||
body: "Nothing answered at that address. Confirm the server is running and the address is right, then try again.",
|
||
focus: "url",
|
||
};
|
||
default:
|
||
return {
|
||
title: "Paperclip couldn't connect",
|
||
body: fallback,
|
||
focus: "none",
|
||
};
|
||
}
|
||
}
|
||
|
||
/**
|
||
* First client-side complaint about the custom headers, or null when they're fine.
|
||
* The API validates these too; checking here means the operator finds out before a
|
||
* round trip rather than after one.
|
||
*/
|
||
export function customHeaderError(rows: CustomHeaderRow[]): string | null {
|
||
const seen = new Set<string>();
|
||
for (const row of rows) {
|
||
const name = row.name.trim();
|
||
if (!name && !row.value.trim()) continue;
|
||
const nameCheck = checkMcpRemoteHeaderName(name);
|
||
if (!nameCheck.ok) return mcpRemoteHeaderRejectionMessage(name, nameCheck.reason!);
|
||
const valueCheck = checkMcpRemoteHeaderValue(row.value);
|
||
if (!valueCheck.ok) return mcpRemoteHeaderRejectionMessage(name, valueCheck.reason!);
|
||
if (!row.value.trim()) return `Add a value for "${name}", or remove the row.`;
|
||
const lower = name.toLowerCase();
|
||
if (seen.has(lower)) return `"${name}" is listed twice.`;
|
||
seen.add(lower);
|
||
}
|
||
return null;
|
||
}
|
||
|
||
export interface GenericConnectDraft {
|
||
link: string;
|
||
name: string;
|
||
authMode: GenericMcpAuthMode;
|
||
/** Simple path: the operator answered "Yes" to "does it need a key?". */
|
||
needsKey: boolean;
|
||
keyValue: string;
|
||
headers: CustomHeaderRow[];
|
||
oauthClientId: string;
|
||
oauthClientSecret: string;
|
||
}
|
||
|
||
export interface GenericConnectPayload {
|
||
link: string;
|
||
name?: string;
|
||
authMode?: GenericMcpAuthMode;
|
||
credentialValues?: Record<string, string>;
|
||
oauthClient?: { clientId: string; clientSecret?: string };
|
||
}
|
||
|
||
/**
|
||
* Build the connect request from wizard state.
|
||
*
|
||
* `authMode` is only sent when the operator made an explicit choice under Advanced
|
||
* authentication; the simple path leaves it off so the server probes and decides.
|
||
* Header values become `headers.<Name>` credential paths, which the server turns
|
||
* into Paperclip secrets — nothing here ever puts a value in the config.
|
||
*/
|
||
export function genericConnectPayload(draft: GenericConnectDraft): GenericConnectPayload {
|
||
const credentialValues: Record<string, string> = {};
|
||
const bearerKey = draft.keyValue.trim();
|
||
const useBearer = draft.authMode === "bearer" || (draft.authMode === "auto" && draft.needsKey);
|
||
if (useBearer && bearerKey) credentialValues["credentials.authorization"] = bearerKey;
|
||
if (draft.authMode === "custom_headers") {
|
||
for (const row of draft.headers) {
|
||
const name = row.name.trim();
|
||
if (!name || !row.value.trim()) continue;
|
||
credentialValues[`headers.${name}`] = row.value;
|
||
}
|
||
}
|
||
const trimmedName = draft.name.trim();
|
||
const clientId = draft.oauthClientId.trim();
|
||
const clientSecret = draft.oauthClientSecret.trim();
|
||
return {
|
||
link: draft.link,
|
||
...(trimmedName ? { name: trimmedName } : {}),
|
||
...(draft.authMode === "auto" ? {} : { authMode: draft.authMode }),
|
||
...(Object.keys(credentialValues).length > 0 ? { credentialValues } : {}),
|
||
...(draft.authMode === "oauth" && clientId
|
||
? { oauthClient: { clientId, ...(clientSecret ? { clientSecret } : {}) } }
|
||
: {}),
|
||
};
|
||
}
|
||
|
||
/** Can "Check link" be pressed? */
|
||
export function canSubmitGenericConnect(draft: GenericConnectDraft): boolean {
|
||
if (!draft.link.trim()) return false;
|
||
if (draft.authMode === "auto") return !draft.needsKey || draft.keyValue.trim().length > 0;
|
||
if (draft.authMode === "bearer") return draft.keyValue.trim().length > 0;
|
||
if (draft.authMode === "custom_headers") {
|
||
const filled = draft.headers.filter((row) => row.name.trim() && row.value.trim());
|
||
return filled.length > 0 && customHeaderError(draft.headers) === null;
|
||
}
|
||
// "oauth" here means the operator is supplying a preregistered client, and
|
||
// "none" means they are asserting the server is public — neither needs a value.
|
||
return true;
|
||
}
|