DottaandPaperclip
a36cbffa9e
## Thinking Path > - Paperclip manages agents and the services they need for work. > - Agents use connection search to discover a setup path before they request access. > - Tool-only filtering hid channel and AI methods from this search. > - Requiring every query word to match rejected normal task descriptions. > - A small aggregator index also omitted supported apps such as Circleback. > - This pull request broadens retrieval and returns purpose-specific setup guidance. > - Agents can choose a relevant result while existing access and provider-choice checks still apply. ## Linked Issues or Issue Description **What happened?** A search such as “AgentMail create an email address and manage an agent mailbox” returned no usable result. “Help me find tools for circle back” also missed Composio's supported Circleback toolkit. Queries longer than 200 characters failed validation. **Expected behavior** Return useful native and verified aggregator matches from natural-language queries. Include channel/email methods when Chat connectors is enabled. Identify each method's purpose and the correct setup path. **Steps to reproduce** Use the queries above with `connections_search` from an active task. Enable Chat connectors for the AgentMail case. The regression suite reproduces these misses before the change. Related routing work: #13941. This change does not change the runner failure path or add channel setup to tool-only connection cards. ## What Changed - Rank name and capability matches. Accept extra words, split names, small spelling errors, and queries up to 4,000 characters. - Include tool, channel/email, and AI methods. Return company-prefix setup links for channel and AI flows. - Add a dated snapshot of 1,583 official Composio toolkit names and a refresh script. Merge duplicate MCP variants for search and link each support claim to official evidence. - Find authorized indexed aggregator namespaces within longer queries. Return multiple app matches when the agent needs to choose. - Prefer exact app names over fuzzy matches for other apps; retain existing AI readiness. - Preserve native preference, company and identity boundaries, administrative denials, and saved provider consent. - Add relevance and database regressions, extend native tool-authority coverage, and document search behavior. ## Verification - Red: 15 new assertions failed against the previous implementation; the existing baseline passed. Added red-green regressions for Motion versus fuzzy Notion and existing AI access during review. A further regression covers mixed ready/unconfigured AI results and their per-result setup guidance. - Green: all 103 tests in the eight focused shared, database, runtime-tool, fixture, and route suites pass on the latest commit. - `pnpm -r typecheck` and `pnpm build` passed. - Latest-commit CI passed: 54 successful checks and two skipped checks, including the full test matrix, browser E2E, typecheck, build, and canary dry run. - The long local `pnpm test:run` attempt began before the review fixes and retained transformed pre-fix search code; it also hit an unrelated timing failure. Fresh serial reruns of the affected search suites and three timeout cases passed all 124 tests. Parallel local route shards hit two additional database setup timeouts; both suites passed all 17 tests on a fresh serial rerun. The complete corresponding CI suites also passed. Duplicate broad local runs were stopped after CI completed. The local UI suite independently passed all 7,007 tests. - Greptile: 5/5 on `cc6a0180d`; all review findings resolved. - Browser verification passed in a disposable local instance through real process-agent search requests: AgentMail opened its setup flow with the requester selected; the saved Circleback choice produced the Composio setup card; a paragraph-length Notion query produced its setup card. No provider credentials or external accounts were created. - The browser test caught an invalid UUID-based setup URL. The fix uses the company prefix and has a regression assertion. - A 3,971-character catalog query found Circleback first in a local 10 ms spot check after sharing query preparation across the catalog scan. This is a single measurement, not a performance guarantee. ## Risks - Broader retrieval can return extra candidates. Named services rank first; agents must select the relevant method. - The public support snapshot can age. It proves catalog support, not account authorization or the availability of every requested action. - Channel and AI methods use existing setup links. The tool connection card still accepts tool methods only. - No schema, migration, credential, or runner lifecycle changes. ## Model Used OpenAI Codex (GPT-6). The session does not expose a more specific model identifier or context-window size. Used reasoning, repository search, code execution, tests, and browser tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
3.6 KiB
Connection search
connections_search accepts a service name or a natural-language query up to
4,000 characters. It ranks names before capabilities, tolerates extra words,
split names such as “Agent Mail,” and single-character spelling errors or
transpositions. Short partial names work too. Capability searches return
overlapping matches without requiring every query word to occur in the catalog.
Results are bounded to 40 entries. Exact app names outrank typo matches for
different apps (for example, Motion is not replaced by Notion). Query preparation
is shared across the catalog scan.
Discovery includes tool, channel/email, and AI methods. Channel methods follow
the instance's Chat connectors experimental setting. Each catalog method names
its purpose. Channel and AI methods include a company-scoped setupPath to
their existing setup flow. connection_request still creates tool setup cards;
channel discovery does not turn an email inbox into an executable MCP tool or
add channel setup to that card.
The agent chooses the relevant match using descriptions and method purposes. It should clarify only when the task remains ambiguous. A search match does not prove that an account is authorized, that a tool is installed, or that a provider can perform every requested action.
Aggregator discovery
The local support index combines reviewed Arcade/Zapier claims with the public Composio toolkit catalog. The Composio snapshot includes names and toolkit identifiers, a source URL, and a verification date. Each result links to its official toolkit page. This covers, for example, Circleback MCP, including queries such as “help me find tools for circle back.”
Refresh the snapshot from the official public page with:
node scripts/update-composio-search-catalog.mjs
The script also accepts a saved HTML file as its first argument for reproducible extraction. Review the resulting diff before committing it. Searches use this local snapshot and authorized indexed tool namespaces; they do not send user queries to a provider. Broad provider descriptions are not evidence of support. Configured connection metadata remains scoped to the current company and identity. Archived and unauthorized catalogs cannot establish an aggregator route.
Built-in connections remain preferred for the same app. Existing AI access is
checked through the agent's AI binding and credential selection. An aggregator route still requires a
saved provider choice or a verified explicit user request. Fuzzy retrieval never
grants consent. A provider connection is not proof that its underlying app is
authorized. Multiple matching aggregator apps are returned as candidates; the
agent searches the selected aggregator.targetService to get its provider question.
Regression coverage
pnpm exec vitest run packages/shared/src/connection-search.test.ts packages/shared/src/connection-routing.test.ts packages/shared/src/validators/connection-intent.test.ts server/src/__tests__/connection-aggregator-fallback.test.ts server/src/__tests__/connection-intents-service.test.ts
The database suite covers the original AgentMail sentence, split names, typos, capability overlap, multiple services, AI discovery, experimental gating, Circleback/Attio/ClickUp, indexed Executor tools, and provider consent. The native tool authority test searches a paragraph longer than the old 200-character limit and then creates a real connection interaction. Existing tests retain checks for private metadata, administrative denials, stale identities, and saved declines.