Files
PaperClipAI/maps/feature-map/connection-setup.md
T
Nicky LeachandPaperclip 190080642e perf(tools): publish catalog refreshes in bounded chunks (#16602)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP discovery publishes tools that setup and access profiles use.
> - Large catalogs currently write one row at a time.
> - Repeated page entries can produce duplicate results, and profile
writes can partly commit.
> - This pull request publishes tools in bounded chunks and makes
profile synchronization atomic.
> - The result supports large catalogs before connector runtime hooks
are enabled.

## Linked Issues or Issue Description

**What happened?**

Catalog refresh performs one database write per tool. Repeated tool
names across discovery pages appear more than once in the result.
Default-profile synchronization uses separate statements, so failure can
leave a profile without its binding. OAuth callbacks also acquire
connection/application locks in the reverse order from publication and
setup, allowing a concurrent refresh to deadlock.

**Expected behavior**

Publish at most 500 tools per catalog statement. Keep one result per
tool name, using its last descriptor. Preserve existing IDs, first-seen
timestamps, disabled actions, and unchanged review decisions. Commit or
roll back default-profile changes together. OAuth persistence must
acquire application then connection locks before secret or connection
writes.

**Steps to reproduce**

1. Refresh a synthetic MCP catalog with 2,500 tools.
2. Return the same tool on two discovery pages with different
descriptions.
3. Fail the profile-binding insert during default-profile creation.
4. Overlap personal, organization, or managed OAuth persistence with
setup finalization or discovery of a new tool.

**Additional context**

This is the first feature extracted from #16503. The separate test-only
maintenance PRs #16601 and #16610 are merged. This PR now targets master
and is the active review. Standing assignments #16603, tool definitions
#16604, and access/credential hooks #16605 remain stacked drafts without
review requests.

## What Changed

- Deduplicate discovery results by tool name and keep the last
descriptor.
- Upsert catalog rows in transactions of at most 500 tools.
- Preserve row identity and operator-disabled status on conflict.
Determine default-access candidates from the rows read under each
publication lock, so an overlapping refresh cannot restore an action
turned Off after another refresh finishes.
- Lock the application before the connection during publication, setup
finalization, removal, and all three OAuth persistence paths. The OAuth
ordering fix is included here so this PR is independently safe; it does
not depend on #16605.
- Synchronize the managed default profile, its entries, bindings, and
managed policies in one transaction.
- Add PostgreSQL publication tests and document partial-failure
recovery.
- Target the existing callback failure injection at profile creation
instead of a transaction ordinal.

## Verification

- `pnpm exec vitest run
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/tool-access-service.test.ts`: 436 tests passed with
the OAuth and overlapping-refresh corrections on top of the merged
#16610 fix.
- The new tests exercise real PostgreSQL with 250 and 2,500 tools,
repeated discovery pages, preserved choices, lock contention, and
failures between chunks and during profile creation.
- Four selected regression cases fail with the production source
restored to the parent commit. The implementation was then restored and
rechecked.
- Six additional real-PostgreSQL regressions cover personal,
organization, and managed OAuth persistence against both setup
finalization and new-tool discovery. All six reproduce PostgreSQL
deadlock `40P01` with the OAuth ordering fix removed; all six pass with
it restored. Discovery uses synthetic credential headers to isolate
publication locking from credential rotation.
- Apex initially returned 3/5 on
`0d6013a9649388041975406fe4b0247cd0e695aa`, identifying an
overlapping-refresh permission race. Two new PostgreSQL regressions
cover both absent and quarantined tools; both prove an agent goes from
deny back to allow without the correction, and remains denied with it.
The test pauses one refresh after its initial catalog snapshot,
completes another refresh, turns the action Off through the service, and
resumes the stale refresh.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `node scripts/check-maps.mjs`: passed (structural validation only).
- `git diff --check`: passed.
- Reviewed the complete diff for secrets and personal data before push.
- Earlier hosted run 38975428064 failed the maximum-size newline
skill-audit watchdog; other interrupted jobs reported runner
shutdown/cancellation. The original watchdog failure did not identify
whether startup or auditing timed out, so its exact cause remains
unknown. The separate merged #16610 isolates the audit deadline from
module startup and preserves the audit hang guard. This branch is
rebased onto that fix.
- Full local build and repository-wide tests were not run. At the
previous head `0d6013a9649388041975406fe4b0247cd0e695aa`, all 55 hosted
checks succeeded (two optional Storybook checks skipped), and standard
Greptile returned 5/5 before Apex identified the permission race above.
At corrected head `dc120d2bc879c54c14347db30e74c753bcff61e6`, all 55
hosted checks succeeded (two optional Storybook checks skipped), and a
fresh standard review returned 5/5 with no open findings. The second
Apex review also returned 5/5 on that exact head (successful Apex check
completed 2026-10-16T23:36:44Z). Both earlier findings are addressed,
there are no open review threads, and the branch is mergeable against
master.

## Risks

Each successful catalog chunk remains committed if a later operation
fails. A retry completes publication. Provider discovery occurs outside
database locks. Default-profile changes commit separately from the
catalog chunks and roll back together on failure. Tests use a controlled
MCP response; they do not qualify a live provider. There is no schema
migration or new plugin API in this PR.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-16 16:40:12 -07:00

185 lines
9.7 KiB
Markdown

---
navigation: {"section":"Apps, channels, and extensions","sectionOrder":50,"order":10}
---
# Connection setup
People connect an app or account, choose who may use it, and recover incomplete
setup from either Apps or a task that needs access. A saved draft, a completed
provider login, a usable connection, and a resumed task are separate checkpoints.
Chat channels additionally require an endpoint and linked sender identity.
Agent access: [companion](../agent-access-map/connection-setup.md).
## Sub-features
- `catalog`: find a provider, inspect its available methods, and open setup;
unavailable methods explain their prerequisite rather than silently switching.
- `identity-and-access`: select personal/shared identity and eligible agents;
an empty explicit selection and company-wide access must not be confused.
- `credentials`: provider sign-in, key, or generic MCP URL follows its supported
setup method; credentials are not retained in browser draft storage.
- `handoff`: provider popup/full-page callback returns to the correct setup
host with the original task intent and identity choice intact.
- `draft-recovery`: cancel, Back, Save & exit, reload, Finish setup, and reconnect
preserve the exact draft rather than creating duplicate connections.
- `task-request`: only the addressed person resolves the requested connection;
usable access releases the pending task request through normal continuation.
- `channel-setup`: provider endpoint, represented agent, linked sender, and
communication instructions are configured before an external message is used.
- `failure`: denied login, unreachable/private URL, expired credentials, and
permission errors leave an actionable state rather than a false success.
## How to get to it (user POV)
### `apps-catalog`
Open Apps (`/apps`), choose an app, and connect from its landing page. Select an
available method and access, then complete setup. Existing connections open
their details; unfinished ones offer a way to finish setup.
### `mcp-link`
Use **Connect with a link** or `/apps/byo` for a generic MCP server. Paste its
URL, select access, and follow the probe's sign-in/key guidance. Provider-specific
catalog setup and a generic endpoint may use different flows.
### `task-request`
Open the task's pending connection request and its setup action as the named
user. Setup can open in a task dialog and may use an OAuth popup. Follow the
request's original link if setup falls back to a standalone page.
### `resume-connection`
From Apps or connection details, use Finish setup, reconnect, or the offered
authentication recovery action. Reload or return from the provider before
checking whether the same saved connection became usable.
### `chat-channel`
With chat connectors enabled, select a supported chat/email provider in Apps
or open `/apps/chat/connect`. Complete that provider's endpoint setup, then
open its settings (`/apps/chat/:endpointId/settings`). Follow identity linking
as the intended sender before messaging the agent externally.
## Driving it with Playwright and the Paperclip CLI
Preconditions: follow the [baseline](./README.md#before-driving-a-journey).
Choose a disposable provider account/resource and record its auth method,
deployment mode, account ownership, and access selection. For live setup, use
the provider's documented prerequisites in the
[connector playbook](../../doc/connections/CONNECTOR-PLAYBOOK.md). Automated fixtures
do not establish OAuth registration, public callbacks, or production provider access.
### `apps-catalog`
Automated: [Browse](../../ui/src/pages/apps/Browse.test.tsx),
[unconnected app](../../ui/src/pages/apps/AppNotConnected.test.tsx), and
[AppsConnect](../../ui/src/pages/apps/AppsConnect.test.tsx) cover catalog entry,
access selection, provider-specific method choices, and setup failures with
mocked APIs. [Connection intents](../../tests/e2e/connection-intents.spec.ts)
drives store and task setup against one fake provider through a useful continuation:
```sh
pnpm exec vitest run ui/src/pages/apps/Browse.test.tsx ui/src/pages/apps/AppsConnect.test.tsx
pnpm exec playwright test -c tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts
```
Manual: connect from an unconnected provider landing. Select the intended
identity and agent access, complete authentication, reload, and inspect the
saved connection. Run a permitted read-only action with an eligible agent and
verify its useful result. Try an ineligible agent and confirm denial. Repeat
with a user who cannot install company-wide; the UI must not imply broad access.
Daytona follows this same key/access wizard. Its native method exposes the
published platform and Toolbox HTTP actions using provider sandbox IDs, without
a Paperclip task or lease requirement. [Daytona browser setup](../../tests/e2e/daytona-connector.spec.ts)
checks the catalog, password field, access controls, and optional organization
pin. It does not submit a key or qualify a real provider account. See
[Daytona](../../doc/connections/DAYTONA.md) for API-key authority and OAuth limits.
### `mcp-link`
Automated: AppsConnect's generic MCP cases cover probe-driven auth, private or
unreachable endpoints, preregistered clients, secret-backed headers, and unsafe
authorization URLs. These are component/API mocks, not remote provider proof.
[Catalog publication](../../server/src/__tests__/tool-catalog-publication.test.ts)
uses PostgreSQL and a controlled MCP response to check bounded writes, repeated
page entries, preserved action choices, rollback, and concurrent publication.
It does not verify a live provider's discovery contract.
Manual: connect a known test MCP URL through the link flow, verify its saved
endpoint and a read-only action, then repeat with a bad URL and wrong credential.
The error should explain the next action and preserve non-secret setup context.
Test no-auth and authenticated endpoints separately; never infer one from the other.
### `task-request`
Automated: [interaction card](../../ui/src/components/IssueThreadInteractionCard.test.tsx)
checks addressed-user controls. AppsConnect tests cover page/dialog parity,
task-bound OAuth, popup handoff, and verifying a saved connection before closing.
Withdrawn requests render a terminal acknowledgement with no setup action; the
interaction-body test and narrow Storybook journey cover this recovery state.
The connection-intents browser suite above exercises fixture-backed continuation.
Manual: ask an agent to use an unavailable app, open its request as the named
user, and complete setup in the dialog. Verify the card settles, the exact task
resumes, and the agent returns a useful provider result. Repeat the page-host
fallback and denied OAuth path. As another user, confirm that merely seeing the
task does not grant the right to resolve the request.
### `resume-connection`
Automated: AppsConnect tests cover exact-draft resume, declined OAuth, reconnect
lookup failure/retry, response-lost creation, archived connection revival, and
saved account reuse. [App detail](../../ui/src/pages/apps/AppDetail.test.tsx) covers
the connection host. These tests do not complete a real provider login.
Manual: leave setup through its footer, reload Apps, choose Finish setup, and
confirm the same connection is resumed. Decline provider login and recover from
that checkpoint. Reconnect an expired disposable account and verify a permitted
action afterwards. Count connections before/after to catch accidental duplicates.
### `chat-channel`
Automated: [setup routing](../../ui/src/pages/apps/chat/ChatEndpointSetup.routing.test.tsx),
[saved setup state](../../ui/src/pages/apps/chat/ChatEndpointSetup.state.test.ts),
[email setup](../../ui/src/pages/apps/chat/EmailEndpointSetup.test.tsx), and
[identity confirmation](../../ui/src/pages/apps/chat/ChatIdentityConfirm.test.tsx)
cover shared UI logic with fixtures. Provider-specific live setup is not implied.
[Channel integration](../../server/src/__tests__/chat-channels.integration.test.ts)
also covers bounded Telegram subscription recovery at startup and after committed
configuration changes, saved retry deadlines, rollback, and parking pending
repair while an endpoint or connection is inactive. The provider responses are
fixtures. Settled maintenance leaves no discovery timer or database scans.
Manual: complete setup for one named provider, reload endpoint settings, link
the intended sender, and send a disposable message. Verify one task and a useful
reply in the originating conversation. Inspect an unlinked sender's denial and
the endpoint's pause behavior. Repeat independently for each provider at issue;
Slack setup does not qualify Discord, Telegram, GitHub, email, or iMessage.
## Gotchas
- Private endpoint rejection depends on deployment: owned loopback MCP servers
are supported in local-trusted/private instances. Public authenticated
deployments apply the private-network restriction.
- A successful OAuth callback is not proof of saved connection access or task
continuation. Verify all three checkpoints.
- Personal identity, shared account access, and per-agent access are distinct.
Connecting an account does not waive an Off action or Ask first review.
- A tool app connection is not a chat endpoint. The same provider can offer both.
- Old Apps/Connections URLs can redirect to Apps. Use the current navigation and
record the actual destination rather than documenting an obsolete screen.
- Full provider matrix, AI-account selection, gateway/profile administration,
and account revocation lifecycles remain outside this seed recipe.
- For governed actions after setup, use [questions and approvals](./questions-and-approvals.md).
### Documentation scope: App catalog and connection setup
Provider-specific authentication/revocation, credential-only accounts, and aggregator administrative variants still need their own detailed matrices.