Nicky LeachandPaperclip
190080642e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP discovery publishes tools that setup and access profiles use. > - Large catalogs currently write one row at a time. > - Repeated page entries can produce duplicate results, and profile writes can partly commit. > - This pull request publishes tools in bounded chunks and makes profile synchronization atomic. > - The result supports large catalogs before connector runtime hooks are enabled. ## Linked Issues or Issue Description **What happened?** Catalog refresh performs one database write per tool. Repeated tool names across discovery pages appear more than once in the result. Default-profile synchronization uses separate statements, so failure can leave a profile without its binding. OAuth callbacks also acquire connection/application locks in the reverse order from publication and setup, allowing a concurrent refresh to deadlock. **Expected behavior** Publish at most 500 tools per catalog statement. Keep one result per tool name, using its last descriptor. Preserve existing IDs, first-seen timestamps, disabled actions, and unchanged review decisions. Commit or roll back default-profile changes together. OAuth persistence must acquire application then connection locks before secret or connection writes. **Steps to reproduce** 1. Refresh a synthetic MCP catalog with 2,500 tools. 2. Return the same tool on two discovery pages with different descriptions. 3. Fail the profile-binding insert during default-profile creation. 4. Overlap personal, organization, or managed OAuth persistence with setup finalization or discovery of a new tool. **Additional context** This is the first feature extracted from #16503. The separate test-only maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Standing assignments #16603, tool definitions #16604, and access/credential hooks #16605 remain stacked drafts without review requests. ## What Changed - Deduplicate discovery results by tool name and keep the last descriptor. - Upsert catalog rows in transactions of at most 500 tools. - Preserve row identity and operator-disabled status on conflict. Determine default-access candidates from the rows read under each publication lock, so an overlapping refresh cannot restore an action turned Off after another refresh finishes. - Lock the application before the connection during publication, setup finalization, removal, and all three OAuth persistence paths. The OAuth ordering fix is included here so this PR is independently safe; it does not depend on #16605. - Synchronize the managed default profile, its entries, bindings, and managed policies in one transaction. - Add PostgreSQL publication tests and document partial-failure recovery. - Target the existing callback failure injection at profile creation instead of a transaction ordinal. ## Verification - `pnpm exec vitest run server/src/__tests__/tool-catalog-publication.test.ts server/src/__tests__/tool-access-service.test.ts`: 436 tests passed with the OAuth and overlapping-refresh corrections on top of the merged #16610 fix. - The new tests exercise real PostgreSQL with 250 and 2,500 tools, repeated discovery pages, preserved choices, lock contention, and failures between chunks and during profile creation. - Four selected regression cases fail with the production source restored to the parent commit. The implementation was then restored and rechecked. - Six additional real-PostgreSQL regressions cover personal, organization, and managed OAuth persistence against both setup finalization and new-tool discovery. All six reproduce PostgreSQL deadlock `40P01` with the OAuth ordering fix removed; all six pass with it restored. Discovery uses synthetic credential headers to isolate publication locking from credential rotation. - Apex initially returned 3/5 on `0d6013a9649388041975406fe4b0247cd0e695aa`, identifying an overlapping-refresh permission race. Two new PostgreSQL regressions cover both absent and quarantined tools; both prove an agent goes from deny back to allow without the correction, and remains denied with it. The test pauses one refresh after its initial catalog snapshot, completes another refresh, turns the action Off through the service, and resumes the stale refresh. - `pnpm --filter @paperclipai/server typecheck`: passed. - `node scripts/check-maps.mjs`: passed (structural validation only). - `git diff --check`: passed. - Reviewed the complete diff for secrets and personal data before push. - Earlier hosted run 38975428064 failed the maximum-size newline skill-audit watchdog; other interrupted jobs reported runner shutdown/cancellation. The original watchdog failure did not identify whether startup or auditing timed out, so its exact cause remains unknown. The separate merged #16610 isolates the audit deadline from module startup and preserves the audit hang guard. This branch is rebased onto that fix. - Full local build and repository-wide tests were not run. At the previous head `0d6013a9649388041975406fe4b0247cd0e695aa`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and standard Greptile returned 5/5 before Apex identified the permission race above. At corrected head `dc120d2bc879c54c14347db30e74c753bcff61e6`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and a fresh standard review returned 5/5 with no open findings. The second Apex review also returned 5/5 on that exact head (successful Apex check completed 2026-10-16T23:36:44Z). Both earlier findings are addressed, there are no open review threads, and the branch is mergeable against master. ## Risks Each successful catalog chunk remains committed if a later operation fails. A retry completes publication. Provider discovery occurs outside database locks. Default-profile changes commit separately from the catalog chunks and roll back together on failure. Tests use a controlled MCP response; they do not qualify a live provider. There is no schema migration or new plugin API in this PR. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
9.7 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Connection setup
People connect an app or account, choose who may use it, and recover incomplete setup from either Apps or a task that needs access. A saved draft, a completed provider login, a usable connection, and a resumed task are separate checkpoints. Chat channels additionally require an endpoint and linked sender identity.
Agent access: companion.
Sub-features
catalog: find a provider, inspect its available methods, and open setup; unavailable methods explain their prerequisite rather than silently switching.identity-and-access: select personal/shared identity and eligible agents; an empty explicit selection and company-wide access must not be confused.credentials: provider sign-in, key, or generic MCP URL follows its supported setup method; credentials are not retained in browser draft storage.handoff: provider popup/full-page callback returns to the correct setup host with the original task intent and identity choice intact.draft-recovery: cancel, Back, Save & exit, reload, Finish setup, and reconnect preserve the exact draft rather than creating duplicate connections.task-request: only the addressed person resolves the requested connection; usable access releases the pending task request through normal continuation.channel-setup: provider endpoint, represented agent, linked sender, and communication instructions are configured before an external message is used.failure: denied login, unreachable/private URL, expired credentials, and permission errors leave an actionable state rather than a false success.
How to get to it (user POV)
apps-catalog
Open Apps (/apps), choose an app, and connect from its landing page. Select an
available method and access, then complete setup. Existing connections open
their details; unfinished ones offer a way to finish setup.
mcp-link
Use Connect with a link or /apps/byo for a generic MCP server. Paste its
URL, select access, and follow the probe's sign-in/key guidance. Provider-specific
catalog setup and a generic endpoint may use different flows.
task-request
Open the task's pending connection request and its setup action as the named user. Setup can open in a task dialog and may use an OAuth popup. Follow the request's original link if setup falls back to a standalone page.
resume-connection
From Apps or connection details, use Finish setup, reconnect, or the offered authentication recovery action. Reload or return from the provider before checking whether the same saved connection became usable.
chat-channel
With chat connectors enabled, select a supported chat/email provider in Apps
or open /apps/chat/connect. Complete that provider's endpoint setup, then
open its settings (/apps/chat/:endpointId/settings). Follow identity linking
as the intended sender before messaging the agent externally.
Driving it with Playwright and the Paperclip CLI
Preconditions: follow the baseline. Choose a disposable provider account/resource and record its auth method, deployment mode, account ownership, and access selection. For live setup, use the provider's documented prerequisites in the connector playbook. Automated fixtures do not establish OAuth registration, public callbacks, or production provider access.
apps-catalog
Automated: Browse, unconnected app, and AppsConnect cover catalog entry, access selection, provider-specific method choices, and setup failures with mocked APIs. Connection intents drives store and task setup against one fake provider through a useful continuation:
pnpm exec vitest run ui/src/pages/apps/Browse.test.tsx ui/src/pages/apps/AppsConnect.test.tsx
pnpm exec playwright test -c tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts
Manual: connect from an unconnected provider landing. Select the intended identity and agent access, complete authentication, reload, and inspect the saved connection. Run a permitted read-only action with an eligible agent and verify its useful result. Try an ineligible agent and confirm denial. Repeat with a user who cannot install company-wide; the UI must not imply broad access.
Daytona follows this same key/access wizard. Its native method exposes the published platform and Toolbox HTTP actions using provider sandbox IDs, without a Paperclip task or lease requirement. Daytona browser setup checks the catalog, password field, access controls, and optional organization pin. It does not submit a key or qualify a real provider account. See Daytona for API-key authority and OAuth limits.
mcp-link
Automated: AppsConnect's generic MCP cases cover probe-driven auth, private or unreachable endpoints, preregistered clients, secret-backed headers, and unsafe authorization URLs. These are component/API mocks, not remote provider proof.
Catalog publication uses PostgreSQL and a controlled MCP response to check bounded writes, repeated page entries, preserved action choices, rollback, and concurrent publication. It does not verify a live provider's discovery contract.
Manual: connect a known test MCP URL through the link flow, verify its saved endpoint and a read-only action, then repeat with a bad URL and wrong credential. The error should explain the next action and preserve non-secret setup context. Test no-auth and authenticated endpoints separately; never infer one from the other.
task-request
Automated: interaction card checks addressed-user controls. AppsConnect tests cover page/dialog parity, task-bound OAuth, popup handoff, and verifying a saved connection before closing. Withdrawn requests render a terminal acknowledgement with no setup action; the interaction-body test and narrow Storybook journey cover this recovery state. The connection-intents browser suite above exercises fixture-backed continuation.
Manual: ask an agent to use an unavailable app, open its request as the named user, and complete setup in the dialog. Verify the card settles, the exact task resumes, and the agent returns a useful provider result. Repeat the page-host fallback and denied OAuth path. As another user, confirm that merely seeing the task does not grant the right to resolve the request.
resume-connection
Automated: AppsConnect tests cover exact-draft resume, declined OAuth, reconnect lookup failure/retry, response-lost creation, archived connection revival, and saved account reuse. App detail covers the connection host. These tests do not complete a real provider login.
Manual: leave setup through its footer, reload Apps, choose Finish setup, and confirm the same connection is resumed. Decline provider login and recover from that checkpoint. Reconnect an expired disposable account and verify a permitted action afterwards. Count connections before/after to catch accidental duplicates.
chat-channel
Automated: setup routing, saved setup state, email setup, and identity confirmation cover shared UI logic with fixtures. Provider-specific live setup is not implied.
Channel integration also covers bounded Telegram subscription recovery at startup and after committed configuration changes, saved retry deadlines, rollback, and parking pending repair while an endpoint or connection is inactive. The provider responses are fixtures. Settled maintenance leaves no discovery timer or database scans.
Manual: complete setup for one named provider, reload endpoint settings, link the intended sender, and send a disposable message. Verify one task and a useful reply in the originating conversation. Inspect an unlinked sender's denial and the endpoint's pause behavior. Repeat independently for each provider at issue; Slack setup does not qualify Discord, Telegram, GitHub, email, or iMessage.
Gotchas
-
Private endpoint rejection depends on deployment: owned loopback MCP servers are supported in local-trusted/private instances. Public authenticated deployments apply the private-network restriction.
-
A successful OAuth callback is not proof of saved connection access or task continuation. Verify all three checkpoints.
-
Personal identity, shared account access, and per-agent access are distinct. Connecting an account does not waive an Off action or Ask first review.
-
A tool app connection is not a chat endpoint. The same provider can offer both.
-
Old Apps/Connections URLs can redirect to Apps. Use the current navigation and record the actual destination rather than documenting an obsolete screen.
-
Full provider matrix, AI-account selection, gateway/profile administration, and account revocation lifecycles remain outside this seed recipe.
-
For governed actions after setup, use questions and approvals.
Documentation scope: App catalog and connection setup
Provider-specific authentication/revocation, credential-only accounts, and aggregator administrative variants still need their own detailed matrices.