mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
Devin FoleyandPaperclip
49b3804830
## Thinking Path > - Paperclip manages AI agents and their access to saved connections. > - The Permissions page loads account details and permitted agent choices. > - Company-wide AI accounts repeat retained installation checks for every candidate agent. > - This makes target checks grow with the square of the roster size. > - The browser waits without a deadline, and connection names do not identify the provider login. > - This pull request batches direct AI checks and bounds loading with recovery controls. > - Account details load separately and show a provider email or an explicit profile-permission limit. ## Linked Issues or Issue Description **What happened?** Opening a saved AI account can leave Permissions on “Loading AI account…” for minutes. The account itself can be connected. The existing capability batch excludes AI connections, so a company-wide install repeats checks for every installed agent for every candidate. Saved names and Paperclip ownership also do not establish which provider login a credential uses. **Expected behavior** Direct AI accounts reuse retained-target checks within one descriptive request. The page shows loaded account controls or a bounded failure with Retry. Permissions shows the provider email and organization when the selected credential permits profile access. An unavailable profile must not block maintenance or disconnect a working inference credential. Writes still authorize current permissions. **Steps to reproduce** 1. Save an AI account in a company with several agents. 2. Install the account company-wide. 3. Open its Permissions page. 4. Delay the account-list or grants response to exercise the loading failure. The maintained benchmark reproduces database work with synthetic local data. Refs #17945, which added batching for non-AI connections. Searches found no open PR for this AI-account loading fix. ## What Changed - Include direct AI accounts in the existing board capability batch. Referenced pool members, delegated actors, complex selectors, and mutation authorization keep their existing evaluator. - Add opt-in per-caller GET deadlines to the API client. A timed-out caller detaches the stuck coalesced request so Retry starts a fresh fetch. Other callers retain their own cancellation and results. - Apply a 15-second deadline to AI account lists and grants reads. The permissions view retries transient failures once and reports other failures immediately. - Share the parent Permissions grants query with AI account details. Keep loaded controls visible after a background error and offer Retry. - Add a grant-scoped identity GET and OpenAPI contract. The service checks current membership and credential audience before it reads the selected credential. It projects only the provider email and organization; provider IDs, tokens, and raw errors stay private. - Show account details independently of maintenance controls. Anthropic profile denial has an explicit message and preserves connected status. Transient profile failures have a separate Retry action. Observations are cached for five minutes under the company, connection, grant, and grant revision. - Extend real PostgreSQL and authenticated HTTP coverage, client/component regressions, map guidance, and the existing benchmark's AI mode. - Address the Apex file-size finding by moving gallery identity lookup into the existing app display helper module. `AppDetail.tsx` is 984 lines; lookup behavior is unchanged. ## Verification Local source checkout on macOS, Node 25.6.1, pnpm 9.15.4, and disposable PostgreSQL 18.1. Provider calls use fixtures. No hosted deployment or provider qualification is claimed. - Passed `pnpm -r typecheck` and `pnpm build`. - Passed all 87 tests in `connection-management.integration.test.ts` and all 33 tests in `connection-install-capabilities.test.ts`. - Passed 196 AI connection, identity-provider, and OpenAPI tests: `ai-connections.test.ts`, `ai-account-identity.test.ts`, and `openapi-routes.test.ts`. - Passed 107 focused UI/client tests: `client.test.ts`, `ManagedAiConnectionDetails.test.tsx`, `AiConnectionAccountDetails.test.tsx`, and `AppDetail.test.tsx`. - After the Apex extraction, passed UI typecheck/build and 106 component tests across `AppDetail.test.tsx`, `AppNotConnected.test.tsx`, `ManagedAiConnectionDetails.test.tsx`, and `AiConnectionAccountDetails.test.tsx`. - The authenticated HTTP tests compare complete grants responses across human roles, for tool and AI connections. They verify a real install mutation succeeds before revocation and fails afterward. - A company-wide personal AI fixture checks 20 targets instead of 400. Pool-member permission loss still denies the complete pool description. - UI states covered: shared cached grants, stuck account/grants fetches, one automatic retry, terminal failure, manual recovery, authorization failure, background failure with retained controls, and unmount cancellation. Client tests also cover stalled response bodies and independent coalesced callers. - Identity coverage includes actual bearer-key/JWT reads, private-user and cross-company denial before provider access, revoked credentials, provider 401/403/429/500, and grant-change isolation. A real local HTTP fixture stalls the response body, verifies the shared 15-second deadline, and keeps unrelated HTTP requests responsive. The test shortens the native deadline to 100 ms. Profile permission denial leaves the saved account connected. - Passed `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check`. - The full local `pnpm test:run` started with the loading-only change. All six chat shards passed. It recorded one failure in the unchanged `heartbeat-retry-scheduling.test.ts` quota-rejection case. That exact case passed a focused rerun on the final head. The initial failure cause remains unconfirmed and its local log is retained; this is not a full local-suite pass. The redundant local run was stopped with SIGINT after final-head CI completed successfully. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41515556725) passed on `ead8d276d4a2023614fff54b3e9a858f28c3178b`: typecheck, build, all 12 server shards, all 3 chat shards, workspace suites, all 9 serialized suites, all 8 end-to-end shards, Runner checks, and the aggregate verify gate. Security checks passed. Optional Storybook checks were skipped by workflow policy. GitHub reports the branch mergeable without conflicts. - Apex reviewed head `ead8d276d4a2023614fff54b3e9a858f28c3178b` at 5/5, confirmed the extraction resolves its earlier file-size finding, and has no unresolved review threads. The existing benchmark ran for the loading-only change now recorded in commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce`, with `--purpose=ai --baseline-ref=026fd9e3452`. The identity follow-up leaves that batch source unchanged. Two warmups and five alternating samples exercise each of eight fixture shapes. Complete positional capabilities, reasons, and operations match the baseline in every sample. | AI fixture | SQL calls before → after | Target reads before → after | | --- | ---: | ---: | | 1 agent, company install | 9 → 9 | 1 → 1 | | 32 agents, company install | 5,248 → 164 | 1,024 → 32 | | 100 agents, company install | 50,400 → 504 | 10,000 → 100 | | 100 agents, partial install | 13,175 → 503 | 2,575 → 100 | These counts measure the real service over local PostgreSQL. They do not predict hosted page latency. To reproduce the standalone benchmark, use a checkout of commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce` and run the command below. Its isolation check intentionally rejects the later identity follow-up's additional runtime changes. ```sh node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --purpose=ai --baseline-ref=026fd9e3452 ``` Run the final-head regression suites from the repository root: ```sh cd server pnpm exec vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/connection-install-capabilities.test.ts src/__tests__/ai-connections.test.ts src/services/ai-account-identity.test.ts src/__tests__/openapi-routes.test.ts cd ../ui pnpm exec vitest run src/api/client.test.ts src/components/ai-connections/ManagedAiConnectionDetails.test.tsx src/components/ai-connections/AiConnectionAccountDetails.test.tsx src/pages/apps/AppDetail.test.tsx ``` ## Risks Descriptive permissions reuse retained facts within one request. A concurrent change can leave a displayed choice stale; the next request and every write still authorize fresh. Pools with referenced members retain their existing cost. The account and grants clients now bound each caller's wait. A browser abort does not cancel an authorization read already admitted by the server. The independent profile request uses a fixed Anthropic origin, a bounded body, no redirects, and a 15-second deadline. Inference-only credentials can return no email; that is reported without guessing from a Paperclip owner or changing connection health. Profiles are fetched only for the caller's authorized selected grant. No ambient CLI identity, persistent identity cache, background polling, or billing-identity change is added. Other API reads keep their existing behavior. No migration is required. Staging has not been redeployed. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, code editing, execution, and test tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
Apps, Connections, and Integrations
Audience: internal engineers and product contributors working on integrations.
Start here when adding a provider: Connection authoring runbook. It is the canonical agent tutorial from provider research and protocol classification through manifest generation, branding, secrets, deterministic tests, real-account proof, and PR submission.
Chat and email channels: conversation-channel contract, question inventory and quality bar, and per-provider question coverage index.
Runtime authentication: AI Connections.
Long-term memory: Experimental memory connectors.
Provider notes: Google Workspace, Gmail, Asana, Sentry, PostHog, Braintrust, Neon, Superagent, Gauge, Ditto, 1Password, Unikraft, AgentMail, Telem.AI, iMessage Photon, Enterpret, Supabase, and Vercel. Optional credential custody: Vercel Connect.
Post-read action: classify a new integration request, pick the right Paperclip layer to change, and avoid creating a parallel connection framework.
Decision Record
Board decisions from PAP-13211 make the Apps v2 substrate on the PAP-10341 branch canonical:
- D1: Apps v2 is the substrate. The active model is
tool_applications,tool_connections, catalog entries, profiles, policy rules, action requests, gateway sessions, audit events, and runtime slots. Connections v1 is retired as an implementation path. - D2: one credential authority per connection, brokered projections. The
default is the Paperclip instance vault: durable third-party credentials live
in
company_secretsas secret refs. A reviewed remote MCP method may instead opt in to Vercel Connect, in which case Vercel is the durable credential authority and Paperclip stores only the connector reference and redacted grant metadata. A connection must never mix those two sources. Adapter config, plugin config, harness credential files, and run environments may receive only brokered or projected credentials. - D3: the vocabulary and three-door IA are product law. The default product doors are Apps, Connections, and Review. Protocol and operator-depth concepts live behind Developer or Advanced surfaces.
- D4: unification lands on PAP-10341. Pages, CircleBack-style harness MCP OAuth, provider gallery work, and plugin-provided integrations converge on this branch instead of spawning new integration substrates.
- D5: inbound stays thin. External clients that call Paperclip use scoped Paperclip tokens and existing profiles/rules. They do not get a separate permission model.
Canonical Object Model
Use connection as the unifying noun. A connection is four things:
- A stored credential reference.
- A capability catalog.
- A governance layer.
- An audit trail.
Everything else is an axis on that object:
| Axis | Values | It answers |
|---|---|---|
| Direction | outbound, inbound | Who is the client? |
| Transport | MCP, native REST/OpenAPI, OAuth app install, webhook | How do bytes move? |
| Auth mode | OAuth, API key/PAT, app installation, none | What does the secret represent? |
| Credential owner | company, user, run | Whose identity acts? |
| Packaging | catalog entry, plugin, skill | How does it ship? |
MCP is a transport, not a product category. "Install the Discord app", "connect Google Drive", and "add an MCP endpoint" all produce governed connections with different transport/auth values.
Layer Stack
When you are unsure where a change belongs, place it on the narrowest layer that solves the problem:
| Layer | Owns | Examples |
|---|---|---|
| Surface | user-facing Apps, Connections, Review, Developer/Advanced screens | gallery cards, setup wizard, review queue |
| Governance | profiles, bindings, allow/ask-first/block rules, quarantine, audit | read-only profile, ask-first write policy |
| Capability | action catalogs, schemas, risk classes, changed-tool review | search_issues, create_comment, schema hash |
| Credential | company_secrets, OAuth broker, credential resolver, token broker |
Slack bot token ref, Google OAuth refresh token ref |
| Identity | actor attribution and token exchange | board user, agent run, first-party service identity |
| Transport | how the external system is reached | remote HTTP MCP, local stdio, REST/OpenAPI, webhook |
The agent should not hold a durable provider credential. It should hold a Paperclip run/session token; the server or broker resolves the connection, checks governance, invokes the provider, and writes audit.
Identity vs. connections
Signing a user in and connecting a resource are different planes with different owners, different token profiles, and different homes. Do not merge them. This section is the public, connections-side statement of the identity model so connector implementers inherit the rule without depending on private identity-service documentation or re-deriving it.
| Plane | Question | Lives where | Token profile |
|---|---|---|---|
| P1. Sign-in methods | Who are you? | paperclip-id (id.paperclip.ing → Account) |
Minimal-scope provider tokens (openid email profile), used once to authenticate, encrypted at rest, never exported |
| P2. Connections (Apps) | What may your agents touch? | Paperclip App instances (tool_connections), acquired via the connect broker for hosted + self-hosted |
Rich-scope, long-lived resource tokens in the instance's encrypted vault; per-agent grants; risk-tier policy defaults |
| P3. Login with Paperclip | Who may authenticate against us? | paperclip-id OIDC provider + DB-backed client registry |
Our ES256 ID/access tokens issued by us to registered RPs (instances, the broker, future third parties) |
Everything in doc/connections/ — the First-30 matrix,
the connection authoring runbook, and the connect-broker work —
lives on plane P2. It never acquires, stores, or brokers a P1 sign-in token.
The standing rule (D7)
Adopted as a standing rule (decision D7) with the identity-model plan. State it verbatim in any P2 design so the app-store work cannot drift into merging the planes:
Sign-in tokens are never reused as resource tokens; id.paperclip.ing never stores resource tokens; no connections hub on the ID service.
P2 tokens flow broker → instance vault as pass-through only; the id.paperclip.ing Account page therefore must not grow a "Connections" hub. The reasons to hold the planes apart (from the plan §3):
- Scope discipline. Sign-in wants the narrowest grant; connections want deliberately broad ones. One button that does both is how you grant repo access just to log in.
- Blast radius. id.paperclip.ing holding every customer's Vercel/Slack/GitHub resource tokens would make it the single juiciest target in the fleet; the broker is intentionally pass-through.
- Self-hosted symmetry. Instances own their vaults, so self-hosters don't depend on our uptime to use their own connections.
- Legibility. Sign-in and connections answer different user questions, and every product we benchmarked (Vercel, Railway, GitHub, Google) keeps them on separate pages with separate names.
The explicit Vercel Connect exception does not change D7 or merge P1 and P2.
The operator chooses Vercel as the P2 credential authority for an individual
connection. id.paperclip.ing is not involved, and neither sign-in tokens nor
provider tokens pass through it. The deployment's Vercel access token or
workload OIDC identity is bootstrap authority for that external vault, not a
provider resource credential.
Naming alignment
Use the surface-correct name for each plane; they intentionally differ:
| Surface | Plane | Name to use |
|---|---|---|
| Paperclip App instances | P2 | "Connections" |
| id.paperclip.ing Account | P1 | "Ways to sign in" |
| id.paperclip.ing admin | P3 | "OIDC clients" (until the app store productizes it) |
Packaging Rule
Default to a catalog entry when an integration can be described as metadata: manifest, auth config, action catalog, resource filters, and policy defaults.
Use a plugin only when the integration needs product code such as custom UI pages, its own tables, workers, migrations, routines, or specialized ingestion. A plugin may bundle catalog entries, but it must not bypass the connection, profile, policy, credential, and audit model.
Use a skill for agent instructions. Skills may use connections; they must not own durable tokens.
Canonical Docs
- Glossary defines product and internal terms.
- Identity vs. connections is the public statement of the P1/P2/P3 boundary and the D7 standing rule for connections work.
- Security threat model harvests the keeper from PAP-2359 and maps it onto Apps v2.
- First-30 matrix harvests the keeper from PAP-2432 and is the source matrix for connector playbook work.
- Connecting any remote MCP server is the baseline: how an operator connects a standards-compliant remote MCP endpoint with no Paperclip code change, and how sign-in resolves a client.
- Connection authoring runbook is the one end-to-end, agent-executable guide for adding a vendor as a catalog entry on Apps v2: research, connection-type selection, OAuth/API-key/generated-URL setup, encrypted credential handling, branding, implementation, browser and live-provider testing, verification, and PR submission.
- Vercel Connect operator guide documents the optional external credential source, deployment flags, runtime resolution, recovery, and smoke requirements.
- MCP access governance remains the operator runbook for the current gateway, profile, policy, approval, runtime, and audit APIs.
Migration Notes
Connections v1 contributed useful policy, UX, and rollout thinking, but its
implementation branch is no longer the target. When you see old tickets or code
using connections, connection_grants, or a provider-directory mental model,
translate the intent into Apps v2:
| Connections v1 intent | Apps v2 home |
|---|---|
| Provider directory | Apps gallery / tool_applications |
| Configured provider instance | Connection / tool_connections |
| Grant allowlist | Profiles, profile bindings, policies |
| Resource filters | Policy/profile conditions plus provider config |
| Tool broker | Tool gateway and runtime supervisor |
| Connection UX tail | Apps, Connections, Review, Developer/Advanced IA |
Do not add new work to the retired v1 branch. If an old ticket still describes a valid product gap, retarget it to an active Apps v2 issue or close it as superseded with a link to the replacement.
For Slack chat bot creation, installation, recovery, and live qualification, see Automatic Slack app setup.