mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
Nicky LeachandPaperclip
190080642e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP discovery publishes tools that setup and access profiles use. > - Large catalogs currently write one row at a time. > - Repeated page entries can produce duplicate results, and profile writes can partly commit. > - This pull request publishes tools in bounded chunks and makes profile synchronization atomic. > - The result supports large catalogs before connector runtime hooks are enabled. ## Linked Issues or Issue Description **What happened?** Catalog refresh performs one database write per tool. Repeated tool names across discovery pages appear more than once in the result. Default-profile synchronization uses separate statements, so failure can leave a profile without its binding. OAuth callbacks also acquire connection/application locks in the reverse order from publication and setup, allowing a concurrent refresh to deadlock. **Expected behavior** Publish at most 500 tools per catalog statement. Keep one result per tool name, using its last descriptor. Preserve existing IDs, first-seen timestamps, disabled actions, and unchanged review decisions. Commit or roll back default-profile changes together. OAuth persistence must acquire application then connection locks before secret or connection writes. **Steps to reproduce** 1. Refresh a synthetic MCP catalog with 2,500 tools. 2. Return the same tool on two discovery pages with different descriptions. 3. Fail the profile-binding insert during default-profile creation. 4. Overlap personal, organization, or managed OAuth persistence with setup finalization or discovery of a new tool. **Additional context** This is the first feature extracted from #16503. The separate test-only maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Standing assignments #16603, tool definitions #16604, and access/credential hooks #16605 remain stacked drafts without review requests. ## What Changed - Deduplicate discovery results by tool name and keep the last descriptor. - Upsert catalog rows in transactions of at most 500 tools. - Preserve row identity and operator-disabled status on conflict. Determine default-access candidates from the rows read under each publication lock, so an overlapping refresh cannot restore an action turned Off after another refresh finishes. - Lock the application before the connection during publication, setup finalization, removal, and all three OAuth persistence paths. The OAuth ordering fix is included here so this PR is independently safe; it does not depend on #16605. - Synchronize the managed default profile, its entries, bindings, and managed policies in one transaction. - Add PostgreSQL publication tests and document partial-failure recovery. - Target the existing callback failure injection at profile creation instead of a transaction ordinal. ## Verification - `pnpm exec vitest run server/src/__tests__/tool-catalog-publication.test.ts server/src/__tests__/tool-access-service.test.ts`: 436 tests passed with the OAuth and overlapping-refresh corrections on top of the merged #16610 fix. - The new tests exercise real PostgreSQL with 250 and 2,500 tools, repeated discovery pages, preserved choices, lock contention, and failures between chunks and during profile creation. - Four selected regression cases fail with the production source restored to the parent commit. The implementation was then restored and rechecked. - Six additional real-PostgreSQL regressions cover personal, organization, and managed OAuth persistence against both setup finalization and new-tool discovery. All six reproduce PostgreSQL deadlock `40P01` with the OAuth ordering fix removed; all six pass with it restored. Discovery uses synthetic credential headers to isolate publication locking from credential rotation. - Apex initially returned 3/5 on `0d6013a9649388041975406fe4b0247cd0e695aa`, identifying an overlapping-refresh permission race. Two new PostgreSQL regressions cover both absent and quarantined tools; both prove an agent goes from deny back to allow without the correction, and remains denied with it. The test pauses one refresh after its initial catalog snapshot, completes another refresh, turns the action Off through the service, and resumes the stale refresh. - `pnpm --filter @paperclipai/server typecheck`: passed. - `node scripts/check-maps.mjs`: passed (structural validation only). - `git diff --check`: passed. - Reviewed the complete diff for secrets and personal data before push. - Earlier hosted run 38975428064 failed the maximum-size newline skill-audit watchdog; other interrupted jobs reported runner shutdown/cancellation. The original watchdog failure did not identify whether startup or auditing timed out, so its exact cause remains unknown. The separate merged #16610 isolates the audit deadline from module startup and preserves the audit hang guard. This branch is rebased onto that fix. - Full local build and repository-wide tests were not run. At the previous head `0d6013a9649388041975406fe4b0247cd0e695aa`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and standard Greptile returned 5/5 before Apex identified the permission race above. At corrected head `dc120d2bc879c54c14347db30e74c753bcff61e6`, all 55 hosted checks succeeded (two optional Storybook checks skipped), and a fresh standard review returned 5/5 with no open findings. The second Apex review also returned 5/5 on that exact head (successful Apex check completed 2026-10-16T23:36:44Z). Both earlier findings are addressed, there are no open review threads, and the branch is mergeable against master. ## Risks Each successful catalog chunk remains committed if a later operation fails. A retry completes publication. Provider discovery occurs outside database locks. Default-profile changes commit separately from the catalog chunks and roll back together on failure. Tests use a controlled MCP response; they do not qualify a live provider. There is no schema migration or new plugin API in this PR. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>