From d0ffa179b45d255af5ad1aed95a852994396d0ed Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 16:13:18 -0600 Subject: [PATCH] perf(tools): index profile selectors per resolution (#18673) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agents and their work. > - Core resolves tool profiles before it prepares runtime MCP assignments. > - A selected profile can contain a rule for each individual tool. > - The resolver and profile summary scan these rules for each catalog row. > - This repeats millions of exact comparisons for a large selected profile. > - This pull request indexes each selector kind once per computation. > - The benefit is less server CPU with the same current access decisions and result contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Effective tool assignment and profile summary computation for large selected profiles. **Current behavior** Each catalog row scans the profile's includes and excludes. The work grows with catalog size times rule count. The runtime assignment path also computes a profile summary with the same scans. **Proposed behavior** Build five selector sets for each include or exclude group. Match catalog rows with exact set membership. Retain all current database reads and the full result contract. **Reason and benefit** Reduce repeated JavaScript comparisons. On the synthetic PostgreSQL fixture below, median process CPU per resolution falls from 50.068 ms to 10.129 ms. This is independent of [#18371](https://github.com/paperclipai/paperclip/pull/18371), which removes unused administrative results and narrows instruction descriptors. This PR retains those results and does not include that change. ## What Changed - Index application, connection, catalog-entry, tool-name, and risk selectors in the profile owner. - Use indexed matching in effective access resolution and profile summaries. - Add a differential test against the original single-rule matcher. Cover null applications, exact special-character names, unknown selectors, and empty rule groups. - Add a PostgreSQL regression for per-profile exclusion precedence, the union of multiple effective profiles, and current rule removal. - Add a real PostgreSQL resolver benchmark and document reproduction and limits. ## Verification Baseline: `410b0e27015495ab8951768b5b0b94727de33785`. Treatment: `895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`. Node 24.22.2 on macOS arm64. Source modules run with `tsx`; these are local service measurements, not production-build or hosted acceptance. The real owning resolver ran against disposable PostgreSQL with 1,890 active catalog tools, 1,575 individual include/exclude rules, one deny-by-default effective profile, and 12 agents. The installation dependency returns an empty list because the fixture has no installed connections. There are no provider calls. Full returned assignments, profile summaries, rules, bindings, and names are identical to baseline. After two warmups per version, seven alternating samples ran ten resolutions each. Other workers paused CPU measurements and builds. Unrelated host work was not controlled. | Per resolution | Baseline median (range) | Change median (range) | | --- | --- | --- | | Process CPU | 50.068 ms (46.380–58.880) | 10.129 ms (8.089–11.072) | | Elapsed time | 50.362 ms (44.799–106.408) | 8.853 ms (7.719–18.590) | | Queries | 6 | 6 | | Returned database rows | 3,480 | 3,480 | Median process CPU falls 79.8%; median elapsed time falls 82.4%. The long baseline elapsed sample had no matching process CPU increase. Query and row counts do not decrease. Small or broadly allowed profiles can save less. This does not establish whole-run latency or hosted performance. Reproduce with `server/scripts/benchmark-tool-profile-matching.ts`; baseline module preparation is in `doc/mcp-discovery-performance.md`. Passed on the treatment source: ```sh node node_modules/vitest/vitest.mjs run server/src/__tests__/tool-profile-matching.test.ts server/src/__tests__/tool-effective-assignments.test.ts server/src/services/native-runtime/runtime-context.test.ts server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts ``` All 45 tests in four suites passed. `git diff --check` passed. The first PostgreSQL test attempt failed at setup because the sandbox blocked its loopback listener (`EPERM`); the authorized unsandboxed retry passed. No behavior failure was discarded. At measured head `895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`, CI typecheck, build, canary, Runner checks, and all eight browser shards passed. Server general shard 8/12 failed an unchanged background-quietness assertion in `heartbeat-run-dispatch-work.test.ts`: expected 235 select calls, observed 236 after advancing fake timers. Its cause remains unverified; no baseline reproduction was run. That head also failed the module-boundary policy because the benchmark inserted synthetic agents outside the test fixture tree. Follow-up head `e20165b6eb` moves the fixture body byte-for-byte into `server/src/__tests__/helpers/tool-profile-matching-fixture.ts`, following the existing listing benchmark pattern. The production matcher and benchmark measurement loop are unchanged. `node scripts/check-module-boundaries.mjs`, a Node 24/tsx fixture import check, and `git diff --check` pass. The benchmark was not rerun for this extraction. New-head CI and review are pending. Greptile gave the measured head 5/5 with no actionable issues. No deployment or live runtime mutation was performed. ## Risks Exact equality must remain identical to the original matcher, especially for null application selectors and exclusions. The differential and PostgreSQL tests cover these cases. Sets add linear temporary memory in rule count. They live only for the current computation. Current company-scoped state reads, profile precedence, revocation, and runtime assignment digest checks remain in place. Revert this commit to roll back. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with source inspection, code execution, tests, and local PostgreSQL benchmarking. The context window size is not available in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/mcp-discovery-performance.md | 32 ++++++++++++ .../benchmark-tool-profile-matching.ts | 51 +++++++++++++++++++ .../helpers/tool-profile-matching-fixture.ts | 15 ++++++ .../tool-effective-assignments.test.ts | 14 +++++ .../__tests__/tool-profile-matching.test.ts | 21 ++++++++ .../tool-access/profiles/projection.ts | 18 +++---- .../tool-access/profiles/resolution.ts | 15 ++---- 7 files changed, 145 insertions(+), 21 deletions(-) create mode 100644 server/scripts/benchmark-tool-profile-matching.ts create mode 100644 server/src/__tests__/helpers/tool-profile-matching-fixture.ts create mode 100644 server/src/__tests__/tool-profile-matching.test.ts diff --git a/doc/mcp-discovery-performance.md b/doc/mcp-discovery-performance.md index 3ec87f17a4c..ac696b01187 100644 --- a/doc/mcp-discovery-performance.md +++ b/doc/mcp-discovery-performance.md @@ -106,3 +106,35 @@ Catalog payload memory and policy-matching CPU still grow with the catalog. The measurements do not establish a safe production heap limit or the share of active installations affected. Verify restart bursts and sustained traffic on the actual deployment before reducing its temporary heap allowance. + +## Profile selector matching + +Effective profile resolution and profile summaries index each profile's include +and exclude selectors once per decision. Catalog rows then use exact set +membership checks for application, connection, catalog entry, tool name, and +risk level. Exclusions still take precedence within a profile; effective profiles +still contribute the union of their allowed tools. Null application selectors +retain their exact-match behavior. + +These indexes live only for the current computation. Each resolver call still +reads current company-scoped bindings, profiles, and rules. Query counts and +returned database rows are unchanged. This reduces JavaScript matching work when +a profile selects many individual tools; it does not cache authorization or +remove the runtime assignment digest check. + +For a real PostgreSQL comparison, save `projection.ts` and `resolution.ts` from +the baseline commit beside their current versions as `.baseline-projection.ts` +and `.baseline-resolution.ts`. Change only the baseline resolver's projection +import to `./.baseline-projection.js`. Then run from `server/`: + +```sh +node --import tsx scripts/benchmark-tool-profile-matching.ts --baseline "$PWD/src/services/tool-access/profiles/.baseline-resolution.ts" +``` + +Remove the two baseline files afterward. The probe uses 1,890 catalog tools, +1,575 individual include/exclude rules, one effective profile, and 12 agents. It +checks full assignment-result equality, warms both versions, and alternates +seven samples of ten resolutions. It reports actual query and returned-row +counts, process CPU, and elapsed time. The installation dependency returns an +empty list for this fixture, which has no installed connections. It does not +measure provider work, whole-run latency, or hosted performance. diff --git a/server/scripts/benchmark-tool-profile-matching.ts b/server/scripts/benchmark-tool-profile-matching.ts new file mode 100644 index 00000000000..f0e0e28ce73 --- /dev/null +++ b/server/scripts/benchmark-tool-profile-matching.ts @@ -0,0 +1,51 @@ +import assert from "node:assert/strict"; +import { pathToFileURL } from "node:url"; +import { and, eq } from "drizzle-orm"; +import { agents, createDb, type Db } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../src/__tests__/helpers/embedded-postgres.js"; +import { createToolProfileMatchingFixture } from "../src/__tests__/helpers/tool-profile-matching-fixture.js"; +import { createToolProfileResolver } from "../src/services/tool-access/profiles/resolution.js"; + +const baselinePath = process.argv[process.argv.indexOf("--baseline") + 1]; +if (!process.argv.includes("--baseline") || !baselinePath) throw new Error("--baseline module required"); +const baselineFactory = (await import(pathToFileURL(baselinePath).href)).createToolProfileResolver as typeof createToolProfileResolver; +const database = await startEmbeddedPostgresTestDatabase("profile-matching-benchmark-"); +const db = createDb(database.connectionString, { maxConnections: 1 }); +let queries = 0, rows = 0; +const observedDb = new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args), execute = selection.execute.bind(selection); + selection.execute = async () => { const result = await execute(); queries++; rows += result.length; return result; }; + return selection; + }) as typeof builder.from; + return builder; + }; +} }); +try { + const { company, agent } = await createToolProfileMatchingFixture(db); + const dependencies = { + assertOptionalAgent: async (companyId: string, agentId: string | null | undefined) => { + const valid = await observedDb.select({ id: agents.id }).from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId!))); + assert.equal(valid.length, 1); + }, + resolveInstalledConnectionsForAgent: async () => [], + toCatalogEntry: () => { throw new Error("Full catalog definitions are not used by this assignment benchmark"); }, + }; + const before = baselineFactory(observedDb, dependencies); + const after = createToolProfileResolver(observedDb, dependencies); + const invoke = (resolver: typeof after) => resolver.getEffectiveToolAssignmentsForAgent(company.id, agent.id); + assert.deepEqual(await invoke(after), await invoke(before)); + for (let i = 0; i < 2; i++) { await invoke(before); await invoke(after); } + for (let round = 0; round < 7; round++) { + for (const [label, resolver] of round % 2 ? [["after", after], ["before", before]] as const : [["before", before], ["after", after]] as const) { + queries = 0; rows = 0; + const cpu = process.cpuUsage(), start = performance.now(); + for (let i = 0; i < 10; i++) await invoke(resolver); + const elapsedMs = performance.now() - start, used = process.cpuUsage(cpu); + console.log(JSON.stringify({ round, label, operations: 10, tools: 1890, rules: 1575, agents: 12, queries, rows, elapsedMs, cpuMs: (used.user + used.system) / 1000 })); + } + } +} finally { await db.$client.end({ timeout: 1 }); await database.cleanup(); } diff --git a/server/src/__tests__/helpers/tool-profile-matching-fixture.ts b/server/src/__tests__/helpers/tool-profile-matching-fixture.ts new file mode 100644 index 00000000000..087e7c8bcb9 --- /dev/null +++ b/server/src/__tests__/helpers/tool-profile-matching-fixture.ts @@ -0,0 +1,15 @@ +import { randomUUID } from "node:crypto"; +import { agents, companies, toolApplications, toolCatalogEntries, toolConnections, toolProfileBindings, toolProfileEntries, toolProfiles, type Db } from "@paperclipai/db"; + +export async function createToolProfileMatchingFixture(db: Db) { + const [company] = await db.insert(companies).values({ name: "Synthetic profile matching", issuePrefix: `BM${randomUUID().slice(0, 6)}` }).returning(); + const companyAgents = await db.insert(agents).values(Array.from({ length: 12 }, (_, i) => ({ companyId: company.id, name: `Agent ${i}` }))).returning(); + const agent = companyAgents[0]!; + const [app] = await db.insert(toolApplications).values({ companyId: company.id, name: "Synthetic app", type: "mcp_http" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId: company.id, applicationId: app.id, name: "Synthetic connection", uid: randomUUID(), transport: "mcp_remote" }).returning(); + const catalog = await db.insert(toolCatalogEntries).values(Array.from({ length: 1890 }, (_, i) => ({ companyId: company.id, applicationId: app.id, connectionId: connection.id, name: `tool_${i}`, toolName: `tool_${i}`, versionHash: `v${i}`, riskLevel: "read" as const }))).returning(); + const [profile] = await db.insert(toolProfiles).values({ companyId: company.id, profileKey: "selected", name: "Selected tools", defaultAction: "deny" }).returning(); + await db.insert(toolProfileBindings).values({ companyId: company.id, profileId: profile.id, targetType: "agent", targetId: agent.id }); + await db.insert(toolProfileEntries).values(catalog.slice(0, 1575).map((entry, i) => ({ companyId: company.id, profileId: profile.id, selectorType: "catalog_entry" as const, catalogEntryId: entry.id, effect: i % 5 === 0 ? "exclude" as const : "include" as const }))); + return { company, agent }; +} diff --git a/server/src/__tests__/tool-effective-assignments.test.ts b/server/src/__tests__/tool-effective-assignments.test.ts index 88534789025..f22b8fd0cb6 100644 --- a/server/src/__tests__/tool-effective-assignments.test.ts +++ b/server/src/__tests__/tool-effective-assignments.test.ts @@ -76,6 +76,20 @@ describeDatabase("effective tool assignments", () => { expect(full.allowedTools[0].annotations).toEqual(f.read.annotations); }); + it("keeps exclusions local to each profile when effective profiles grant the same tool", async () => { + const f = await fixture(); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "catalog_entry", effect: "exclude", catalogEntryId: f.read.id }); + const [second] = await db.insert(toolProfiles).values({ companyId: f.company.id, profileKey: "second", name: "Second", defaultAction: "deny" }).returning(); + await db.insert(toolProfileBindings).values({ companyId: f.company.id, profileId: second.id, targetType: "agent", targetId: f.agent.id }); + const [include] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: second.id, + selectorType: "catalog_entry", effect: "include", catalogEntryId: f.read.id }).returning(); + expect((await compare(f)).assignments.allowedTools.map(tool => tool.id)).toEqual([f.read.id, f.write.id]); + await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, include.id)); + expect((await compare(f)).assignments.allowedTools.map(tool => tool.id)).toEqual([f.write.id]); + }); + it("keeps unresolved exact names and observes changed catalog, profile and binding state", async () => { const f = await fixture(); await db.insert(toolProfileEntries).values([ diff --git a/server/src/__tests__/tool-profile-matching.test.ts b/server/src/__tests__/tool-profile-matching.test.ts new file mode 100644 index 00000000000..4c8498ed6bd --- /dev/null +++ b/server/src/__tests__/tool-profile-matching.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from "vitest"; +import { profileCatalogMatcher, profileEntryMatchesCatalog, type ProfileCatalogMatch, type ProfileEntryMatch } from "../services/tool-access/profiles/projection.js"; + +describe("indexed profile catalog matching", () => { + it("preserves all selector kinds, null matches and exact strings in mixed rule sets", () => { + const catalog: ProfileCatalogMatch[] = Array.from({ length: 75 }, (_, index) => ({ + id: `tool-${index}`, applicationId: index % 3 ? `app-${index % 5}` : null, + connectionId: `connection-${index % 7}`, toolName: index % 4 ? `tool-${index}` : "quoted' path\\tool_🙂", + riskLevel: index % 2 ? "read" : "write", + })); + const selectors = ["application", "connection", "catalog_entry", "tool_name", "risk_level", "unknown"]; + const rules: ProfileEntryMatch[] = catalog.flatMap(entry => selectors.map(selectorType => ({ + selectorType, applicationId: entry.applicationId, connectionId: entry.connectionId, + catalogEntryId: entry.id, toolName: entry.toolName, riskLevel: entry.riskLevel, + }))); + for (const selected of [[], ...rules.map(rule => [rule]), rules.slice(0, 25), rules]) { + const matches = profileCatalogMatcher(selected); + expect(catalog.filter(matches)).toEqual(catalog.filter(entry => selected.some(rule => profileEntryMatchesCatalog(rule, entry)))); + } + }); +}); diff --git a/server/src/services/tool-access/profiles/projection.ts b/server/src/services/tool-access/profiles/projection.ts index 58f64d0f20e..877c2606fd4 100644 --- a/server/src/services/tool-access/profiles/projection.ts +++ b/server/src/services/tool-access/profiles/projection.ts @@ -128,22 +128,18 @@ export function summarizeProfile(input: { catalog: ProfileCatalogEntry[]; agentIds: string[]; }): ToolProfileSummary { - const includes = input.entries.filter((entry) => entry.effect === "include"); - const excludes = input.entries.filter((entry) => entry.effect === "exclude"); + const included = profileCatalogMatcher(input.entries.filter((entry) => entry.effect === "include")); + const excluded = profileCatalogMatcher(input.entries.filter((entry) => entry.effect === "exclude")); const allowedCatalogIds = new Set(); const allowedApplicationIds = new Set(); const excludedCatalogIds = new Set(); for (const catalogEntry of input.catalog) { - const excluded = excludes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ); - if (excluded) excludedCatalogIds.add(catalogEntry.id); - if (excluded) continue; - const included = includes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ); - if (input.profile.defaultAction === "allow" || included) { + if (excluded(catalogEntry)) { + excludedCatalogIds.add(catalogEntry.id); + continue; + } + if (input.profile.defaultAction === "allow" || included(catalogEntry)) { allowedCatalogIds.add(catalogEntry.id); if (catalogEntry.applicationId) allowedApplicationIds.add(catalogEntry.applicationId); diff --git a/server/src/services/tool-access/profiles/resolution.ts b/server/src/services/tool-access/profiles/resolution.ts index c237636eb2b..b06683b8203 100644 --- a/server/src/services/tool-access/profiles/resolution.ts +++ b/server/src/services/tool-access/profiles/resolution.ts @@ -3,7 +3,7 @@ import type { Db } from "@paperclipai/db"; import { agents, toolCatalogEntries, toolProfileBindings, toolProfileEntries, toolProfiles } from "@paperclipai/db"; import type { ToolCatalogEntry, ToolConnection, ToolProfileEffectiveSummary, ToolProfileWithDetails } from "@paperclipai/shared"; import { effectiveToolProfileBindings, profileIdsInBindingOrder } from "../../tool-profile-binding-precedence.js"; -import { buildProfileDetails, profileCatalogColumns, profileEntryMatchesCatalog, toProfileBinding, toProfileEntry, type ProfileCatalogEntry, type ProfileEntryMatch, type ProfileCatalogMatch } from "./projection.js"; +import { buildProfileDetails, profileCatalogColumns, profileCatalogMatcher, toProfileBinding, toProfileEntry, type ProfileCatalogEntry, type ProfileEntryMatch, type ProfileCatalogMatch } from "./projection.js"; type ProfileAssignmentDependencies = { assertOptionalAgent(companyId: string, agentId: string | null | undefined, label: string): Promise; resolveInstalledConnectionsForAgent(companyId: string, agentId: string): Promise; @@ -37,18 +37,13 @@ function resolveCatalogAccess( const excludes = profileEntries.filter( (entry) => entry.effect === "exclude", ); + const included = profileCatalogMatcher(includes); + const excluded = profileCatalogMatcher(excludes); for (const catalogEntry of catalog) { - if ( - excludes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) - ) - continue; + if (excluded(catalogEntry)) continue; if ( profile.defaultAction === "allow" || - includes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) + included(catalogEntry) ) { allowedCatalogIds.add(catalogEntry.id); allowedToolNames.add(catalogEntry.toolName);