mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).
**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).
**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).
**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.
**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.
**Breaking changes**
None. The policy data model, API, and listing contract do not change.
## What Changed
- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.
## Verification
- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.
| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
| --- | --- | --- |
| Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
| Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
| Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
| Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
| Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
| Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
| Selector evaluations | 305,600 | 1,164 |
| Raw selector property reads | 6,155,830 | 9,490 (compile only) |
```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
```
- Not verified: the hosted staging profile after deploy.
## Risks
- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
248 lines
14 KiB
Markdown
248 lines
14 KiB
Markdown
# MCP discovery resource bounds
|
||
|
||
Tool discovery shares policy inputs for one listing and projects context IDs in SQL.
|
||
It does not load task descriptions or run results to decide access. The cache ends
|
||
with the listing. Tool execution reads current policy and current rate counters.
|
||
|
||
The control-plane process admits two whole listings at a time and queues at most
|
||
32 more. Excess requests receive HTTP 503 with `tool_discovery_busy`. Disconnected
|
||
listing requests leave the queue or stop scheduling new decisions. Already-started
|
||
reads drain before the admission slot is released. This applies only to read-only
|
||
discovery; a disconnected tool invocation is not automatically cancelled.
|
||
|
||
Discovery audit rows retain the visible tool count and a SHA-256 digest of sorted
|
||
names. They do not retain the full name list. Named gateway tokens expire normally;
|
||
startup and scheduler sweeps delete at most 500 expired tokens per pass using the
|
||
expiry index. Tokens with no expiry and unexpired tokens remain available. The
|
||
separate access and activity audit records remain available.
|
||
Audit inserts resolve the token reference atomically and lock a surviving token
|
||
row for that statement. If cleanup already removed it, the reference is null and
|
||
the original token ID remains in audit details. An admitted request can complete;
|
||
later requests still fail authentication after expiry.
|
||
|
||
The stateless gateway and runtime-tools MCP endpoints return HTTP 405 with
|
||
`Allow: POST` for GET instead of returning JSON as if it were an SSE stream.
|
||
The runtime-tools endpoint still validates its caller before that response.
|
||
See the [MCP transport contract](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports).
|
||
|
||
## New-tool review projection
|
||
|
||
Pending-tool review builds covered application and connection sets from each
|
||
profile's include rules. An included catalog entry covers its connection and its
|
||
non-null application only while that entry exists in the supplied catalog.
|
||
Explicit application selectors retain null application matching. A second index
|
||
records all exact selectors, including exclusions, so already reviewed tools do
|
||
not appear as new. The catalog filter retains its order, status check, strict
|
||
review watermark, and returned metadata.
|
||
|
||
The indexes live only for one projection. Profile listings, new-tool review, and
|
||
app attention still read current company-scoped database state on each request.
|
||
This replaces nested catalog-by-rule scans with linear indexing and filtering.
|
||
It changes neither database query counts nor response contracts.
|
||
|
||
The PostgreSQL reproduction uses 200 selected profiles, 1,900 active tools, and
|
||
76,000 rules. Profiles alternate between 65 and 695 rules. Half reference 695
|
||
removed catalog entries, while half select current tools. This exercises both
|
||
scope checks and explicit-review checks. It compares complete ordered results
|
||
from the real profile listing and app-attention services before timing them.
|
||
|
||
Save baseline `profiles/projection.ts`, `profiles/service.ts`, and
|
||
`catalog/attention.ts` beside their current versions as `.baseline-projection.ts`,
|
||
`.baseline-service.ts`, and `.baseline-attention.ts`. Change the two baseline
|
||
consumers' projection imports to `.baseline-projection.js` with the same relative
|
||
path. Then run from `server/`:
|
||
|
||
```sh
|
||
node --expose-gc --import tsx scripts/benchmark-profile-review-projection.ts --baseline-profiles "$PWD/src/services/tool-access/profiles/.baseline-service.ts" --baseline-attention "$PWD/src/services/tool-access/catalog/.baseline-attention.ts"
|
||
```
|
||
|
||
Remove the three baseline files after the comparison. The probe alternates five
|
||
samples per implementation and reports process CPU, elapsed time, actual query
|
||
count, and returned rows. No provider calls or hosted traffic are generated.
|
||
|
||
## Reproduce the measurements
|
||
|
||
Build workspace dependencies, then run from `server/`:
|
||
|
||
```sh
|
||
node --expose-gc --max-old-space-size=4096 --import tsx scripts/benchmark-tool-gateway-listing.ts --tools 50,300,900 --parallel 1,16 --repeat 3
|
||
node --expose-gc --max-old-space-size=4096 --import tsx scripts/benchmark-tool-gateway-listing.ts --tools 900 --connections 4 --parallel 16 --repeat 3
|
||
```
|
||
|
||
The harness uses a throwaway embedded PostgreSQL database. It makes no model or
|
||
provider calls. Each fixture has a 400 KB run snapshot, 140 KB result, approximately
|
||
5 KB input schemas, 15 KB connection configurations, one selector per catalog tool,
|
||
and 373 policies. Use `--implementation /absolute/path/to/tool-gateway.ts` to replay
|
||
an earlier implementation, with its corresponding policy-service import.
|
||
|
||
## Verification on 2026-10-01
|
||
|
||
Node 26.4.0 on macOS; forced GC before each measurement; heap/RSS sampled every
|
||
2 ms. Baseline: `f2e0f196308629ef05c7f65782243e1713b208f0`.
|
||
[Raw measurements](benchmarks/2026-10-01-mcp-discovery.json) retain every sample.
|
||
|
||
| Scenario | Queries | Peak extra heap | Elapsed time |
|
||
| --- | ---: | ---: | ---: |
|
||
| Baseline, 900 tools, one listing | 11,489 | 1,680–1,698 MiB | 3.54–3.63 s |
|
||
| Fixed, 900 tools, one listing | 36 | 37–37 MiB | 0.24–0.24 s |
|
||
| Fixed, 900 tools, 16 listings | 576 total | 169–187 MiB | 3.63–3.65 s |
|
||
| Fixed, 900 tools, four connections, 16 listings | 624 total | 146–174 MiB | 3.69–3.72 s |
|
||
|
||
Before the listing optimization, the 50-versus-500-tool regression test failed
|
||
with 722 versus 6,422 statements. The connection-row duplication test also failed.
|
||
The follow-up regressions failed for full run-row discovery reads, abandoned
|
||
requests, full name-list audit records, and expired named tokens. They pass with
|
||
the fixes. HTTP coverage exercises initialize, GET/SSE rejection, 16 concurrent
|
||
500-tool listings, an allowed remote call, policy revocation after discovery, and
|
||
the denied subsequent call. The remote provider response is deterministic.
|
||
|
||
The eight runnable MCP browser stories pass against a throwaway production UI
|
||
and server, including governed execution and approval journeys.
|
||
|
||
These are fixture measurements, not a replay of the incident database. RSS
|
||
includes module/runtime overhead and can stay high after V8 frees objects. Query
|
||
counts depend on connection/profile sets; uncached rate-limit checks add queries.
|
||
Catalog payload memory and policy-matching CPU still grow with the catalog.
|
||
The measurements do not establish a safe production heap limit or the share of
|
||
active installations affected. Verify restart bursts and sustained traffic on the
|
||
actual deployment before reducing its temporary heap allowance.
|
||
|
||
## Profile selector matching
|
||
|
||
Effective profile resolution and profile summaries index each profile's include
|
||
and exclude selectors once per decision. Catalog rows then use exact set
|
||
membership checks for application, connection, catalog entry, tool name, and
|
||
risk level. Exclusions still take precedence within a profile; effective profiles
|
||
still contribute the union of their allowed tools. Null application selectors
|
||
retain their exact-match behavior.
|
||
|
||
These indexes live only for the current computation. Each resolver call still
|
||
reads current company-scoped bindings, profiles, and rules. Query counts and
|
||
returned database rows are unchanged. This reduces JavaScript matching work when
|
||
a profile selects many individual tools; it does not cache authorization or
|
||
remove the runtime assignment digest check.
|
||
|
||
For a real PostgreSQL comparison, save `projection.ts` and `resolution.ts` from
|
||
the baseline commit beside their current versions as `.baseline-projection.ts`
|
||
and `.baseline-resolution.ts`. Change only the baseline resolver's projection
|
||
import to `./.baseline-projection.js`. Then run from `server/`:
|
||
|
||
```sh
|
||
node --import tsx scripts/benchmark-tool-profile-matching.ts --baseline "$PWD/src/services/tool-access/profiles/.baseline-resolution.ts"
|
||
```
|
||
|
||
Remove the two baseline files afterward. The probe uses 1,890 catalog tools,
|
||
1,575 individual include/exclude rules, one effective profile, and 12 agents. It
|
||
checks full assignment-result equality, warms both versions, and alternates
|
||
seven samples of ten resolutions. It reports actual query and returned-row
|
||
counts, process CPU, and elapsed time. The installation dependency returns an
|
||
empty list for this fixture, which has no installed connections. It does not
|
||
measure provider work, whole-run latency, or hosted performance.
|
||
|
||
## Policy selector matching
|
||
|
||
A listing decides access once per tool and shares one decision cache. The cache
|
||
holds the company's enabled policies compiled once: selectors are parsed once,
|
||
and policies are indexed by catalog entry, then by tool name for policies
|
||
without a catalog entry selector. Each decision checks only the policies in the
|
||
matching buckets plus the policies that select neither key, in the original
|
||
priority and creation order. Conditions are parsed once per policy. Decisions
|
||
without a cache, including those that consume a rate limit, compile the policies
|
||
they read in the same way. Decisions are unchanged.
|
||
|
||
To compare against a baseline commit on real PostgreSQL, run from `server/`:
|
||
|
||
```sh
|
||
pnpm exec tsx scripts/benchmark-tool-policy-matching.ts --baseline-ref origin/master --tools 800 --rounds 9
|
||
```
|
||
|
||
The probe uses 800 catalog tools and 365 enabled policies shaped like the
|
||
staging company that showed this cost (342 select a catalog entry, 18 a catalog
|
||
entry and connection, 3 several other keys, 17 carry conditions). It checks that
|
||
both versions return the same decisions, then alternates one listing per version
|
||
per round and reports elapsed time, process CPU, the longest event-loop block,
|
||
and selector work per listing.
|
||
|
||
|
||
## Local profile comparison on 2026-11-02
|
||
|
||
The profile and app-attention paths were re-profiled together after
|
||
[#18681](https://github.com/paperclipai/paperclip/pull/18681) and
|
||
[#18673](https://github.com/paperclipai/paperclip/pull/18673). Both fixes replace
|
||
nested selector scans with indexes that live for one computation.
|
||
|
||
The comparison used Node 24.22.2 on macOS arm64. Service modules were compiled
|
||
as ES modules for Node 24 before measurement. No TypeScript loader ran during
|
||
measurement. All variants used the same database, fixture, and database client.
|
||
Only the service and projection implementation changed between variants.
|
||
The fixture had 200 profiles, 1,900 active tools, 695 removed tools, 76,000
|
||
rules, and 12 agents. Complete ordered responses matched before timing.
|
||
|
||
Baseline: `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The pending-review fix was
|
||
`fd59c0183ce64c44696f79604516f4d51ca5662b`. The summary fix was
|
||
`e20165b6eb871c29e504a3837e8fe92a17ef969f`; its later benchmark helper change
|
||
does not change production code. Five unprofiled samples alternated order,
|
||
with forced GC before each operation. Values below are medians.
|
||
[All samples](benchmarks/2026-11-02-profile-projection.json) retain each measurement
|
||
and the pending-review-only treatment.
|
||
|
||
| Service | Baseline CPU | Combined CPU | Baseline elapsed | Combined elapsed |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| `listProfiles` | 3,164 ms | 385 ms | 3,045 ms | 215 ms |
|
||
| `listAppsNeedingAttention` | 2,466 ms | 221 ms | 2,378 ms | 159 ms |
|
||
|
||
Each version made seven queries per service. Profile listing returned 78,114
|
||
database rows; app attention returned 78,101. These counts did not change.
|
||
The gains came from less JavaScript work after the reads.
|
||
|
||
Separate Inspector CPU captures used a 1 ms sampling interval. The baseline
|
||
showed `profileCoversCatalogScope` and `summarizeProfile` as major CPU costs.
|
||
Their repeated scans were absent from the combined capture. Residual work
|
||
included pending projection, row conversion, and garbage collection.
|
||
Profile capture time was excluded from the timing samples above.
|
||
|
||
The committed harness is in `server/scripts/profile-tool-projection/`.
|
||
Use Node 24.22.2, install workspace dependencies, and run these commands from
|
||
the repository root in a disposable worktree. The compile step writes local
|
||
`dist/` output. It is not the full application build. The Git history must
|
||
contain the baseline and pending-fix commits named above.
|
||
|
||
```sh
|
||
git fetch https://github.com/paperclipai/paperclip.git refs/pull/18681/head
|
||
pnpm install --frozen-lockfile
|
||
node server/scripts/profile-tool-projection/compile.mjs
|
||
node --expose-gc --import ./server/scripts/profile-tool-projection/register.mjs server/scripts/profile-tool-projection/benchmark.mjs --output-dir /tmp/paperclip-tool-projection --repeat 5
|
||
node --expose-gc --import ./server/scripts/profile-tool-projection/register.mjs server/scripts/profile-tool-projection/responsiveness.mjs --output-dir /tmp/paperclip-tool-projection
|
||
```
|
||
|
||
The compile step writes the baseline and pending-only owner modules beside
|
||
the current compiled modules. A build manifest records the current Git HEAD
|
||
and the SHA-256 hashes of the four current owner source files. The benchmark
|
||
records this manifest, validates complete output equality, then measures
|
||
`profiles` and `attention` separately for `baseline`, `pending`, and `combined`.
|
||
It writes `measurements.json` and six separate `.cpuprofile` files to the
|
||
output directory. The HTTP probe writes `responsiveness.json` there. Keep
|
||
CPU profiles private because they contain local source paths. Rebuild and
|
||
rerun if the owner files change. Serialize these commands with other heavy
|
||
builds and benchmarks on the same host.
|
||
|
||
A separate process also sent simple HTTP requests every 20 ms while each
|
||
real service operation ran. The local liveness handler shared the service's
|
||
Node event loop. There was at most one outstanding probe request. Across
|
||
three alternating trials, the median largest probe delay fell from 3,008 ms
|
||
to 78 ms during profile listing, and from 2,103 ms to 64 ms during app
|
||
attention. All 48 probe requests completed without error.
|
||
|
||
These are synthetic service measurements on a shared developer machine.
|
||
They are not a full application build, a browser test, or hosted acceptance.
|
||
The liveness handler is a minimal test handler, not the application's health
|
||
route. Workload, JIT warmup, and host activity affect absolute timings. The
|
||
comparison does not establish a page-load percentage or production capacity.
|
||
Use a matched deployment window to confirm the effect on user requests.
|
||
|
||
Both benchmark scripts use the test database helper. Set
|
||
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use a local PostgreSQL server; otherwise
|
||
they create and remove an embedded test database. The helper creates a unique
|
||
disposable database and removes it afterward. Benchmark setup and fixture
|
||
insertion are outside the timing samples.
|