Files
PaperClipAI/server/scripts/benchmark-assigned-mcp-tools-memory.ts
T
DottaandClaude Opus 5.5 89e6575418 test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:25 -06:00

71 lines
3.2 KiB
TypeScript

import type { ToolGatewayDescriptor, ToolGatewayService } from "../src/services/tool-gateway.js";
import { createAssignedMcpTools } from "../src/services/native-runtime/assigned-mcp-tools.js";
function option(name: string, fallback: number) {
const index = process.argv.indexOf(name);
const value = Number(index < 0 ? fallback : process.argv[index + 1]);
if (!Number.isInteger(value) || value < 1) throw new Error(`Expected a positive integer for ${name}`);
return value;
}
const toolCount = option("--tools", 80);
const schemaBytes = option("--schema-bytes", 2048);
const sessions = option("--sessions", 200);
const turns = option("--turns", 20);
const gc = (globalThis as { gc?: () => void }).gc;
if (!gc) throw new Error("Run with node --expose-gc");
function listing(): ToolGatewayDescriptor[] {
return Array.from({ length: toolCount }, (_, index) => {
const properties: Record<string, unknown> = {};
for (let field = 0; JSON.stringify(properties).length < schemaBytes; field += 1) {
properties[`field_${field}`] = { type: "string", description: `Field ${field} of tool ${index} with representative documentation text.` };
}
return {
name: `fixture_app:tool_${index}`, displayName: `Tool ${index}`, description: `Fixture tool ${index}`,
parametersSchema: { type: "object", properties, required: ["field_0"] },
pluginId: "fixture", providerType: "mcp_remote_http", risk: "read",
} as ToolGatewayDescriptor;
});
}
const gateway = { listToolsForNamedGateway: async () => listing() } as unknown as ToolGatewayService;
const create = () => createAssignedMcpTools({ gateway, gatewayPublicId: "benchmark", bearerToken: "benchmark" });
function heap() {
gc!(); gc!();
return process.memoryUsage().heapUsed;
}
function schemaIdentities(definitions: Array<Record<string, unknown>>) {
return definitions.map(definition => definition.inputSchema);
}
async function measureSessions(baseline: number) {
const held: Array<Awaited<ReturnType<typeof create>>> = [];
for (let index = 0; index < sessions; index += 1) held.push(await create());
const retainedBytesPerSession = Math.round((heap() - baseline) / sessions);
const turnResults: Array<Array<Record<string, unknown>>> = [];
const beforeTurns = heap();
for (let index = 0; index < turns; index += 1) turnResults.push(held[0]!.definitions());
const retainedBytesPerTurn = Math.round((heap() - beforeTurns) / turns);
const first = schemaIdentities(turnResults[0]!);
const schemaObjectsAfterTurns = new Set(turnResults.flatMap(schemaIdentities)).size;
const sharedAcrossTurns = turnResults.every(result => schemaIdentities(result).every((schema, index) => schema === first[index]));
return { retainedBytesPerSession, retainedBytesPerTurn, schemaObjectsAfterTurns, sharedAcrossTurns };
}
async function main() {
await create();
const baseline = heap();
const measured = await measureSessions(baseline);
const released = heap();
console.log(JSON.stringify({
tools: toolCount, schemaBytes, sessions, turns,
...measured,
schemaObjectsPerTurn: measured.schemaObjectsAfterTurns / turns,
bytesStillHeldAfterRelease: Math.max(0, released - baseline),
}, null, 2));
}
await main();