mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs get their tools from a session tool authority, which holds the run's assigned app (MCP) tool catalog > - #21415 releases a finished session's authority, so the server heap no longer grows with each finished session > - Two behaviors of that fix had no test: a released session fails closed, and the release when a follow-up run retires its warm predecessor and then fails before it admits its own authority > - The memory saving was measured once, by hand, with no script in the repository > - This pull request adds the two tests and a benchmark script. It changes no product code > - The benefit is that a later change cannot silently bring back the leak or let a released session reach its tools ## Linked Issues or Issue Description Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate; a reviewer comment there named the follow-up failure path). **What is missing?** #21415 tests that finished sessions return the authority map to its starting size. No test checks that a released session refuses late tool calls. No test checks the follow-up path, where a follow-up run retires the previous warm session (so the release at forget time is skipped while the follow-up owns the scope) and then fails before it admits its own authority. **Expected behavior** A late tool call or completion feedback for a released session fails with `native_session_tool_authority_unavailable`. The next run of the same scope gets a new authority with only its own assignments. When a follow-up fails after it retires its predecessor, the authority map is back at its starting size after that run settles. **Paperclip version or commit** master `edb6945ff3` (after #21415, `85f4a91e70`). ## What Changed - Test "refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments". - Test "releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own". A follow-up of the same warm scope with changed configuration retires the predecessor. Then its assigned tool listing fails. The test asserts that the predecessor was closed, that the follow-up failed with that error, and that `sessionToolAuthorityCount()` is back at its starting value. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained bytes of one assigned tool catalog per session, and schema copies per `definitions()` call. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "released native session tool authority|retired warm predecessor"`: 2 passed. - Red check: with the release at settle removed from `native-session-executor.ts`, both tests fail (`expected 1 to be +0`, and the late tool call is not refused). - Retained bytes per finished per-turn session were measured through `executePaperclipNativeSession` with the real `createAssignedMcpTools` (80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On `85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794 bytes (test-harness noise), and the authority map stayed at its starting size. - `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes 2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per assigned tool catalog. Each `definitions()` call makes 80 new schema copies (about 296 KB), and none are kept. - Full `native-session-executor.test.ts` locally, with a load average near 78 from other work on the machine: 1,483 passed and 5 failed. All 5 failures were timeouts in tests this change does not touch (the slowest took 163 s). CI gives the clean full-suite result. ## Risks - Low risk. Tests and one benchmark script only. No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
71 lines
3.2 KiB
TypeScript
71 lines
3.2 KiB
TypeScript
import type { ToolGatewayDescriptor, ToolGatewayService } from "../src/services/tool-gateway.js";
|
|
import { createAssignedMcpTools } from "../src/services/native-runtime/assigned-mcp-tools.js";
|
|
|
|
function option(name: string, fallback: number) {
|
|
const index = process.argv.indexOf(name);
|
|
const value = Number(index < 0 ? fallback : process.argv[index + 1]);
|
|
if (!Number.isInteger(value) || value < 1) throw new Error(`Expected a positive integer for ${name}`);
|
|
return value;
|
|
}
|
|
const toolCount = option("--tools", 80);
|
|
const schemaBytes = option("--schema-bytes", 2048);
|
|
const sessions = option("--sessions", 200);
|
|
const turns = option("--turns", 20);
|
|
const gc = (globalThis as { gc?: () => void }).gc;
|
|
if (!gc) throw new Error("Run with node --expose-gc");
|
|
|
|
function listing(): ToolGatewayDescriptor[] {
|
|
return Array.from({ length: toolCount }, (_, index) => {
|
|
const properties: Record<string, unknown> = {};
|
|
for (let field = 0; JSON.stringify(properties).length < schemaBytes; field += 1) {
|
|
properties[`field_${field}`] = { type: "string", description: `Field ${field} of tool ${index} with representative documentation text.` };
|
|
}
|
|
return {
|
|
name: `fixture_app:tool_${index}`, displayName: `Tool ${index}`, description: `Fixture tool ${index}`,
|
|
parametersSchema: { type: "object", properties, required: ["field_0"] },
|
|
pluginId: "fixture", providerType: "mcp_remote_http", risk: "read",
|
|
} as ToolGatewayDescriptor;
|
|
});
|
|
}
|
|
|
|
const gateway = { listToolsForNamedGateway: async () => listing() } as unknown as ToolGatewayService;
|
|
const create = () => createAssignedMcpTools({ gateway, gatewayPublicId: "benchmark", bearerToken: "benchmark" });
|
|
|
|
function heap() {
|
|
gc!(); gc!();
|
|
return process.memoryUsage().heapUsed;
|
|
}
|
|
|
|
function schemaIdentities(definitions: Array<Record<string, unknown>>) {
|
|
return definitions.map(definition => definition.inputSchema);
|
|
}
|
|
|
|
async function measureSessions(baseline: number) {
|
|
const held: Array<Awaited<ReturnType<typeof create>>> = [];
|
|
for (let index = 0; index < sessions; index += 1) held.push(await create());
|
|
const retainedBytesPerSession = Math.round((heap() - baseline) / sessions);
|
|
const turnResults: Array<Array<Record<string, unknown>>> = [];
|
|
const beforeTurns = heap();
|
|
for (let index = 0; index < turns; index += 1) turnResults.push(held[0]!.definitions());
|
|
const retainedBytesPerTurn = Math.round((heap() - beforeTurns) / turns);
|
|
const first = schemaIdentities(turnResults[0]!);
|
|
const schemaObjectsAfterTurns = new Set(turnResults.flatMap(schemaIdentities)).size;
|
|
const sharedAcrossTurns = turnResults.every(result => schemaIdentities(result).every((schema, index) => schema === first[index]));
|
|
return { retainedBytesPerSession, retainedBytesPerTurn, schemaObjectsAfterTurns, sharedAcrossTurns };
|
|
}
|
|
|
|
async function main() {
|
|
await create();
|
|
const baseline = heap();
|
|
const measured = await measureSessions(baseline);
|
|
const released = heap();
|
|
console.log(JSON.stringify({
|
|
tools: toolCount, schemaBytes, sessions, turns,
|
|
...measured,
|
|
schemaObjectsPerTurn: measured.schemaObjectsAfterTurns / turns,
|
|
bytesStillHeldAfterRelease: Math.max(0, released - baseline),
|
|
}, null, 2));
|
|
}
|
|
|
|
await main();
|