test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 12:17:25 -06:00
1 parent df1b12ad8e
commit 89e6575418
2 files changed
+181

No files matched your search

@@ -0,0 +1,70 @@
import type { ToolGatewayDescriptor, ToolGatewayService } from "../src/services/tool-gateway.js";
import { createAssignedMcpTools } from "../src/services/native-runtime/assigned-mcp-tools.js";
function option(name: string, fallback: number) {
const index = process.argv.indexOf(name);
const value = Number(index < 0 ? fallback : process.argv[index + 1]);
if (!Number.isInteger(value) || value < 1) throw new Error(`Expected a positive integer for ${name}`);
return value;
}
const toolCount = option("--tools", 80);
const schemaBytes = option("--schema-bytes", 2048);
const sessions = option("--sessions", 200);
const turns = option("--turns", 20);
const gc = (globalThis as { gc?: () => void }).gc;
if (!gc) throw new Error("Run with node --expose-gc");
function listing(): ToolGatewayDescriptor[] {
return Array.from({ length: toolCount }, (_, index) => {
const properties: Record<string, unknown> = {};
for (let field = 0; JSON.stringify(properties).length < schemaBytes; field += 1) {
properties[`field_${field}`] = { type: "string", description: `Field ${field} of tool ${index} with representative documentation text.` };
}
return {
name: `fixture_app:tool_${index}`, displayName: `Tool ${index}`, description: `Fixture tool ${index}`,
parametersSchema: { type: "object", properties, required: ["field_0"] },
pluginId: "fixture", providerType: "mcp_remote_http", risk: "read",
} as ToolGatewayDescriptor;
});
}
const gateway = { listToolsForNamedGateway: async () => listing() } as unknown as ToolGatewayService;
const create = () => createAssignedMcpTools({ gateway, gatewayPublicId: "benchmark", bearerToken: "benchmark" });
function heap() {
gc!(); gc!();
return process.memoryUsage().heapUsed;
}
function schemaIdentities(definitions: Array<Record<string, unknown>>) {
return definitions.map(definition => definition.inputSchema);
}
async function measureSessions(baseline: number) {
const held: Array<Awaited<ReturnType<typeof create>>> = [];
for (let index = 0; index < sessions; index += 1) held.push(await create());
const retainedBytesPerSession = Math.round((heap() - baseline) / sessions);
const turnResults: Array<Array<Record<string, unknown>>> = [];
const beforeTurns = heap();
for (let index = 0; index < turns; index += 1) turnResults.push(held[0]!.definitions());
const retainedBytesPerTurn = Math.round((heap() - beforeTurns) / turns);
const first = schemaIdentities(turnResults[0]!);
const schemaObjectsAfterTurns = new Set(turnResults.flatMap(schemaIdentities)).size;
const sharedAcrossTurns = turnResults.every(result => schemaIdentities(result).every((schema, index) => schema === first[index]));
return { retainedBytesPerSession, retainedBytesPerTurn, schemaObjectsAfterTurns, sharedAcrossTurns };
}
async function main() {
await create();
const baseline = heap();
const measured = await measureSessions(baseline);
const released = heap();
console.log(JSON.stringify({
tools: toolCount, schemaBytes, sessions, turns,
...measured,
schemaObjectsPerTurn: measured.schemaObjectsAfterTurns / turns,
bytesStillHeldAfterRelease: Math.max(0, released - baseline),
}, null, 2));
}
await main();
@@ -12513,6 +12513,52 @@ describe("native warm session supervision", () => {
}
});
it("releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own", async () => {
const { base } = fixture("authority-failed-follow-up");
const next = { ...base, binding: { ...base.binding, runId: "authority-failed-follow-up-next" } };
const root = await mkdtemp(join(tmpdir(), "native-authority-failed-follow-up-"));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", root);
const close = vi.fn(async () => undefined);
const issueService = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({
update: vi.fn(async () => ({ status: "blocked", statusVersion: 1 })),
} as unknown as ReturnType<typeof issueServiceModule.issueService>);
const assigned = { PAPERCLIP_NATIVE_MCP_NAME: "assigned", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-follow-up",
PAPERCLIP_NATIVE_MCP_TOKEN: "follow-up-fixture" };
state.createAssignedMcpTools.mockResolvedValueOnce({ definitions: () => [], has: () => false, execute: vi.fn() })
.mockRejectedValueOnce(new Error("assigned tools unavailable"));
state.execute.mockReset().mockImplementationOnce(async options => {
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const factory = state.createTransport.mock.calls.at(-1)![0];
const identity = { ...factory.prpIdentity, normalizedSessionId: base.session.normalizedSessionId };
await mkdir(join(factory.stateDirectory!, "control-plane"), { recursive: true });
await mkdir(join(factory.stateDirectory!, "runner"), { recursive: true });
await writeFile(join(factory.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
await writeFile(join(factory.stateDirectory!, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(identity, "ready")));
await options.onSession?.({ close }); return result;
});
try {
const before = sessionToolAuthorityCount();
await executePaperclipNativeSession({ db: leaseDb(base), execution: base, runnerInstanceId: "runner", useRunnerd: true,
runnerEnvironment: assigned });
expect(sessionToolAuthorityCount()).toBe(before + 1);
const current = leaseDb(next);
const nextDb = { ...current, select: (columns?: Record<string, unknown>) => columns?.status && columns?.runnerProfileJson
? { from: () => ({ where: () => ({ limit: () => Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: base } }]) }) }) }
: current.select(columns as never) } as unknown as Db;
const failure = await executePaperclipNativeSession({ db: nextDb, execution: next, runnerInstanceId: "runner", useRunnerd: true,
runnerEnvironment: { ...assigned, PAPERCLIP_AGENT_KEY_ID: "sha256:follow-up-key" } }).catch((error: Error) => error);
expect(failure).toMatchObject({ message: "assigned tools unavailable" });
expect(state.createAssignedMcpTools).toHaveBeenCalledTimes(2);
expect(close).toHaveBeenCalledWith({ reason: "warm native session configuration changed" });
expect(state.execute).toHaveBeenCalledOnce();
expect(sessionToolAuthorityCount()).toBe(before);
} finally {
issueService.mockRestore();
vi.unstubAllEnvs();
await rm(root, { recursive: true, force: true });
}
});
it("awaits prior idle ownership retirement before launching the accepted-plan session", async () => {
const { base, next, run } = fixture("identity-handoff");
let finishClose!: () => void;
@@ -23099,3 +23145,68 @@ describe("isTransientRemoteCheckpointReadFailure", () => {
await expect(retryTransientRemoteCheckpointRead(async () => { if (drops-- > 0) throw reset(); return "settled"; }, vi.fn(), 0)).resolves.toBe("settled");
});
});
describe("released native session tool authority", () => {
const assignedTool = (label: string) => ({ name: `app_released_${label}`, description: `Released ${label}`, inputSchema: { type: "object" } });
const assignedTools = (label: string) => {
const tool = assignedTool(label);
return { definitions: () => [tool], has: (name: string) => name === tool.name,
execute: vi.fn(async (_call: unknown) => ({ content: [{ type: "text", text: label }] })) };
};
const scoped = (runId: string) => ({ ...execution,
binding: { ...execution.binding, runId, issueId: "released-authority-issue" },
session: { ...execution.session, normalizedSessionId: "released-authority-session",
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as NativeExecutionInputV1);
const environment = { PAPERCLIP_NATIVE_MCP_NAME: "assigned", PAPERCLIP_NATIVE_MCP_URL: "http://localhost/mcp/gateways/assigned-released",
PAPERCLIP_NATIVE_MCP_TOKEN: "released-token" };
const call = (tool: string) => ({ tool, callId: `call-${tool}`, arguments: {} }) as never;
beforeEach(() => {
state.toolAuthorityDefinitions.mockImplementation(async binding =>
((binding.assignedMcpTools as ReturnType<typeof assignedTools> | undefined)?.definitions() ?? []) as never);
state.toolAuthorityExecute.mockReset().mockImplementation(async (binding: Record<string, unknown>, value: unknown) => {
const tools = binding.assignedMcpTools as ReturnType<typeof assignedTools>;
const { tool, arguments: args } = value as { tool: string; arguments: unknown };
if (!tools.has(tool)) throw new Error("unknown tool");
return tools.execute({ tool, arguments: args });
});
});
afterEach(() => {
state.toolAuthorityDefinitions.mockImplementation(async () => []);
state.toolAuthorityExecute.mockReset();
});
it("refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments", async () => {
const first = assignedTools("first"), second = assignedTools("second");
const current = scoped("released-authority-first");
state.createAssignedMcpTools.mockResolvedValueOnce(first);
state.execute.mockReset().mockImplementationOnce(async options => {
await options.onSession?.({ close: async () => {} });
return { result: { summary: "done" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn",
normalizedSessionId: current.session.normalizedSessionId, providerSessionId: null, driverKind: "test", driverVersion: "1",
nativeEventCount: 0, highestContiguousSourceSeq: 0 };
});
await executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: "runner", useRunnerd: true,
runnerEnvironment: environment });
const released = state.createBackend.mock.calls.at(-1)![1];
await expect(Promise.resolve().then(() => released.dynamicToolHandler!(call(assignedTool("first").name))))
.rejects.toThrow("native_session_tool_authority_unavailable");
await expect(released.completionFeedback!({} as never)).rejects.toThrow("native_session_tool_authority_unavailable");
expect(first.execute).not.toHaveBeenCalled();
const next = scoped("released-authority-second");
state.createAssignedMcpTools.mockResolvedValueOnce(second);
const directory = await mkdtemp(join(tmpdir(), "native-authority-readmission-"));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", directory);
try {
await createRunnerdBackend({ db: leaseDb(next), execution: next, runnerInstanceId: "runner", runnerEnvironment: environment });
} finally {
vi.unstubAllEnvs();
await rm(directory, { recursive: true, force: true });
}
const admitted = state.createBackend.mock.calls.at(-1)![1];
await expect(admitted.dynamicToolHandler!(call(assignedTool("second").name))).resolves.toEqual({ content: [{ type: "text", text: "second" }] });
await expect(admitted.dynamicToolHandler!(call(assignedTool("first").name))).rejects.toThrow("unknown tool");
expect(first.execute).not.toHaveBeenCalled();
expect(state.createAssignedMcpTools).toHaveBeenCalledTimes(2);
});
});