mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs get their tools from a session tool authority, which holds the run's assigned app (MCP) tool catalog > - #21415 releases a finished session's authority, so the server heap no longer grows with each finished session > - Two behaviors of that fix had no test: a released session fails closed, and the release when a follow-up run retires its warm predecessor and then fails before it admits its own authority > - The memory saving was measured once, by hand, with no script in the repository > - This pull request adds the two tests and a benchmark script. It changes no product code > - The benefit is that a later change cannot silently bring back the leak or let a released session reach its tools ## Linked Issues or Issue Description Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate; a reviewer comment there named the follow-up failure path). **What is missing?** #21415 tests that finished sessions return the authority map to its starting size. No test checks that a released session refuses late tool calls. No test checks the follow-up path, where a follow-up run retires the previous warm session (so the release at forget time is skipped while the follow-up owns the scope) and then fails before it admits its own authority. **Expected behavior** A late tool call or completion feedback for a released session fails with `native_session_tool_authority_unavailable`. The next run of the same scope gets a new authority with only its own assignments. When a follow-up fails after it retires its predecessor, the authority map is back at its starting size after that run settles. **Paperclip version or commit** master `edb6945ff3` (after #21415, `85f4a91e70`). ## What Changed - Test "refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments". - Test "releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own". A follow-up of the same warm scope with changed configuration retires the predecessor. Then its assigned tool listing fails. The test asserts that the predecessor was closed, that the follow-up failed with that error, and that `sessionToolAuthorityCount()` is back at its starting value. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained bytes of one assigned tool catalog per session, and schema copies per `definitions()` call. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "released native session tool authority|retired warm predecessor"`: 2 passed. - Red check: with the release at settle removed from `native-session-executor.ts`, both tests fail (`expected 1 to be +0`, and the late tool call is not refused). - Retained bytes per finished per-turn session were measured through `executePaperclipNativeSession` with the real `createAssignedMcpTools` (80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On `85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794 bytes (test-harness noise), and the authority map stayed at its starting size. - `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes 2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per assigned tool catalog. Each `definitions()` call makes 80 new schema copies (about 296 KB), and none are kept. - Full `native-session-executor.test.ts` locally, with a load average near 78 from other work on the machine: 1,483 passed and 5 failed. All 5 failures were timeouts in tests this change does not touch (the slowest took 163 s). CI gives the clean full-suite result. ## Risks - Low risk. Tests and one benchmark script only. No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
df1b12ad8e
commit
89e6575418
2 files changed
+181
No files matched your search
@@ -0,0 +1,70 @@
|
||||
import type { ToolGatewayDescriptor, ToolGatewayService } from "../src/services/tool-gateway.js";
|
||||
import { createAssignedMcpTools } from "../src/services/native-runtime/assigned-mcp-tools.js";
|
||||
|
||||
function option(name: string, fallback: number) {
|
||||
const index = process.argv.indexOf(name);
|
||||
const value = Number(index < 0 ? fallback : process.argv[index + 1]);
|
||||
if (!Number.isInteger(value) || value < 1) throw new Error(`Expected a positive integer for ${name}`);
|
||||
return value;
|
||||
}
|
||||
const toolCount = option("--tools", 80);
|
||||
const schemaBytes = option("--schema-bytes", 2048);
|
||||
const sessions = option("--sessions", 200);
|
||||
const turns = option("--turns", 20);
|
||||
const gc = (globalThis as { gc?: () => void }).gc;
|
||||
if (!gc) throw new Error("Run with node --expose-gc");
|
||||
|
||||
function listing(): ToolGatewayDescriptor[] {
|
||||
return Array.from({ length: toolCount }, (_, index) => {
|
||||
const properties: Record<string, unknown> = {};
|
||||
for (let field = 0; JSON.stringify(properties).length < schemaBytes; field += 1) {
|
||||
properties[`field_${field}`] = { type: "string", description: `Field ${field} of tool ${index} with representative documentation text.` };
|
||||
}
|
||||
return {
|
||||
name: `fixture_app:tool_${index}`, displayName: `Tool ${index}`, description: `Fixture tool ${index}`,
|
||||
parametersSchema: { type: "object", properties, required: ["field_0"] },
|
||||
pluginId: "fixture", providerType: "mcp_remote_http", risk: "read",
|
||||
} as ToolGatewayDescriptor;
|
||||
});
|
||||
}
|
||||
|
||||
const gateway = { listToolsForNamedGateway: async () => listing() } as unknown as ToolGatewayService;
|
||||
const create = () => createAssignedMcpTools({ gateway, gatewayPublicId: "benchmark", bearerToken: "benchmark" });
|
||||
|
||||
function heap() {
|
||||
gc!(); gc!();
|
||||
return process.memoryUsage().heapUsed;
|
||||
}
|
||||
|
||||
function schemaIdentities(definitions: Array<Record<string, unknown>>) {
|
||||
return definitions.map(definition => definition.inputSchema);
|
||||
}
|
||||
|
||||
async function measureSessions(baseline: number) {
|
||||
const held: Array<Awaited<ReturnType<typeof create>>> = [];
|
||||
for (let index = 0; index < sessions; index += 1) held.push(await create());
|
||||
const retainedBytesPerSession = Math.round((heap() - baseline) / sessions);
|
||||
const turnResults: Array<Array<Record<string, unknown>>> = [];
|
||||
const beforeTurns = heap();
|
||||
for (let index = 0; index < turns; index += 1) turnResults.push(held[0]!.definitions());
|
||||
const retainedBytesPerTurn = Math.round((heap() - beforeTurns) / turns);
|
||||
const first = schemaIdentities(turnResults[0]!);
|
||||
const schemaObjectsAfterTurns = new Set(turnResults.flatMap(schemaIdentities)).size;
|
||||
const sharedAcrossTurns = turnResults.every(result => schemaIdentities(result).every((schema, index) => schema === first[index]));
|
||||
return { retainedBytesPerSession, retainedBytesPerTurn, schemaObjectsAfterTurns, sharedAcrossTurns };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
await create();
|
||||
const baseline = heap();
|
||||
const measured = await measureSessions(baseline);
|
||||
const released = heap();
|
||||
console.log(JSON.stringify({
|
||||
tools: toolCount, schemaBytes, sessions, turns,
|
||||
...measured,
|
||||
schemaObjectsPerTurn: measured.schemaObjectsAfterTurns / turns,
|
||||
bytesStillHeldAfterRelease: Math.max(0, released - baseline),
|
||||
}, null, 2));
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -12513,6 +12513,52 @@ describe("native warm session supervision", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own", async () => {
|
||||
const { base } = fixture("authority-failed-follow-up");
|
||||
const next = { ...base, binding: { ...base.binding, runId: "authority-failed-follow-up-next" } };
|
||||
const root = await mkdtemp(join(tmpdir(), "native-authority-failed-follow-up-"));
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", root);
|
||||
const close = vi.fn(async () => undefined);
|
||||
const issueService = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({
|
||||
update: vi.fn(async () => ({ status: "blocked", statusVersion: 1 })),
|
||||
} as unknown as ReturnType<typeof issueServiceModule.issueService>);
|
||||
const assigned = { PAPERCLIP_NATIVE_MCP_NAME: "assigned", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-follow-up",
|
||||
PAPERCLIP_NATIVE_MCP_TOKEN: "follow-up-fixture" };
|
||||
state.createAssignedMcpTools.mockResolvedValueOnce({ definitions: () => [], has: () => false, execute: vi.fn() })
|
||||
.mockRejectedValueOnce(new Error("assigned tools unavailable"));
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const factory = state.createTransport.mock.calls.at(-1)![0];
|
||||
const identity = { ...factory.prpIdentity, normalizedSessionId: base.session.normalizedSessionId };
|
||||
await mkdir(join(factory.stateDirectory!, "control-plane"), { recursive: true });
|
||||
await mkdir(join(factory.stateDirectory!, "runner"), { recursive: true });
|
||||
await writeFile(join(factory.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
|
||||
await writeFile(join(factory.stateDirectory!, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(identity, "ready")));
|
||||
await options.onSession?.({ close }); return result;
|
||||
});
|
||||
try {
|
||||
const before = sessionToolAuthorityCount();
|
||||
await executePaperclipNativeSession({ db: leaseDb(base), execution: base, runnerInstanceId: "runner", useRunnerd: true,
|
||||
runnerEnvironment: assigned });
|
||||
expect(sessionToolAuthorityCount()).toBe(before + 1);
|
||||
const current = leaseDb(next);
|
||||
const nextDb = { ...current, select: (columns?: Record<string, unknown>) => columns?.status && columns?.runnerProfileJson
|
||||
? { from: () => ({ where: () => ({ limit: () => Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: base } }]) }) }) }
|
||||
: current.select(columns as never) } as unknown as Db;
|
||||
const failure = await executePaperclipNativeSession({ db: nextDb, execution: next, runnerInstanceId: "runner", useRunnerd: true,
|
||||
runnerEnvironment: { ...assigned, PAPERCLIP_AGENT_KEY_ID: "sha256:follow-up-key" } }).catch((error: Error) => error);
|
||||
expect(failure).toMatchObject({ message: "assigned tools unavailable" });
|
||||
expect(state.createAssignedMcpTools).toHaveBeenCalledTimes(2);
|
||||
expect(close).toHaveBeenCalledWith({ reason: "warm native session configuration changed" });
|
||||
expect(state.execute).toHaveBeenCalledOnce();
|
||||
expect(sessionToolAuthorityCount()).toBe(before);
|
||||
} finally {
|
||||
issueService.mockRestore();
|
||||
vi.unstubAllEnvs();
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("awaits prior idle ownership retirement before launching the accepted-plan session", async () => {
|
||||
const { base, next, run } = fixture("identity-handoff");
|
||||
let finishClose!: () => void;
|
||||
@@ -23099,3 +23145,68 @@ describe("isTransientRemoteCheckpointReadFailure", () => {
|
||||
await expect(retryTransientRemoteCheckpointRead(async () => { if (drops-- > 0) throw reset(); return "settled"; }, vi.fn(), 0)).resolves.toBe("settled");
|
||||
});
|
||||
});
|
||||
|
||||
describe("released native session tool authority", () => {
|
||||
const assignedTool = (label: string) => ({ name: `app_released_${label}`, description: `Released ${label}`, inputSchema: { type: "object" } });
|
||||
const assignedTools = (label: string) => {
|
||||
const tool = assignedTool(label);
|
||||
return { definitions: () => [tool], has: (name: string) => name === tool.name,
|
||||
execute: vi.fn(async (_call: unknown) => ({ content: [{ type: "text", text: label }] })) };
|
||||
};
|
||||
const scoped = (runId: string) => ({ ...execution,
|
||||
binding: { ...execution.binding, runId, issueId: "released-authority-issue" },
|
||||
session: { ...execution.session, normalizedSessionId: "released-authority-session",
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as NativeExecutionInputV1);
|
||||
const environment = { PAPERCLIP_NATIVE_MCP_NAME: "assigned", PAPERCLIP_NATIVE_MCP_URL: "http://localhost/mcp/gateways/assigned-released",
|
||||
PAPERCLIP_NATIVE_MCP_TOKEN: "released-token" };
|
||||
const call = (tool: string) => ({ tool, callId: `call-${tool}`, arguments: {} }) as never;
|
||||
|
||||
beforeEach(() => {
|
||||
state.toolAuthorityDefinitions.mockImplementation(async binding =>
|
||||
((binding.assignedMcpTools as ReturnType<typeof assignedTools> | undefined)?.definitions() ?? []) as never);
|
||||
state.toolAuthorityExecute.mockReset().mockImplementation(async (binding: Record<string, unknown>, value: unknown) => {
|
||||
const tools = binding.assignedMcpTools as ReturnType<typeof assignedTools>;
|
||||
const { tool, arguments: args } = value as { tool: string; arguments: unknown };
|
||||
if (!tools.has(tool)) throw new Error("unknown tool");
|
||||
return tools.execute({ tool, arguments: args });
|
||||
});
|
||||
});
|
||||
afterEach(() => {
|
||||
state.toolAuthorityDefinitions.mockImplementation(async () => []);
|
||||
state.toolAuthorityExecute.mockReset();
|
||||
});
|
||||
|
||||
it("refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments", async () => {
|
||||
const first = assignedTools("first"), second = assignedTools("second");
|
||||
const current = scoped("released-authority-first");
|
||||
state.createAssignedMcpTools.mockResolvedValueOnce(first);
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close: async () => {} });
|
||||
return { result: { summary: "done" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn",
|
||||
normalizedSessionId: current.session.normalizedSessionId, providerSessionId: null, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 0, highestContiguousSourceSeq: 0 };
|
||||
});
|
||||
await executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: "runner", useRunnerd: true,
|
||||
runnerEnvironment: environment });
|
||||
const released = state.createBackend.mock.calls.at(-1)![1];
|
||||
await expect(Promise.resolve().then(() => released.dynamicToolHandler!(call(assignedTool("first").name))))
|
||||
.rejects.toThrow("native_session_tool_authority_unavailable");
|
||||
await expect(released.completionFeedback!({} as never)).rejects.toThrow("native_session_tool_authority_unavailable");
|
||||
expect(first.execute).not.toHaveBeenCalled();
|
||||
const next = scoped("released-authority-second");
|
||||
state.createAssignedMcpTools.mockResolvedValueOnce(second);
|
||||
const directory = await mkdtemp(join(tmpdir(), "native-authority-readmission-"));
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", directory);
|
||||
try {
|
||||
await createRunnerdBackend({ db: leaseDb(next), execution: next, runnerInstanceId: "runner", runnerEnvironment: environment });
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
const admitted = state.createBackend.mock.calls.at(-1)![1];
|
||||
await expect(admitted.dynamicToolHandler!(call(assignedTool("second").name))).resolves.toEqual({ content: [{ type: "text", text: "second" }] });
|
||||
await expect(admitted.dynamicToolHandler!(call(assignedTool("first").name))).rejects.toThrow("unknown tool");
|
||||
expect(first.execute).not.toHaveBeenCalled();
|
||||
expect(state.createAssignedMcpTools).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user