From cf89f0436c7d87e6f1ac86b9f626296cfd562b54 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:29:30 -0600 Subject: [PATCH] fix(native-runtime): name the closest assigned tools when a tool name is mistyped (#22327) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runners give agents their assigned app tools through two wrapper tools: `paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`. > - Search returns each tool with a hashed alias (`app__`). #22213 also accepts an unambiguous gateway name or display name. > - Some models copy the alias with an error: they drop or change one digest character, change the characters of the bot id, or attach the digest of a different tool. > - The wrapper correctly rejects these names, because a guess could select a write tool when the agent meant a read tool. > - But the error only said "name must be an exact name". The agent searched again and often repeated the same mistake. > - This pull request adds the closest names that the agent can use to the error. It does not resolve any name by approximation. > - The benefit is a faster recovery after a mistyped name, with no change to authorization. ## Linked Issues or Issue Description Refs #22213 **What happened?** An agent searched its assigned tools and then called `paperclip_call_assigned_tool` with a name that is almost the returned alias. For example, one digest character was missing, or the agent used the digest of the read tool with the name of the submit tool. The call failed with `assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools. Search assigned tools again and use the returned name.` In observed runs, the agent made more failed calls with other wrong names. **Expected behavior** The call is still rejected. The error also shows the closest assigned names that the agent can use, so that the agent can correct the name directly. **Paperclip version or commit** `master` at c0aa7751f8. **Steps to reproduce** 1. Assign the tools `github_bot:read_pull_request` (read) and `github_bot:begin_review` (write) to an agent on an on-demand native runner. 2. Call `paperclip_search_assigned_tools` with `{"query":"github_bot"}` and copy the alias of `begin_review`. 3. Remove one character from the digest and call `paperclip_call_assigned_tool` with that name. 4. See `assigned_mcp_tool_unknown` with no information about the correct name. ## What Changed - `server/src/services/native-runtime/assigned-mcp-tools.ts`: - Add `closestNames`. It returns up to three candidate names whose edit distance (case-insensitive) is not more than one third of the longer name. It skips the distance computation when the length difference is already more than this limit. Ties sort by name. - The unknown-name error of `paperclip_call_assigned_tool` now includes `Closest assigned names: , , .` when a near name exists. If no name is near, the error text is the same as before. - The unknown `schemaTool` error of search includes the same list. - If the fresh grant list fails on the call path (for example, a revoked token), the call still returns `assigned_mcp_tool_unknown`, with no hint. The hint never replaces the error. - Both lists use one new helper, `authorizedTools`. It is the fresh-grant and work-mode filter that search already used. A revoked tool and a tool that the current work mode does not permit are never shown. - `editDistance` is a local helper in the same module. It has the same algorithm as the private helper in `packages/shared/src/config-schema.ts`. That shared file is pinned as a reviewed SDK reference source, so an export from it would require a new pin review. A second commit tried this, and the runner support suite failed on the stale pin, so the third commit reverted it. - `server/src/services/native-runtime/assigned-mcp-tools.test.ts`: add a regression test. ## Verification - Failing before the first fix: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/assigned-mcp-tools.test.ts -t "closest permitted"` failed with `AssertionError: expected 'assigned_mcp_tool_unknown: name must …' to contain 'Closest assigned names: app_github_bo…'`. - Failing before the review fixes (second commit). I added each assertion before its fix: - Distance limit: the planning-mode error was expected to have no hint. It failed with `Closest assigned names: app_github_bot_read_pull_request_…, app_calendar_event_0_…, app_calendar_event_1_…`. - Length bound: a 4 MiB name on the call path and on the `schemaTool` path took 20045 ms. The test now uses an 8 MiB name and the 15 s test timeout, not a wall-clock assertion. Without the length bound it fails with `Test timed out in 15000ms`. - Fresh grants: after the gateway revoked `begin_review`, the call error still listed `app_github_bot_begin_review_…`. - Failing before the fourth commit: with a gateway listing that rejects with `gateway_token_revoked`, the call error was `gateway_token_revoked` instead of `assigned_mcp_tool_unknown: name must be an exact name ...`. - Passing after the fixes: the full file `src/services/native-runtime/assigned-mcp-tools.test.ts` passes (60 of 60 tests). The oversized-name checks finish in a few milliseconds. - CI on the second commit: only `Runner support suite` failed for a code reason (the stale SDK pin above). `Build`, `Canary Dry Run`, and three server test shards stopped because the CI runner received a shutdown signal. No test failed. - The new test checks these conditions: - A digest with one missing character lists the correct alias first. Unrelated tools are not in the list. - A write-tool name with the digest of the read tool lists the write tool first. - In planning mode, a typo of a write tool gives the old error with no hint. Write tools and distant read tools are not shown. - An unknown `schemaTool` lists the correct alias first. - A very long name gets no hint and finishes quickly on both paths. - A revoked tool is not shown. - A failed grant listing gives the plain unknown-name error, not the gateway error. - The gateway receives no call for any of these names. - No other code or test depends on the changed error text (`git grep`). - Not run locally: the full server `tsc --noEmit`. The sandbox stopped it for memory. CI runs the typecheck. It passed on the first commit. ## Risks - Low risk. The change affects only the text of errors that are already returned. No name is resolved by approximation, and every call still needs an exact name. - The list shows only names that search would return to the caller now: fresh grants and the current work mode apply. - A name that is not near any tool gets the old error, so the hint does not send the agent to an unrelated tool. - The cost is one fresh grant list and one bounded edit-distance pass over the session catalog, only when a name is unknown. A long name skips the distance computation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-runtime/assigned-mcp-tools.test.ts | 38 +++++++++++++ .../native-runtime/assigned-mcp-tools.ts | 57 +++++++++++++++---- 2 files changed, 83 insertions(+), 12 deletions(-) diff --git a/server/src/services/native-runtime/assigned-mcp-tools.test.ts b/server/src/services/native-runtime/assigned-mcp-tools.test.ts index 96a6f1ce4b9..5f4b653f4d9 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.test.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.test.ts @@ -131,6 +131,44 @@ describe("assigned MCP runner tools", () => { expect(f.executeTool).not.toHaveBeenCalled(); }); + it("names the closest permitted assigned tools when a name is mistyped, without calling any of them", async () => { + const read = descriptor("github_bot:read_pull_request"); + const begin = descriptor("github_bot:begin_review", "write"); + const submit = descriptor("github_bot:submit_review", "write"); + const f = fixture([read, begin, submit, ...Array.from({ length: 6 }, (_, i) => descriptor(`calendar:event_${i}`))]); + const assigned = await createAssignedMcpTools({ ...f, delivery: "on_demand" }); + const page = await assigned.execute({ tool: searchName, arguments: { query: "github_bot", limit: 20 } }) as { tools: Array<{ name: string }> }; + const alias = (tool: ToolGatewayDescriptor) => page.tools.find(entry => entry.name.includes(tool.name.split(":")[1]!))!.name; + const unknown = (name: string, mode?: "planning") => + assigned.execute({ tool: callName, arguments: { name, arguments: {} } }, mode).then(() => "", (error: Error) => error.message); + + const dropped = await unknown(`${alias(begin).slice(0, -6)}${alias(begin).slice(-5)}`); + expect(dropped).toMatch(/^assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools\./); + expect(dropped).toContain(`Closest assigned names: ${alias(begin)}`); + expect(dropped).not.toContain("calendar"); + + const borrowed = await unknown(`${alias(submit).slice(0, -12)}${alias(read).slice(-12)}`); + expect(borrowed).toContain(`Closest assigned names: ${alias(submit)}`); + + const planning = await unknown(`${alias(begin).slice(0, -1)}0`, "planning"); + expect(planning).toBe("assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools. Search assigned tools again and use the returned name."); + + await expect(assigned.execute({ tool: searchName, arguments: { query: "", schemaTool: `${alias(read)}x` } })) + .rejects.toThrow(`Closest assigned names: ${alias(read)}`); + + const oversized = `${alias(read)}${"x".repeat(8 * 1024 * 1024)}`; + expect(await unknown(oversized)).not.toContain("Closest assigned names"); + await expect(assigned.execute({ tool: searchName, arguments: { query: "", schemaTool: oversized } })) + .rejects.toThrow("schemaTool must match an authorized inputSchemaRef. Omit schemaTool"); + + f.listToolsForNamedGateway.mockResolvedValue([read, submit]); + expect(await unknown(`${alias(begin).slice(0, -1)}0`)).not.toContain(alias(begin)); + + f.listToolsForNamedGateway.mockRejectedValue(new Error("gateway_token_revoked")); + expect(await unknown(`${alias(read).slice(0, -1)}0`)).toBe("assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools. Search assigned tools again and use the returned name."); + expect(f.executeTool).not.toHaveBeenCalled(); + }); + it.each([undefined, "on_demand"] as const)("advertises usable page limits through MCP discovery (%s)", async delivery => { const f = fixture(Array.from({ length: 25 }, (_, i) => descriptor(`operation_${i}`))); const assigned = await createAssignedMcpTools({ ...f, delivery }); diff --git a/server/src/services/native-runtime/assigned-mcp-tools.ts b/server/src/services/native-runtime/assigned-mcp-tools.ts index 7bea0dc5c1c..129c579c6a4 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.ts @@ -85,6 +85,32 @@ export function assignedToolName(name: string): string { return `app_${readable}_${digest}`; } +function editDistance(left: string, right: string): number { + let previous = Array.from({ length: right.length + 1 }, (_, index) => index); + for (let leftIndex = 1; leftIndex <= left.length; leftIndex += 1) { + const current = [leftIndex]; + for (let rightIndex = 1; rightIndex <= right.length; rightIndex += 1) { + const substitutionCost = left[leftIndex - 1] === right[rightIndex - 1] ? 0 : 1; + current[rightIndex] = Math.min(current[rightIndex - 1]! + 1, previous[rightIndex]! + 1, previous[rightIndex - 1]! + substitutionCost); + } + previous = current; + } + return previous[right.length]!; +} + +function closestNames(reference: string, names: string[]): string { + const target = reference.trim().toLowerCase(); + const closest = names + .map(name => ({ name, limit: Math.floor(Math.max(target.length, name.length) / 3) })) + .filter(({ name, limit }) => Math.abs(target.length - name.length) <= limit) + .map(({ name, limit }) => ({ name, limit, distance: editDistance(target, name.toLowerCase()) })) + .filter(({ distance, limit }) => distance <= limit) + .sort((a, b) => a.distance - b.distance || (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) + .slice(0, 3) + .map(({ name }) => name); + return closest.length > 0 ? ` Closest assigned names: ${closest.join(", ")}.` : ""; +} + export const ASSIGNED_SCHEMA_LISTING_BUDGET_BYTES = 768 * 1024; export const ASSIGNED_SCHEMA_ADMISSION_CACHE_ENTRIES = 1024; @@ -316,6 +342,19 @@ export async function createAssignedMcpTools(input: { return matches[0]?.[0]; } + // Intersect fresh grants with this session's pinned catalog. Revocations + // take effect immediately; newly assigned tools require a new session. + async function authorizedTools(currentWorkMode?: WorkMode) { + const current = new Map(deduplicateToolDescriptors(await input.gateway.listToolsForNamedGateway({ + gatewayPublicId: input.gatewayPublicId, bearerToken: input.bearerToken, + })).map(tool => [tool.name, tool])); + return [...tools].filter(([, { tool }]) => { + const fresh = current.get(tool.name); + return fresh && permits(tool) && permits(tool, currentWorkMode) && + permits(fresh) && permits(fresh, currentWorkMode); + }); + } + async function search(argumentsValue: unknown, currentWorkMode?: WorkMode) { const args = object(argumentsValue); const offset = args.offset ?? 0; @@ -332,21 +371,14 @@ export async function createAssignedMcpTools(input: { if (schemaOffset > 0 && !hasSchemaTool) { throw new Error("assigned_mcp_tool_invalid_arguments: A positive schemaOffset requires an exact inputSchemaRef in schemaTool. Omit schemaTool and schemaOffset to search assigned tools."); } - // Intersect fresh grants with this session's pinned catalog. Revocations - // take effect immediately; newly assigned tools require a new session. - const current = new Map(deduplicateToolDescriptors(await input.gateway.listToolsForNamedGateway({ - gatewayPublicId: input.gatewayPublicId, bearerToken: input.bearerToken, - })).map(tool => [tool.name, tool])); const terms = args.query.toLowerCase().trim().split(/\s+/).filter(Boolean); - const authorized = [...tools].filter(([, { tool }]) => { - const fresh = current.get(tool.name); - return fresh && permits(tool) && permits(tool, currentWorkMode) && - permits(fresh) && permits(fresh, currentWorkMode); - }); + const authorized = await authorizedTools(currentWorkMode); if (hasSchemaTool) { const reference = resolveAssignedName(args.schemaTool as string, "schemaTool"); const selected = authorized.find(([name]) => name === reference); - if (!selected) throw new Error("assigned_mcp_tool_unknown: schemaTool must match an authorized inputSchemaRef. Omit schemaTool to search assigned tools, then use an exact inputSchemaRef from the results."); + if (!selected) { + throw new Error(`assigned_mcp_tool_unknown: schemaTool must match an authorized inputSchemaRef.${closestNames(args.schemaTool as string, authorized.map(([name]) => name))} Omit schemaTool to search assigned tools, then use an exact inputSchemaRef from the results.`); + } const schema = JSON.stringify(selected[1].schema); const end = schemaOffset + schemaChunkCharacters; return { @@ -405,7 +437,8 @@ export async function createAssignedMcpTools(input: { parameters = object(args.arguments); const resolved = resolveAssignedName(args.name, "name"); if (!resolved) { - throw new Error("assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools. Search assigned tools again and use the returned name."); + const authorized = await authorizedTools(currentWorkMode).then(entries => entries.map(([candidate]) => candidate), () => []); + throw new Error(`assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools.${closestNames(args.name, authorized)} Search assigned tools again and use the returned name.`); } name = resolved; }