mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-05 12:29:37 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use assigned connection tools through a governed gateway. > - OpenCode received the gateway catalog directly, outside the bounded server tool projection. > - A large catalog can exceed a provider grammar compiler's capacity even when it fits the transport limits. > - Synthetic requests reproduced the Fireworks grammar timeout on the configured DeepSeek model. > - This pull request gives OpenCode two assigned-tool discovery and call schemas through the existing server relay. > - The gateway keeps the same grants, validation, approvals, revocation, and audit rules. ## Linked Issues or Issue Description **What happened?** A native OpenCode run could fail during provider grammar compilation before its first assigned-tool call. OpenCode advertised the full assigned gateway catalog. A synthetic catalog of 741 tool schemas reproduced `Grammar processing took too much time` on Fireworks through OpenRouter with `deepseek/deepseek-v4.1-flash-1017`. The same model and backend accepted a compact two-tool discovery/call catalog. This reproduces the mechanism; it does not replay a historical private request. **Expected behavior** Assigned-tool discovery must keep provider grammar size bounded. Every call must retain the run's original gateway binding and permissions. **Steps to reproduce** 1. Give a native OpenCode run a large assigned connection catalog. 2. Use a provider that compiles tool schemas into a constrained output grammar. 3. Observe the full catalog in the provider tool surface before this change. 4. After this change, observe only `paperclip_search_assigned_tools` and `paperclip_call_assigned_tool` for assigned tools. Discover and call a permitted tool through its pinned alias. **Paperclip version or commit** Source checkout based on current `master`. **Deployment mode** Native local or sandbox execution with OpenCode 1.18.34. Related work: Refs #14258 (discovery paging), #16678 (local adapter MCP launch), and #17688 (gateway capability advertisement). This change covers native OpenCode catalog projection. ## What Changed - Route native OpenCode assigned tools through the existing server-owned authenticated relay. - Use bounded discovery/call schemas for OpenCode, including small catalogs. Preserve transport capacity checks. - Keep assigned gateway credentials on the controller. Ignore gateway and credential values in tool arguments. - Preserve exact declared MCP operation names through admission and driver dispatch. Resolve transport prefixes only against the closed catalog. Reject unknown or shortened names. - Refresh former OpenCode direct-catalog and shortened-name checkpoints. Keep the existing fingerprints for other providers. - Add local and remote wiring, catalog capacity, alias, scope, work-mode, approval, and revocation tests. - Allow the native test files to use a validated dedicated external test database when embedded PostgreSQL is unavailable. - Document the assigned-tool execution contract. - Bound OpenCode discovery pages to 32 KiB and large-schema chunks to 16K characters. Keep other harness budgets unchanged. - Test medium pages and escaped large schemas through the actual HTTP MCP bridge on the explicit Runner testing surface. ## Verification - Red before the catalog fix: OpenCode did not create the server relay, and forced compact delivery advertised 1 or 741 direct schemas. - Red before the alias repair: HTTP MCP changed declared operation names, and the driver sent `search_assigned_tools` instead of the authorized `paperclip_search_assigned_tools`. - Current repair tests: 112 OpenCode/proxy tests pass, plus one actual Rust/proxy/HTTP MCP regression. Two checkpoint regressions fail with the former scoped revision and pass with the new revision. Other providers keep their prior fingerprints. - The combined QA candidate with this exact PR head passes all 916 native tests and all 113 transport tests on supported Node 24.22. An earlier Node 26.4/macOS run passed the 113 assertions but had an uncaught networking error. That non-clean run remains recorded; its stack matches [the upstream Undici defect](https://github.com/nodejs/undici/issues/5544). No production workaround or error suppression was added. - `pnpm -r typecheck`, `pnpm build`, and `git diff --check` pass on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`. The maps structural check passes. - The local `pnpm test:run` attempt stopped after host embedded PostgreSQL initialization failures. The relevant database suites pass with isolated PostgreSQL 18. All repository CI gates, including aggregate `ci / verify`, pass on the current head. - Synthetic live provider comparison: 741 invented schemas produced the exact Fireworks grammar timeout in 20.9 seconds. Two compact invented schemas produced a discovery call in 585 ms on the same model and backend. - The first hosted QA attempt exposed the alias failure described above. On the patched source, a real local Rust runner → proxy → OpenCode 1.18.34 → exact DeepSeek smoke completed discovery, one synthetic gateway read, and `paperclip_finish`, then emitted `turn/completed`. It used the production compact projection and unchanged Rust allowlist. - Two earlier local smoke attempts exposed missing terminal authority and completion-contract fields in the smoke harness. Those failures remain recorded. The corrected ordinary bindings produced the clean result. - Fresh Greptile: 5/5 on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`, with no new actionable findings or unresolved threads. - Hosted QA publication and lifecycle acceptance remain pending. The separate deployed-agent `Paperclip Review` check is also pending; it follows a task that failed on the earlier deployed runtime. - Local OpenCode/OpenRouter acceptance at `bc41ba72`: 530 synthetic tools were authorized and installed. A simple model-ID reply completed in 17.3 seconds with no app calls. A discovery task completed in 37.6 seconds, used the search/call relay, made one upstream call, and returned the exact synthetic value. This is local source acceptance, not a staging sandbox timing comparison. - The large-result review finding has a failing-before/passing-after HTTP MCP regression. Five medium schemas now page without invalid JSON. A large schema with escaped text reconstructs exactly through bounded chunks. All 35 assigned-tool tests and the server typecheck pass after this follow-up. - Final head `d84ce4856a6e65a35ad5fb8b561492598b525f4e`: all CI gates pass. Greptile and Paperclip Review both rate the change 5/5. No review threads remain open. The final full build, server typecheck, and 35 assigned-tool tests pass locally. ## Risks - OpenCode uses discovery before it calls an assigned tool, including for small catalogs. This can add a model turn. - An existing OpenCode provider checkpoint rotates once because it retains the former tool declarations. Paperclip task history remains available. - The historical filtered catalog was not captured. The synthetic comparison proves the failure mechanism, not the exact cardinality of an earlier private request. - No model, provider route, permission, or automatic retry policy changes. ## Model Used The bounded-result follow-up also used OpenAI Codex (GPT-6 family). The exact backend model ID and context-window size were not exposed for that session. It used reasoning, code execution, and tool use. OpenAI `gpt-6.1-sol` in Codex, with reasoning, tool use, code execution, and source inspection. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
212 lines
11 KiB
TypeScript
212 lines
11 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { deduplicateToolDescriptors, ToolGatewayHttpError, type ToolGatewayDescriptor, type ToolGatewayService } from "../tool-gateway.js";
|
|
|
|
type WorkMode = "standard" | "planning" | "ask";
|
|
type ToolDefinition = Record<string, unknown>;
|
|
|
|
const SEARCH_TOOL = "paperclip_search_assigned_tools";
|
|
const CALL_TOOL = "paperclip_call_assigned_tool";
|
|
// A single valid runner schema can be 512 KiB. Leave room for its description
|
|
// while keeping the complete result below the 768 KiB provider-result bound.
|
|
const SEARCH_PAGE_BYTES = 640 * 1024;
|
|
const SCHEMA_CHUNK_CHARACTERS = 64 * 1024;
|
|
const ON_DEMAND_TOOLS: ToolDefinition[] = [
|
|
{
|
|
name: SEARCH_TOOL,
|
|
description: "Search your assigned app tools by name or description and read their input schemas. Pass {\"query\":\"search terms\"}, or an empty query to browse. Omit unused optional fields instead of sending null. To fetch the next page, pass the returned nextOffset as offset. A large definition returns inputSchemaRef: set schemaTool to that name with an empty query, then pass the returned nextSchemaOffset as schemaOffset until null. Concatenate schemaJson chunks and parse JSON. Use paperclip_call_assigned_tool with a returned name and arguments matching its inputSchema.",
|
|
inputSchema: {
|
|
type: "object", additionalProperties: false,
|
|
properties: {
|
|
query: { type: "string", maxLength: 200 },
|
|
offset: { type: "integer", minimum: 0 },
|
|
limit: { type: "integer", minimum: 1, maximum: 20 },
|
|
schemaTool: { type: "string", description: "Exact inputSchemaRef from a search result." },
|
|
schemaOffset: { type: "integer", minimum: 0 },
|
|
},
|
|
required: ["query"],
|
|
},
|
|
},
|
|
{
|
|
name: CALL_TOOL,
|
|
description: "Call an assigned app tool discovered with paperclip_search_assigned_tools. Use its exact returned name and arguments matching its inputSchema. The same permissions, approvals, and audit rules apply as for direct app tools.",
|
|
inputSchema: {
|
|
type: "object", additionalProperties: false,
|
|
properties: { name: { type: "string" }, arguments: { type: "object", additionalProperties: true } },
|
|
required: ["name", "arguments"],
|
|
},
|
|
},
|
|
];
|
|
|
|
function object(value: unknown): Record<string, unknown> {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
|
throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
}
|
|
return value as Record<string, unknown>;
|
|
}
|
|
|
|
function definition(name: string, tool: ToolGatewayDescriptor): ToolDefinition {
|
|
return { name, description: `${tool.displayName}: ${tool.description}`, inputSchema: structuredClone(tool.parametersSchema) };
|
|
}
|
|
|
|
// Execution must use the app's configured gateway, including deployment
|
|
// restrictions, OAuth refresh, and approval delivery. Never fall back to an
|
|
// isolated gateway whose defaults differ from the running instance.
|
|
const assignedMcpGateways = new WeakMap<Db, ToolGatewayService>();
|
|
|
|
export function registerAssignedMcpGateway(db: Db, gateway: ToolGatewayService): void {
|
|
assignedMcpGateways.set(db, gateway);
|
|
}
|
|
|
|
export function getAssignedMcpGateway(db: Db): ToolGatewayService {
|
|
const gateway = assignedMcpGateways.get(db);
|
|
if (!gateway) throw new Error("assigned_mcp_gateway_unavailable");
|
|
return gateway;
|
|
}
|
|
|
|
/** No configured gateway means no authorized connection tools or instructions. */
|
|
export async function resolveAssignedConnectionInstructionsForRun(db: Db, binding: { companyId: string; agentId: string; runId: string }) {
|
|
const gateway = assignedMcpGateways.get(db);
|
|
return gateway ? gateway.resolveConnectionInstructionsForRun(binding) : null;
|
|
}
|
|
|
|
function assignedToolName(name: string): string {
|
|
const sanitize = (value: string) => value.replace(/[^a-zA-Z0-9_]+/g, "_").replace(/^_+|_+$/g, "");
|
|
const separator = name.lastIndexOf(":");
|
|
const readable = separator >= 0
|
|
? `${sanitize(name.slice(0, separator)).slice(0, 20) || "tool"}_${sanitize(name.slice(separator + 1)).slice(-26) || "action"}`
|
|
: sanitize(name).slice(0, 47) || "tool";
|
|
const digest = createHash("sha256").update(name).digest("hex").slice(0, 12);
|
|
return `app_${readable}_${digest}`;
|
|
}
|
|
|
|
/** Server-owned MCP bindings projected into the runner's existing tool channel. */
|
|
export async function createAssignedMcpTools(input: {
|
|
gateway: ToolGatewayService;
|
|
gatewayPublicId: string;
|
|
bearerToken: string;
|
|
workMode?: WorkMode;
|
|
/** Bound provider grammar complexity independently of protocol byte capacity. */
|
|
delivery?: "on_demand";
|
|
}) {
|
|
const listed = await input.gateway.listToolsForNamedGateway({
|
|
gatewayPublicId: input.gatewayPublicId,
|
|
bearerToken: input.bearerToken,
|
|
});
|
|
const tools = new Map<string, ToolGatewayDescriptor>();
|
|
for (const descriptor of deduplicateToolDescriptors(listed)) {
|
|
const name = assignedToolName(descriptor.name);
|
|
const previous = tools.get(name);
|
|
if (previous) {
|
|
throw new ToolGatewayHttpError(409,
|
|
`Assigned MCP tool name collision for ${JSON.stringify([previous.name, descriptor.name])}.`,
|
|
"assigned_mcp_tool_name_collision");
|
|
}
|
|
tools.set(name, descriptor);
|
|
}
|
|
const permits = (tool: ToolGatewayDescriptor, mode: WorkMode = input.workMode ?? "standard") => mode === "standard" || tool.risk === "read";
|
|
// OpenCode's HTTP bridge returns at most 64K JSON characters. Leave room
|
|
// for the envelope and for JSON escaping when returning a schema chunk.
|
|
// Other harnesses retain the larger PRP page and chunk budgets.
|
|
const searchPageBytes = input.delivery === "on_demand" ? 32 * 1024 : SEARCH_PAGE_BYTES;
|
|
const schemaChunkCharacters = input.delivery === "on_demand" ? 16 * 1024 : SCHEMA_CHUNK_CHARACTERS;
|
|
|
|
async function search(argumentsValue: unknown, currentWorkMode?: WorkMode) {
|
|
const args = object(argumentsValue);
|
|
const offset = args.offset ?? 0;
|
|
const limit = args.limit ?? 5;
|
|
const schemaOffset = args.schemaOffset ?? 0;
|
|
if (typeof args.query !== "string" || args.query.length > 200 ||
|
|
typeof offset !== "number" || !Number.isSafeInteger(offset) || offset < 0 ||
|
|
typeof limit !== "number" || !Number.isSafeInteger(limit) || limit < 1 || limit > 20 ||
|
|
(args.schemaTool !== undefined && typeof args.schemaTool !== "string") ||
|
|
typeof schemaOffset !== "number" || !Number.isSafeInteger(schemaOffset) || schemaOffset < 0 ||
|
|
(args.schemaOffset !== undefined && args.schemaTool === undefined)) {
|
|
throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
}
|
|
// Intersect fresh grants with this session's pinned catalog. Revocations
|
|
// take effect immediately; newly assigned tools require a new session.
|
|
const current = new Map(deduplicateToolDescriptors(await input.gateway.listToolsForNamedGateway({
|
|
gatewayPublicId: input.gatewayPublicId, bearerToken: input.bearerToken,
|
|
})).map(tool => [tool.name, tool]));
|
|
const terms = args.query.toLowerCase().trim().split(/\s+/).filter(Boolean);
|
|
const authorized = [...tools].filter(([, tool]) => {
|
|
const fresh = current.get(tool.name);
|
|
return fresh && permits(tool) && permits(tool, currentWorkMode) &&
|
|
permits(fresh) && permits(fresh, currentWorkMode);
|
|
});
|
|
if (typeof args.schemaTool === "string") {
|
|
const selected = authorized.find(([name]) => name === args.schemaTool);
|
|
if (!selected) throw new Error("assigned_mcp_tool_unknown");
|
|
const schema = JSON.stringify(selected[1].parametersSchema);
|
|
const end = schemaOffset + schemaChunkCharacters;
|
|
return {
|
|
name: selected[0], schemaJson: schema.slice(schemaOffset, end),
|
|
nextSchemaOffset: end < schema.length ? end : null,
|
|
};
|
|
}
|
|
const matches = authorized.filter(([name, tool]) =>
|
|
terms.every(term => `${name} ${tool.displayName} ${tool.description}`.toLowerCase().includes(term)))
|
|
.sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0);
|
|
const page: ToolDefinition[] = [];
|
|
for (const [name, tool] of matches.slice(offset, offset + limit)) {
|
|
let next = definition(name, tool);
|
|
if (Buffer.byteLength(JSON.stringify([next]), "utf8") > searchPageBytes) {
|
|
// One large schema must not block browsing the tools after it. Expose
|
|
// a reference whose schema can be fetched in bounded, reauthorized
|
|
// chunks, without weakening the gateway's argument validation.
|
|
next = { name, description: String(next.description).slice(0, 2000), inputSchemaRef: name };
|
|
}
|
|
if (Buffer.byteLength(JSON.stringify([...page, next]), "utf8") > searchPageBytes) {
|
|
break;
|
|
}
|
|
page.push(next);
|
|
}
|
|
return { tools: page, nextOffset: offset + page.length < matches.length ? offset + page.length : null };
|
|
}
|
|
|
|
return {
|
|
definitions(fits?: (definitions: ToolDefinition[]) => boolean): ToolDefinition[] {
|
|
const direct = [...tools].filter(([, tool]) => permits(tool)).map(([name, tool]) => definition(name, tool));
|
|
if (input.delivery !== "on_demand" && (!fits || fits(direct))) return direct;
|
|
const compact = structuredClone(ON_DEMAND_TOOLS);
|
|
if (fits && !fits(compact)) throw new Error("assigned_mcp_tool_catalog_capacity_exceeded");
|
|
return compact;
|
|
},
|
|
has(name: string): boolean {
|
|
return tools.has(name) || name === SEARCH_TOOL || name === CALL_TOOL;
|
|
},
|
|
async execute(call: { tool: string; arguments: unknown }, currentWorkMode?: WorkMode): Promise<unknown> {
|
|
if (call.tool === SEARCH_TOOL) return search(call.arguments, currentWorkMode);
|
|
let name = call.tool;
|
|
let parameters = call.arguments;
|
|
if (name === CALL_TOOL) {
|
|
const args = object(parameters);
|
|
if (typeof args.name !== "string") throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
name = args.name;
|
|
parameters = object(args.arguments);
|
|
}
|
|
const descriptor = tools.get(name);
|
|
if (!descriptor) throw new Error("assigned_mcp_tool_unknown");
|
|
if (!permits(descriptor) || !permits(descriptor, currentWorkMode)) throw new Error("paperclip_runner_tool_mode_denied");
|
|
// Reauthorize through the existing gateway on every call. Discovery is
|
|
// not a grant: revocation, policy, approval, and audit remain server-owned.
|
|
const result = await input.gateway.executeTool({
|
|
gatewayPublicId: input.gatewayPublicId,
|
|
sessionToken: input.bearerToken,
|
|
tool: descriptor.name,
|
|
parameters,
|
|
});
|
|
if (result.status !== "completed" && result.status !== "replayed") {
|
|
throw new Error("assigned_mcp_tool_execution_incomplete");
|
|
}
|
|
// The gateway normally throws provider errors. Keep a malformed or older
|
|
// provider response from being reported as successful through PRP.
|
|
if (result.result && typeof result.result === "object" && "isError" in result.result && result.result.isError === true) {
|
|
throw new ToolGatewayHttpError(502, "The assigned tool returned an error.", "tool_error");
|
|
}
|
|
return result.result;
|
|
},
|
|
};
|
|
}
|