diff --git a/doc/plans/2026-08-20-runner-tool-search.md b/doc/plans/2026-08-20-runner-tool-search.md new file mode 100644 index 00000000000..8200392a073 --- /dev/null +++ b/doc/plans/2026-08-20-runner-tool-search.md @@ -0,0 +1,178 @@ +# Paperclip Runner lazy tool discovery + +Date: 2026-08-20 +Status: fallback implemented and evaluated; native adapter pending transport support + +Reference: [OpenAI Tool Search](https://developers.openai.com/api/docs/guides/tools-tool-search) + +## Outcome + +Keep the model's initial protocol surface small while allowing it to discover optional +Paperclip operations at runtime. Discovery never grants authority: the runner filters +what may be discovered, and the dispatcher independently authorizes every invocation. + +OpenAI Tool Search is the preferred provider capability where the selected model and +driver support it. The runner also owns a provider-neutral fallback so Paperclip's +protocol and security semantics do not depend on one transport. + +## Product contract + +The initial model context contains: + +- a deliberately small always-present task core; +- high-level descriptions of searchable namespaces, not every operation schema; +- either native `tool_search` or runner-managed `discover_capabilities`; +- explicit guidance that discovery should happen only when the task requires it. + +Candidate namespace projection, kept below ten operations per namespace: + +| Namespace | Purpose | Initial operations | +| --- | --- | --- | +| `discovery` | Find company tasks, agents, projects, and goals | `search_tasks`, `list_agents`, `get_agent`, `list_projects`, `list_goals` | +| `delegation` | Create bounded child work and dependencies | `create_task`, `set_dependencies` | +| `governance` | Read, request, discuss, and decide approvals | approval operations | +| `workspace` | Inspect and control task workspace services | workspace operations | +| `cases` | Read and maintain case records | case operations | +| `routines` | Read and maintain routines | routine operations | +| `company_skills` | Inspect and synchronize company skills | skill operations | +| `secrets` | Discover metadata and perform tightly governed secret reads | secret operations | +| `company_admin` | Portability and company administration | admin operations | + +The exact always-present core is a measured product decision. Start with active-task +context/history, progress/status reporting, and terminal/disposition operations; use +eval results to decide whether document reads and writes remain eager or become +deferred. + +## Architecture + +### Canonical registry projection + +Extend the canonical operation registry with discovery metadata: namespace, concise +search description, searchable aliases, sensitivity classification, and whether an +operation's name may be disclosed when unauthorized. Continue deriving provider +schemas and scenario descriptors from the same canonical operation identity. + +### Discovery policy + +Given a query and session identity, compute candidates from the canonical registry, +then filter by company, actor, task, mode, role, claims, enabled capabilities, scenario +policy, disabled-by-default status, and driver support. Rank only the permitted set. +Sensitive undiscoverable operations must not leak through names, counts, descriptions, +errors, or timing where practical. + +Return trusted registry definitions only. Never accept a model-authored schema or an +untrusted remote definition as executable protocol authority. + +### Invocation policy + +Loaded tools receive no durable authority grant. Every call goes through the existing +semantic dispatcher and repeats the current policy decision. Revoked claims or changed +task state therefore take effect even if the schema remains in conversation context. + +### Provider adapters + +1. **Native adapter:** project namespaces/deferred functions and consume provider + tool-search call/output events. +2. **Runner-managed adapter:** expose `discover_capabilities`; resolve the query + locally; add selected trusted definitions on the next model continuation. +3. Normalize both paths into PRP events so Runner Lab and Evalbook do not depend on + provider-specific event shapes. + +### Protocol evidence + +Add evidence events for discovery requested, candidates considered (redacted), policy +decision, definitions loaded, namespace activated, and discovery failure. Preserve the +query and returned operation IDs when safe, plus model/driver/configuration, token and +cost measurements. Link the eventual invocation back to the discovery event. + +## Delivery phases + +### D0 — transport spike + +Prove whether the Codex app-server driver can send deferred definitions and surface +native tool-search events for the configured GPT-5.4-class model. Produce one recorded +attempt. Do not change default exposure. + +Result: the current Codex app-server driver accepts dynamic tools at thread creation but +does not expose Responses API native Tool Search events or a supported mid-turn schema +append operation. Native support is therefore not claimed. The provider-neutral gateway +below is the implemented path. + +### D1 — provider-neutral domain layer + +Add discovery metadata and namespace validation to the canonical registry. Implement +the policy-filtered search service and normalized discovery event contract. Unit-test +ranking, nondisclosure, mode/role/claim filtering, and registry drift. + +### D2 — runner-managed fallback + +Implement `discover_capabilities`, continuation-time schema loading, policy recheck on +invocation, limits on results/searches/loaded tools, and deterministic errors. Ship +behind `toolExposure: "eager" | "lazy"`, defaulting to `eager`. + +Implemented: lazy threads receive the always-present core plus +`discover_capabilities` and `invoke_discovered_capability`. Search is deterministic, +namespace-aware, limited to eight results, filtered by live exposure policy, and returns +only trusted catalog schemas. Invocation requires prior discovery and is reauthorized by +the semantic dispatcher. Loaded operation identities survive runner snapshots. + +### D3 — native Tool Search adapter + +If D0 succeeds, implement native namespace/deferred-tool projection and normalize its +events to the same domain contract. If D0 fails, retain the evidence and defer this +adapter without blocking the fallback. + +### D4 — Runner Lab and reports + +Render discovery inline in the issue thread, expose policy detail in Devtools, connect +loaded definitions to subsequent calls, and report eager/lazy configuration plus token, +latency, and cost deltas. + +Implemented: discovery is retained as redacted `tool_discovery` evidence, appears on the +semantic-tool boundary, and Devtools reports exposure mode and loaded operation IDs. + +### D5 — eval and rollout + +Implement the lazy-discovery eval track in the sibling `paperclip-evals` repository. +Run deterministic checks first and live comparisons on GPT-5.4 mini only. Switch the +default to lazy after semantic parity, authorization/nondisclosure, observability, and +efficiency gates pass. Keep eager mode as a temporary diagnostic control until one +release cycle completes. + +Initial result: all 35 GPT-5.4-mini live roster cases passed on +`20260820-full-protocol-lazy-01`, including three lazy-discovery cases and four new +multi-operation workflows. Eager remains the default pending a broader repetition and +efficiency comparison; this single green run is functional evidence, not a reliability +estimate. + +## Guardrails and limits + +- Maximum returned definitions per search and maximum loaded definitions per session. +- Maximum discovery calls per turn/session, with loop detection. +- No `generic_api_request` outside explicit skill-test scenarios. +- No disclosure of secret/admin tools to unauthorized actors. +- Namespace and operation descriptions are trusted code-reviewed content. +- Search results are company-scoped and deterministic for identical policy state. +- Tool invocation always rechecks live policy and terminal task state. +- Cache behavior is measured but never allowed to weaken authorization freshness. + +## Acceptance criteria + +- Lazy sessions begin without optional operation schemas. +- A permitted optional operation can be found and invoked end to end. +- An unauthorized operation is neither disclosed nor invokable. +- Revocation after loading prevents invocation. +- Native and fallback routes produce equivalent semantic PRP evidence. +- Runner Lab makes discovery and its causal invocation inspectable. +- Evalbook compares eager and lazy correctness, tokens, latency, and cost. +- No regression in the existing deterministic capability suite or protocol eval roster. + +## Open decisions to resolve with evidence + +1. Exact always-present core, especially document operations. +2. Lexical versus embedding-assisted local ranking; begin lexical and deterministic. +3. Whether sensitive namespaces are hidden entirely or return a generic unavailable + result; prefer complete nondisclosure. +4. Session persistence rules for loaded schemas across compaction and continuation. +5. Whether native Tool Search is available through Codex app-server with the required + event fidelity; D0 decides this rather than assuming it. diff --git a/packages/paperclip-runner/src/devtools/index.ts b/packages/paperclip-runner/src/devtools/index.ts index 894c668562c..09bfb1b73ea 100644 --- a/packages/paperclip-runner/src/devtools/index.ts +++ b/packages/paperclip-runner/src/devtools/index.ts @@ -96,7 +96,7 @@ function publicCompanyState(serialized: string): CapabilityJsonValue { } function boundary(kind: string): string { - if (kind === "tool_call" || kind === "tool_result") return "semantic tool"; + if (kind === "tool_call" || kind === "tool_result" || kind === "tool_discovery") return "semantic tool"; if (kind === "provider_event") return "Codex app-server"; if (kind === "process") return "runnerd process"; if (kind === "interaction") return "mock control plane"; @@ -157,6 +157,8 @@ export function projectCapabilityDevtools(snapshot: CapabilityLiveSessionSnapsho currentAttemptId: snapshot.currentAttemptId ?? null, terminalTurns: snapshot.terminalTurns ?? [], usageLedger: snapshot.usageLedger ?? [], + toolExposure: snapshot.config.toolExposure ?? "eager", + loadedOperationIds: snapshot.loadedOperationIds ?? [], createdAt: snapshot.createdAt, updatedAt: snapshot.updatedAt, }), @@ -166,6 +168,7 @@ export function projectCapabilityDevtools(snapshot: CapabilityLiveSessionSnapsho effectiveCapabilities: snapshot.config.capabilities, explicitClaims: snapshot.config.explicitClaims, turnTimeoutMs: snapshot.config.turnTimeoutMs, + toolExposure: snapshot.config.toolExposure ?? "eager", workingDirectory: "[withheld]", }), }; diff --git a/packages/paperclip-runner/src/live/evidence-redaction.ts b/packages/paperclip-runner/src/live/evidence-redaction.ts index 203a69d47cc..a90b19cc366 100644 --- a/packages/paperclip-runner/src/live/evidence-redaction.ts +++ b/packages/paperclip-runner/src/live/evidence-redaction.ts @@ -32,6 +32,7 @@ export type CapabilityEvidenceKind = | "session" | "provider_event" | "tool_exposure" + | "tool_discovery" | "tool_call" | "tool_result" | "interaction" @@ -313,6 +314,15 @@ export function redactCapabilityEvidenceData( .sort(), scenarioId: clamp(asString(data.scenarioId), 120), }; + case "tool_discovery": + return { + action: clamp(asString(data.action), 80), + query: clamp(asString(data.query), 500), + namespace: clamp(asString(data.namespace), 120), + operationIds: (Array.isArray(data.operationIds) ? data.operationIds : []) + .filter((id): id is string => typeof id === "string" && CATALOG_OPERATION_IDS.has(id)) + .sort(), + }; case "tool_call": { const operationId = asString(data.operationId); return { @@ -374,6 +384,10 @@ export function capabilityEvidenceDetail( const ids = Array.isArray(data.operationIds) ? data.operationIds.length : 0; return `tool exposure · ${ids} operation${ids === 1 ? "" : "s"}`; } + case "tool_discovery": { + const ids = Array.isArray(data.operationIds) ? data.operationIds.length : 0; + return `tool discovery · ${value("action") || "searched"} · ${ids} loaded`; + } case "tool_call": { const fields = asRecord(data.input).fields; const count = Array.isArray(fields) ? fields.length : 0; @@ -450,6 +464,12 @@ export function capabilityEvidenceDetails( add("Scenario", data.scenarioId); addList("Operations", data.operationIds); break; + case "tool_discovery": + add("Action", data.action); + add("Query", data.query); + add("Namespace", data.namespace); + addList("Operations", data.operationIds); + break; case "tool_call": { const input = asRecord(data.input); add("Call", data.callId); diff --git a/packages/paperclip-runner/src/live/live-session.test.ts b/packages/paperclip-runner/src/live/live-session.test.ts index 12c39b4abff..6376fc6dc65 100644 --- a/packages/paperclip-runner/src/live/live-session.test.ts +++ b/packages/paperclip-runner/src/live/live-session.test.ts @@ -273,6 +273,24 @@ function providerState(): FakeProviderState { } describe("Capability live runnerd and Codex session", () => { + it("opens lazy sessions with core and discovery gateways instead of optional schemas", async () => { + const state = providerState(); + const claim = "discovery:agents:read"; + const service = new CapabilityLiveSessionService({ transportFactory: fakeTransportFactory(state) }); + const session = await service.create({ + runId: "run-live-lazy", sessionId: "session-live-lazy", toolExposure: "lazy", + capabilities: [claim], explicitClaims: [claim], scenario: { id: "lazy", claims: [claim] }, + }); + const start = state.transports[0]!.requests.find((entry) => entry.method === "thread/start")!; + const names = (start.params.dynamicTools as Array<{ name: string }>).map((tool) => tool.name); + expect(names).toContain("get_task_context"); + expect(names).toContain("discover_capabilities"); + expect(names).toContain("invoke_discovered_capability"); + expect(names).not.toContain("list_agents"); + expect(session.snapshot().config.toolExposure).toBe("lazy"); + await service.shutdown(session.id); + }); + it("returns typed mock results to the same multi-turn Codex thread without Paperclip network calls", async () => { const state = providerState(); const fetchSpy = vi.spyOn(globalThis, "fetch"); diff --git a/packages/paperclip-runner/src/live/live-session.ts b/packages/paperclip-runner/src/live/live-session.ts index 6207c009731..c1e13710a5c 100644 --- a/packages/paperclip-runner/src/live/live-session.ts +++ b/packages/paperclip-runner/src/live/live-session.ts @@ -17,6 +17,7 @@ import type { } from "../mock-core/capability-control-plane-types.js"; import { CapabilityMockControlPlaneAdapter } from "../mock-core/capability-mock-control-plane-adapter.js"; import { CapabilitySemanticDispatcher } from "../semantic-tools/dispatcher.js"; +import { CAPABILITY_DISCOVERY_NAMESPACES, type CapabilityToolExposureMode } from "../semantic-tools/discovery.js"; import type { CapabilitySemanticAuthorizationRecord, CapabilitySemanticScenarioPolicy, @@ -44,6 +45,8 @@ const LIVE_BASE_INSTRUCTIONS = [ "Do not discover skills, credentials, endpoints, or hidden control-plane capabilities.", ].join(" "); const CODEX_PERMISSION_PROFILE = "paperclip-runner-workspace-only"; +const DISCOVER_TOOL = "discover_capabilities"; +const INVOKE_DISCOVERED_TOOL = "invoke_discovered_capability"; export type CapabilityLiveSessionStatus = | "starting" @@ -95,6 +98,7 @@ export interface CapabilityLiveSessionConfigSnapshot { explicitClaims: string[]; seedState: string; turnTimeoutMs: number; + toolExposure?: CapabilityToolExposureMode; } export type CapabilityLiveAttemptStatus = "running" | "succeeded" | "failed" | "terminated"; @@ -183,6 +187,7 @@ export interface CapabilityLiveSessionSnapshot { usageLedger?: CapabilityLiveUsageReceipt[]; /** Bounded mock-control-plane snapshots for DevTools time travel and diffs. */ stateHistory?: CapabilityLiveStateRevision[]; + loadedOperationIds?: string[]; } export interface CreateCapabilityLiveSessionInput { @@ -199,6 +204,7 @@ export interface CreateCapabilityLiveSessionInput { sessionId?: string; attemptId?: string; turnTimeoutMs?: number; + toolExposure?: CapabilityToolExposureMode; } export interface ResumeCapabilityLiveSessionInput { @@ -534,6 +540,18 @@ function dynamicToolSpec(definition: CapabilitySemanticToolDefinition): Record[] { + return [{ + name: DISCOVER_TOOL, + description: `Search authorized optional Paperclip capabilities. Namespaces: ${CAPABILITY_DISCOVERY_NAMESPACES.map((item) => `${item.name} (${item.description})`).join("; ")}`, + inputSchema: { type: "object", properties: { query: { type: "string", minLength: 1, maxLength: 500 }, namespace: { type: "string" }, limit: { type: "integer", minimum: 1, maximum: 8 } }, required: ["query"], additionalProperties: false }, + }, { + name: INVOKE_DISCOVERED_TOOL, + description: "Invoke one optional Paperclip operation returned by discover_capabilities. Authority is rechecked at invocation.", + inputSchema: { type: "object", properties: { operationId: { type: "string" }, input: { type: "object", additionalProperties: true } }, required: ["operationId", "input"], additionalProperties: false }, + }]; +} + function userInput(message: string): Record { return { type: "text", text: message, text_elements: [] }; } @@ -608,6 +626,7 @@ export class CapabilityLiveSessionService { explicitClaims: authority.explicitClaims, seedState, turnTimeoutMs: input.turnTimeoutMs ?? 120_000, + toolExposure: input.toolExposure ?? "eager", }, store: this.#store, transportFactory: this.#transportFactory, @@ -742,6 +761,7 @@ export class CapabilityLiveSessionService { capabilities: config.capabilities, explicitClaims: config.explicitClaims, turnTimeoutMs: config.turnTimeoutMs, + toolExposure: config.toolExposure ?? "eager", }); } @@ -810,6 +830,7 @@ export class CapabilityLiveSession { #eventSeq = 0; #turnEventCount = 0; readonly #rawUsageByTurn = new Map(); + readonly #loadedOperationIds = new Set(); private constructor(options: OpenSessionOptions & { snapshot?: CapabilityLiveSessionSnapshot }) { this.#port = options.port; @@ -847,6 +868,7 @@ export class CapabilityLiveSession { operationId: "session.open", state: this.#port.serialize(), }]); + for (const operationId of options.snapshot?.loadedOperationIds ?? []) this.#loadedOperationIds.add(operationId); this.#providerThreadId = options.snapshot?.providerThreadId ?? ""; this.#providerSessionId = options.snapshot?.providerSessionId ?? null; this.#providerModel = options.snapshot?.providerModel === undefined @@ -1015,6 +1037,7 @@ export class CapabilityLiveSession { terminalTurns: structuredClone(this.#terminalTurns), usageLedger: structuredClone(this.#usageLedger), stateHistory: structuredClone(this.#stateHistory), + loadedOperationIds: [...this.#loadedOperationIds].sort(), }; } @@ -1360,10 +1383,14 @@ export class CapabilityLiveSession { capabilities: { experimentalApi: true, requestAttestation: false }, }); this.#transport.notify("initialized"); - const tools = this.#dispatcher.listTools(this.#authority.runId); + const authorizedTools = this.#dispatcher.listTools(this.#authority.runId); + const tools = (this.#config.toolExposure ?? "eager") === "lazy" + ? authorizedTools.filter((tool) => tool.annotations.exposure === "always") + : authorizedTools; this.#appendEvidence("tool_exposure", null, { operationIds: tools.map((tool) => tool.name), scenarioId: this.#config.scenario.id, + toolExposure: this.#config.toolExposure ?? "eager", }); if (resume) { const read = await this.#transport.request("thread/read", { @@ -1427,7 +1454,10 @@ export class CapabilityLiveSession { runtimeWorkspaceRoots: [this.#config.workingDirectory], approvalPolicy: "never", baseInstructions: LIVE_BASE_INSTRUCTIONS, - dynamicTools: tools.map(dynamicToolSpec), + dynamicTools: [ + ...tools.map(dynamicToolSpec), + ...((this.#config.toolExposure ?? "eager") === "lazy" ? discoveryToolSpecs() : []), + ], experimentalRawEvents: true, persistExtendedHistory: true, }); @@ -1515,25 +1545,55 @@ export class CapabilityLiveSession { contentItems: [{ type: "inputText", text: "Tool call was outside the active Capability thread and turn." }], }; } + if (operationId === DISCOVER_TOOL) { + const args = record(request.params.arguments); + try { + const discovered = this.#dispatcher.discoverTools( + this.#authority.runId, + text(args.query), + { ...(text(args.namespace) ? { namespace: text(args.namespace) } : {}), ...(typeof args.limit === "number" ? { limit: args.limit } : {}) }, + ); + for (const operation of discovered.operations) this.#loadedOperationIds.add(operation.name); + this.#appendEvidence("tool_discovery", turnId, { + action: "loaded", query: discovered.query, namespace: discovered.namespace ?? "", + operationIds: discovered.operations.map((operation) => operation.name), + }); + await this.#persist(); + return { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(discovered) }] }; + } catch (error) { + this.#appendEvidence("tool_discovery", turnId, { action: "rejected", query: text(args.query), namespace: text(args.namespace), operationIds: [] }); + return { success: false, contentItems: [{ type: "inputText", text: error instanceof Error ? error.message : "Capability discovery failed." }] }; + } + } + let semanticOperationId = operationId; + let semanticInput = request.params.arguments; + if (operationId === INVOKE_DISCOVERED_TOOL) { + const args = record(request.params.arguments); + semanticOperationId = text(args.operationId); + semanticInput = args.input; + if (!this.#loadedOperationIds.has(semanticOperationId)) { + return { success: false, contentItems: [{ type: "inputText", text: "Operation was not loaded by discover_capabilities." }] }; + } + } const beforeRevision = this.#port.snapshot().revision; this.#appendEvidence("tool_call", turnId, { callId, - operationId, - input: jsonValue(request.params.arguments), + operationId: semanticOperationId, + input: jsonValue(semanticInput), beforeRevision, }); this.#emit({ turnId, kind: "activity", reason: "tool_call" }); const result = await this.#dispatcher.dispatch({ runId: this.#authority.runId, callId, - operationId, - input: request.params.arguments, + operationId: semanticOperationId, + input: semanticInput, }); - this.#recordStateRevision(turnId, operationId); + this.#recordStateRevision(turnId, semanticOperationId); const replayDisposition = result.ok ? text(record(result.result).disposition) : ""; this.#appendEvidence("tool_result", turnId, { callId, - operationId, + operationId: semanticOperationId, result: jsonValue(result), beforeRevision, afterRevision: result.stateRevision, diff --git a/packages/paperclip-runner/src/semantic-tools/discovery.ts b/packages/paperclip-runner/src/semantic-tools/discovery.ts new file mode 100644 index 00000000000..499347e7de2 --- /dev/null +++ b/packages/paperclip-runner/src/semantic-tools/discovery.ts @@ -0,0 +1,98 @@ +import { CAPABILITY_SEMANTIC_TOOL_CATALOG } from "./catalog.js"; +import { decideCapabilitySemanticAuthorization } from "./policy.js"; +import type { + CapabilitySemanticOperationId, + CapabilitySemanticPolicyContext, + CapabilitySemanticToolDefinition, + CapabilitySemanticToolDescriptor, +} from "./types.js"; + +export type CapabilityToolExposureMode = "eager" | "lazy"; + +export interface CapabilityDiscoveryResult { + readonly schema: "paperclip.capability-discovery.v1"; + readonly query: string; + readonly namespace: string | null; + readonly operations: readonly CapabilitySemanticToolDefinition[]; + readonly truncated: boolean; +} + +const NAMESPACE: Readonly> = Object.freeze({ + get_task_context: "active_task", get_task_history: "active_task", + list_documents: "documents", read_document: "documents", list_document_revisions: "documents", + report_progress: "active_task", answer_status_question: "active_task", write_document: "documents", + request_human_input: "documents", register_deliverable: "documents", finish_task: "active_task", + block_task: "active_task", request_review: "active_task", search_tasks: "discovery", + list_agents: "discovery", get_agent: "discovery", create_task: "delegation", + set_dependencies: "delegation", list_approvals: "governance", get_approval: "governance", + get_approval_context: "governance", request_approval: "governance", + decide_approval: "governance", comment_on_approval: "governance", + get_workspace_runtime: "workspace", control_workspace_service: "workspace", + schedule_wake: "continuation", generic_api_request: "test_infrastructure", +}); + +export const CAPABILITY_DISCOVERY_NAMESPACES = Object.freeze([ + { name: "discovery", description: "Find company tasks and agents." }, + { name: "delegation", description: "Create bounded child work and task dependencies." }, + { name: "governance", description: "Read, request, discuss, and decide approvals." }, + { name: "workspace", description: "Inspect and control active-task workspace services." }, + { name: "continuation", description: "Schedule a bounded continuation wake." }, +]); + +export function capabilityToolNamespace(operationId: CapabilitySemanticOperationId): string { + return NAMESPACE[operationId]; +} + +export function alwaysCapabilityDefinitions(): readonly CapabilitySemanticToolDefinition[] { + return CAPABILITY_SEMANTIC_TOOL_CATALOG.filter((item) => item.exposure === "always").map(toDefinition); +} + +export function discoverCapabilityDefinitions( + query: string, + context: CapabilitySemanticPolicyContext, + options: { namespace?: string; limit?: number } = {}, +): CapabilityDiscoveryResult { + const normalized = query.trim().toLowerCase(); + if (normalized.length === 0) throw new Error("discovery_query_empty"); + const limit = Math.max(1, Math.min(options.limit ?? 5, 8)); + const tokens = normalized.split(/[^a-z0-9]+/).filter((token) => token.length > 1); + const permitted = CAPABILITY_SEMANTIC_TOOL_CATALOG + .filter((descriptor) => descriptor.exposure === "optional") + .filter((descriptor) => options.namespace === undefined || capabilityToolNamespace(descriptor.operationId) === options.namespace) + .filter((descriptor) => decideCapabilitySemanticAuthorization(descriptor, context, "exposure").allowed) + .map((descriptor) => ({ descriptor, score: score(descriptor, tokens, normalized) })) + .filter((candidate) => candidate.score > 0) + .sort((left, right) => right.score - left.score || left.descriptor.operationId.localeCompare(right.descriptor.operationId)); + return Object.freeze({ + schema: "paperclip.capability-discovery.v1", + query, + namespace: options.namespace ?? null, + operations: Object.freeze(permitted.slice(0, limit).map(({ descriptor }) => toDefinition(descriptor))), + truncated: permitted.length > limit, + }); +} + +function score(descriptor: CapabilitySemanticToolDescriptor, tokens: readonly string[], query: string): number { + const namespace = capabilityToolNamespace(descriptor.operationId); + const id = descriptor.operationId.toLowerCase(); + const haystack = `${id} ${descriptor.title} ${descriptor.description} ${namespace}`.toLowerCase(); + let value = id === query ? 100 : namespace === query ? 50 : 0; + for (const token of tokens) { + if (id.includes(token)) value += 12; + if (namespace.includes(token)) value += 8; + if (haystack.includes(token)) value += 3; + } + return value; +} + +function toDefinition(descriptor: CapabilitySemanticToolDescriptor): CapabilitySemanticToolDefinition { + return { + name: descriptor.operationId, description: descriptor.description, + inputSchema: descriptor.inputSchema, outputSchema: descriptor.outputSchema, + annotations: { + semanticContract: descriptor.schema, operationId: descriptor.operationId, + version: descriptor.version, exposure: descriptor.exposure, + requiredClaims: descriptor.requiredClaims, + }, + }; +} diff --git a/packages/paperclip-runner/src/semantic-tools/dispatcher.ts b/packages/paperclip-runner/src/semantic-tools/dispatcher.ts index 44e8c67eb74..9851fe8d97c 100644 --- a/packages/paperclip-runner/src/semantic-tools/dispatcher.ts +++ b/packages/paperclip-runner/src/semantic-tools/dispatcher.ts @@ -19,6 +19,7 @@ import { containsProtectedSemanticData, redactSemanticValue, } from "./redaction.js"; +import { discoverCapabilityDefinitions } from "./discovery.js"; import type { CapabilitySemanticAuthorizationDecision, CapabilitySemanticAuthorizationRecord, @@ -89,6 +90,14 @@ export class CapabilitySemanticDispatcher { return deepFreeze(definitions); } + discoverTools( + runId: string, + query: string, + options: { namespace?: string; limit?: number } = {}, + ) { + return discoverCapabilityDefinitions(query, this.#policyContext(runId), options); + } + authorizationRecords(): readonly CapabilitySemanticAuthorizationRecord[] { return deepFreeze(structuredClone(this.#records)); } diff --git a/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts b/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts index 29b6ae04f9c..ffa4b5efe32 100644 --- a/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts +++ b/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts @@ -84,6 +84,26 @@ describe("Capability semantic catalog and authorization", () => { expect(adapter.snapshot().revision).toBe(before); }); + it("discovers only authorized optional tools and returns trusted schemas", async () => { + const claims = ["discovery:agents:read", "governance:approvals:read"]; + const adapter = await running(claims); + const dispatcher = new CapabilitySemanticDispatcher(adapter, { + scenario: { id: "lazy-search", claims }, explicitClaims: claims, + }); + const found = dispatcher.discoverTools(OPEN.identity.runId, "find company agents", { namespace: "discovery" }); + expect(found.operations.map((tool) => tool.name)).toEqual(["list_agents", "get_agent"]); + expect(found.operations.every((tool) => tool.annotations.semanticContract === "paperclip.semantic-tool.v1")).toBe(true); + expect(JSON.stringify(found)).not.toContain("list_approvals"); + }); + + it("does not disclose optional tools that current authority cannot invoke", async () => { + const adapter = await running(); + const dispatcher = new CapabilitySemanticDispatcher(adapter); + const found = dispatcher.discoverTools(OPEN.identity.runId, "create child task approval secret admin"); + expect(found.operations).toEqual([]); + expect(JSON.stringify(found.operations)).not.toMatch(/create_task|approval|secret|administer_company/); + }); + it("executes a granted optional operation through the mock port", async () => { const claim = "dependencies:write"; const adapter = await running([claim], {