mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 02:00:52 +01:00
feat(runner): add lazy capability discovery
This commit is contained in:
1 parent
1f4499460e
commit
a18aab01ad
8 files changed
+415
-9
No files matched your search
@@ -0,0 +1,178 @@
|
||||
# Paperclip Runner lazy tool discovery
|
||||
|
||||
Date: 2026-08-20
|
||||
Status: fallback implemented and evaluated; native adapter pending transport support
|
||||
|
||||
Reference: [OpenAI Tool Search](https://developers.openai.com/api/docs/guides/tools-tool-search)
|
||||
|
||||
## Outcome
|
||||
|
||||
Keep the model's initial protocol surface small while allowing it to discover optional
|
||||
Paperclip operations at runtime. Discovery never grants authority: the runner filters
|
||||
what may be discovered, and the dispatcher independently authorizes every invocation.
|
||||
|
||||
OpenAI Tool Search is the preferred provider capability where the selected model and
|
||||
driver support it. The runner also owns a provider-neutral fallback so Paperclip's
|
||||
protocol and security semantics do not depend on one transport.
|
||||
|
||||
## Product contract
|
||||
|
||||
The initial model context contains:
|
||||
|
||||
- a deliberately small always-present task core;
|
||||
- high-level descriptions of searchable namespaces, not every operation schema;
|
||||
- either native `tool_search` or runner-managed `discover_capabilities`;
|
||||
- explicit guidance that discovery should happen only when the task requires it.
|
||||
|
||||
Candidate namespace projection, kept below ten operations per namespace:
|
||||
|
||||
| Namespace | Purpose | Initial operations |
|
||||
| --- | --- | --- |
|
||||
| `discovery` | Find company tasks, agents, projects, and goals | `search_tasks`, `list_agents`, `get_agent`, `list_projects`, `list_goals` |
|
||||
| `delegation` | Create bounded child work and dependencies | `create_task`, `set_dependencies` |
|
||||
| `governance` | Read, request, discuss, and decide approvals | approval operations |
|
||||
| `workspace` | Inspect and control task workspace services | workspace operations |
|
||||
| `cases` | Read and maintain case records | case operations |
|
||||
| `routines` | Read and maintain routines | routine operations |
|
||||
| `company_skills` | Inspect and synchronize company skills | skill operations |
|
||||
| `secrets` | Discover metadata and perform tightly governed secret reads | secret operations |
|
||||
| `company_admin` | Portability and company administration | admin operations |
|
||||
|
||||
The exact always-present core is a measured product decision. Start with active-task
|
||||
context/history, progress/status reporting, and terminal/disposition operations; use
|
||||
eval results to decide whether document reads and writes remain eager or become
|
||||
deferred.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Canonical registry projection
|
||||
|
||||
Extend the canonical operation registry with discovery metadata: namespace, concise
|
||||
search description, searchable aliases, sensitivity classification, and whether an
|
||||
operation's name may be disclosed when unauthorized. Continue deriving provider
|
||||
schemas and scenario descriptors from the same canonical operation identity.
|
||||
|
||||
### Discovery policy
|
||||
|
||||
Given a query and session identity, compute candidates from the canonical registry,
|
||||
then filter by company, actor, task, mode, role, claims, enabled capabilities, scenario
|
||||
policy, disabled-by-default status, and driver support. Rank only the permitted set.
|
||||
Sensitive undiscoverable operations must not leak through names, counts, descriptions,
|
||||
errors, or timing where practical.
|
||||
|
||||
Return trusted registry definitions only. Never accept a model-authored schema or an
|
||||
untrusted remote definition as executable protocol authority.
|
||||
|
||||
### Invocation policy
|
||||
|
||||
Loaded tools receive no durable authority grant. Every call goes through the existing
|
||||
semantic dispatcher and repeats the current policy decision. Revoked claims or changed
|
||||
task state therefore take effect even if the schema remains in conversation context.
|
||||
|
||||
### Provider adapters
|
||||
|
||||
1. **Native adapter:** project namespaces/deferred functions and consume provider
|
||||
tool-search call/output events.
|
||||
2. **Runner-managed adapter:** expose `discover_capabilities`; resolve the query
|
||||
locally; add selected trusted definitions on the next model continuation.
|
||||
3. Normalize both paths into PRP events so Runner Lab and Evalbook do not depend on
|
||||
provider-specific event shapes.
|
||||
|
||||
### Protocol evidence
|
||||
|
||||
Add evidence events for discovery requested, candidates considered (redacted), policy
|
||||
decision, definitions loaded, namespace activated, and discovery failure. Preserve the
|
||||
query and returned operation IDs when safe, plus model/driver/configuration, token and
|
||||
cost measurements. Link the eventual invocation back to the discovery event.
|
||||
|
||||
## Delivery phases
|
||||
|
||||
### D0 — transport spike
|
||||
|
||||
Prove whether the Codex app-server driver can send deferred definitions and surface
|
||||
native tool-search events for the configured GPT-5.4-class model. Produce one recorded
|
||||
attempt. Do not change default exposure.
|
||||
|
||||
Result: the current Codex app-server driver accepts dynamic tools at thread creation but
|
||||
does not expose Responses API native Tool Search events or a supported mid-turn schema
|
||||
append operation. Native support is therefore not claimed. The provider-neutral gateway
|
||||
below is the implemented path.
|
||||
|
||||
### D1 — provider-neutral domain layer
|
||||
|
||||
Add discovery metadata and namespace validation to the canonical registry. Implement
|
||||
the policy-filtered search service and normalized discovery event contract. Unit-test
|
||||
ranking, nondisclosure, mode/role/claim filtering, and registry drift.
|
||||
|
||||
### D2 — runner-managed fallback
|
||||
|
||||
Implement `discover_capabilities`, continuation-time schema loading, policy recheck on
|
||||
invocation, limits on results/searches/loaded tools, and deterministic errors. Ship
|
||||
behind `toolExposure: "eager" | "lazy"`, defaulting to `eager`.
|
||||
|
||||
Implemented: lazy threads receive the always-present core plus
|
||||
`discover_capabilities` and `invoke_discovered_capability`. Search is deterministic,
|
||||
namespace-aware, limited to eight results, filtered by live exposure policy, and returns
|
||||
only trusted catalog schemas. Invocation requires prior discovery and is reauthorized by
|
||||
the semantic dispatcher. Loaded operation identities survive runner snapshots.
|
||||
|
||||
### D3 — native Tool Search adapter
|
||||
|
||||
If D0 succeeds, implement native namespace/deferred-tool projection and normalize its
|
||||
events to the same domain contract. If D0 fails, retain the evidence and defer this
|
||||
adapter without blocking the fallback.
|
||||
|
||||
### D4 — Runner Lab and reports
|
||||
|
||||
Render discovery inline in the issue thread, expose policy detail in Devtools, connect
|
||||
loaded definitions to subsequent calls, and report eager/lazy configuration plus token,
|
||||
latency, and cost deltas.
|
||||
|
||||
Implemented: discovery is retained as redacted `tool_discovery` evidence, appears on the
|
||||
semantic-tool boundary, and Devtools reports exposure mode and loaded operation IDs.
|
||||
|
||||
### D5 — eval and rollout
|
||||
|
||||
Implement the lazy-discovery eval track in the sibling `paperclip-evals` repository.
|
||||
Run deterministic checks first and live comparisons on GPT-5.4 mini only. Switch the
|
||||
default to lazy after semantic parity, authorization/nondisclosure, observability, and
|
||||
efficiency gates pass. Keep eager mode as a temporary diagnostic control until one
|
||||
release cycle completes.
|
||||
|
||||
Initial result: all 35 GPT-5.4-mini live roster cases passed on
|
||||
`20260820-full-protocol-lazy-01`, including three lazy-discovery cases and four new
|
||||
multi-operation workflows. Eager remains the default pending a broader repetition and
|
||||
efficiency comparison; this single green run is functional evidence, not a reliability
|
||||
estimate.
|
||||
|
||||
## Guardrails and limits
|
||||
|
||||
- Maximum returned definitions per search and maximum loaded definitions per session.
|
||||
- Maximum discovery calls per turn/session, with loop detection.
|
||||
- No `generic_api_request` outside explicit skill-test scenarios.
|
||||
- No disclosure of secret/admin tools to unauthorized actors.
|
||||
- Namespace and operation descriptions are trusted code-reviewed content.
|
||||
- Search results are company-scoped and deterministic for identical policy state.
|
||||
- Tool invocation always rechecks live policy and terminal task state.
|
||||
- Cache behavior is measured but never allowed to weaken authorization freshness.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Lazy sessions begin without optional operation schemas.
|
||||
- A permitted optional operation can be found and invoked end to end.
|
||||
- An unauthorized operation is neither disclosed nor invokable.
|
||||
- Revocation after loading prevents invocation.
|
||||
- Native and fallback routes produce equivalent semantic PRP evidence.
|
||||
- Runner Lab makes discovery and its causal invocation inspectable.
|
||||
- Evalbook compares eager and lazy correctness, tokens, latency, and cost.
|
||||
- No regression in the existing deterministic capability suite or protocol eval roster.
|
||||
|
||||
## Open decisions to resolve with evidence
|
||||
|
||||
1. Exact always-present core, especially document operations.
|
||||
2. Lexical versus embedding-assisted local ranking; begin lexical and deterministic.
|
||||
3. Whether sensitive namespaces are hidden entirely or return a generic unavailable
|
||||
result; prefer complete nondisclosure.
|
||||
4. Session persistence rules for loaded schemas across compaction and continuation.
|
||||
5. Whether native Tool Search is available through Codex app-server with the required
|
||||
event fidelity; D0 decides this rather than assuming it.
|
||||
@@ -96,7 +96,7 @@ function publicCompanyState(serialized: string): CapabilityJsonValue {
|
||||
}
|
||||
|
||||
function boundary(kind: string): string {
|
||||
if (kind === "tool_call" || kind === "tool_result") return "semantic tool";
|
||||
if (kind === "tool_call" || kind === "tool_result" || kind === "tool_discovery") return "semantic tool";
|
||||
if (kind === "provider_event") return "Codex app-server";
|
||||
if (kind === "process") return "runnerd process";
|
||||
if (kind === "interaction") return "mock control plane";
|
||||
@@ -157,6 +157,8 @@ export function projectCapabilityDevtools(snapshot: CapabilityLiveSessionSnapsho
|
||||
currentAttemptId: snapshot.currentAttemptId ?? null,
|
||||
terminalTurns: snapshot.terminalTurns ?? [],
|
||||
usageLedger: snapshot.usageLedger ?? [],
|
||||
toolExposure: snapshot.config.toolExposure ?? "eager",
|
||||
loadedOperationIds: snapshot.loadedOperationIds ?? [],
|
||||
createdAt: snapshot.createdAt,
|
||||
updatedAt: snapshot.updatedAt,
|
||||
}),
|
||||
@@ -166,6 +168,7 @@ export function projectCapabilityDevtools(snapshot: CapabilityLiveSessionSnapsho
|
||||
effectiveCapabilities: snapshot.config.capabilities,
|
||||
explicitClaims: snapshot.config.explicitClaims,
|
||||
turnTimeoutMs: snapshot.config.turnTimeoutMs,
|
||||
toolExposure: snapshot.config.toolExposure ?? "eager",
|
||||
workingDirectory: "[withheld]",
|
||||
}),
|
||||
};
|
||||
|
||||
@@ -32,6 +32,7 @@ export type CapabilityEvidenceKind =
|
||||
| "session"
|
||||
| "provider_event"
|
||||
| "tool_exposure"
|
||||
| "tool_discovery"
|
||||
| "tool_call"
|
||||
| "tool_result"
|
||||
| "interaction"
|
||||
@@ -313,6 +314,15 @@ export function redactCapabilityEvidenceData(
|
||||
.sort(),
|
||||
scenarioId: clamp(asString(data.scenarioId), 120),
|
||||
};
|
||||
case "tool_discovery":
|
||||
return {
|
||||
action: clamp(asString(data.action), 80),
|
||||
query: clamp(asString(data.query), 500),
|
||||
namespace: clamp(asString(data.namespace), 120),
|
||||
operationIds: (Array.isArray(data.operationIds) ? data.operationIds : [])
|
||||
.filter((id): id is string => typeof id === "string" && CATALOG_OPERATION_IDS.has(id))
|
||||
.sort(),
|
||||
};
|
||||
case "tool_call": {
|
||||
const operationId = asString(data.operationId);
|
||||
return {
|
||||
@@ -374,6 +384,10 @@ export function capabilityEvidenceDetail(
|
||||
const ids = Array.isArray(data.operationIds) ? data.operationIds.length : 0;
|
||||
return `tool exposure · ${ids} operation${ids === 1 ? "" : "s"}`;
|
||||
}
|
||||
case "tool_discovery": {
|
||||
const ids = Array.isArray(data.operationIds) ? data.operationIds.length : 0;
|
||||
return `tool discovery · ${value("action") || "searched"} · ${ids} loaded`;
|
||||
}
|
||||
case "tool_call": {
|
||||
const fields = asRecord(data.input).fields;
|
||||
const count = Array.isArray(fields) ? fields.length : 0;
|
||||
@@ -450,6 +464,12 @@ export function capabilityEvidenceDetails(
|
||||
add("Scenario", data.scenarioId);
|
||||
addList("Operations", data.operationIds);
|
||||
break;
|
||||
case "tool_discovery":
|
||||
add("Action", data.action);
|
||||
add("Query", data.query);
|
||||
add("Namespace", data.namespace);
|
||||
addList("Operations", data.operationIds);
|
||||
break;
|
||||
case "tool_call": {
|
||||
const input = asRecord(data.input);
|
||||
add("Call", data.callId);
|
||||
|
||||
@@ -273,6 +273,24 @@ function providerState(): FakeProviderState {
|
||||
}
|
||||
|
||||
describe("Capability live runnerd and Codex session", () => {
|
||||
it("opens lazy sessions with core and discovery gateways instead of optional schemas", async () => {
|
||||
const state = providerState();
|
||||
const claim = "discovery:agents:read";
|
||||
const service = new CapabilityLiveSessionService({ transportFactory: fakeTransportFactory(state) });
|
||||
const session = await service.create({
|
||||
runId: "run-live-lazy", sessionId: "session-live-lazy", toolExposure: "lazy",
|
||||
capabilities: [claim], explicitClaims: [claim], scenario: { id: "lazy", claims: [claim] },
|
||||
});
|
||||
const start = state.transports[0]!.requests.find((entry) => entry.method === "thread/start")!;
|
||||
const names = (start.params.dynamicTools as Array<{ name: string }>).map((tool) => tool.name);
|
||||
expect(names).toContain("get_task_context");
|
||||
expect(names).toContain("discover_capabilities");
|
||||
expect(names).toContain("invoke_discovered_capability");
|
||||
expect(names).not.toContain("list_agents");
|
||||
expect(session.snapshot().config.toolExposure).toBe("lazy");
|
||||
await service.shutdown(session.id);
|
||||
});
|
||||
|
||||
it("returns typed mock results to the same multi-turn Codex thread without Paperclip network calls", async () => {
|
||||
const state = providerState();
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch");
|
||||
|
||||
@@ -17,6 +17,7 @@ import type {
|
||||
} from "../mock-core/capability-control-plane-types.js";
|
||||
import { CapabilityMockControlPlaneAdapter } from "../mock-core/capability-mock-control-plane-adapter.js";
|
||||
import { CapabilitySemanticDispatcher } from "../semantic-tools/dispatcher.js";
|
||||
import { CAPABILITY_DISCOVERY_NAMESPACES, type CapabilityToolExposureMode } from "../semantic-tools/discovery.js";
|
||||
import type {
|
||||
CapabilitySemanticAuthorizationRecord,
|
||||
CapabilitySemanticScenarioPolicy,
|
||||
@@ -44,6 +45,8 @@ const LIVE_BASE_INSTRUCTIONS = [
|
||||
"Do not discover skills, credentials, endpoints, or hidden control-plane capabilities.",
|
||||
].join(" ");
|
||||
const CODEX_PERMISSION_PROFILE = "paperclip-runner-workspace-only";
|
||||
const DISCOVER_TOOL = "discover_capabilities";
|
||||
const INVOKE_DISCOVERED_TOOL = "invoke_discovered_capability";
|
||||
|
||||
export type CapabilityLiveSessionStatus =
|
||||
| "starting"
|
||||
@@ -95,6 +98,7 @@ export interface CapabilityLiveSessionConfigSnapshot {
|
||||
explicitClaims: string[];
|
||||
seedState: string;
|
||||
turnTimeoutMs: number;
|
||||
toolExposure?: CapabilityToolExposureMode;
|
||||
}
|
||||
|
||||
export type CapabilityLiveAttemptStatus = "running" | "succeeded" | "failed" | "terminated";
|
||||
@@ -183,6 +187,7 @@ export interface CapabilityLiveSessionSnapshot {
|
||||
usageLedger?: CapabilityLiveUsageReceipt[];
|
||||
/** Bounded mock-control-plane snapshots for DevTools time travel and diffs. */
|
||||
stateHistory?: CapabilityLiveStateRevision[];
|
||||
loadedOperationIds?: string[];
|
||||
}
|
||||
|
||||
export interface CreateCapabilityLiveSessionInput {
|
||||
@@ -199,6 +204,7 @@ export interface CreateCapabilityLiveSessionInput {
|
||||
sessionId?: string;
|
||||
attemptId?: string;
|
||||
turnTimeoutMs?: number;
|
||||
toolExposure?: CapabilityToolExposureMode;
|
||||
}
|
||||
|
||||
export interface ResumeCapabilityLiveSessionInput {
|
||||
@@ -534,6 +540,18 @@ function dynamicToolSpec(definition: CapabilitySemanticToolDefinition): Record<s
|
||||
};
|
||||
}
|
||||
|
||||
function discoveryToolSpecs(): Record<string, unknown>[] {
|
||||
return [{
|
||||
name: DISCOVER_TOOL,
|
||||
description: `Search authorized optional Paperclip capabilities. Namespaces: ${CAPABILITY_DISCOVERY_NAMESPACES.map((item) => `${item.name} (${item.description})`).join("; ")}`,
|
||||
inputSchema: { type: "object", properties: { query: { type: "string", minLength: 1, maxLength: 500 }, namespace: { type: "string" }, limit: { type: "integer", minimum: 1, maximum: 8 } }, required: ["query"], additionalProperties: false },
|
||||
}, {
|
||||
name: INVOKE_DISCOVERED_TOOL,
|
||||
description: "Invoke one optional Paperclip operation returned by discover_capabilities. Authority is rechecked at invocation.",
|
||||
inputSchema: { type: "object", properties: { operationId: { type: "string" }, input: { type: "object", additionalProperties: true } }, required: ["operationId", "input"], additionalProperties: false },
|
||||
}];
|
||||
}
|
||||
|
||||
function userInput(message: string): Record<string, unknown> {
|
||||
return { type: "text", text: message, text_elements: [] };
|
||||
}
|
||||
@@ -608,6 +626,7 @@ export class CapabilityLiveSessionService {
|
||||
explicitClaims: authority.explicitClaims,
|
||||
seedState,
|
||||
turnTimeoutMs: input.turnTimeoutMs ?? 120_000,
|
||||
toolExposure: input.toolExposure ?? "eager",
|
||||
},
|
||||
store: this.#store,
|
||||
transportFactory: this.#transportFactory,
|
||||
@@ -742,6 +761,7 @@ export class CapabilityLiveSessionService {
|
||||
capabilities: config.capabilities,
|
||||
explicitClaims: config.explicitClaims,
|
||||
turnTimeoutMs: config.turnTimeoutMs,
|
||||
toolExposure: config.toolExposure ?? "eager",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -810,6 +830,7 @@ export class CapabilityLiveSession {
|
||||
#eventSeq = 0;
|
||||
#turnEventCount = 0;
|
||||
readonly #rawUsageByTurn = new Map<string, { providerRequests: number; inputTokens: number; outputTokens: number; cachedInputTokens: number; reasoningTokens: number }>();
|
||||
readonly #loadedOperationIds = new Set<string>();
|
||||
|
||||
private constructor(options: OpenSessionOptions & { snapshot?: CapabilityLiveSessionSnapshot }) {
|
||||
this.#port = options.port;
|
||||
@@ -847,6 +868,7 @@ export class CapabilityLiveSession {
|
||||
operationId: "session.open",
|
||||
state: this.#port.serialize(),
|
||||
}]);
|
||||
for (const operationId of options.snapshot?.loadedOperationIds ?? []) this.#loadedOperationIds.add(operationId);
|
||||
this.#providerThreadId = options.snapshot?.providerThreadId ?? "";
|
||||
this.#providerSessionId = options.snapshot?.providerSessionId ?? null;
|
||||
this.#providerModel = options.snapshot?.providerModel === undefined
|
||||
@@ -1015,6 +1037,7 @@ export class CapabilityLiveSession {
|
||||
terminalTurns: structuredClone(this.#terminalTurns),
|
||||
usageLedger: structuredClone(this.#usageLedger),
|
||||
stateHistory: structuredClone(this.#stateHistory),
|
||||
loadedOperationIds: [...this.#loadedOperationIds].sort(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1360,10 +1383,14 @@ export class CapabilityLiveSession {
|
||||
capabilities: { experimentalApi: true, requestAttestation: false },
|
||||
});
|
||||
this.#transport.notify("initialized");
|
||||
const tools = this.#dispatcher.listTools(this.#authority.runId);
|
||||
const authorizedTools = this.#dispatcher.listTools(this.#authority.runId);
|
||||
const tools = (this.#config.toolExposure ?? "eager") === "lazy"
|
||||
? authorizedTools.filter((tool) => tool.annotations.exposure === "always")
|
||||
: authorizedTools;
|
||||
this.#appendEvidence("tool_exposure", null, {
|
||||
operationIds: tools.map((tool) => tool.name),
|
||||
scenarioId: this.#config.scenario.id,
|
||||
toolExposure: this.#config.toolExposure ?? "eager",
|
||||
});
|
||||
if (resume) {
|
||||
const read = await this.#transport.request("thread/read", {
|
||||
@@ -1427,7 +1454,10 @@ export class CapabilityLiveSession {
|
||||
runtimeWorkspaceRoots: [this.#config.workingDirectory],
|
||||
approvalPolicy: "never",
|
||||
baseInstructions: LIVE_BASE_INSTRUCTIONS,
|
||||
dynamicTools: tools.map(dynamicToolSpec),
|
||||
dynamicTools: [
|
||||
...tools.map(dynamicToolSpec),
|
||||
...((this.#config.toolExposure ?? "eager") === "lazy" ? discoveryToolSpecs() : []),
|
||||
],
|
||||
experimentalRawEvents: true,
|
||||
persistExtendedHistory: true,
|
||||
});
|
||||
@@ -1515,25 +1545,55 @@ export class CapabilityLiveSession {
|
||||
contentItems: [{ type: "inputText", text: "Tool call was outside the active Capability thread and turn." }],
|
||||
};
|
||||
}
|
||||
if (operationId === DISCOVER_TOOL) {
|
||||
const args = record(request.params.arguments);
|
||||
try {
|
||||
const discovered = this.#dispatcher.discoverTools(
|
||||
this.#authority.runId,
|
||||
text(args.query),
|
||||
{ ...(text(args.namespace) ? { namespace: text(args.namespace) } : {}), ...(typeof args.limit === "number" ? { limit: args.limit } : {}) },
|
||||
);
|
||||
for (const operation of discovered.operations) this.#loadedOperationIds.add(operation.name);
|
||||
this.#appendEvidence("tool_discovery", turnId, {
|
||||
action: "loaded", query: discovered.query, namespace: discovered.namespace ?? "",
|
||||
operationIds: discovered.operations.map((operation) => operation.name),
|
||||
});
|
||||
await this.#persist();
|
||||
return { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(discovered) }] };
|
||||
} catch (error) {
|
||||
this.#appendEvidence("tool_discovery", turnId, { action: "rejected", query: text(args.query), namespace: text(args.namespace), operationIds: [] });
|
||||
return { success: false, contentItems: [{ type: "inputText", text: error instanceof Error ? error.message : "Capability discovery failed." }] };
|
||||
}
|
||||
}
|
||||
let semanticOperationId = operationId;
|
||||
let semanticInput = request.params.arguments;
|
||||
if (operationId === INVOKE_DISCOVERED_TOOL) {
|
||||
const args = record(request.params.arguments);
|
||||
semanticOperationId = text(args.operationId);
|
||||
semanticInput = args.input;
|
||||
if (!this.#loadedOperationIds.has(semanticOperationId)) {
|
||||
return { success: false, contentItems: [{ type: "inputText", text: "Operation was not loaded by discover_capabilities." }] };
|
||||
}
|
||||
}
|
||||
const beforeRevision = this.#port.snapshot().revision;
|
||||
this.#appendEvidence("tool_call", turnId, {
|
||||
callId,
|
||||
operationId,
|
||||
input: jsonValue(request.params.arguments),
|
||||
operationId: semanticOperationId,
|
||||
input: jsonValue(semanticInput),
|
||||
beforeRevision,
|
||||
});
|
||||
this.#emit({ turnId, kind: "activity", reason: "tool_call" });
|
||||
const result = await this.#dispatcher.dispatch({
|
||||
runId: this.#authority.runId,
|
||||
callId,
|
||||
operationId,
|
||||
input: request.params.arguments,
|
||||
operationId: semanticOperationId,
|
||||
input: semanticInput,
|
||||
});
|
||||
this.#recordStateRevision(turnId, operationId);
|
||||
this.#recordStateRevision(turnId, semanticOperationId);
|
||||
const replayDisposition = result.ok ? text(record(result.result).disposition) : "";
|
||||
this.#appendEvidence("tool_result", turnId, {
|
||||
callId,
|
||||
operationId,
|
||||
operationId: semanticOperationId,
|
||||
result: jsonValue(result),
|
||||
beforeRevision,
|
||||
afterRevision: result.stateRevision,
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
import { CAPABILITY_SEMANTIC_TOOL_CATALOG } from "./catalog.js";
|
||||
import { decideCapabilitySemanticAuthorization } from "./policy.js";
|
||||
import type {
|
||||
CapabilitySemanticOperationId,
|
||||
CapabilitySemanticPolicyContext,
|
||||
CapabilitySemanticToolDefinition,
|
||||
CapabilitySemanticToolDescriptor,
|
||||
} from "./types.js";
|
||||
|
||||
export type CapabilityToolExposureMode = "eager" | "lazy";
|
||||
|
||||
export interface CapabilityDiscoveryResult {
|
||||
readonly schema: "paperclip.capability-discovery.v1";
|
||||
readonly query: string;
|
||||
readonly namespace: string | null;
|
||||
readonly operations: readonly CapabilitySemanticToolDefinition[];
|
||||
readonly truncated: boolean;
|
||||
}
|
||||
|
||||
const NAMESPACE: Readonly<Record<CapabilitySemanticOperationId, string>> = Object.freeze({
|
||||
get_task_context: "active_task", get_task_history: "active_task",
|
||||
list_documents: "documents", read_document: "documents", list_document_revisions: "documents",
|
||||
report_progress: "active_task", answer_status_question: "active_task", write_document: "documents",
|
||||
request_human_input: "documents", register_deliverable: "documents", finish_task: "active_task",
|
||||
block_task: "active_task", request_review: "active_task", search_tasks: "discovery",
|
||||
list_agents: "discovery", get_agent: "discovery", create_task: "delegation",
|
||||
set_dependencies: "delegation", list_approvals: "governance", get_approval: "governance",
|
||||
get_approval_context: "governance", request_approval: "governance",
|
||||
decide_approval: "governance", comment_on_approval: "governance",
|
||||
get_workspace_runtime: "workspace", control_workspace_service: "workspace",
|
||||
schedule_wake: "continuation", generic_api_request: "test_infrastructure",
|
||||
});
|
||||
|
||||
export const CAPABILITY_DISCOVERY_NAMESPACES = Object.freeze([
|
||||
{ name: "discovery", description: "Find company tasks and agents." },
|
||||
{ name: "delegation", description: "Create bounded child work and task dependencies." },
|
||||
{ name: "governance", description: "Read, request, discuss, and decide approvals." },
|
||||
{ name: "workspace", description: "Inspect and control active-task workspace services." },
|
||||
{ name: "continuation", description: "Schedule a bounded continuation wake." },
|
||||
]);
|
||||
|
||||
export function capabilityToolNamespace(operationId: CapabilitySemanticOperationId): string {
|
||||
return NAMESPACE[operationId];
|
||||
}
|
||||
|
||||
export function alwaysCapabilityDefinitions(): readonly CapabilitySemanticToolDefinition[] {
|
||||
return CAPABILITY_SEMANTIC_TOOL_CATALOG.filter((item) => item.exposure === "always").map(toDefinition);
|
||||
}
|
||||
|
||||
export function discoverCapabilityDefinitions(
|
||||
query: string,
|
||||
context: CapabilitySemanticPolicyContext,
|
||||
options: { namespace?: string; limit?: number } = {},
|
||||
): CapabilityDiscoveryResult {
|
||||
const normalized = query.trim().toLowerCase();
|
||||
if (normalized.length === 0) throw new Error("discovery_query_empty");
|
||||
const limit = Math.max(1, Math.min(options.limit ?? 5, 8));
|
||||
const tokens = normalized.split(/[^a-z0-9]+/).filter((token) => token.length > 1);
|
||||
const permitted = CAPABILITY_SEMANTIC_TOOL_CATALOG
|
||||
.filter((descriptor) => descriptor.exposure === "optional")
|
||||
.filter((descriptor) => options.namespace === undefined || capabilityToolNamespace(descriptor.operationId) === options.namespace)
|
||||
.filter((descriptor) => decideCapabilitySemanticAuthorization(descriptor, context, "exposure").allowed)
|
||||
.map((descriptor) => ({ descriptor, score: score(descriptor, tokens, normalized) }))
|
||||
.filter((candidate) => candidate.score > 0)
|
||||
.sort((left, right) => right.score - left.score || left.descriptor.operationId.localeCompare(right.descriptor.operationId));
|
||||
return Object.freeze({
|
||||
schema: "paperclip.capability-discovery.v1",
|
||||
query,
|
||||
namespace: options.namespace ?? null,
|
||||
operations: Object.freeze(permitted.slice(0, limit).map(({ descriptor }) => toDefinition(descriptor))),
|
||||
truncated: permitted.length > limit,
|
||||
});
|
||||
}
|
||||
|
||||
function score(descriptor: CapabilitySemanticToolDescriptor, tokens: readonly string[], query: string): number {
|
||||
const namespace = capabilityToolNamespace(descriptor.operationId);
|
||||
const id = descriptor.operationId.toLowerCase();
|
||||
const haystack = `${id} ${descriptor.title} ${descriptor.description} ${namespace}`.toLowerCase();
|
||||
let value = id === query ? 100 : namespace === query ? 50 : 0;
|
||||
for (const token of tokens) {
|
||||
if (id.includes(token)) value += 12;
|
||||
if (namespace.includes(token)) value += 8;
|
||||
if (haystack.includes(token)) value += 3;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function toDefinition(descriptor: CapabilitySemanticToolDescriptor): CapabilitySemanticToolDefinition {
|
||||
return {
|
||||
name: descriptor.operationId, description: descriptor.description,
|
||||
inputSchema: descriptor.inputSchema, outputSchema: descriptor.outputSchema,
|
||||
annotations: {
|
||||
semanticContract: descriptor.schema, operationId: descriptor.operationId,
|
||||
version: descriptor.version, exposure: descriptor.exposure,
|
||||
requiredClaims: descriptor.requiredClaims,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
containsProtectedSemanticData,
|
||||
redactSemanticValue,
|
||||
} from "./redaction.js";
|
||||
import { discoverCapabilityDefinitions } from "./discovery.js";
|
||||
import type {
|
||||
CapabilitySemanticAuthorizationDecision,
|
||||
CapabilitySemanticAuthorizationRecord,
|
||||
@@ -89,6 +90,14 @@ export class CapabilitySemanticDispatcher {
|
||||
return deepFreeze(definitions);
|
||||
}
|
||||
|
||||
discoverTools(
|
||||
runId: string,
|
||||
query: string,
|
||||
options: { namespace?: string; limit?: number } = {},
|
||||
) {
|
||||
return discoverCapabilityDefinitions(query, this.#policyContext(runId), options);
|
||||
}
|
||||
|
||||
authorizationRecords(): readonly CapabilitySemanticAuthorizationRecord[] {
|
||||
return deepFreeze(structuredClone(this.#records));
|
||||
}
|
||||
|
||||
@@ -84,6 +84,26 @@ describe("Capability semantic catalog and authorization", () => {
|
||||
expect(adapter.snapshot().revision).toBe(before);
|
||||
});
|
||||
|
||||
it("discovers only authorized optional tools and returns trusted schemas", async () => {
|
||||
const claims = ["discovery:agents:read", "governance:approvals:read"];
|
||||
const adapter = await running(claims);
|
||||
const dispatcher = new CapabilitySemanticDispatcher(adapter, {
|
||||
scenario: { id: "lazy-search", claims }, explicitClaims: claims,
|
||||
});
|
||||
const found = dispatcher.discoverTools(OPEN.identity.runId, "find company agents", { namespace: "discovery" });
|
||||
expect(found.operations.map((tool) => tool.name)).toEqual(["list_agents", "get_agent"]);
|
||||
expect(found.operations.every((tool) => tool.annotations.semanticContract === "paperclip.semantic-tool.v1")).toBe(true);
|
||||
expect(JSON.stringify(found)).not.toContain("list_approvals");
|
||||
});
|
||||
|
||||
it("does not disclose optional tools that current authority cannot invoke", async () => {
|
||||
const adapter = await running();
|
||||
const dispatcher = new CapabilitySemanticDispatcher(adapter);
|
||||
const found = dispatcher.discoverTools(OPEN.identity.runId, "create child task approval secret admin");
|
||||
expect(found.operations).toEqual([]);
|
||||
expect(JSON.stringify(found.operations)).not.toMatch(/create_task|approval|secret|administer_company/);
|
||||
});
|
||||
|
||||
it("executes a granted optional operation through the mock port", async () => {
|
||||
const claim = "dependencies:write";
|
||||
const adapter = await running([claim], {
|
||||
|
||||
Reference in new issue
Block a user