mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use installed skills and exact version pins during execution. > - Managed skills still depended on mutable directories on the web server. > - A replacement server could lose acknowledged edits or remove an unassigned skill whose directory was absent. > - This pull request makes immutable versions the authority for managed content and installed repository reads. > - The server can rebuild delivery files from durable content after replacement. > - This establishes skills readiness for ephemeral web servers. General server draining and workspace recovery remain separate work. ## Linked Issues or Issue Description **Subsystem affected** Skills services, shared contracts, database receipts, Skill Studio, and runtime delivery. **Problem or motivation** A fresh web server must discover, read, edit, and deliver protected installed skills using the same database, object storage, secrets, and supported application artifacts. It must not need its predecessor's files or process memory. **Proposed solution** Use `managed` ownership with a null source locator. Read installed content by immutable version. Prepare packages before a short publication transaction. Persist execution selections before materialization. Recover exact pinned runtime assets and reject deletion while active or recovering pins exist. **Alternatives considered** Reconstructing a mutable installed directory preserves the old source-of-truth ambiguity. Permanent dual writes create two competing authorities. This change uses one managed model after a bounded legacy conversion. **Roadmap alignment** This improves the existing Skills Manager, Studio, version history, and Cloud execution capabilities. The maintainer supplied the implementation contract. This does not add a separate skills product or deploy fleet changes. Related work: #18082 supplies immutable package storage. #17273 addresses edits outside the API for linked local skills; this PR keeps that source category separate. #2648 is a closed earlier ephemeral-filesystem proposal. ## What Changed - Add managed ownership and focused content publication and execution materialization modules. Keep the archive format and package storage tables. - Publish managed create, save, delete, fork, rename, restore, and catalog content with version checks. Add durable create receipts. Preserve file-save receipt responses. - Return file version identities. Keep Studio drafts after a conflict and provide an explicit reload action. A delayed reload cannot overwrite newer typing or navigation. - Persist exact runtime selections, including empty selections. Rebuild pinned assets and verify their admitted digests on a replacement server. - Preserve source access on forks and disconnected copies. Keep the existing workspace policy category for managed skills. - Add bounded conversion, eligibility, and independent verification phases. Load persisted Cloud configuration in the standalone CLI without seeding or changing it. Reject partial current versions against installed inventories; recover only complete saved content or the exact source. Document storage migration blockers and the compatible rollback floor. - Add separate-process PostgreSQL/S3 qualification and update domain, retention, policy, and UI regressions. - Pass the plugin worker manager into Studio execution, admit restricted `skill_test` mode through native Runner contracts, and preserve admitted scope across native and REST tools. Native tests cannot create ordinary tasks, browse company data, or use external tools. - Select the exact durable Studio revision even when unassigned or Beta skills is off. Persist Studio output and audit atomically with native task completion, preserving the result on replay. ## Verification - Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full CI: all 49 workflow jobs completed successfully ([run 41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)). This includes typecheck, test suites, Runner checks, browser checks and build. Greptile review 15 is 5/5 on that same revision, with zero unresolved threads. The PR is ready for review. - Local workspace typecheck and build passed after integrating master. Focused checks passed: 33 execution/completion cases, 52 Runner HTTP cases, 16 managed-content cases, 17 Studio cases, the new real-process CLI regression, map checks and UI token gates. The CLI regression proves inspection does not seed missing Cloud configuration. Server typecheck passed after that fix. - The independent-process PostgreSQL/S3 matrix passed through normal HTTP skill routes. It covers A → B → C replacement, lost commit responses, upload crashes, concurrent saves, binary contents and executable delivery. Metadata listing and warm selected delivery retain zero object downloads. Local timing measurements are recorded; they do not establish hosted latency. - Hosted A → B → C passed on the owned disposable test tenant with external Postgres, S3 and a real remote Paperclip Computer. Predecessor temporary homes were absent. C read and executed B's acknowledged managed save. New runs used repository v1 while v2 was installed. Hashes matched an independently checked exact upstream commit. - Compatible image rollback to `6518b323` passed. Studio retained the unassigned fork's C save. A fresh remote run executed the newer managed content and historical repository pin. The actual shell event confirmed expected hashes, stdout, exit 0 and mode 0555. - Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run 41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)). The final Studio test delivered only the unassigned fork's exact v3, executed its script and persisted a successful output snapshot. Independent command-event inspection confirmed all three hashes, stdout, exit 0 and mode 0555. All 14 installations were version-backed. Independent verification checked all 25 versions: 23 remote objects, two bundled database snapshots, zero blockers. - The deployed image is `9200e3fcb4`. Later master integration, the client-only delayed-reload draft guard and the CLI bootstrap have separate automated proof. The CLI bootstrap was also exercised against the tenant's stored configuration and storage through the same read-only services. This PR does not claim that the deployed image is the final PR revision. - Live Studio checks exposed worker wiring, admission, access scope, exact selection and completion-publication gaps. They are fixed. Failed attempts remain in the evidence. The full local suite exceeded 80 minutes under host contention and was stopped. Later combined/isolated local attempts also timed out; they are not counted as passes. Full CI covers the integrated suites. Earlier CI found an obsolete restart-scope fixture, which was corrected and passed all 52 HTTP cases. - Reproduction and migration: [`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md). Source revisions, deployments, versions, run IDs, hashes and failure history: [`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md). ## Risks - Legacy conversion requires stopped admission and retired filesystem-authoritative writers. Pre-conversion binaries are not supported rollback targets afterward. - Cloud readiness requires remote object storage. Local provider receipts and provider mismatches remain blockers until objects are copied and verified. - Linked local sources retain filesystem semantics and are outside the cloud guarantee. - Cold reads still fetch the existing archive format. Warm runtime preparation retains the verified-cache path. - Bundled reconciliation preserves an installed head. Existing release snapshots remain selectable. An older server must not replace a newer head with its shipped files. - No production volumes, fleet defaults, or general deployment coordination change in this PR. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and browser use. The exact backend model ID and context-window size are not exposed in this session. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
222 lines
11 KiB
TypeScript
222 lines
11 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { deduplicateToolDescriptors, ToolGatewayHttpError, type ToolGatewayDescriptor, type ToolGatewayService } from "../tool-gateway.js";
|
|
|
|
type WorkMode = "standard" | "planning" | "ask" | "skill_test";
|
|
type ToolDefinition = Record<string, unknown>;
|
|
|
|
const SEARCH_TOOL = "paperclip_search_assigned_tools";
|
|
const CALL_TOOL = "paperclip_call_assigned_tool";
|
|
// A single valid runner schema can be 512 KiB. Leave room for its description
|
|
// while keeping the complete result below the 768 KiB provider-result bound.
|
|
const SEARCH_PAGE_BYTES = 640 * 1024;
|
|
const SCHEMA_CHUNK_CHARACTERS = 64 * 1024;
|
|
const ON_DEMAND_TOOLS: ToolDefinition[] = [
|
|
{
|
|
name: SEARCH_TOOL,
|
|
description: "Search your assigned app tools by name or description and read their input schemas. Pass {\"query\":\"search terms\"}, or an empty query to browse. Use limit from 1 to 20 (default 5). Omit unused optional fields instead of sending null. To fetch the next page, pass the returned nextOffset as offset. A large definition returns inputSchemaRef: set schemaTool to that name with an empty query, then pass the returned nextSchemaOffset as schemaOffset until null. Concatenate schemaJson chunks and parse JSON. Use paperclip_call_assigned_tool with a returned name and arguments matching its inputSchema.",
|
|
inputSchema: {
|
|
type: "object", additionalProperties: false,
|
|
properties: {
|
|
query: { type: "string", maxLength: 200 },
|
|
offset: { type: "integer", minimum: 0 },
|
|
limit: { type: "integer", minimum: 1, maximum: 20, default: 5, description: "Maximum results per page (1 to 20; default 5)." },
|
|
schemaTool: { type: "string", description: "Exact inputSchemaRef from a search result." },
|
|
schemaOffset: { type: "integer", minimum: 0 },
|
|
},
|
|
required: ["query"],
|
|
},
|
|
},
|
|
{
|
|
name: CALL_TOOL,
|
|
description: "Call an assigned app tool discovered with paperclip_search_assigned_tools. Use its exact returned name and arguments matching its inputSchema. The same permissions, approvals, and audit rules apply as for direct app tools.",
|
|
inputSchema: {
|
|
type: "object", additionalProperties: false,
|
|
properties: { name: { type: "string" }, arguments: { type: "object", additionalProperties: true } },
|
|
required: ["name", "arguments"],
|
|
},
|
|
},
|
|
];
|
|
|
|
function object(value: unknown): Record<string, unknown> {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
|
throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
}
|
|
return value as Record<string, unknown>;
|
|
}
|
|
|
|
function definition(name: string, tool: ToolGatewayDescriptor): ToolDefinition {
|
|
return { name, description: `${tool.displayName}: ${tool.description}`, inputSchema: structuredClone(tool.parametersSchema) };
|
|
}
|
|
|
|
// Execution must use the app's configured gateway, including deployment
|
|
// restrictions, OAuth refresh, and approval delivery. Never fall back to an
|
|
// isolated gateway whose defaults differ from the running instance.
|
|
const assignedMcpGateways = new WeakMap<Db, ToolGatewayService>();
|
|
|
|
export function registerAssignedMcpGateway(db: Db, gateway: ToolGatewayService): void {
|
|
assignedMcpGateways.set(db, gateway);
|
|
}
|
|
|
|
export function getAssignedMcpGateway(db: Db): ToolGatewayService {
|
|
const gateway = assignedMcpGateways.get(db);
|
|
if (!gateway) throw new Error("assigned_mcp_gateway_unavailable");
|
|
return gateway;
|
|
}
|
|
|
|
/** No configured gateway means no authorized connection tools or instructions. */
|
|
export async function resolveAssignedConnectionInstructionsForRun(db: Db, binding: { companyId: string; agentId: string; runId: string }) {
|
|
const gateway = assignedMcpGateways.get(db);
|
|
return gateway ? gateway.resolveConnectionInstructionsForRun(binding) : null;
|
|
}
|
|
|
|
function assignedToolName(name: string): string {
|
|
const sanitize = (value: string) => value.replace(/[^a-zA-Z0-9_]+/g, "_").replace(/^_+|_+$/g, "");
|
|
const separator = name.lastIndexOf(":");
|
|
const readable = separator >= 0
|
|
? `${sanitize(name.slice(0, separator)).slice(0, 20) || "tool"}_${sanitize(name.slice(separator + 1)).slice(-26) || "action"}`
|
|
: sanitize(name).slice(0, 47) || "tool";
|
|
const digest = createHash("sha256").update(name).digest("hex").slice(0, 12);
|
|
return `app_${readable}_${digest}`;
|
|
}
|
|
|
|
/** Server-owned MCP bindings projected into the runner's existing tool channel. */
|
|
export async function createAssignedMcpTools(input: {
|
|
gateway: ToolGatewayService;
|
|
gatewayPublicId: string;
|
|
bearerToken: string;
|
|
workMode?: WorkMode;
|
|
/** Bound provider grammar complexity independently of protocol byte capacity. */
|
|
delivery?: "on_demand";
|
|
}) {
|
|
const listed = await input.gateway.listToolsForNamedGateway({
|
|
gatewayPublicId: input.gatewayPublicId,
|
|
bearerToken: input.bearerToken,
|
|
});
|
|
const tools = new Map<string, ToolGatewayDescriptor>();
|
|
for (const descriptor of deduplicateToolDescriptors(listed)) {
|
|
const name = assignedToolName(descriptor.name);
|
|
const previous = tools.get(name);
|
|
if (previous) {
|
|
throw new ToolGatewayHttpError(409,
|
|
`Assigned MCP tool name collision for ${JSON.stringify([previous.name, descriptor.name])}.`,
|
|
"assigned_mcp_tool_name_collision");
|
|
}
|
|
tools.set(name, descriptor);
|
|
}
|
|
const permits = (tool: ToolGatewayDescriptor, mode: WorkMode = input.workMode ?? "standard") => mode === "standard" || tool.risk === "read";
|
|
// OpenCode's HTTP bridge returns at most 64K JSON characters. Leave room
|
|
// for the envelope and for JSON escaping when returning a schema chunk.
|
|
// Other harnesses retain the larger PRP page and chunk budgets.
|
|
const searchPageBytes = input.delivery === "on_demand" ? 32 * 1024 : SEARCH_PAGE_BYTES;
|
|
const schemaChunkCharacters = input.delivery === "on_demand" ? 16 * 1024 : SCHEMA_CHUNK_CHARACTERS;
|
|
|
|
async function search(argumentsValue: unknown, currentWorkMode?: WorkMode) {
|
|
const args = object(argumentsValue);
|
|
const offset = args.offset ?? 0;
|
|
const limit = args.limit ?? 5;
|
|
const schemaOffset = args.schemaOffset ?? 0;
|
|
if (typeof args.query !== "string" || args.query.length > 200 ||
|
|
typeof offset !== "number" || !Number.isSafeInteger(offset) || offset < 0 ||
|
|
typeof limit !== "number" || !Number.isSafeInteger(limit) || limit < 1 || limit > 20 ||
|
|
(args.schemaTool !== undefined && typeof args.schemaTool !== "string") ||
|
|
typeof schemaOffset !== "number" || !Number.isSafeInteger(schemaOffset) || schemaOffset < 0 ||
|
|
(args.schemaOffset !== undefined && args.schemaTool === undefined)) {
|
|
throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
}
|
|
// Intersect fresh grants with this session's pinned catalog. Revocations
|
|
// take effect immediately; newly assigned tools require a new session.
|
|
const current = new Map(deduplicateToolDescriptors(await input.gateway.listToolsForNamedGateway({
|
|
gatewayPublicId: input.gatewayPublicId, bearerToken: input.bearerToken,
|
|
})).map(tool => [tool.name, tool]));
|
|
const terms = args.query.toLowerCase().trim().split(/\s+/).filter(Boolean);
|
|
const authorized = [...tools].filter(([, tool]) => {
|
|
const fresh = current.get(tool.name);
|
|
return fresh && permits(tool) && permits(tool, currentWorkMode) &&
|
|
permits(fresh) && permits(fresh, currentWorkMode);
|
|
});
|
|
if (typeof args.schemaTool === "string") {
|
|
const selected = authorized.find(([name]) => name === args.schemaTool);
|
|
if (!selected) throw new Error("assigned_mcp_tool_unknown");
|
|
const schema = JSON.stringify(selected[1].parametersSchema);
|
|
const end = schemaOffset + schemaChunkCharacters;
|
|
return {
|
|
name: selected[0], schemaJson: schema.slice(schemaOffset, end),
|
|
nextSchemaOffset: end < schema.length ? end : null,
|
|
};
|
|
}
|
|
// Keep offsets anchored to the pinned catalog while fresh grants only
|
|
// decide which entries are returned from that stable sequence.
|
|
const authorizedNames = new Set(authorized.map(([name]) => name));
|
|
const matches = [...tools].filter(([, tool]) => permits(tool)).filter(([name, tool]) =>
|
|
terms.every(term => `${name} ${tool.displayName} ${tool.description}`.toLowerCase().includes(term)))
|
|
.sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0);
|
|
const page: ToolDefinition[] = [];
|
|
let nextOffset = offset;
|
|
while (nextOffset < matches.length && page.length < limit) {
|
|
const [name, tool] = matches[nextOffset]!;
|
|
if (!authorizedNames.has(name)) {
|
|
nextOffset += 1;
|
|
continue;
|
|
}
|
|
let next = definition(name, tool);
|
|
if (Buffer.byteLength(JSON.stringify([next]), "utf8") > searchPageBytes) {
|
|
// One large schema must not block browsing the tools after it. Expose
|
|
// a reference whose schema can be fetched in bounded, reauthorized
|
|
// chunks, without weakening the gateway's argument validation.
|
|
next = { name, description: String(next.description).slice(0, 2000), inputSchemaRef: name };
|
|
}
|
|
if (Buffer.byteLength(JSON.stringify([...page, next]), "utf8") > searchPageBytes) {
|
|
break;
|
|
}
|
|
page.push(next);
|
|
nextOffset += 1;
|
|
}
|
|
return { tools: page, nextOffset: nextOffset < matches.length ? nextOffset : null };
|
|
}
|
|
|
|
return {
|
|
definitions(fits?: (definitions: ToolDefinition[]) => boolean): ToolDefinition[] {
|
|
const direct = [...tools].filter(([, tool]) => permits(tool)).map(([name, tool]) => definition(name, tool));
|
|
if (input.delivery !== "on_demand" && (!fits || fits(direct))) return direct;
|
|
const compact = structuredClone(ON_DEMAND_TOOLS);
|
|
if (fits && !fits(compact)) throw new Error("assigned_mcp_tool_catalog_capacity_exceeded");
|
|
return compact;
|
|
},
|
|
has(name: string): boolean {
|
|
return tools.has(name) || name === SEARCH_TOOL || name === CALL_TOOL;
|
|
},
|
|
async execute(call: { tool: string; arguments: unknown }, currentWorkMode?: WorkMode): Promise<unknown> {
|
|
if (call.tool === SEARCH_TOOL) return search(call.arguments, currentWorkMode);
|
|
let name = call.tool;
|
|
let parameters = call.arguments;
|
|
if (name === CALL_TOOL) {
|
|
const args = object(parameters);
|
|
if (typeof args.name !== "string") throw new Error("assigned_mcp_tool_invalid_arguments");
|
|
name = args.name;
|
|
parameters = object(args.arguments);
|
|
}
|
|
const descriptor = tools.get(name);
|
|
if (!descriptor) throw new Error("assigned_mcp_tool_unknown");
|
|
if (!permits(descriptor) || !permits(descriptor, currentWorkMode)) throw new Error("paperclip_runner_tool_mode_denied");
|
|
// Reauthorize through the existing gateway on every call. Discovery is
|
|
// not a grant: revocation, policy, approval, and audit remain server-owned.
|
|
const result = await input.gateway.executeTool({
|
|
gatewayPublicId: input.gatewayPublicId,
|
|
sessionToken: input.bearerToken,
|
|
tool: descriptor.name,
|
|
parameters,
|
|
});
|
|
if (result.status !== "completed" && result.status !== "replayed") {
|
|
throw new Error("assigned_mcp_tool_execution_incomplete");
|
|
}
|
|
// The gateway normally throws provider errors. Keep a malformed or older
|
|
// provider response from being reported as successful through PRP.
|
|
if (result.result && typeof result.result === "object" && "isError" in result.result && result.result.isError === true) {
|
|
throw new ToolGatewayHttpError(502, "The assigned tool returned an error.", "tool_error");
|
|
}
|
|
return result.result;
|
|
},
|
|
};
|
|
}
|