Files
PaperClipAI/packages/shared/src/validators/plugin.test.ts
T
Nicky LeachandPaperclip 4ff493fb40 feat(plugins): include managed skills for selected environments (#17774)
## Thinking Path

> - Paperclip manages agents and their execution environments.
> - Plugins can vendor company skills and provide environments.
> - Environment tools need their instructions before the provider
starts.
> - Persistent computers also need those skills before any run begins.
> - This change adds environment requirements to normal skill selection
and lifecycle synchronization.
> - Plugin names prevent ambiguity, and required skills take delivery
precedence without deleting company data.

## Linked Issues or Issue Description

**Problem or motivation**

An environment can provide desktop tools while its agent receives no
instructions for those tools. A first-run-only import also leaves idle
computers without required skills. Same-name company skills can make
provider discovery ambiguous.

**Proposed solution**

Let plugin environment drivers declare their managed `skillKeys`.
Install missing requirements during lifecycle reconciliation or new-run
preparation. Deliver updates through the durable runtime event journal.
Give managed imports stable plugin names and report shadowed
assignments.

**Alternatives considered**

ROM links do not make a skill a normal Paperclip selection. Polling
duplicates the existing lifecycle delivery queue. Replacing a same-name
company record would destroy independent data and assignments.

**Roadmap alignment**

This extends the shipped Plugin system and Skills Manager. The existing
related-PR search found no duplicate contract.

## What Changed

- Validate `environmentDrivers[].skillKeys` as unique references to the
same plugin's managed skills. Require a ready, enabled plugin with
`skills.managed` authority.
- Resolve company-scoped skills before version capture. Preserve
explicit pins, company edits, and saved recovery selections. Serialize
concurrent first installs.
- Allow lifecycle callers to request missing-skill installation through
`agents.getRuntime`. Ordinary task and recovery reads do not install new
declarations. Pass task-selected environment IDs through the SDK.
- Add SQL migration `0378_environment_skill_runtime_hooks`. Required
skill edits, effective default changes, environment changes, and plugin
requirement changes enqueue durable runtime events. Reuse lifecycle
delivery and retries; add no polling.
- Import managed defaults as `plugin-<short-plugin-id>-<skill-slug>`,
including the SKILL.md name. Preserve canonical keys. Keep same-name
company records separate without counter suffixes.
- Required environment skills win delivery over same-name company
assignments. Preserve saved assignments and library content. Record
shadowed and winning keys in run warnings, run snapshots, and plugin
runtime state. Removing the environment requirement restores the
ordinary assignment.
- Document the contract for `driver: "plugin"`. Other drivers and
plugins without requirements keep existing behavior. Existing managed
copies retain edits and names until reset.

## Verification

- Latest targeted local checks: environment skills (5), managed skills
(9), heartbeat execution configuration (27), and SDK worker RPC host
(23): **65 tests passed** (including the targeted schema-drift check).
Disposable PostgreSQL exercised the migration, first install before any
run, concurrent lifecycle reads, implicit edit events, inherited
defaults, rollback, plugin requirement changes, shadowing, preserved
records, version capture, and recovery.
- Review follow-up also verifies case-insensitive/runtime-name aliases,
recovery of a task-mentioned shadowed skill, preserved quoted/multiline
YAML, and zero runtime events for repeated local-folder settings writes.
- Migration numbering and safety checks passed. Whitespace and
secret/PII scans passed.
- Earlier targeted manifest validation passed (43 tests). No validator
behavior changed in this follow-up.
- Broad tests, typecheck, build and end-to-end verification passed in
[GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/40387647219)
for current head `ce57f2a4d`. Greptile is 5/5 with no unresolved
threads. The [commitperclip review
check](https://github.com/paperclipai/paperclip/actions/runs/40387646208)
also passed after restoring its required duplicate-search checkbox.
- Earlier attempts found a SQL alias collision, an unparenthesized CASE
expression, and fixture assumptions about bundled catalog rows and YAML
quoting. These were corrected before the passing checks. The first broad
CI run also exposed a schema-drift check that assumed every journal
entry had a schema snapshot; it now reads the latest actual snapshot, so
SQL-only migrations need no invented snapshot. A test callback
replacement also affected an unrelated fixture; it was restored before
the final passing execution-configuration suite. Sandbox runs could not
start PostgreSQL; the reported database tests used local socket access.

## Risks

- Apply migration 0378 before using the updated lifecycle contract. It
adds trigger functions and event producers; it does not rewrite agent
assignments or skill content.
- Required environment skills take precedence over same-name
assignments, including task-selected skills. Both library records
remain. Run warnings identify omissions.
- Company edits are preserved. Plugin upgrades do not reset installed
content or rename existing copies. Explicit reset adopts current
defaults.
- Namespaced library slugs can be shared by distinct canonical records.
Callers must use IDs or canonical keys when a slug is ambiguous.
- No live provider request or deployment was performed. The downstream
plugin must be built against this Core revision. Persistent-home
synchronization retains its existing edit-conflict and retry behavior.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code
execution. The exact model identifier and context-window size are not
exposed in this task context.

## Checklist

- [x] Thinking path and issue description included
- [x] Model and available capabilities recorded
- [x] Roadmap and related PRs checked
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] Public references only; no internal issue IDs or instance URLs
- [x] Targeted local tests pass; broad verification delegated to GitHub
CI
- [x] New behavior covered and documentation updated
- [x] Current-head CI green and Greptile 5/5 with no unresolved comments

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 14:11:35 -07:00

493 lines
18 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { PLUGIN_CAPABILITIES } from "../constants.js";
import { resolveDeclaredSandboxCapabilities } from "../environment-support.js";
import { pluginApiRouteDeclarationSchema, pluginManagedRoutineDeclarationSchema, pluginManifestV1Schema, pluginUiSlotDeclarationSchema } from "./plugin.js";
function buildSandboxProviderManifest(driver: Record<string, unknown>) {
return {
id: "paperclip.capability-provider",
apiVersion: 1,
version: "0.1.0",
displayName: "Capability Provider",
description: "Sandbox provider that declares fine-grained capabilities.",
author: "Paperclip",
categories: ["automation"],
capabilities: ["environment.drivers.register"],
entrypoints: { worker: "./dist/worker.js" },
environmentDrivers: [
{
driverKey: "capability-provider",
kind: "sandbox_provider",
displayName: "Capability Provider",
configSchema: { type: "object" },
...driver,
},
],
};
}
describe("plugin capability constants", () => {
it("exposes each capability once", () => {
expect(new Set(PLUGIN_CAPABILITIES).size).toBe(PLUGIN_CAPABILITIES.length);
});
});
describe("plugin manifest validators", () => {
it("limits environment skills to unique declarations owned by the plugin", () => {
const manifest = { ...buildSandboxProviderManifest({ skillKeys: ["desktop"] }),
capabilities: ["environment.drivers.register", "skills.managed"],
skills: [{ skillKey: "desktop", displayName: "Desktop" }] };
expect(pluginManifestV1Schema.parse(manifest).environmentDrivers![0].skillKeys).toEqual(["desktop"]);
for (const keys of [["missing"], ["desktop", "desktop"]]) {
expect(pluginManifestV1Schema.safeParse({ ...manifest,
environmentDrivers: [{ ...manifest.environmentDrivers[0], skillKeys: keys }] }).success).toBe(false);
}
expect(pluginManifestV1Schema.safeParse({ ...manifest, capabilities: ["environment.drivers.register"] }).success).toBe(false);
});
it("requires authority for required agent lifecycle participants", () => {
const manifest = { id: "example.lifecycle", apiVersion: 1, version: "0.1.0", displayName: "Lifecycle", description: "Lifecycle tests", author: "Tests", categories: ["automation"], entrypoints: { worker: "worker.js" }, agentLifecycle: true };
expect(pluginManifestV1Schema.safeParse({ ...manifest, capabilities: ["agents.read"] }).success).toBe(false);
expect(pluginManifestV1Schema.parse({ ...manifest, capabilities: ["agents.lifecycle.manage"] }).agentLifecycle).toBe(true);
});
it("requires subscription authority and known resource types for durable delivery", () => {
const manifest = { id: "example.lifecycle", apiVersion: 1, version: "1.0.0", displayName: "Lifecycle", description: "Lifecycle tests", author: "Tests", categories: ["automation"], entrypoints: { worker: "worker.js" }, resourceLifecycle: ["project"] };
expect(pluginManifestV1Schema.safeParse({ ...manifest, capabilities: ["projects.read"] }).success).toBe(false);
expect(pluginManifestV1Schema.parse({ ...manifest, capabilities: ["events.subscribe"] }).resourceLifecycle).toEqual(["project"]);
for (const resourceLifecycle of [[], ["issue"]]) {
expect(pluginManifestV1Schema.safeParse({ ...manifest, resourceLifecycle, capabilities: ["events.subscribe"] }).success).toBe(false);
}
});
it("requires routing authority for native pooled connector declarations without a custom UI bundle", () => {
const manifest = { id: "example.pool", apiVersion: 1, version: "0.1.0", displayName: "Pool", description: "Pool", author: "Tests", categories: ["connector"], entrypoints: { worker: "worker.js" }, aiConnectionRouter: { name: "AI connection pool", description: "Use saved connections" } };
expect(pluginManifestV1Schema.safeParse({ ...manifest, capabilities: ["ui.page.register"] }).success).toBe(false);
expect(pluginManifestV1Schema.parse({ ...manifest, capabilities: ["ai.connections.route"] }).aiConnectionRouter).toEqual(manifest.aiConnectionRouter);
});
it("accepts existing-style plugins that do not request access or authorization capabilities", () => {
const parsed = pluginManifestV1Schema.parse({
id: "paperclip.compat-dashboard",
apiVersion: 1,
version: "0.1.0",
displayName: "Compat Dashboard",
description: "Dashboard-only plugin without access or authorization host APIs.",
author: "Paperclip",
categories: ["ui"],
capabilities: ["ui.dashboardWidget.register"],
entrypoints: {
worker: "./dist/worker.js",
ui: "./dist/ui.js",
},
ui: {
slots: [
{
type: "dashboardWidget",
id: "compat-dashboard",
displayName: "Compat Dashboard",
exportName: "CompatDashboard",
},
],
},
});
expect(parsed.capabilities).toEqual(["ui.dashboardWidget.register"]);
});
it("accepts sandbox provider template config bindings", () => {
const parsed = pluginManifestV1Schema.parse({
id: "paperclip.template-provider",
apiVersion: 1,
version: "0.1.0",
displayName: "Template Provider",
description: "Sandbox provider with captured template config binding.",
author: "Paperclip",
categories: ["automation"],
capabilities: ["environment.drivers.register"],
entrypoints: { worker: "./dist/worker.js" },
environmentDrivers: [
{
driverKey: "template-provider",
kind: "sandbox_provider",
displayName: "Template Provider",
supportsTemplateCapture: true,
templateRefKind: "provider_template",
templateConfigBinding: {
field: "templateId",
unsetFields: ["image"],
},
configSchema: { type: "object" },
},
],
});
expect(parsed.environmentDrivers?.[0]?.templateConfigBinding).toEqual({
field: "templateId",
unsetFields: ["image"],
});
});
it("rejects template config bindings that replace provider identity", () => {
const parsed = pluginManifestV1Schema.safeParse({
id: "paperclip.bad-template-provider",
apiVersion: 1,
version: "0.1.0",
displayName: "Bad Template Provider",
categories: ["automation"],
capabilities: ["environment.drivers.register"],
entrypoints: { worker: "./dist/worker.js" },
environmentDrivers: [
{
driverKey: "bad-template-provider",
kind: "sandbox_provider",
displayName: "Bad Template Provider",
templateConfigBinding: {
field: "provider",
},
configSchema: { type: "object" },
},
],
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues.some((issue) => issue.message.includes("provider key"))).toBe(true);
});
});
describe("plugin managed routine validators", () => {
it("accepts core issue surface visibility values in routine templates", () => {
const parsed = pluginManagedRoutineDeclarationSchema.parse({
routineKey: "wiki.refresh",
title: "Refresh Wiki",
activityGatePolicy: "require_external_activity",
activityGateScope: "project",
issueTemplate: { surfaceVisibility: "default" },
});
expect(parsed.issueTemplate?.surfaceVisibility).toBe("default");
expect(parsed.activityGatePolicy).toBe("require_external_activity");
expect(parsed.activityGateScope).toBe("project");
});
it("rejects non-core issue surface visibility values in routine templates", () => {
const parsed = pluginManagedRoutineDeclarationSchema.safeParse({
routineKey: "wiki.refresh",
title: "Refresh Wiki",
issueTemplate: { surfaceVisibility: "normal" },
});
expect(parsed.success).toBe(false);
});
});
describe("plugin managed skill validators", () => {
const baseManifest = {
id: "paperclip.test-managed-skills",
apiVersion: 1,
version: "0.1.0",
displayName: "Managed Skills",
description: "Managed skills test plugin.",
author: "Paperclip",
categories: ["automation"],
entrypoints: { worker: "./dist/worker.js" },
} as const;
it("requires skills.managed when managed skills are declared", () => {
const parsed = pluginManifestV1Schema.safeParse({
...baseManifest,
capabilities: [],
skills: [{ skillKey: "wiki-maintainer", displayName: "Wiki Maintainer" }],
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues.some((issue) => issue.message.includes("skills.managed"))).toBe(true);
});
it("accepts managed skills with the skills.managed capability", () => {
const parsed = pluginManifestV1Schema.parse({
...baseManifest,
capabilities: ["skills.managed"],
skills: [{ skillKey: "wiki-maintainer", displayName: "Wiki Maintainer" }],
});
expect(parsed.skills?.[0]?.skillKey).toBe("wiki-maintainer");
});
});
describe("plugin UI slot validators", () => {
it("accepts route-scoped sidebar slots with a routePath", () => {
const parsed = pluginUiSlotDeclarationSchema.parse({
type: "routeSidebar",
id: "wiki-route-sidebar",
displayName: "Wiki Sidebar",
exportName: "WikiSidebar",
routePath: "wiki",
});
expect(parsed.routePath).toBe("wiki");
});
it("requires route-scoped sidebar slots to declare a routePath", () => {
const parsed = pluginUiSlotDeclarationSchema.safeParse({
type: "routeSidebar",
id: "wiki-route-sidebar",
displayName: "Wiki Sidebar",
exportName: "WikiSidebar",
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues[0]?.message).toBe("routeSidebar slots require routePath");
});
it("keeps reserved company route protection for route-scoped sidebars", () => {
const parsed = pluginUiSlotDeclarationSchema.safeParse({
type: "routeSidebar",
id: "settings-route-sidebar",
displayName: "Settings Sidebar",
exportName: "SettingsSidebar",
routePath: "settings",
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues.some((issue) => issue.message.includes("reserved by the host"))).toBe(true);
});
it("accepts workspace entity types as detailTab targets", () => {
const parsed = pluginUiSlotDeclarationSchema.parse({
type: "detailTab",
id: "workspace-diff-viewer",
displayName: "Diff",
exportName: "WorkspaceDiffViewer",
entityTypes: ["execution_workspace", "project_workspace"],
});
expect(parsed.entityTypes).toEqual(["execution_workspace", "project_workspace"]);
});
it("accepts execution_workspace as a toolbarButton entityType", () => {
const parsed = pluginUiSlotDeclarationSchema.parse({
type: "toolbarButton",
id: "workspace-open-diff",
displayName: "Open diff",
exportName: "OpenWorkspaceDiffButton",
entityTypes: ["execution_workspace"],
});
expect(parsed.entityTypes).toEqual(["execution_workspace"]);
});
it("accepts company settings page slots with a non-core settings route", () => {
const parsed = pluginUiSlotDeclarationSchema.parse({
type: "companySettingsPage",
id: "permissions-settings",
displayName: "Permissions",
exportName: "PermissionsSettingsPage",
routePath: "permissions",
});
expect(parsed.routePath).toBe("permissions");
});
it("prevents company settings page slots from shadowing core settings routes", () => {
const parsed = pluginUiSlotDeclarationSchema.safeParse({
type: "companySettingsPage",
id: "instance-settings",
displayName: "Instance",
exportName: "InstanceSettingsPage",
routePath: "instance",
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues.some((issue) => issue.message.includes("reserved by the host"))).toBe(true);
});
});
describe("sandbox provider capability declaration validators", () => {
it("preserves a declared acquisition budget and keeps it optional", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({ defaultAcquireTimeoutMs: 300_000 }),
);
expect(parsed.environmentDrivers?.[0]?.defaultAcquireTimeoutMs).toBe(300_000);
const legacy = pluginManifestV1Schema.parse(buildSandboxProviderManifest({}));
expect(legacy.environmentDrivers?.[0]?.defaultAcquireTimeoutMs).toBeUndefined();
});
it.each([0, -1, 1.5, Infinity, NaN, "300000", 86_400_001])(
"rejects an invalid acquisition budget: %s",
(defaultAcquireTimeoutMs) => {
expect(pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({ defaultAcquireTimeoutMs }),
).success).toBe(false);
},
);
it("test_manifest_accepts_sandbox_capabilities_and_rejects_unknown_capability_keys", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({
sandboxCapabilities: {
reusableLeases: true,
nativeSyncIn: true,
nativeSyncOut: false,
persistentProcessSessions: true,
independentControlCommands: false,
incrementalSessionOutput: true,
},
}),
);
expect(parsed.environmentDrivers?.[0]?.sandboxCapabilities).toEqual({
reusableLeases: true,
nativeSyncIn: true,
nativeSyncOut: false,
persistentProcessSessions: true,
independentControlCommands: false,
incrementalSessionOutput: true,
});
const rejected = pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({
sandboxCapabilities: {
reusableLeases: true,
// A typo or unknown capability name must fail validation, not drop
// silently. The nested schema is `.strict()`.
nativeSync: true,
},
}),
);
expect(rejected.success).toBe(false);
});
it("test_manifest_accepts_concurrent_sync_operations_capability", () => {
// A provider opts in to parallel bidirectional file sync with this key. The
// strict schema accepts it and keeps the declared value.
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({
sandboxCapabilities: { concurrentSyncOperations: true },
}),
);
expect(parsed.environmentDrivers?.[0]?.sandboxCapabilities).toEqual({
concurrentSyncOperations: true,
});
});
it("test_manifest_rejects_unknown_sync_concurrency_capability_key", () => {
// A neighboring but unknown concurrency key must fail validation, not drop
// silently. The strict schema rejects a capability the host does not honor.
const rejected = pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({
sandboxCapabilities: { concurrentSyncAndExec: true },
}),
);
expect(rejected.success).toBe(false);
});
it("test_supports_reusable_leases_compat_maps_to_reusable_leases", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({ supportsReusableLeases: true }),
);
const driver = parsed.environmentDrivers?.[0];
expect(driver?.sandboxCapabilities).toBeUndefined();
expect(resolveDeclaredSandboxCapabilities(driver!).reusableLeases).toBe(true);
});
it("test_sandbox_capabilities_reusable_leases_wins_over_compat_field", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({
supportsReusableLeases: true,
sandboxCapabilities: { reusableLeases: false },
}),
);
const driver = parsed.environmentDrivers?.[0];
// The nested declaration wins over the legacy compat flag when both exist.
expect(resolveDeclaredSandboxCapabilities(driver!).reusableLeases).toBe(false);
});
});
describe("login pty transport capability and legacy alias", () => {
it("test_login_pty_new_field_parses_and_carries_the_flag", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({ supportsLoginPty: true }),
);
expect(parsed.environmentDrivers?.[0]?.supportsLoginPty).toBe(true);
});
it("test_legacy_alias_only_canonicalizes_onto_login_pty", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({ supportsSetupTokenLogin: true }),
);
const driver = parsed.environmentDrivers?.[0];
// The validator maps the deprecated alias onto the canonical field at parse
// time, and it drops the alias so a downstream reader cannot read the old
// name.
expect(driver?.supportsLoginPty).toBe(true);
expect(driver).not.toHaveProperty("supportsSetupTokenLogin");
});
it("test_conflicting_alias_and_new_field_reject", () => {
const rejected = pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({
supportsLoginPty: true,
supportsSetupTokenLogin: false,
}),
);
expect(rejected.success).toBe(false);
});
it("test_matching_alias_and_new_field_pass", () => {
const parsed = pluginManifestV1Schema.parse(
buildSandboxProviderManifest({
supportsLoginPty: true,
supportsSetupTokenLogin: true,
}),
);
expect(parsed.environmentDrivers?.[0]?.supportsLoginPty).toBe(true);
});
it("test_unknown_login_field_misspelling_rejects", () => {
const rejected = pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({ supportsLoginPTY: true }),
);
expect(rejected.success).toBe(false);
});
it("test_login_pty_accepts_literal_booleans_only", () => {
const rejected = pluginManifestV1Schema.safeParse(
buildSandboxProviderManifest({ supportsLoginPty: "true" }),
);
expect(rejected.success).toBe(false);
});
});
describe("plugin WebSocket company resolution", () => {
const route = { routeKey: "channel.connect", method: "GET", path: "/channels/:channelId", auth: "board", capability: "api.routes.register" };
it("rejects body resolution only for WebSocket routes", () => {
const companyResolution = { from: "body", key: "companyId" };
expect(pluginApiRouteDeclarationSchema.safeParse({ ...route, companyResolution }).success).toBe(true);
expect(pluginApiRouteDeclarationSchema.safeParse({ ...route, companyResolution, transport: "websocket" }).success).toBe(false);
});
it.each([
undefined,
{ from: "query", key: "companyId" },
{ from: "issue", param: "issueId" },
{ from: "agent", param: "agentId" },
])("accepts body-free resolution %j", (companyResolution) => {
expect(pluginApiRouteDeclarationSchema.safeParse({ ...route, companyResolution, transport: "websocket" }).success).toBe(true);
});
});