mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 04:02:18 +01:00
## Thinking Path > - Paperclip manages agents and their execution environments. > - Plugins can vendor company skills and provide environments. > - Environment tools need their instructions before the provider starts. > - Persistent computers also need those skills before any run begins. > - This change adds environment requirements to normal skill selection and lifecycle synchronization. > - Plugin names prevent ambiguity, and required skills take delivery precedence without deleting company data. ## Linked Issues or Issue Description **Problem or motivation** An environment can provide desktop tools while its agent receives no instructions for those tools. A first-run-only import also leaves idle computers without required skills. Same-name company skills can make provider discovery ambiguous. **Proposed solution** Let plugin environment drivers declare their managed `skillKeys`. Install missing requirements during lifecycle reconciliation or new-run preparation. Deliver updates through the durable runtime event journal. Give managed imports stable plugin names and report shadowed assignments. **Alternatives considered** ROM links do not make a skill a normal Paperclip selection. Polling duplicates the existing lifecycle delivery queue. Replacing a same-name company record would destroy independent data and assignments. **Roadmap alignment** This extends the shipped Plugin system and Skills Manager. The existing related-PR search found no duplicate contract. ## What Changed - Validate `environmentDrivers[].skillKeys` as unique references to the same plugin's managed skills. Require a ready, enabled plugin with `skills.managed` authority. - Resolve company-scoped skills before version capture. Preserve explicit pins, company edits, and saved recovery selections. Serialize concurrent first installs. - Allow lifecycle callers to request missing-skill installation through `agents.getRuntime`. Ordinary task and recovery reads do not install new declarations. Pass task-selected environment IDs through the SDK. - Add SQL migration `0378_environment_skill_runtime_hooks`. Required skill edits, effective default changes, environment changes, and plugin requirement changes enqueue durable runtime events. Reuse lifecycle delivery and retries; add no polling. - Import managed defaults as `plugin-<short-plugin-id>-<skill-slug>`, including the SKILL.md name. Preserve canonical keys. Keep same-name company records separate without counter suffixes. - Required environment skills win delivery over same-name company assignments. Preserve saved assignments and library content. Record shadowed and winning keys in run warnings, run snapshots, and plugin runtime state. Removing the environment requirement restores the ordinary assignment. - Document the contract for `driver: "plugin"`. Other drivers and plugins without requirements keep existing behavior. Existing managed copies retain edits and names until reset. ## Verification - Latest targeted local checks: environment skills (5), managed skills (9), heartbeat execution configuration (27), and SDK worker RPC host (23): **65 tests passed** (including the targeted schema-drift check). Disposable PostgreSQL exercised the migration, first install before any run, concurrent lifecycle reads, implicit edit events, inherited defaults, rollback, plugin requirement changes, shadowing, preserved records, version capture, and recovery. - Review follow-up also verifies case-insensitive/runtime-name aliases, recovery of a task-mentioned shadowed skill, preserved quoted/multiline YAML, and zero runtime events for repeated local-folder settings writes. - Migration numbering and safety checks passed. Whitespace and secret/PII scans passed. - Earlier targeted manifest validation passed (43 tests). No validator behavior changed in this follow-up. - Broad tests, typecheck, build and end-to-end verification passed in [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/40387647219) for current head `ce57f2a4d`. Greptile is 5/5 with no unresolved threads. The [commitperclip review check](https://github.com/paperclipai/paperclip/actions/runs/40387646208) also passed after restoring its required duplicate-search checkbox. - Earlier attempts found a SQL alias collision, an unparenthesized CASE expression, and fixture assumptions about bundled catalog rows and YAML quoting. These were corrected before the passing checks. The first broad CI run also exposed a schema-drift check that assumed every journal entry had a schema snapshot; it now reads the latest actual snapshot, so SQL-only migrations need no invented snapshot. A test callback replacement also affected an unrelated fixture; it was restored before the final passing execution-configuration suite. Sandbox runs could not start PostgreSQL; the reported database tests used local socket access. ## Risks - Apply migration 0378 before using the updated lifecycle contract. It adds trigger functions and event producers; it does not rewrite agent assignments or skill content. - Required environment skills take precedence over same-name assignments, including task-selected skills. Both library records remain. Run warnings identify omissions. - Company edits are preserved. Plugin upgrades do not reset installed content or rename existing copies. Explicit reset adopts current defaults. - Namespaced library slugs can be shared by distinct canonical records. Callers must use IDs or canonical keys when a slug is ambiguous. - No live provider request or deployment was performed. The downstream plugin must be built against this Core revision. Persistent-home synchronization retains its existing edit-conflict and retry behavior. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The exact model identifier and context-window size are not exposed in this task context. ## Checklist - [x] Thinking path and issue description included - [x] Model and available capabilities recorded - [x] Roadmap and related PRs checked - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] Public references only; no internal issue IDs or instance URLs - [x] Targeted local tests pass; broad verification delegated to GitHub CI - [x] New behavior covered and documentation updated - [x] Current-head CI green and Greptile 5/5 with no unresolved comments --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
365 lines
12 KiB
TypeScript
365 lines
12 KiB
TypeScript
import { and, eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
pluginManagedResources,
|
|
} from "@paperclipai/db";
|
|
import { joinFrontmatterBlock, normalizeAgentUrlKey, splitFrontmatterBlock } from "@paperclipai/shared";
|
|
import type {
|
|
CompanySkill,
|
|
PaperclipPluginManifestV1,
|
|
PluginManagedSkillDeclaration,
|
|
PluginManagedSkillResolution,
|
|
} from "@paperclipai/shared";
|
|
import { notFound } from "../errors.js";
|
|
import { logActivity } from "./activity-log.js";
|
|
import { companySkillService } from "./company-skills.js";
|
|
|
|
const MANAGED_SKILL_RESOURCE_KIND = "skill";
|
|
|
|
interface PluginManagedSkillServiceOptions {
|
|
pluginId: string;
|
|
pluginKey: string;
|
|
manifest?: PaperclipPluginManifestV1 | null;
|
|
}
|
|
|
|
function pluginKeySlug(pluginKey: string) {
|
|
return normalizeAgentUrlKey(pluginKey) ?? "plugin";
|
|
}
|
|
|
|
function canonicalSkillKey(pluginKey: string, skillKey: string) {
|
|
return `plugin/${pluginKeySlug(pluginKey)}/${skillKey}`;
|
|
}
|
|
|
|
function managedSkillName(pluginKey: string, declaration: PluginManagedSkillDeclaration) {
|
|
const shortPlugin = pluginKeySlug(pluginKey.split(".").at(-1)!).replace(/^plugin-/, "");
|
|
return `plugin-${shortPlugin}-${normalizeAgentUrlKey(declaration.slug ?? declaration.skillKey) ?? declaration.skillKey}`;
|
|
}
|
|
|
|
function yamlString(value: string) {
|
|
return JSON.stringify(value);
|
|
}
|
|
|
|
function buildDefaultMarkdown(
|
|
pluginKey: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
) {
|
|
const description = declaration.description?.trim() || `${declaration.displayName} plugin skill.`;
|
|
return [
|
|
"---",
|
|
`name: ${yamlString(managedSkillName(pluginKey, declaration))}`,
|
|
`slug: ${yamlString(managedSkillName(pluginKey, declaration))}`,
|
|
`description: ${yamlString(description)}`,
|
|
`key: ${yamlString(canonicalSkillKey(pluginKey, declaration.skillKey))}`,
|
|
"---",
|
|
"",
|
|
`# ${declaration.displayName}`,
|
|
"",
|
|
description,
|
|
"",
|
|
].join("\n");
|
|
}
|
|
|
|
function withManagedSkillKey(markdown: string, canonicalKey: string, name: string) {
|
|
const block = splitFrontmatterBlock(markdown.replace(/\r\n/g, "\n"));
|
|
const fields: Record<string, string> = { key: canonicalKey, name, slug: name };
|
|
const seen = new Set<string>();
|
|
let continuation = false;
|
|
const lines = block.frontmatterText.split("\n").flatMap(line => {
|
|
const field = /^(key|name|slug|"(?:key|name|slug)"|'(?:key|name|slug)')\s*:/.exec(line);
|
|
if (field) {
|
|
const key = field[1].replace(/["']/g, "");
|
|
seen.add(key); continuation = true;
|
|
return [`${key}: ${yamlString(fields[key])}`];
|
|
}
|
|
if (continuation && /^(?:\s|$)/.test(line)) return [];
|
|
continuation = false;
|
|
return [line];
|
|
});
|
|
for (const [key, value] of Object.entries(fields)) if (!seen.has(key)) lines.push(`${key}: ${yamlString(value)}`);
|
|
return joinFrontmatterBlock({ ...block, hasFrontmatter: true, frontmatterText: lines.join("\n") });
|
|
}
|
|
|
|
function buildPackageFiles(
|
|
pluginKey: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
) {
|
|
const root = declaration.skillKey;
|
|
const canonicalKey = canonicalSkillKey(pluginKey, declaration.skillKey);
|
|
const files: Record<string, string> = {
|
|
[`${root}/SKILL.md`]: declaration.markdown?.trim()
|
|
? withManagedSkillKey(declaration.markdown, canonicalKey, managedSkillName(pluginKey, declaration))
|
|
: buildDefaultMarkdown(pluginKey, declaration),
|
|
};
|
|
for (const file of declaration.files ?? []) {
|
|
files[`${root}/${file.path}`] = file.content;
|
|
}
|
|
return files;
|
|
}
|
|
|
|
function buildDeclaredSkillFiles(
|
|
pluginKey: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
) {
|
|
const packageFiles = buildPackageFiles(pluginKey, declaration);
|
|
const root = declaration.skillKey;
|
|
const prefix = `${root}/`;
|
|
const files: Record<string, string> = {};
|
|
for (const [filePath, content] of Object.entries(packageFiles)) {
|
|
files[filePath.startsWith(prefix) ? filePath.slice(prefix.length) : filePath] = content;
|
|
}
|
|
return files;
|
|
}
|
|
|
|
function buildSkillDefaults(
|
|
pluginKey: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
) {
|
|
return {
|
|
skillKey: declaration.skillKey,
|
|
displayName: declaration.displayName,
|
|
slug: managedSkillName(pluginKey, declaration),
|
|
description: declaration.description ?? null,
|
|
canonicalKey: canonicalSkillKey(pluginKey, declaration.skillKey),
|
|
files: [
|
|
"SKILL.md",
|
|
...(declaration.files ?? []).map((file) => file.path),
|
|
],
|
|
};
|
|
}
|
|
|
|
function stableJson(value: unknown): string {
|
|
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
|
|
if (value && typeof value === "object") {
|
|
return `{${Object.entries(value as Record<string, unknown>)
|
|
.sort(([left], [right]) => left.localeCompare(right))
|
|
.map(([key, entry]) => `${JSON.stringify(key)}:${stableJson(entry)}`)
|
|
.join(",")}}`;
|
|
}
|
|
return JSON.stringify(value);
|
|
}
|
|
|
|
function resolution(
|
|
pluginKey: string,
|
|
companyId: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
skill: CompanySkill | null,
|
|
status: PluginManagedSkillResolution["status"],
|
|
defaultDrift: PluginManagedSkillResolution["defaultDrift"] = null,
|
|
): PluginManagedSkillResolution {
|
|
return {
|
|
pluginKey,
|
|
resourceKind: "skill",
|
|
resourceKey: declaration.skillKey,
|
|
companyId,
|
|
skillId: skill?.id ?? null,
|
|
skill,
|
|
status,
|
|
defaultDrift,
|
|
};
|
|
}
|
|
|
|
export function pluginManagedSkillService(
|
|
db: Db,
|
|
options: PluginManagedSkillServiceOptions,
|
|
) {
|
|
const skills = companySkillService(db);
|
|
|
|
function declarationFor(skillKey: string) {
|
|
const declaration = options.manifest?.skills?.find((skill) => skill.skillKey === skillKey);
|
|
if (!declaration) {
|
|
throw notFound(`Managed skill declaration not found: ${skillKey}`);
|
|
}
|
|
return declaration;
|
|
}
|
|
|
|
async function getBinding(companyId: string, skillKey: string) {
|
|
return db
|
|
.select()
|
|
.from(pluginManagedResources)
|
|
.where(and(
|
|
eq(pluginManagedResources.companyId, companyId),
|
|
eq(pluginManagedResources.pluginId, options.pluginId),
|
|
eq(pluginManagedResources.resourceKind, MANAGED_SKILL_RESOURCE_KIND),
|
|
eq(pluginManagedResources.resourceKey, skillKey),
|
|
))
|
|
.then((rows) => rows[0] ?? null);
|
|
}
|
|
|
|
async function upsertBinding(
|
|
companyId: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
skillId: string,
|
|
) {
|
|
const defaultsJson = buildSkillDefaults(options.pluginKey, declaration);
|
|
const existing = await getBinding(companyId, declaration.skillKey);
|
|
if (existing) {
|
|
if (
|
|
existing.resourceId === skillId &&
|
|
stableJson(existing.defaultsJson) === stableJson(defaultsJson)
|
|
) {
|
|
return existing;
|
|
}
|
|
return db
|
|
.update(pluginManagedResources)
|
|
.set({
|
|
resourceId: skillId,
|
|
defaultsJson,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(pluginManagedResources.id, existing.id))
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
}
|
|
return db
|
|
.insert(pluginManagedResources)
|
|
.values({
|
|
companyId,
|
|
pluginId: options.pluginId,
|
|
pluginKey: options.pluginKey,
|
|
resourceKind: MANAGED_SKILL_RESOURCE_KIND,
|
|
resourceKey: declaration.skillKey,
|
|
resourceId: skillId,
|
|
defaultsJson,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
}
|
|
|
|
async function getSkill(companyId: string, skillId: string) {
|
|
return skills.getById(companyId, skillId);
|
|
}
|
|
|
|
async function managedSkillDefaultDrift(
|
|
companyId: string,
|
|
skill: CompanySkill | null,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
): Promise<PluginManagedSkillResolution["defaultDrift"]> {
|
|
if (!skill) return null;
|
|
const declaredFiles = buildDeclaredSkillFiles(options.pluginKey, declaration);
|
|
const currentFiles: Record<string, string | null> = {};
|
|
const paths = new Set([
|
|
...Object.keys(declaredFiles),
|
|
...skill.fileInventory.map((entry) => entry.path),
|
|
]);
|
|
|
|
for (const filePath of paths) {
|
|
if (filePath === "SKILL.md") {
|
|
currentFiles[filePath] = skill.markdown;
|
|
continue;
|
|
}
|
|
try {
|
|
currentFiles[filePath] = (await skills.readFile(companyId, skill.id, filePath))?.content ?? null;
|
|
} catch {
|
|
currentFiles[filePath] = null;
|
|
}
|
|
}
|
|
|
|
const changedFiles = [...paths]
|
|
.filter((filePath) => (currentFiles[filePath] ?? null) !== (declaredFiles[filePath] ?? null))
|
|
.sort((left, right) => left.localeCompare(right));
|
|
return changedFiles.length > 0 ? { changedFiles } : null;
|
|
}
|
|
|
|
async function resolvedSkill(
|
|
companyId: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
skill: CompanySkill | null,
|
|
status: PluginManagedSkillResolution["status"],
|
|
) {
|
|
return resolution(
|
|
options.pluginKey,
|
|
companyId,
|
|
declaration,
|
|
skill,
|
|
status,
|
|
await managedSkillDefaultDrift(companyId, skill, declaration),
|
|
);
|
|
}
|
|
|
|
async function importDeclaredSkill(
|
|
companyId: string,
|
|
declaration: PluginManagedSkillDeclaration,
|
|
mode: "reconcile" | "reset",
|
|
) {
|
|
const beforeByKey = mode === "reconcile"
|
|
? await skills.getByKey(companyId, canonicalSkillKey(options.pluginKey, declaration.skillKey))
|
|
: null;
|
|
if (beforeByKey) {
|
|
await upsertBinding(companyId, declaration, beforeByKey.id);
|
|
return { skill: beforeByKey, status: "relinked" as const };
|
|
}
|
|
const results = await skills.importPackageFiles(
|
|
companyId,
|
|
buildPackageFiles(options.pluginKey, declaration),
|
|
{ onConflict: "replace", allowSlugCollision: true },
|
|
);
|
|
const imported = results.find((result) =>
|
|
result.skill.key === canonicalSkillKey(options.pluginKey, declaration.skillKey)
|
|
|| result.originalSlug === (declaration.slug ?? declaration.skillKey)
|
|
|| result.originalSlug === declaration.skillKey
|
|
)?.skill ?? results[0]?.skill ?? null;
|
|
if (!imported) {
|
|
throw notFound(`Managed skill was not imported: ${declaration.skillKey}`);
|
|
}
|
|
await upsertBinding(companyId, declaration, imported.id);
|
|
const status: PluginManagedSkillResolution["status"] = mode === "reset" ? "reset" : "created";
|
|
return { skill: imported, status };
|
|
}
|
|
|
|
async function get(skillKey: string, companyId: string) {
|
|
const declaration = declarationFor(skillKey);
|
|
const binding = await getBinding(companyId, skillKey);
|
|
if (!binding) return resolvedSkill(companyId, declaration, null, "missing");
|
|
const skill = await getSkill(companyId, binding.resourceId);
|
|
return resolvedSkill(companyId, declaration, skill, skill ? "resolved" : "missing");
|
|
}
|
|
|
|
async function reconcile(skillKey: string, companyId: string) {
|
|
const declaration = declarationFor(skillKey);
|
|
const current = await get(skillKey, companyId);
|
|
if (current.skill) {
|
|
await upsertBinding(companyId, declaration, current.skill.id);
|
|
return current;
|
|
}
|
|
const imported = await importDeclaredSkill(companyId, declaration, "reconcile");
|
|
await logActivity(db, {
|
|
companyId,
|
|
actorType: "plugin",
|
|
actorId: options.pluginId,
|
|
action: "plugin.managed_skill.reconciled",
|
|
entityType: "company_skill",
|
|
entityId: imported.skill.id,
|
|
details: {
|
|
sourcePluginKey: options.pluginKey,
|
|
managedResourceKey: declaration.skillKey,
|
|
status: imported.status,
|
|
},
|
|
});
|
|
return resolvedSkill(companyId, declaration, imported.skill, imported.status);
|
|
}
|
|
|
|
async function reset(skillKey: string, companyId: string) {
|
|
const declaration = declarationFor(skillKey);
|
|
const imported = await importDeclaredSkill(companyId, declaration, "reset");
|
|
await logActivity(db, {
|
|
companyId,
|
|
actorType: "plugin",
|
|
actorId: options.pluginId,
|
|
action: "plugin.managed_skill.reset",
|
|
entityType: "company_skill",
|
|
entityId: imported.skill.id,
|
|
details: {
|
|
sourcePluginKey: options.pluginKey,
|
|
managedResourceKey: declaration.skillKey,
|
|
},
|
|
});
|
|
return resolvedSkill(companyId, declaration, imported.skill, "reset");
|
|
}
|
|
|
|
return {
|
|
get,
|
|
reconcile,
|
|
reset,
|
|
};
|
|
}
|