Files
PaperClipAI/server/src/services/distribution-plugin-catalog.ts
T
Devin FoleyandPaperclip b70641f23f feat(plugins): support image catalogs and persistent application overlays (#13646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Plugins extend the application without adding each integration to
Core.
> - A downstream image needs a way to supply prebuilt plugins.
> - Some plugin UI must stay mounted as users move between pages.
> - This change adds an image catalog and a persistent application slot.
> - Operators can upgrade or remove these plugins through their image
and configuration.

## Linked Issues or Issue Description

**Subsystem affected**

Plugin packaging, activation and application UI.

**Problem or motivation**

The built-in plugin catalog is fixed in Core source. Downstream images
cannot add entries through an explicit catalog. Existing page slots also
cannot preserve a small application overlay across route changes.

**Proposed solution**

Read a bounded catalog of prebuilt plugins from the image. Verify its
files before importing manifests. Use the existing managed selection and
plugin lifecycle. Add an `appShellOverlay` slot with account and company
cleanup.

**Alternatives considered**

A downstream fork adds merge work. Script injection provides no plugin
lifecycle. A separate runtime download system adds a second distribution
channel.

**Roadmap alignment**

This extends the existing plugin system. Related PR #9006 covers runtime
install replication; this change covers immutable image contents. PR
#12555 covers CLI scaffolding. Neither provides this catalog or
application slot. The maintainer requested this work directly.

## What Changed

- Validate catalog identities, confined paths, package versions and
bundle hashes before importing code.
- Apply image selection to persisted plugin installs, including removal
and rollback. Adopt the verified image path from existing npm/local
installs and bind runtime worker/UI entrypoints to verified package
declarations.
- Mount application overlays in both UI shells. Preserve route state and
clear it on account, company and onboarding changes.
- Restrict service-worker offline storage/fallback to hashed public
assets in a separate cache namespace; exclude application HTML and
extension/API data, including after worker restart.
- Document the packaging contract, trust model and rollback
requirements.

## Verification

- Passed `pnpm -r typecheck`, `pnpm build`, and `pnpm
check:token-gates`. Affected server/UI typechecks and builds, plus token
gates, passed again after rebasing onto current master; the 124 focused
tests also passed after rebase.
- Latest focused verification: 124 tests in nine files passed for
catalog/reconciliation/loader, overlay lifecycle, Layout and
service-worker policy. The broader UI/shared/SDK run passed 7,204 tests
in 690 files with canonical `TMPDIR`.
- Real disposable Core/PostgreSQL: catalog install, selection removal,
0.1.0→0.1.1→0.1.0, same-version npm/legacy-path adoption, and
preservation of disabled status passed. Added permissions entered
`upgrade_pending`, withheld UI across restart, and activated only after
explicit operator enable.
- Real Chromium: desktop/mobile layout, route draft retention and
Escape/focus passed with mocked extension responses. A persistent
browser restart retained public hashed-asset offline fallback while
refusing seeded legacy/current private entries and legacy HTML.
- Full `pnpm test:run`: 12,539 passed; 17 failed across six existing
files, stopping later phases. macOS read-only directory renames fail in
runtime-skill-cache and company-skills-service; email tests require an
absent local AgentMail fixture. Native runner/comment-redaction passed
in isolation after temporary Rust setup; agent-conversations also passed
in isolation. No unrelated source was changed to hide failures.
- After rebase, two unchanged chat timing tests failed in CI and passed
locally in isolation. Their CI shard passed on its single retry. All
other current-head CI jobs passed on the initial run; review is 5/5 with
no unresolved threads.
- No live deployment or external plugin service was used.

## Risks

- Plugins are trusted code. The catalog detects packaging errors; it
does not authenticate an untrusted image builder.
- Invalid catalogs fail startup. Images must contain the catalog and
bundles together, with stable directories.
- A host older than this contract lacks the activation guard. Disable
added plugins and remove their configuration keys before reverting to
it.
- Offline navigation now returns 503 instead of replaying cached
application HTML. Only public build assets have offline fallback.
- Rolling back an unapproved permission change retains the approval
gate; review the current manifest and explicitly enable it. A reduced
permission set cannot establish prior approval or prior enabled status.
- Plugin data migrations need their own rollback policy. This change
retains installed records and does not reverse migrations.

## Model Used

- OpenAI GPT-6 (Codex), model ID `gpt-6`, with repository inspection,
code execution and browser verification. The runtime does not expose an
exact context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (relevant suites; broad
macOS server-run exceptions are documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (fresh run on 488b3754ae; chat
shard passed its single retry)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(fresh review on 488b3754ae)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-19 09:18:10 -07:00

166 lines
9.0 KiB
TypeScript

import fs from "node:fs";
import path from "node:path";
import { createHash } from "node:crypto";
import { z } from "zod";
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
import { pluginCapabilityValidator } from "./plugin-capability-validator.js";
const segment = z.string().regex(/^[a-z][a-z0-9.-]{0,99}$/);
export const distributionPluginCatalogSchema = z.object({
schemaVersion: z.literal(1),
plugins: z.array(z.object({
key: segment,
pluginKey: segment,
version: z.string().regex(/^\d+\.\d+\.\d+(?:-[a-zA-Z0-9.-]+)?(?:\+[a-zA-Z0-9.-]+)?$/),
directory: segment,
digest: z.string().regex(/^sha256:[a-f0-9]{64}$/),
}).strict()).max(100),
}).strict();
export type DistributionPlugin = z.infer<typeof distributionPluginCatalogSchema>["plugins"][number] & {
localPath: string;
/** Normalized paths from the digest-verified package metadata. */
entrypoints: { worker: string; ui?: string };
};
function bundleEntrypoint(declared: unknown): string {
const relative = typeof declared === "string" ? declared.replace(/^\.\//, "").replace(/\/$/, "") : "";
if (!relative || path.posix.isAbsolute(relative) || path.win32.isAbsolute(relative) || relative.includes("\\") || relative.split("/").some((part) => !part || part === "." || part === "..")) {
throw new Error("Distribution entrypoint must stay inside its bundle");
}
return relative;
}
export function assertDistributionManifestCapabilities(manifest: PaperclipPluginManifestV1): void {
const result = pluginCapabilityValidator().validateManifestCapabilities(manifest);
if (!result.allowed) {
throw new Error(`Distribution manifest is missing required capabilities: ${result.missing.join(", ")}`);
}
}
export function distributionPluginsRoot(catalogRoot: string): string {
// Match canonical paths persisted by local-path installs (for example,
// macOS /tmp -> /private/tmp), without permitting a symlinked catalog itself.
try { return path.join(fs.realpathSync(catalogRoot), "distribution"); }
catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
return path.resolve(catalogRoot, "distribution");
}
}
/** Guard all activation paths, including persisted installs after a rollback. */
export function distributionPluginActivationGuard(
catalogRoot: string,
entries: readonly DistributionPlugin[],
selectedKeys: readonly string[] | null,
) {
const root = distributionPluginsRoot(catalogRoot);
return (input: { pluginKey?: string; packageRoot: string; installedPackagePath?: string | null; manifest?: PaperclipPluginManifestV1; previousManifest?: PaperclipPluginManifestV1 }) => {
let packageRoot: string;
try { packageRoot = fs.realpathSync(input.packageRoot); }
catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
packageRoot = path.resolve(input.packageRoot);
}
const entry = entries.find((candidate) => input.pluginKey ? candidate.pluginKey === input.pluginKey : candidate.localPath === packageRoot);
const relative = path.relative(root, packageRoot);
const inside = relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
// A missing image directory can make package resolution fall back to npm.
// Retain the persisted image-path provenance even when that directory no
// longer exists; the resolved fallback is not an ordinary installation.
const installedRelative = input.installedPackagePath ? path.relative(root, path.resolve(input.installedPackagePath)) : null;
const installedInside = installedRelative !== null && (installedRelative === "" || (!installedRelative.startsWith("..") && !path.isAbsolute(installedRelative)));
if (!inside && !installedInside && !entry) return;
if (!entry || (selectedKeys !== null && !selectedKeys.includes(entry.key)) || packageRoot !== entry.localPath) {
throw new Error("Distribution plugin is absent or not selected in this deployment");
}
if (input.manifest && (input.manifest.id !== entry.pluginKey || input.manifest.version !== entry.version)) {
throw new Error("Distribution manifest does not match its catalog identity/version");
}
if (input.manifest) {
assertDistributionManifestCapabilities(input.manifest);
if (input.previousManifest) {
const approved = new Set(input.previousManifest.capabilities);
const added = input.manifest.capabilities.filter((capability) => !approved.has(capability));
if (added.length) throw new Error(`Distribution plugin capabilities require approval: ${added.join(", ")}`);
}
const worker = bundleEntrypoint(input.manifest.entrypoints.worker);
const ui = input.manifest.entrypoints.ui === undefined ? undefined : bundleEntrypoint(input.manifest.entrypoints.ui);
if (worker !== entry.entrypoints.worker || ui !== entry.entrypoints.ui) {
throw new Error("Distribution manifest entrypoints do not match the verified package");
}
}
};
}
/** Same portable file inventory used by image builders: path, SHA-256, mode. */
export function distributionBundleDigest(root: string): string {
const inventory: Array<[string, string, number]> = [];
let bytesRead = 0;
const hash = (value: Buffer | string) => `sha256:${createHash("sha256").update(value).digest("hex")}`;
function walk(directory: string, relative = "") {
if (relative.split("/").length > 32) throw new Error("Distribution bundle exceeds directory depth limit");
for (const name of fs.readdirSync(directory).sort()) {
const relativePath = relative ? `${relative}/${name}` : name;
const file = path.join(directory, name);
const stat = fs.lstatSync(file);
if (stat.isSymbolicLink()) throw new Error("Distribution bundles cannot contain symlinks");
if (stat.isDirectory()) walk(file, relativePath);
else if (stat.isFile()) {
bytesRead += stat.size;
if (bytesRead > 256 * 1024 * 1024 || inventory.length >= 10_000) throw new Error("Distribution bundle exceeds verification limits");
inventory.push([relativePath, hash(fs.readFileSync(file)), stat.mode & 0o777]);
} else throw new Error("Distribution bundles contain only regular files and directories");
}
}
walk(root);
if (!inventory.length) throw new Error("Distribution bundle is empty");
return hash(JSON.stringify(inventory));
}
/**
* Optional, image-owned extension catalog. No URLs, executable configuration,
* runtime dependency installation, or replacement of a built-in entry.
* Validate bytes before importing any manifest code.
*/
export function readDistributionPluginCatalog(
catalogRoot: string,
builtins: readonly { key: string; pluginKey: string }[],
): DistributionPlugin[] {
const root = distributionPluginsRoot(catalogRoot);
const file = path.join(root, "catalog.json");
const rootStat = fs.lstatSync(root, { throwIfNoEntry: false });
if (!rootStat) return [];
if (rootStat.isSymbolicLink() || !rootStat.isDirectory()) throw new Error("Distribution catalog root must be a regular directory, not a symlink");
const stat = fs.lstatSync(file, { throwIfNoEntry: false });
if (!stat) return [];
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 128 * 1024) throw new Error("Invalid distribution catalog file");
const catalog = distributionPluginCatalogSchema.parse(JSON.parse(fs.readFileSync(file, "utf8")));
const keys = new Set(builtins.map((entry) => entry.key));
const pluginKeys = new Set(builtins.map((entry) => entry.pluginKey));
const directories = new Set<string>();
return catalog.plugins.map((entry) => {
if (keys.has(entry.key) || pluginKeys.has(entry.pluginKey) || directories.has(entry.directory)) {
throw new Error("Duplicate or built-in distribution plugin identity");
}
keys.add(entry.key); pluginKeys.add(entry.pluginKey); directories.add(entry.directory);
const localPath = path.join(root, entry.directory);
const info = fs.lstatSync(localPath);
if (!info.isDirectory() || info.isSymbolicLink()) throw new Error("Invalid distribution bundle directory");
if (distributionBundleDigest(localPath) !== entry.digest) throw new Error(`Distribution bundle digest mismatch: ${entry.key}`);
const pkg = JSON.parse(fs.readFileSync(path.join(localPath, "package.json"), "utf8"));
if (pkg.version !== entry.version || !pkg.paperclipPlugin) throw new Error("Distribution package version or entrypoints missing");
for (const name of ["manifest", "worker", "ui"] as const) {
const declared = pkg.paperclipPlugin[name];
if (name === "ui" && declared === undefined) continue;
const relative = bundleEntrypoint(declared);
const target = fs.statSync(path.join(localPath, relative));
if (name === "ui" ? !target.isDirectory() : !target.isFile()) throw new Error("Distribution entrypoint is not prebuilt");
}
return { ...entry, localPath, entrypoints: {
worker: bundleEntrypoint(pkg.paperclipPlugin.worker),
...(pkg.paperclipPlugin.ui === undefined ? {} : { ui: bundleEntrypoint(pkg.paperclipPlugin.ui) }),
} };
});
}