mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 09:07:46 +02:00
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environments give each agent run an execution target, and sandbox providers (Daytona, E2B, Novita, exe.dev) run as plugin workers > - A managed deployment provisions one platform-managed sandbox row with no credential in config; the provider is documented to fall back to its process env var (for example `DAYTONA_API_KEY`) > - Plugin workers spawn with a scrubbed environment, so that fallback never sees the host env var — probe and lease acquisition fail with "require an API key in config or DAYTONA_API_KEY" even when the deployment sets the var > - Separately, the managed-sandbox-only mode hides local rows from every list, but the instance Default picker renders a hardcoded synthetic "Local" option that no filter touches > - This pull request forwards each bundled provider's documented credential env var to its own plugin worker, and gates the synthetic Local option on the flag > - The benefit is that the documented host-env credential fallback works for plugin-backed providers, and managed-sandbox-only instances no longer offer Local anywhere ## Linked Issues or Issue Description **Subsystem affected** Plugin worker environment construction (`server/src/services/plugin-loader.ts`) and the environments UI (instance Default picker, agent form inherited-environment label). **Problem or motivation** Two follow-ups to the managed-sandbox-only mode (#11200), both found on a live managed deployment: 1. The deployment sets `DAYTONA_API_KEY` as a server env var and the managed sandbox row omits `config.apiKey` by contract. "Test Connection" fails with `Sandbox environment probe failed for provider "daytona". Daytona sandbox environments require an API key in config or DAYTONA_API_KEY.` A real agent run fails the same way at lease acquisition. The cause: sandbox providers run as plugin workers, and `buildPluginWorkerEnv` passes only model-provider keys and in-cluster Kubernetes vars. The provider's own documented credential env var never reaches the worker, so the in-plugin `process.env` fallback reads nothing. The self-hosted path has the same gap: the Daytona plugin README documents `DAYTONA_API_KEY` as a host-level fallback, and it does not work today. 2. With `enableManagedSandboxOnly` on, the instance Default environment picker still shows "Local". The server filters local *rows* out of the list, and the client filter mirrors that for cached lists, but this option is a hardcoded `<option value="">Local</option>` — not a list row — so no filter removes it. Selecting it writes a null default, which run selection then rejects fail-closed. **Proposed solution** Forward each bundled sandbox provider's documented credential env var into its plugin worker, keyed by the manifest's declared `environmentDrivers[].driverKey` so a worker only receives its own provider's credential (daytona → `DAYTONA_API_KEY`, e2b → `E2B_API_KEY`, exe-dev → `EXE_API_KEY`, novita → `NOVITA_API_KEY`). Keep the existing gate: only plugins that declare `environment.drivers.register` receive any passthrough. In the UI, render the synthetic Local option only when managed-sandbox-only is off; under the flag show a disabled "Select environment" placeholder only while no default is stamped yet, and stop the agent form's inherited label from reading "Local". **Alternatives considered** Adding `DAYTONA_API_KEY` to the existing `ADAPTER_ENV_PASSTHROUGH` list was rejected: that list goes to every environment-driver plugin, so each provider would receive every other provider's credential. A manifest schema field for declared credential env vars was rejected as heavier than needed: the bundled providers are known, and the mapping lives next to the two existing passthrough lists. ## What Changed - `server/src/services/plugin-loader.ts`: new `SANDBOX_PROVIDER_CREDENTIAL_ENV_PASSTHROUGH` map (driverKey → documented credential env vars). `buildPluginWorkerEnv` reads the manifest's `environmentDrivers` and forwards only the matching vars, after the existing `environment.drivers.register` gate. Blank values stay excluded. - `server/src/__tests__/plugin-database.test.ts`: the daytona worker receives `DAYTONA_API_KEY` and not another provider's key; a plugin whose drivers have no mapping (kubernetes) receives no credential var. - `ui/src/pages/CompanyEnvironments.tsx`: the Default picker's synthetic Local option renders only when managed-sandbox-only is off. Under the flag, a disabled "Select environment" placeholder renders only while the default is unset. - `ui/src/pages/CompanyEnvironments.test.tsx`: the Local option is present by default and absent under the flag; saved non-local environments stay selectable. - `ui/src/components/AgentConfigForm.tsx`: the inherited-environment label falls back to "Managed sandbox" instead of "Local" under the flag. ## Verification - `server`: `npx vitest run src/__tests__/plugin-database.test.ts -t buildPluginWorkerEnv` — 5 passed (3 existing, 2 new). - `ui`: `npx vitest run src/pages/CompanyEnvironments.test.tsx` — 22 passed (2 new); `npx vitest run src/components/AgentConfigForm.render.test.tsx` — 10 passed. - `tsc --noEmit` clean in `server` and `ui`. - Live managed deployment: confirmed the tenant service env carries `DAYTONA_API_KEY` while the probe fails with the exact message above, which pins the root cause to the worker env, not delivery. ## Risks - The worker env grows by exactly one var per matching bundled provider, only when the deployment sets it and only for plugins that declare a matching environment driver. Plugins without a mapping see no change. - Self-hosted behavioral shift is the fix itself: a host-level `DAYTONA_API_KEY` (or E2B/EXE/NOVITA equivalent) now reaches the provider as its README documents. Deployments that set the var but expected it to stay inert had no working configuration to preserve — the provider errored on every keyless probe and run. - UI change is inert unless `enableManagedSandboxOnly` is on (default false everywhere). ## Model Used Claude Fable 5 (`claude-fable-5`) via Claude Code — extended thinking, tool use, parallel read-only subagents for the two root-cause traces. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
767 lines
26 KiB
TypeScript
767 lines
26 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
companies,
|
|
createDb,
|
|
issueRelations,
|
|
issues,
|
|
pluginDatabaseNamespaces,
|
|
pluginMigrations,
|
|
plugins,
|
|
} from "@paperclipai/db";
|
|
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
import {
|
|
derivePluginDatabaseNamespace,
|
|
pluginDatabaseService,
|
|
validatePluginMigrationStatement,
|
|
validatePluginRuntimeExecute,
|
|
validatePluginRuntimeQuery,
|
|
} from "../services/plugin-database.js";
|
|
import { buildPluginWorkerEnv, pluginLoader } from "../services/plugin-loader.js";
|
|
|
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
const multiMigrationPluginKey = "paperclip.dbfixture";
|
|
const llmWikiPluginKey = "paperclipai.plugin-llm-wiki";
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres plugin database tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
describe("plugin database SQL validation", () => {
|
|
it("allows namespace migrations with whitelisted public foreign keys", () => {
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"CREATE TABLE plugin_test.rows (id uuid PRIMARY KEY, issue_id uuid REFERENCES public.issues(id))",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).not.toThrow();
|
|
});
|
|
|
|
it("allows qualified index creation and namespace-scoped migration backfills", () => {
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"CREATE INDEX IF NOT EXISTS rows_issue_idx ON plugin_test.rows (issue_id)",
|
|
"plugin_test",
|
|
)
|
|
).not.toThrow();
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
`
|
|
WITH source_rows AS (
|
|
SELECT id FROM plugin_test.rows
|
|
)
|
|
INSERT INTO plugin_test.row_copies (id)
|
|
SELECT id FROM source_rows
|
|
ON CONFLICT (id) DO NOTHING
|
|
`,
|
|
"plugin_test",
|
|
)
|
|
).not.toThrow();
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
`
|
|
UPDATE plugin_test.rows r
|
|
SET copied_from_id = s.id
|
|
FROM plugin_test.source_rows s
|
|
WHERE s.id = r.id
|
|
`,
|
|
"plugin_test",
|
|
)
|
|
).not.toThrow();
|
|
});
|
|
|
|
it("keeps migration backfill writes scoped to the plugin namespace", () => {
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"CREATE TABLE rows (id uuid PRIMARY KEY, issue_id uuid REFERENCES public.issues(id))",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).toThrow(/fully qualified/i);
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"WITH source_rows AS (SELECT id FROM plugin_test.rows) INSERT INTO public.issues (id) SELECT id FROM source_rows",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).toThrow(/public/i);
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"UPDATE public.issues SET title = 'bad'",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).toThrow(/public/i);
|
|
});
|
|
|
|
it("rejects migrations that create public objects", () => {
|
|
expect(() =>
|
|
validatePluginMigrationStatement(
|
|
"CREATE TABLE public.rows (id uuid PRIMARY KEY)",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).toThrow(/public/i);
|
|
});
|
|
|
|
it("allows whitelisted runtime reads but rejects public writes", () => {
|
|
expect(() =>
|
|
validatePluginRuntimeQuery(
|
|
"SELECT r.id FROM plugin_test.rows r JOIN public.issues i ON i.id = r.issue_id",
|
|
"plugin_test",
|
|
["issues"],
|
|
)
|
|
).not.toThrow();
|
|
expect(() =>
|
|
validatePluginRuntimeExecute("UPDATE public.issues SET title = $1", "plugin_test")
|
|
).toThrow(/namespace/i);
|
|
});
|
|
|
|
it("targets anonymous DO blocks without rejecting do-prefixed aliases", () => {
|
|
expect(() =>
|
|
validatePluginRuntimeQuery(
|
|
"SELECT EXTRACT(DOW FROM created_at) AS do_flag FROM plugin_test.rows",
|
|
"plugin_test",
|
|
)
|
|
).not.toThrow();
|
|
expect(() =>
|
|
validatePluginMigrationStatement("DO $$ BEGIN END $$;", "plugin_test")
|
|
).toThrow(/disallowed/i);
|
|
});
|
|
});
|
|
|
|
describe("buildPluginWorkerEnv", () => {
|
|
const instanceInfo = {
|
|
deploymentMode: "authenticated",
|
|
deploymentExposure: "public",
|
|
};
|
|
|
|
it("passes only model provider keys through to environment driver plugins", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: { capabilities: ["environment.drivers.register"] },
|
|
instanceInfo,
|
|
processEnv: {
|
|
ANTHROPIC_API_KEY: "anthropic-token",
|
|
OPENAI_API_KEY: "openai-token",
|
|
GEMINI_API_KEY: " ",
|
|
AWS_SECRET_ACCESS_KEY: "aws-secret",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
ANTHROPIC_API_KEY: "anthropic-token",
|
|
OPENAI_API_KEY: "openai-token",
|
|
});
|
|
});
|
|
|
|
it("passes in-cluster Kubernetes service-discovery vars to environment driver plugins", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: { capabilities: ["environment.drivers.register"] },
|
|
instanceInfo,
|
|
processEnv: {
|
|
KUBERNETES_SERVICE_HOST: "10.0.0.1",
|
|
KUBERNETES_SERVICE_PORT: "443",
|
|
KUBERNETES_SERVICE_PORT_HTTPS: " ",
|
|
AWS_SECRET_ACCESS_KEY: "aws-secret",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
KUBERNETES_SERVICE_HOST: "10.0.0.1",
|
|
KUBERNETES_SERVICE_PORT: "443",
|
|
});
|
|
});
|
|
|
|
it("does not pass provider keys to non-environment plugins", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: { capabilities: ["ui.slots.register"] },
|
|
instanceInfo,
|
|
processEnv: {
|
|
OPENAI_API_KEY: "openai-token",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
});
|
|
});
|
|
|
|
it("passes a first-party sandbox provider's documented credential env var to its own worker", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: {
|
|
capabilities: ["environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "daytona" }],
|
|
},
|
|
packageName: "@paperclipai/plugin-daytona",
|
|
packagePath: null,
|
|
instanceInfo,
|
|
processEnv: {
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
NOVITA_API_KEY: "novita-token",
|
|
E2B_API_KEY: " ",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
});
|
|
});
|
|
|
|
it("passes the credential to a first-party plugin installed from the bundled catalog", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: {
|
|
capabilities: ["environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "daytona" }],
|
|
},
|
|
packageName: "@paperclipai/plugin-daytona",
|
|
packagePath: "/app/packages/plugins/sandbox-providers/daytona",
|
|
trustedLocalPluginRoots: ["/app/packages/plugins"],
|
|
instanceInfo,
|
|
processEnv: {
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
});
|
|
});
|
|
|
|
it("does not pass the credential to a local plugin that self-declares the first-party name", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: {
|
|
capabilities: ["environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "daytona" }],
|
|
},
|
|
packageName: "@paperclipai/plugin-daytona",
|
|
packagePath: "/home/operator/.paperclip/plugins/fake-daytona",
|
|
trustedLocalPluginRoots: ["/app/packages/plugins"],
|
|
instanceInfo,
|
|
processEnv: {
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
});
|
|
});
|
|
|
|
it("does not pass a credential to a third-party plugin that claims a first-party driver key", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: {
|
|
capabilities: ["environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "daytona" }],
|
|
},
|
|
packageName: "@acme/plugin-fake-daytona",
|
|
instanceInfo,
|
|
processEnv: {
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
});
|
|
});
|
|
|
|
it("does not pass a credential when the first-party package omits its expected driver key", () => {
|
|
const env = buildPluginWorkerEnv({
|
|
manifest: {
|
|
capabilities: ["environment.drivers.register"],
|
|
environmentDrivers: [{ driverKey: "kubernetes" }],
|
|
},
|
|
packageName: "@paperclipai/plugin-daytona",
|
|
instanceInfo,
|
|
processEnv: {
|
|
DAYTONA_API_KEY: "daytona-token",
|
|
},
|
|
});
|
|
|
|
expect(env).toEqual({
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
});
|
|
});
|
|
});
|
|
|
|
describeEmbeddedPostgres("plugin database namespaces", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let packageRoots: string[] = [];
|
|
|
|
beforeAll(async () => {
|
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-plugin-db-");
|
|
db = createDb(tempDb.connectionString);
|
|
}, 20_000);
|
|
|
|
afterEach(async () => {
|
|
for (const pluginKey of ["paperclip.dbtest", "paperclip.escape", "paperclip.refresh", multiMigrationPluginKey, llmWikiPluginKey]) {
|
|
const namespace = derivePluginDatabaseNamespace(pluginKey);
|
|
await db.execute(sql.raw(`DROP SCHEMA IF EXISTS "${namespace}" CASCADE`));
|
|
}
|
|
await db.execute(sql.raw(`DROP SCHEMA IF EXISTS "${derivePluginDatabaseNamespace(llmWikiPluginKey, "llm_wiki")}" CASCADE`));
|
|
await db.delete(pluginMigrations);
|
|
await db.delete(pluginDatabaseNamespaces);
|
|
await db.delete(plugins);
|
|
await db.delete(issueRelations);
|
|
await db.delete(issues);
|
|
await db.delete(companies);
|
|
await Promise.all(packageRoots.map((root) => rm(root, { recursive: true, force: true })));
|
|
packageRoots = [];
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function createPluginPackage(manifest: PaperclipPluginManifestV1, migrationSql: string) {
|
|
const packageRoot = await mkdtemp(path.join(os.tmpdir(), "paperclip-plugin-package-"));
|
|
packageRoots.push(packageRoot);
|
|
const migrationsDir = path.join(packageRoot, manifest.database!.migrationsDir);
|
|
await mkdir(migrationsDir, { recursive: true });
|
|
await writeFile(path.join(migrationsDir, "001_init.sql"), migrationSql, "utf8");
|
|
return packageRoot;
|
|
}
|
|
|
|
function llmWikiManifest(): PaperclipPluginManifestV1 {
|
|
return {
|
|
id: llmWikiPluginKey,
|
|
apiVersion: 1,
|
|
version: "0.1.0",
|
|
displayName: "LLM Wiki",
|
|
description: "Local-file LLM Wiki plugin.",
|
|
author: "Paperclip",
|
|
categories: ["automation", "ui"],
|
|
capabilities: [
|
|
"database.namespace.migrate",
|
|
"database.namespace.read",
|
|
"database.namespace.write",
|
|
],
|
|
entrypoints: { worker: "./dist/worker.js" },
|
|
database: {
|
|
namespaceSlug: "llm_wiki",
|
|
migrationsDir: "migrations",
|
|
coreReadTables: ["companies", "issues", "projects", "agents"],
|
|
},
|
|
};
|
|
}
|
|
|
|
async function createInstallablePluginPackage(
|
|
pluginManifest: PaperclipPluginManifestV1,
|
|
migrationSql: string,
|
|
) {
|
|
const packageRoot = await createPluginPackage(pluginManifest, migrationSql);
|
|
await writeFile(
|
|
path.join(packageRoot, "package.json"),
|
|
JSON.stringify({
|
|
name: pluginManifest.id,
|
|
version: pluginManifest.version,
|
|
type: "module",
|
|
paperclipPlugin: { manifest: "./manifest.js" },
|
|
}),
|
|
"utf8",
|
|
);
|
|
await writeFile(
|
|
path.join(packageRoot, "manifest.js"),
|
|
`export default ${JSON.stringify(pluginManifest, null, 2)};\n`,
|
|
"utf8",
|
|
);
|
|
await mkdir(path.join(packageRoot, "dist"), { recursive: true });
|
|
await writeFile(path.join(packageRoot, "dist", "worker.js"), "export {};\n", "utf8");
|
|
return packageRoot;
|
|
}
|
|
|
|
async function installPluginRecord(manifest: PaperclipPluginManifestV1) {
|
|
const pluginId = randomUUID();
|
|
await db.insert(plugins).values({
|
|
id: pluginId,
|
|
pluginKey: manifest.id,
|
|
packageName: manifest.id,
|
|
version: manifest.version,
|
|
apiVersion: manifest.apiVersion,
|
|
categories: manifest.categories,
|
|
manifestJson: manifest,
|
|
status: "installed",
|
|
installOrder: 1,
|
|
});
|
|
return pluginId;
|
|
}
|
|
|
|
function manifest(pluginKey = "paperclip.dbtest"): PaperclipPluginManifestV1 {
|
|
return {
|
|
id: pluginKey,
|
|
apiVersion: 1,
|
|
version: "1.0.0",
|
|
displayName: "DB Test",
|
|
description: "Exercises restricted plugin database access.",
|
|
author: "Paperclip",
|
|
categories: ["automation"],
|
|
capabilities: [
|
|
"database.namespace.migrate",
|
|
"database.namespace.read",
|
|
"database.namespace.write",
|
|
],
|
|
entrypoints: { worker: "./dist/worker.js" },
|
|
database: {
|
|
migrationsDir: "migrations",
|
|
coreReadTables: ["issues"],
|
|
},
|
|
};
|
|
}
|
|
|
|
it("applies multi-file plugin migrations through the production validator", async () => {
|
|
const pluginManifest = manifest(multiMigrationPluginKey);
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
|
const packageRoot = await createPluginPackage(
|
|
pluginManifest,
|
|
`CREATE TABLE ${namespace}.source_rows (id uuid PRIMARY KEY, label text NOT NULL);`,
|
|
);
|
|
await writeFile(
|
|
path.join(packageRoot, pluginManifest.database!.migrationsDir, "002_derived.sql"),
|
|
`CREATE TABLE ${namespace}.derived_rows (
|
|
id uuid PRIMARY KEY,
|
|
source_id uuid NOT NULL REFERENCES ${namespace}.source_rows(id)
|
|
);`,
|
|
"utf8",
|
|
);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
await pluginDatabaseService(db).applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
|
|
const migrations = await db
|
|
.select()
|
|
.from(pluginMigrations)
|
|
.where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.status, "applied")));
|
|
expect(migrations).toHaveLength(2);
|
|
});
|
|
|
|
it("applies the bundled LLM Wiki migrations through the production validator", async () => {
|
|
const pluginManifest = llmWikiManifest();
|
|
const repoRoot = path.basename(process.cwd()) === "server" ? path.resolve(process.cwd(), "..") : process.cwd();
|
|
const packageRoot = path.join(repoRoot, "packages", "plugins", "plugin-llm-wiki");
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id, pluginManifest.database?.namespaceSlug);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
|
|
await pluginDatabaseService(db).applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
|
|
const migrations = await db
|
|
.select()
|
|
.from(pluginMigrations)
|
|
.where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.status, "applied")));
|
|
expect(migrations.map((migration) => migration.migrationKey)).toEqual([
|
|
"001_llm_wiki.sql",
|
|
"002_paperclip_distillation.sql",
|
|
"003_spaces.sql",
|
|
]);
|
|
|
|
const constraintRows = Array.from(
|
|
await db.execute(
|
|
sql<{ table_name: string; conname: string; columns: string[] }>`
|
|
SELECT t.relname AS table_name, c.conname, array_agg(a.attname ORDER BY constraint_columns.ordinality)::text[] AS columns
|
|
FROM pg_constraint c
|
|
JOIN pg_class t ON t.oid = c.conrelid
|
|
JOIN unnest(c.conkey) WITH ORDINALITY AS constraint_columns(attnum, ordinality) ON true
|
|
JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = constraint_columns.attnum
|
|
WHERE c.connamespace = ${namespace}::regnamespace AND c.contype = 'u'
|
|
GROUP BY t.relname, c.conname
|
|
ORDER BY t.relname, c.conname
|
|
`,
|
|
) as Iterable<{ table_name: string; conname: string; columns: string[] }>,
|
|
);
|
|
const constraints = constraintRows.map((row) => row.conname);
|
|
const uniqueColumnSets = new Set(
|
|
constraintRows.map((row) => `${row.table_name}:${row.columns.join(",")}`),
|
|
);
|
|
expect(constraints).toEqual(
|
|
expect.arrayContaining([
|
|
"wiki_pages_company_wiki_space_path_key",
|
|
"distillation_cursors_company_wiki_space_scope_key",
|
|
"distillation_work_items_company_wiki_space_idempotency_key",
|
|
"page_bindings_company_wiki_space_page_path_key",
|
|
]),
|
|
);
|
|
expect(constraints).not.toContain("wiki_pages_company_id_wiki_id_path_key");
|
|
expect(constraints).not.toContain("paperclip_distillation_cursor_company_id_wiki_id_source_sco_key");
|
|
expect(constraints).not.toContain("paperclip_distillation_work_i_company_id_wiki_id_idempotenc_key");
|
|
expect(constraints).not.toContain("paperclip_page_bindings_company_id_wiki_id_page_path_key");
|
|
expect(uniqueColumnSets).not.toContain("wiki_pages:company_id,wiki_id,path");
|
|
expect(uniqueColumnSets).not.toContain("paperclip_distillation_cursors:company_id,wiki_id,source_scope,scope_key,source_kind");
|
|
expect(uniqueColumnSets).not.toContain("paperclip_distillation_work_items:company_id,wiki_id,idempotency_key");
|
|
expect(uniqueColumnSets).not.toContain("paperclip_page_bindings:company_id,wiki_id,page_path");
|
|
});
|
|
|
|
it("applies migrations once and allows whitelisted core joins at runtime", async () => {
|
|
const pluginManifest = manifest();
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
|
const packageRoot = await createPluginPackage(
|
|
pluginManifest,
|
|
`
|
|
CREATE TABLE ${namespace}.mission_rows (
|
|
id uuid PRIMARY KEY,
|
|
issue_id uuid NOT NULL REFERENCES public.issues(id),
|
|
label text NOT NULL
|
|
);
|
|
`,
|
|
);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
const companyId = randomUUID();
|
|
const issueId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Paperclip",
|
|
issuePrefix: "TST",
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Joined issue",
|
|
status: "todo",
|
|
priority: "medium",
|
|
identifier: "TST-1",
|
|
});
|
|
|
|
const pluginDb = pluginDatabaseService(db);
|
|
await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
|
|
await pluginDb.execute(
|
|
pluginId,
|
|
`INSERT INTO ${namespace}.mission_rows (id, issue_id, label) VALUES ($1, $2, $3)`,
|
|
[randomUUID(), issueId, "alpha"],
|
|
);
|
|
const rows = await pluginDb.query<{ label: string; title: string }>(
|
|
pluginId,
|
|
`SELECT m.label, i.title FROM ${namespace}.mission_rows m JOIN public.issues i ON i.id = m.issue_id`,
|
|
);
|
|
expect(rows).toEqual([{ label: "alpha", title: "Joined issue" }]);
|
|
|
|
const migrations = await db
|
|
.select()
|
|
.from(pluginMigrations)
|
|
.where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.status, "applied")));
|
|
expect(migrations).toHaveLength(1);
|
|
});
|
|
|
|
it("rejects runtime writes to public core tables", async () => {
|
|
const pluginManifest = manifest();
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
|
const packageRoot = await createPluginPackage(
|
|
pluginManifest,
|
|
`CREATE TABLE ${namespace}.notes (id uuid PRIMARY KEY, body text NOT NULL);`,
|
|
);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
const pluginDb = pluginDatabaseService(db);
|
|
await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
|
|
await expect(
|
|
pluginDb.execute(pluginId, "UPDATE public.issues SET title = $1", ["bad"]),
|
|
).rejects.toThrow(/plugin namespace/i);
|
|
});
|
|
|
|
it("records a failed migration when SQL escapes the plugin namespace", async () => {
|
|
const pluginManifest = manifest("paperclip.escape");
|
|
const packageRoot = await createPluginPackage(
|
|
pluginManifest,
|
|
"CREATE TABLE public.plugin_escape (id uuid PRIMARY KEY);",
|
|
);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
|
|
await expect(
|
|
pluginDatabaseService(db).applyMigrations(pluginId, pluginManifest, packageRoot),
|
|
).rejects.toThrow(/public\.plugin_escape|public/i);
|
|
|
|
const [migration] = await db
|
|
.select()
|
|
.from(pluginMigrations)
|
|
.where(eq(pluginMigrations.pluginId, pluginId));
|
|
expect(migration?.status).toBe("failed");
|
|
});
|
|
|
|
it("rolls back plugin install when migration validation fails", async () => {
|
|
const pluginManifest = manifest("paperclip.escape");
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
|
const packageRoot = await createInstallablePluginPackage(
|
|
pluginManifest,
|
|
"CREATE TABLE public.plugin_escape (id uuid PRIMARY KEY);",
|
|
);
|
|
const loader = pluginLoader(db, {
|
|
enableLocalFilesystem: false,
|
|
enableNpmDiscovery: false,
|
|
});
|
|
|
|
await expect(loader.installPlugin({ localPath: packageRoot }))
|
|
.rejects.toThrow(/public\.plugin_escape|public/i);
|
|
|
|
const installedPlugins = await db
|
|
.select()
|
|
.from(plugins)
|
|
.where(eq(plugins.pluginKey, pluginManifest.id));
|
|
const namespaces = await db
|
|
.select()
|
|
.from(pluginDatabaseNamespaces)
|
|
.where(eq(pluginDatabaseNamespaces.pluginKey, pluginManifest.id));
|
|
const migrations = await db
|
|
.select()
|
|
.from(pluginMigrations)
|
|
.where(eq(pluginMigrations.pluginKey, pluginManifest.id));
|
|
const schemaRows = Array.from(
|
|
await db.execute(
|
|
sql<{ schema_name: string }>`SELECT schema_name FROM information_schema.schemata WHERE schema_name = ${namespace}`,
|
|
) as Iterable<{ schema_name: string }>,
|
|
);
|
|
|
|
expect(installedPlugins).toHaveLength(0);
|
|
expect(namespaces).toHaveLength(0);
|
|
expect(migrations).toHaveLength(0);
|
|
expect(schemaRows).toHaveLength(0);
|
|
});
|
|
|
|
it("refreshes persisted manifests from disk before activation", async () => {
|
|
const staleManifest = manifest("paperclip.refresh");
|
|
const refreshedManifest: PaperclipPluginManifestV1 = {
|
|
...staleManifest,
|
|
capabilities: [...staleManifest.capabilities, "agent.tools.register"],
|
|
database: {
|
|
...staleManifest.database!,
|
|
coreReadTables: ["companies"],
|
|
},
|
|
tools: [
|
|
{
|
|
name: "db-smoke",
|
|
displayName: "DB Smoke",
|
|
description: "Exercises plugin tool registration worker lookup.",
|
|
parametersSchema: { type: "object", properties: {} },
|
|
},
|
|
],
|
|
};
|
|
const namespace = derivePluginDatabaseNamespace(refreshedManifest.id);
|
|
const packageRoot = await createInstallablePluginPackage(
|
|
refreshedManifest,
|
|
`
|
|
CREATE TABLE ${namespace}.company_refs (
|
|
id uuid PRIMARY KEY,
|
|
company_id uuid NOT NULL REFERENCES public.companies(id)
|
|
);
|
|
`,
|
|
);
|
|
const pluginId = await installPluginRecord(staleManifest);
|
|
await db
|
|
.update(plugins)
|
|
.set({
|
|
packagePath: packageRoot,
|
|
status: "ready",
|
|
})
|
|
.where(eq(plugins.id, pluginId));
|
|
|
|
const workerManager = {
|
|
startWorker: vi.fn().mockResolvedValue(undefined),
|
|
stopAll: vi.fn().mockResolvedValue(undefined),
|
|
};
|
|
const toolDispatcher = {
|
|
registerPluginTools: vi.fn(),
|
|
};
|
|
const loader = pluginLoader(db, {
|
|
enableLocalFilesystem: false,
|
|
enableNpmDiscovery: false,
|
|
}, {
|
|
workerManager,
|
|
eventBus: {
|
|
forPlugin: vi.fn(() => ({})),
|
|
subscriptionCount: vi.fn(() => 0),
|
|
},
|
|
jobScheduler: {
|
|
registerPlugin: vi.fn().mockResolvedValue(undefined),
|
|
stop: vi.fn(),
|
|
},
|
|
jobStore: {
|
|
syncJobDeclarations: vi.fn().mockResolvedValue(undefined),
|
|
},
|
|
toolDispatcher,
|
|
lifecycleManager: {
|
|
markError: vi.fn().mockResolvedValue(undefined),
|
|
},
|
|
buildHostHandlers: vi.fn(() => ({})),
|
|
instanceInfo: {
|
|
instanceId: "test-instance",
|
|
hostVersion: "1.0.0",
|
|
deploymentMode: "authenticated",
|
|
deploymentExposure: "public",
|
|
},
|
|
} as never);
|
|
|
|
const result = await loader.loadSingle(pluginId);
|
|
|
|
expect(result.success).toBe(true);
|
|
expect(workerManager.startWorker).toHaveBeenCalledWith(
|
|
pluginId,
|
|
expect.objectContaining({
|
|
databaseNamespace: namespace,
|
|
env: {
|
|
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
|
},
|
|
manifest: expect.objectContaining({
|
|
database: expect.objectContaining({ coreReadTables: ["companies"] }),
|
|
}),
|
|
}),
|
|
);
|
|
expect(toolDispatcher.registerPluginTools).toHaveBeenCalledWith(
|
|
refreshedManifest.id,
|
|
expect.objectContaining({
|
|
tools: refreshedManifest.tools,
|
|
}),
|
|
pluginId,
|
|
);
|
|
const [plugin] = await db
|
|
.select()
|
|
.from(plugins)
|
|
.where(eq(plugins.id, pluginId));
|
|
expect(plugin?.manifestJson.database?.coreReadTables).toEqual(["companies"]);
|
|
});
|
|
|
|
it("rejects checksum changes for already applied migrations", async () => {
|
|
const pluginManifest = manifest();
|
|
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
|
const packageRoot = await createPluginPackage(
|
|
pluginManifest,
|
|
`CREATE TABLE ${namespace}.checksum_rows (id uuid PRIMARY KEY);`,
|
|
);
|
|
const pluginId = await installPluginRecord(pluginManifest);
|
|
const pluginDb = pluginDatabaseService(db);
|
|
await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
|
|
|
|
await writeFile(
|
|
path.join(packageRoot, "migrations", "001_init.sql"),
|
|
`CREATE TABLE ${namespace}.checksum_rows (id uuid PRIMARY KEY, note text);`,
|
|
"utf8",
|
|
);
|
|
|
|
await expect(pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot))
|
|
.rejects.toThrow(/checksum mismatch/i);
|
|
});
|
|
});
|