mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
Forward provider credentials from verified image catalogs (#21186)
## Thinking Path > - Paperclip manages AI agents for work. > - Application images can ship plugins that provide task environments. > - Paperclip starts plugin workers with a restricted process environment. > - An image author can know which provider credentials an image-bundled worker needs. > - The image catalog already identifies selected bundles and verifies their content. > - This change adds a way for the image catalog to declare the provider keys that a verified worker receives. ## Linked Issues or Issue Description **What existing behavior does this improve?** Image-bundled environment drivers can use the worker credential delivery already available to built-in providers. **Current behavior** The host provides fixed credential forwarding for certain built-in providers. **Proposed behavior** A verified image bundle can receive provider API keys declared on its own catalog entry. **Reason and benefit** Image authors can scope deployment credentials to the verified worker that needs them. ## What Changed - Add an optional provider credential declaration to the image-owned distribution catalog. - Scope forwarding to the verified worker for the selected bundle. - Publish distribution catalog support for image builders. ## Verification - The new loader regression failed before the change because the old catalog rejected the credential declaration. - Ran six focused Vitest suites for bundled activation, package identity, distribution catalog validation, worker environment, and managed configuration. 166 tests passed and one existing test was skipped. - Ran `pnpm --filter @paperclipai/server typecheck`. It passed. - Reviewed the diff for secret values and personal information. Test keys are synthetic. - A downstream image build fixture exercised the catalog reader, bundle verification, and worker environment together. The supported build passed only the declared provider key to the verified worker. The build for an older target omitted the declaration. - Local verification used embedded PostgreSQL and prebuilt fixtures. ## Risks - Deploy images with catalog credential declarations only to Core versions that support the field. Older hosts reject unknown catalog fields. - The image author controls which verified bundle receives each provider API key. ## Model Used - Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex. Independent Codex review and verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue in the PR - [x] I have used public references - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
267983356c
commit
2168f50741
9 files changed
+127
-9
No files matched your search
@@ -41,6 +41,26 @@ in operation. This detects packaging errors but does not authenticate an
|
||||
untrusted image builder. Image provenance and signatures remain deployment
|
||||
responsibilities.
|
||||
|
||||
### Provider credentials
|
||||
|
||||
An image author can set `workerCredentialEnv` on a distribution catalog
|
||||
entry to pass provider API keys from the host environment. For example,
|
||||
`"workerCredentialEnv": ["EXAMPLE_API_KEY"]` lets the worker receive
|
||||
`EXAMPLE_API_KEY` when the deployment sets it. List 1 to 16 distinct
|
||||
uppercase environment variable names ending in `_API_KEY`, each at most
|
||||
128 characters. Store the names in the catalog and set the values through
|
||||
the deployment.
|
||||
|
||||
Core grants these variables to the selected, digest-verified worker when
|
||||
it declares `environment.drivers.register`. This grant is available through
|
||||
managed image activation. The ordinary `plugins.workerEnv` credential
|
||||
restriction applies independently.
|
||||
|
||||
Before emitting this field, image builders can check for
|
||||
`workerCredentialEnv` in the capability list at
|
||||
`server/src/services/distribution-plugin-catalog-capabilities.json` in the
|
||||
target Core source revision. Older versions reject the unknown catalog field.
|
||||
|
||||
## Selection, upgrades and rollback
|
||||
|
||||
Managed instances select a distribution key through the existing
|
||||
|
||||
@@ -32,7 +32,11 @@ describePostgres("bundled image registration in PostgreSQL", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let peer: ReturnType<typeof createDb>;
|
||||
const roots: string[] = [];
|
||||
const fixture = () => { const result = bundledImageFixture(); roots.push(result.root); return result; };
|
||||
const fixture = (...args: Parameters<typeof bundledImageFixture>) => {
|
||||
const result = bundledImageFixture(...args);
|
||||
roots.push(result.root);
|
||||
return result;
|
||||
};
|
||||
beforeAll(async () => {
|
||||
if (externalDatabaseUrl) {
|
||||
await applyPendingMigrations(externalDatabaseUrl);
|
||||
@@ -72,6 +76,55 @@ describePostgres("bundled image registration in PostgreSQL", () => {
|
||||
}, services);
|
||||
}
|
||||
|
||||
it("forwards catalog credentials only to the verified bundle with the declaration", async () => {
|
||||
const manifestOverrides = {
|
||||
capabilities: ["environment.drivers.register"],
|
||||
environmentDrivers: [{
|
||||
driverKey: "fixture-tasks",
|
||||
kind: "environment_driver",
|
||||
displayName: "Fixture tasks",
|
||||
configSchema: { type: "object", additionalProperties: false },
|
||||
}],
|
||||
} satisfies Partial<ReturnType<typeof fixture>["manifest"]>;
|
||||
const image = fixture(manifestOverrides, { workerCredentialEnv: ["TEST_PROVIDER_API_KEY"] });
|
||||
const imageWithoutGrant = fixture(manifestOverrides);
|
||||
const { services } = runtime();
|
||||
const managed = loaderFor(image, db, services as unknown as PluginRuntimeServices);
|
||||
const managedWithoutGrant = loaderFor(imageWithoutGrant, db, services as unknown as PluginRuntimeServices);
|
||||
const unmanaged = pluginLoader(db, {}, services as unknown as PluginRuntimeServices);
|
||||
await managed.installPlugin({ localPath: image.packageRoot });
|
||||
const registry = pluginRegistryService(db);
|
||||
const row = (await registry.getByKey(image.manifest.id))!;
|
||||
await registry.updateStatus(row.id, { status: "ready" });
|
||||
|
||||
vi.stubEnv("TEST_PROVIDER_API_KEY", "fixture-provider-key");
|
||||
vi.stubEnv("TEST_OTHER_API_KEY", "fixture-other-key");
|
||||
try {
|
||||
expect((await managed.loadSingle(row.id)).success).toBe(true);
|
||||
const worker = services.workerManager.startWorker.mock.lastCall?.[1];
|
||||
expect(worker).toMatchObject({
|
||||
entrypointPath: path.join(image.packageRoot, "dist", "worker.js"),
|
||||
env: { TEST_PROVIDER_API_KEY: "fixture-provider-key" },
|
||||
});
|
||||
expect(worker).not.toHaveProperty("env.TEST_OTHER_API_KEY");
|
||||
|
||||
expect((await managedWithoutGrant.loadSingle(row.id)).success).toBe(true);
|
||||
const workerWithoutGrant = services.workerManager.startWorker.mock.lastCall?.[1];
|
||||
expect(workerWithoutGrant).toMatchObject({ entrypointPath: path.join(imageWithoutGrant.packageRoot, "dist", "worker.js") });
|
||||
expect(workerWithoutGrant).not.toHaveProperty("env.TEST_PROVIDER_API_KEY");
|
||||
|
||||
const copy = path.join(image.root, "operator-copy");
|
||||
cpSync(image.packageRoot, copy, { recursive: true });
|
||||
await registry.update(row.id, { packagePath: copy });
|
||||
expect((await unmanaged.loadSingle(row.id)).success).toBe(true);
|
||||
const unverifiedWorker = services.workerManager.startWorker.mock.lastCall?.[1];
|
||||
expect(unverifiedWorker).toMatchObject({ entrypointPath: path.join(copy, "dist", "worker.js") });
|
||||
expect(unverifiedWorker).not.toHaveProperty("env.TEST_PROVIDER_API_KEY");
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
|
||||
it("starts current-image workers after losing the old home and retains each company's config", async () => {
|
||||
const old = fixture();
|
||||
const first = fixture();
|
||||
|
||||
@@ -29,6 +29,19 @@ describe("image-owned plugin catalogs", () => {
|
||||
expect(installs.map(({ key }) => key)).toEqual(["daytona", "acme.widget"]);
|
||||
expect(installs[1]?.localPath).toBe(localPath);
|
||||
});
|
||||
it("accepts only bounded and distinct provider API key names as worker credentials", () => {
|
||||
const { root, entry, save } = fixture();
|
||||
save([{ ...entry, workerCredentialEnv: ["EXAMPLE_API_KEY", "SECOND_API_KEY"] }]);
|
||||
expect(readDistributionPluginCatalog(root, BUNDLED_PLUGIN_CATALOG)[0]?.workerCredentialEnv)
|
||||
.toEqual(["EXAMPLE_API_KEY", "SECOND_API_KEY"]);
|
||||
for (const invalid of [
|
||||
[], ["NODE_OPTIONS"], ["DATABASE_URL"], ["example_API_KEY"], ["EXAMPLE_API_KEY=value"],
|
||||
["EXAMPLE_API_KEY", "EXAMPLE_API_KEY"], Array.from({ length: 17 }, (_, i) => `EXAMPLE_${i}_API_KEY`),
|
||||
]) {
|
||||
save([{ ...entry, workerCredentialEnv: invalid }]);
|
||||
expect(() => readDistributionPluginCatalog(root, BUNDLED_PLUGIN_CATALOG)).toThrow(/workerCredentialEnv/);
|
||||
}
|
||||
});
|
||||
it("rejects tampered bundle bytes before importing manifest code", () => {
|
||||
const { root, localPath } = fixture();
|
||||
writeFileSync(path.join(localPath, "dist", "manifest.js"), "throw new Error('must never execute');");
|
||||
|
||||
@@ -7,7 +7,7 @@ import { distributionBundleDigest, readDistributionPluginCatalog } from "../../s
|
||||
import { derivePluginDatabaseNamespace } from "../../services/plugin-database.js";
|
||||
import { createBundledPluginPackageResolver } from "../../services/bundled-plugin-package.js";
|
||||
|
||||
export function bundledImageFixture(manifestOverrides: Partial<PaperclipPluginManifestV1> = {}, options: { workerPrelude?: string } = {}) {
|
||||
export function bundledImageFixture(manifestOverrides: Partial<PaperclipPluginManifestV1> = {}, options: { workerPrelude?: string; workerCredentialEnv?: string[] } = {}) {
|
||||
const root = realpathSync(mkdtempSync(path.join(os.tmpdir(), "bundled-image-")));
|
||||
const packageRoot = path.join(root, "distribution", "fixture");
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
@@ -41,7 +41,7 @@ else {
|
||||
mkdirSync(path.join(packageRoot, manifest.database.migrationsDir), { recursive: true });
|
||||
writeFileSync(path.join(packageRoot, manifest.database.migrationsDir, "001_fixture.sql"), `CREATE TABLE ${derivePluginDatabaseNamespace(manifest.id)}.image_rows (id uuid PRIMARY KEY);`);
|
||||
}
|
||||
const entry = { key: "fixture", pluginKey: manifest.id, version: manifest.version, directory: "fixture", digest: distributionBundleDigest(packageRoot) };
|
||||
const entry = { key: "fixture", pluginKey: manifest.id, version: manifest.version, directory: "fixture", digest: distributionBundleDigest(packageRoot), workerCredentialEnv: options.workerCredentialEnv };
|
||||
writeFileSync(path.join(root, "distribution", "catalog.json"), JSON.stringify({ schemaVersion: 1, plugins: [entry] }));
|
||||
const catalog = readDistributionPluginCatalog(root, BUNDLED_PLUGIN_CATALOG);
|
||||
const installs = resolveBundledPluginInstalls(["fixture"], { catalogRoot: root, enforceCatalogRoot: true, env: {}, distributionPlugins: catalog });
|
||||
|
||||
@@ -152,8 +152,9 @@ describe("buildPluginWorkerEnv", () => {
|
||||
const env = buildPluginWorkerEnv({
|
||||
manifest: { capabilities: [] },
|
||||
instanceInfo,
|
||||
verifiedDistribution: { workerCredentialEnv: ["TEST_PROVIDER_API_KEY"] },
|
||||
grantedEnvVars: ["PROVIDER_ORIGIN", "PROVIDER_BLANK", "PROVIDER_MISSING"],
|
||||
processEnv: { PROVIDER_ORIGIN: "https://provider.test", PROVIDER_BLANK: " ", DATABASE_URL: "postgres://host", OPENAI_API_KEY: "openai-token" },
|
||||
processEnv: { PROVIDER_ORIGIN: "https://provider.test", PROVIDER_BLANK: " ", DATABASE_URL: "postgres://host", OPENAI_API_KEY: "openai-token", TEST_PROVIDER_API_KEY: "fixture-provider-key" },
|
||||
});
|
||||
expect(env).toEqual({
|
||||
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
||||
@@ -288,6 +289,22 @@ describe("buildPluginWorkerEnv", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each([null, "/app/packages/plugins/distribution/fixture", "/operator/plugins/fixture"])(
|
||||
"denies catalog credentials without verified image provenance for %s",
|
||||
(packagePath) => {
|
||||
expect(buildPluginWorkerEnv({
|
||||
manifest: {
|
||||
capabilities: ["environment.drivers.register"],
|
||||
environmentDrivers: [{ driverKey: "fixture-tasks" }],
|
||||
},
|
||||
packageName: "@paperclipai/plugin-image-fixture",
|
||||
packagePath,
|
||||
instanceInfo,
|
||||
processEnv: { TEST_PROVIDER_API_KEY: "fixture-provider-key" },
|
||||
})).not.toHaveProperty("TEST_PROVIDER_API_KEY");
|
||||
},
|
||||
);
|
||||
|
||||
it("passes the credential to a first-party plugin installed from the bundled catalog", () => {
|
||||
const env = buildPluginWorkerEnv({
|
||||
manifest: {
|
||||
|
||||
@@ -102,7 +102,9 @@ export function createBundledPluginPackageResolver(installs: readonly ResolvedBu
|
||||
const install = input.pluginKey ? byKey.get(input.pluginKey) : installs.find((candidate) => candidate.localPath === input.packageRoot);
|
||||
if (!install || path.resolve(input.packageRoot) !== install.localPath) throw new Error("Bundled plugin is absent or not selected in this deployment");
|
||||
read(install);
|
||||
if (input.manifest) assertManifest(install, input.manifest, input.previousManifest);
|
||||
if (!input.manifest) return;
|
||||
assertManifest(install, input.manifest, input.previousManifest);
|
||||
return install.distribution;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{ "schemaVersion": 1, "capabilities": ["workerCredentialEnv"] }
|
||||
@@ -6,6 +6,8 @@ import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import { pluginCapabilityValidator } from "./plugin-capability-validator.js";
|
||||
|
||||
const segment = z.string().regex(/^[a-z][a-z0-9.-]{0,99}$/);
|
||||
const workerCredentialEnv = z.array(z.string().max(128).regex(/^[A-Z][A-Z0-9_]*_API_KEY$/))
|
||||
.min(1).max(16).refine((names) => new Set(names).size === names.length, "Worker credential names must be distinct");
|
||||
export const distributionPluginCatalogSchema = z.object({
|
||||
schemaVersion: z.literal(1),
|
||||
plugins: z.array(z.object({
|
||||
@@ -14,6 +16,7 @@ export const distributionPluginCatalogSchema = z.object({
|
||||
version: z.string().regex(/^\d+\.\d+\.\d+(?:-[a-zA-Z0-9.-]+)?(?:\+[a-zA-Z0-9.-]+)?$/),
|
||||
directory: segment,
|
||||
digest: z.string().regex(/^sha256:[a-f0-9]{64}$/),
|
||||
workerCredentialEnv: workerCredentialEnv.optional(),
|
||||
}).strict()).max(100),
|
||||
}).strict();
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@ import type { PluginJobStore } from "./plugin-job-store.js";
|
||||
import type { PluginToolDispatcher } from "./plugin-tool-dispatcher.js";
|
||||
import type { PluginLifecycleManager } from "./plugin-lifecycle.js";
|
||||
import { pluginDatabaseService } from "./plugin-database.js";
|
||||
import type { DistributionPlugin } from "./distribution-plugin-catalog.js";
|
||||
import { resolveBundledCatalogRoot } from "./bundled-plugins.js";
|
||||
import { managedPluginWorkerEnvGrant } from "./managed-plugin-worker-env.js";
|
||||
import { createWorkScheduler } from "./work-scheduler.js";
|
||||
@@ -160,6 +161,8 @@ export function buildPluginWorkerEnv(input: {
|
||||
packageName?: string;
|
||||
/** Local install path (`PluginRecord.packagePath`); null for registry installs. */
|
||||
packagePath?: string | null;
|
||||
/** Selected image distribution verified during managed package activation. */
|
||||
verifiedDistribution?: Pick<DistributionPlugin, "workerCredentialEnv">;
|
||||
/** Test seam; defaults to the repo plugin tree and the bundled catalog root. */
|
||||
trustedLocalPluginRoots?: readonly string[];
|
||||
grantedEnvVars?: readonly string[];
|
||||
@@ -194,7 +197,12 @@ export function buildPluginWorkerEnv(input: {
|
||||
)
|
||||
? credentialEntry.envVars
|
||||
: [];
|
||||
for (const key of [...ADAPTER_ENV_PASSTHROUGH, ...K8S_IN_CLUSTER_ENV_PASSTHROUGH, ...credentialKeys]) {
|
||||
for (const key of [
|
||||
...ADAPTER_ENV_PASSTHROUGH,
|
||||
...K8S_IN_CLUSTER_ENV_PASSTHROUGH,
|
||||
...credentialKeys,
|
||||
...(input.verifiedDistribution?.workerCredentialEnv ?? []),
|
||||
]) {
|
||||
const value = processEnv[key];
|
||||
if (value && value.trim().length > 0) {
|
||||
env[key] = value;
|
||||
@@ -281,7 +289,7 @@ function getDeclaredPageRoutePaths(manifest: PaperclipPluginManifestV1): string[
|
||||
export interface PluginLoaderOptions {
|
||||
resolvePackageRoot?: (plugin: PluginRecord) => string;
|
||||
prebuiltOnly?: boolean;
|
||||
/** Image-owned deployment policy, checked before importing code and starting workers. */
|
||||
/** Check image policy and return verified distribution evidence when available. */
|
||||
assertPackageActivation?: (input: {
|
||||
pluginKey?: string;
|
||||
packageRoot: string;
|
||||
@@ -290,7 +298,7 @@ export interface PluginLoaderOptions {
|
||||
manifest?: PaperclipPluginManifestV1;
|
||||
/** Persisted grants, supplied before a runtime manifest refresh is saved. */
|
||||
previousManifest?: PaperclipPluginManifestV1;
|
||||
}) => void;
|
||||
}) => DistributionPlugin | void;
|
||||
/**
|
||||
* Path to the local plugin directory to scan.
|
||||
* Defaults to ~/.paperclip/plugins/
|
||||
@@ -2370,7 +2378,7 @@ export function pluginLoader(
|
||||
assertPackageActivation?.({ pluginKey, packageRoot, installedPackagePath: activePlugin.packagePath });
|
||||
activePlugin = await refreshPluginManifestFromPackage(activePlugin, packageRoot);
|
||||
manifest = activePlugin.manifestJson;
|
||||
assertPackageActivation?.({ pluginKey, packageRoot, installedPackagePath: activePlugin.packagePath, manifest });
|
||||
const verifiedDistribution = assertPackageActivation?.({ pluginKey, packageRoot, installedPackagePath: activePlugin.packagePath, manifest });
|
||||
const workerEntrypoint = resolvePackageRoot
|
||||
? path.resolve(packageRoot, manifest.entrypoints.worker)
|
||||
: resolveWorkerEntrypoint(activePlugin, localPluginDir);
|
||||
@@ -2441,6 +2449,7 @@ export function pluginLoader(
|
||||
manifest,
|
||||
packageName: activePlugin.packageName,
|
||||
packagePath: resolvePackageRoot ? packageRoot : activePlugin.packagePath,
|
||||
verifiedDistribution: resolvePackageRoot && verifiedDistribution ? verifiedDistribution : undefined,
|
||||
instanceInfo,
|
||||
grantedEnvVars: managedPluginWorkerEnvGrant(activePlugin.pluginKey),
|
||||
}),
|
||||
|
||||
Reference in new issue
Block a user