mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 14:16:39 +01:00
feat(cloud): record configuration migration and package qualification
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
9dcf3f9904
commit
c190103824
16 files changed
+1029
-11
No files matched your search
@@ -755,6 +755,27 @@ The focused PostgreSQL suite can use an owned disposable PostgreSQL database via
|
||||
unavailable. It applies migrations and inserts test data, so never point this
|
||||
variable at an existing application database.
|
||||
|
||||
## Cloud configuration migration receipt
|
||||
|
||||
The private `instance_settings` singleton `cloud-configuration-migration:v1`
|
||||
records atomic import completion, semantic source digests, imported revisions,
|
||||
and metadata-only external adapter blockers. It composes the adapter and
|
||||
startup-config owners without replacing an existing authoritative row. Receipt
|
||||
locking and the import transaction handle concurrent attempts and retries; no
|
||||
local completion marker participates in conflict resolution.
|
||||
|
||||
Qualification records current configuration revisions and selected image package
|
||||
evidence before normal plugin reconciliation. Image inspection does not hold
|
||||
admin write locks; configuration and registration snapshots are rechecked in a
|
||||
short transaction before the report is saved. A historical successful report
|
||||
never grants current eligibility. External metadata blockers survive loss of the
|
||||
legacy home and require a deliberate revision-checked recheck of a present valid
|
||||
original source file. Package mismatches remain recorded after maintenance
|
||||
reconciliation. See the [bounded migration and qualification procedure](cloud-configuration-migration.md)
|
||||
for the stopped-writer prerequisite, commands, retry semantics, integrity limits,
|
||||
and the explicit exclusion of secret bytes, whole-server zero downtime, and
|
||||
volume removal.
|
||||
|
||||
## Cloud runtime identity singleton
|
||||
|
||||
The private `instance_settings` row whose singleton key is
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# Cloud configuration migration and package qualification
|
||||
|
||||
This procedure covers saved adapter preferences, the allowlisted nonsecret
|
||||
startup configuration, and selected bundled plugin packages. The result is a
|
||||
point-in-time report. It does not authorize a deployment, guarantee zero
|
||||
downtime, establish general Railway admission, or prove that a volume can be
|
||||
removed. Secret and key bytes, database credentials, artifacts, logs, local
|
||||
storage, workspaces, and plugin state outside PostgreSQL need their own storage
|
||||
and replacement checks. This includes the decision signing key and local
|
||||
encryption keys. Their byte authority is unchanged by this procedure.
|
||||
|
||||
## Migration prerequisite
|
||||
|
||||
Stop every legacy file writer before the first import. This includes old
|
||||
application replicas and operator commands that can write configuration or
|
||||
adapter settings. One maintenance window is sufficient. The importer cannot
|
||||
make an old file writer participate in its database transaction. Keep those
|
||||
writers stopped after import; later file edits do not update database authority.
|
||||
|
||||
Keep the normal database backup and recovery procedure for the instance. Use the
|
||||
new release with the same managed environment and database connection. The
|
||||
server runs migration before constructing configuration-dependent services.
|
||||
For an explicit source-checkout invocation, use:
|
||||
|
||||
```sh
|
||||
pnpm exec tsx server/src/qualify-cloud-configuration.ts --migrate
|
||||
```
|
||||
|
||||
The command uses `DATABASE_URL` and the normal Cloud environment. It does not
|
||||
accept credentials as arguments. It writes the receipt to PostgreSQL and prints
|
||||
a JSON report; no report file or local completion marker is required. Exit code
|
||||
0 means the requested configuration/package check passed, 2 means the report has
|
||||
blockers, and 1 means the operation failed. A fresh installation can report a
|
||||
missing plugin registration before normal boot installation completes. Run the
|
||||
check again after that installation settles.
|
||||
|
||||
## Database authority and retry behavior
|
||||
|
||||
The private `instance_settings` singleton `cloud-configuration-migration:v1`
|
||||
contains the versioned receipt. The transaction locks that row and composes the
|
||||
adapter and startup-config owners' atomic seed methods. Completion and both
|
||||
seeds commit together. A failed write or crash before commit leaves no completed
|
||||
receipt. A retry uses committed database state and never replaces a later
|
||||
settings revision. Earlier independently seeded owner records remain authoritative.
|
||||
|
||||
The receipt records source kind, a digest of the nonsecret semantic candidate,
|
||||
the imported revision, and bounded legacy external-registration evidence.
|
||||
An absent startup configuration remains an authoritative `config: null`; it is
|
||||
distinct from an uninitialized database record. No raw secret values are copied.
|
||||
Once the receipt exists, ordinary boots do not reread legacy source files.
|
||||
A replacement home can be empty or absent without changing saved authority.
|
||||
|
||||
All legacy readers have a 64 KiB limit. They require regular files, reject
|
||||
symlinks and FIFOs, and use bounded reads. External registration inspection reads
|
||||
metadata only. It never loads a registered module or follows a registered package
|
||||
path. A nonempty, invalid, or unreadable registration file creates a durable
|
||||
blocker. Removing the old home does not clear that blocker.
|
||||
|
||||
To resolve an external-registration blocker, stop its writer, deliberately
|
||||
remove the unsupported registration metadata at the original source location,
|
||||
and leave a present valid empty JSON array file. Read the current receipt revision
|
||||
from the report, then run:
|
||||
|
||||
```sh
|
||||
pnpm exec tsx server/src/qualify-cloud-configuration.ts --recheck-external-revision 3
|
||||
```
|
||||
|
||||
Use the actual current revision in place of `3`. A changed revision, a different
|
||||
source location, or a missing/invalid source file rejects the recheck. Reusing
|
||||
the same container path with an empty home is insufficient. This is an explicit
|
||||
operator resolution; qualification never clears the blocker automatically.
|
||||
|
||||
## Current packages and overlap comparison
|
||||
|
||||
Every qualification reads current settings revisions and plugin registrations.
|
||||
It resolves selected packages from this image with the same bundled package
|
||||
identity rules as normal loading. It verifies package bytes before importing a
|
||||
trusted image manifest, then compares that manifest with the approved database
|
||||
registration. Missing packages, missing evidence, incompatible versions,
|
||||
unselected registrations, and integrity failures prevent qualification. Legacy
|
||||
package paths and npm discovery cannot supply missing image bytes.
|
||||
|
||||
Image inspection happens outside database write locks. A short transaction then
|
||||
locks and rereads the current configuration and registration records. A change
|
||||
during inspection produces a blocker and requires a new check. The report gives
|
||||
`adapterRevision`, `configRevision`, `configurationDigest`, and
|
||||
`packageEvidenceDigest`. A historical completed receipt is evidence, not current
|
||||
authorization. Call qualification again for a current result.
|
||||
|
||||
The server records this report before ordinary bundled-plugin reconciliation.
|
||||
It retains `lastPackageMismatch` after later successful checks, so reconciliation
|
||||
cannot erase the evidence seen before an upgrade. The report does not block
|
||||
normal maintenance upgrades or replace capability approval. After an upgrade
|
||||
settles, its current packages can qualify normally.
|
||||
|
||||
For a same-package comparison, capture the current release's
|
||||
`packageEvidenceDigest` and pass it explicitly on the replacement:
|
||||
|
||||
```sh
|
||||
pnpm exec tsx server/src/qualify-cloud-configuration.ts --expected-package-evidence sha256:REPLACE_WITH_THE_64_HEX_DIGEST
|
||||
```
|
||||
|
||||
`configurationPackagesEligible` describes current database/configuration/package
|
||||
compatibility. `samePackageOverlapEligible` is true only when an explicit expected
|
||||
digest also matches. There is no permanent first-release baseline that would
|
||||
prevent future maintenance releases from establishing their own evidence.
|
||||
|
||||
The package evidence digest covers selected packages, not container image bytes.
|
||||
Distribution packages reuse and reverify their catalog digest. Builtin packages
|
||||
have a bounded inventory of physical file bytes, modes, and contained symlink
|
||||
text, including standalone pnpm dependency trees. Dangling or escaping links,
|
||||
special files, more than 10,000 entries, depth above 32, or more than 256 MiB per
|
||||
package fail verification. Inventory work is synchronous and linear in selected
|
||||
package bytes. It runs only during startup before the application is constructed
|
||||
or in the explicit operator process, with no idle polling. The report includes
|
||||
`packageInspectionDurationMs`. Packages must remain immutable throughout a check.
|
||||
|
||||
## Verification scope
|
||||
|
||||
The focused PostgreSQL tests use two independent clients and separate home/image
|
||||
roots. They import legacy settings, change saved revisions, remove the old roots,
|
||||
reconstruct from the database, and start a real plugin worker from the replacement
|
||||
image fixture. They also cover concurrent migration, commit failure and retry,
|
||||
database failure, stale sources, durable external blockers, package mismatch,
|
||||
maintenance reconciliation, and changes during manifest inspection.
|
||||
|
||||
Set `PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL` only to an owned disposable
|
||||
PostgreSQL database when embedded PostgreSQL is unavailable. The suite applies
|
||||
migrations and writes/deletes test records. A complete release image, hosted
|
||||
replacement, overlapping production processes, secret-byte migration, and volume
|
||||
removal are not established by these local fixtures.
|
||||
@@ -112,6 +112,17 @@ exercises persistence, concurrent writers, admission freshness, activity audit,
|
||||
and failed writes; [authorization tests](../../server/src/__tests__/adapter-routes-authz.test.ts)
|
||||
cover admin and hidden-setting gates. These are local tests, not deployed proof.
|
||||
|
||||
Cloud configuration migration and package qualification are operator commands,
|
||||
not agent-accessible REST capabilities. They require the managed environment and
|
||||
database operator access. The [command](../../server/src/qualify-cloud-configuration.ts)
|
||||
composes transaction-safe owner seeds and writes a private receipt. Every
|
||||
qualification rereads current revisions and image package evidence; a completed
|
||||
historical receipt grants no deployment or execution authority. [PostgreSQL coverage](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts)
|
||||
uses independent clients, lost-home reconstruction, a real replacement fixture
|
||||
worker, rollback/retry, and durable blockers. [Reader coverage](../../server/src/__tests__/cloud-configuration-readers.test.ts)
|
||||
checks bounded sources and pnpm-style link integrity. These are local fixtures;
|
||||
see the [scope and operator procedure](../../doc/cloud-configuration-migration.md).
|
||||
|
||||
Cloud stack Admins may update experimental settings through the trusted Cloud
|
||||
actor without receiving other instance-admin powers. This uses the current
|
||||
stack role on each request; company membership alone does not grant it.
|
||||
|
||||
@@ -75,6 +75,16 @@ These controls do not qualify a deployed Cloud rollout or durable file storage.
|
||||
|
||||
Automated: [experimental settings](../../ui/src/pages/InstanceExperimentalSettings.test.tsx) and [database backup routes](../../server/src/__tests__/instance-database-backups-routes.test.ts) cover their UI/API contracts.
|
||||
|
||||
Cloud migration: in one maintenance window, stop legacy file writers and run the
|
||||
[configuration/package qualification command](../../doc/cloud-configuration-migration.md).
|
||||
Inspect its database receipt, update a saved preference, then repeat against a
|
||||
second owned application/image root with the first home unavailable. Use an
|
||||
explicit package evidence digest for same-package comparison. Confirm stale
|
||||
qualification and missing legacy files cannot clear blockers. This recipe covers
|
||||
configuration/packages and does not qualify whole-server zero downtime, secret
|
||||
bytes, or volume removal. [Local PostgreSQL evidence path](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts)
|
||||
includes a real fixture worker; it does not establish a hosted replacement.
|
||||
|
||||
Managed adapters: change a visible adapter preference as an instance admin,
|
||||
then use a second application process against the same disposable database to
|
||||
confirm catalog and new-agent selection reflect it. Verify saved agents remain
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { applyPendingMigrations, closeRegisteredClients, createDb, instanceSettings, plugins, type Db } from "@paperclipai/db";
|
||||
import { eq, inArray, sql } from "drizzle-orm";
|
||||
import { paperclipConfigSchema } from "@paperclipai/shared";
|
||||
import { adapterSettingsService, initializeAdapterSettings } from "../services/adapter-settings.js";
|
||||
import { cloudRuntimeConfigService, initializeCloudRuntimeConfig } from "../services/cloud-runtime-config.js";
|
||||
import { cloudConfigurationMigrationService } from "../services/cloud-configuration-migration.js";
|
||||
import { qualifyCloudConfigurationForCurrentImage } from "../services/cloud-configuration-qualification.js";
|
||||
import { runCloudConfigurationQualification } from "../qualify-cloud-configuration.js";
|
||||
import { pluginLoader, type PluginRuntimeServices } from "../services/plugin-loader.js";
|
||||
import { pluginRegistryService } from "../services/plugin-registry.js";
|
||||
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { ensureBundledPlugins } from "../services/bundled-plugins.js";
|
||||
import { bundledImageFixture } from "./helpers/bundled-image-fixture.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const externalDatabaseUrl = process.env.PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL?.trim();
|
||||
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configuration-migration:v1"];
|
||||
(support.supported ? describe : describe.skip)("Cloud configuration migration and current-image qualification", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
let peer: Db;
|
||||
let homeA: string;
|
||||
let homeB: string;
|
||||
const roots: string[] = [];
|
||||
const fixture = (version = "1.0.0") => { const image = bundledImageFixture({ version }); roots.push(image.root); return image; };
|
||||
const env = (home = homeA) => ({ PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "test-cloud-token", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: path.join(home, "config.json") });
|
||||
const migration = (client = db, home = homeA) => cloudConfigurationMigrationService(client, { runtimeEnv: env(home) });
|
||||
beforeAll(async () => {
|
||||
if (externalDatabaseUrl) {
|
||||
await applyPendingMigrations(externalDatabaseUrl);
|
||||
database = { connectionString: externalDatabaseUrl, cleanup: () => closeRegisteredClients(externalDatabaseUrl) };
|
||||
} else database = await startEmbeddedPostgresTestDatabase("cloud-configuration-migration-");
|
||||
db = createDb(database.connectionString);
|
||||
peer = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
beforeEach(async () => {
|
||||
homeA = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-legacy-A-"));
|
||||
homeB = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-replacement-B-"));
|
||||
roots.push(homeA, homeB);
|
||||
await db.delete(instanceSettings).where(inArray(instanceSettings.singletonKey, keys));
|
||||
await db.delete(plugins);
|
||||
});
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
for (const root of roots.splice(0)) await fs.rm(root, { recursive: true, force: true });
|
||||
});
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
|
||||
async function legacy() {
|
||||
const config = paperclipConfigSchema.parse({
|
||||
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
|
||||
database: { mode: "postgres" }, logging: { mode: "cloud" },
|
||||
server: { port: 3199 }, telemetry: { enabled: false },
|
||||
});
|
||||
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
|
||||
await fs.writeFile(path.join(homeA, "adapter-settings.json"), JSON.stringify({ disabledTypes: ["claude_local", "future_adapter"] }));
|
||||
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]");
|
||||
return config;
|
||||
}
|
||||
|
||||
function loaderFor(image: ReturnType<typeof fixture>, client = db, services?: PluginRuntimeServices) {
|
||||
return pluginLoader(client, {
|
||||
localPluginDir: path.join(homeB, "plugins"), prebuiltOnly: true,
|
||||
resolvePackageRoot: image.resolver.resolve, assertPackageActivation: image.resolver.assertActivation,
|
||||
enableLocalFilesystem: false, enableNpmDiscovery: false,
|
||||
}, services);
|
||||
}
|
||||
async function install(image: ReturnType<typeof fixture>) {
|
||||
await loaderFor(image).installPlugin({ localPath: image.packageRoot });
|
||||
const registry = pluginRegistryService(db);
|
||||
const row = (await registry.getByKey(image.manifest.id))!;
|
||||
await registry.updateStatus(row.id, { status: "ready" });
|
||||
return row;
|
||||
}
|
||||
function qualify(image: ReturnType<typeof fixture>, client = db, home = homeA, expectedPackageEvidenceDigest?: string) {
|
||||
const loader = loaderFor(image, client);
|
||||
return migration(client, home).qualify({ installs: image.installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest });
|
||||
}
|
||||
|
||||
it("imports A atomically, retains later DB revisions, and starts B's real image worker after A disappears", async () => {
|
||||
await legacy();
|
||||
const oldImage = fixture();
|
||||
const newImage = fixture();
|
||||
const row = await install(oldImage);
|
||||
const first = await migration().migrate();
|
||||
expect(first.migration.adapter.authority).toBe("legacy_file");
|
||||
expect(first.migration.config.authority).toBe("legacy_file");
|
||||
const original = await qualify(oldImage);
|
||||
expect(original.configurationPackagesEligible).toBe(true);
|
||||
expect(original.samePackageOverlapEligible).toBe(false);
|
||||
await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true);
|
||||
const config = (await cloudRuntimeConfigService(peer).inspectStored())!;
|
||||
expect(config.config).not.toBeNull();
|
||||
await cloudRuntimeConfigService(peer).replace({ ...config.config!, server: { ...config.config!.server, port: 3299 } }, config.revision);
|
||||
await fs.rm(homeA, { recursive: true });
|
||||
await fs.rm(oldImage.root, { recursive: true });
|
||||
expect((await migration(peer, homeB).migrate()).migration).toEqual(first.migration);
|
||||
await initializeAdapterSettings(peer, { runtimeEnv: env(homeB) });
|
||||
const restored = await initializeCloudRuntimeConfig(peer, { runtimeEnv: env(homeB) });
|
||||
expect(restored?.current.config?.server.port).toBe(3299);
|
||||
expect(await adapterSettingsService(peer, { runtimeEnv: env(homeB) }).getDisabledTypes()).toEqual(["claude_local", "codex_local", "future_adapter"]);
|
||||
const replacement = await qualify(newImage, peer, homeB, original.packageEvidenceDigest);
|
||||
expect(replacement).toMatchObject({ adapterRevision: 1, configRevision: 1, configurationPackagesEligible: true, samePackageOverlapEligible: true });
|
||||
expect(replacement.configurationDigest).not.toBe(original.configurationDigest);
|
||||
const manager = createPluginWorkerManager();
|
||||
try {
|
||||
const services = {
|
||||
workerManager: manager, lifecycleManager: { markError: vi.fn() },
|
||||
jobStore: { syncJobDeclarations: vi.fn() }, jobScheduler: { registerPlugin: vi.fn() },
|
||||
toolDispatcher: {}, eventBus: { forPlugin: vi.fn(), subscriptionCount: vi.fn(() => 0) },
|
||||
buildHostHandlers: vi.fn(() => ({})), instanceInfo: { instanceId: "qualification-fixture", hostVersion: "1.0.0" },
|
||||
} as unknown as PluginRuntimeServices;
|
||||
expect((await loaderFor(newImage, peer, services).loadSingle(row.id)).success).toBe(true);
|
||||
expect(await manager.call(row.id, "health", {})).toMatchObject({ status: "ok", details: { source: expect.stringContaining(newImage.packageRoot) } });
|
||||
} finally { await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => {
|
||||
await legacy();
|
||||
const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]);
|
||||
expect(first).toEqual(second);
|
||||
await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("claude_local", false);
|
||||
await fs.writeFile(path.join(homeA, "config.json"), "invalid stale file");
|
||||
await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid stale file");
|
||||
expect(await migration().migrate()).toEqual(first);
|
||||
expect(await adapterSettingsService(db, { runtimeEnv: env() }).getDisabledTypes()).toEqual(["future_adapter"]);
|
||||
});
|
||||
|
||||
it("rolls back both seeds and the receipt if completion fails, then retries after the crash boundary", async () => {
|
||||
await legacy();
|
||||
await db.execute(sql`CREATE FUNCTION reject_cloud_migration_receipt() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.singleton_key = 'cloud-configuration-migration:v1' THEN RAISE EXCEPTION 'test receipt failure'; END IF; RETURN NEW; END $$`);
|
||||
await db.execute(sql`CREATE TRIGGER reject_cloud_migration_receipt BEFORE UPDATE ON instance_settings FOR EACH ROW EXECUTE FUNCTION reject_cloud_migration_receipt()`);
|
||||
try {
|
||||
await expect(migration().migrate()).rejects.toMatchObject({ cause: { message: expect.stringContaining("test receipt failure") } });
|
||||
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
|
||||
} finally {
|
||||
await db.execute(sql`DROP TRIGGER reject_cloud_migration_receipt ON instance_settings`);
|
||||
await db.execute(sql`DROP FUNCTION reject_cloud_migration_receipt()`);
|
||||
}
|
||||
expect((await migration(peer).migrate()).revision).toBe(0);
|
||||
expect((await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).length).toBe(3);
|
||||
});
|
||||
|
||||
it("fails without partial authority for malformed legacy config and preserves already-authoritative rows", async () => {
|
||||
await legacy();
|
||||
await fs.writeFile(path.join(homeA, "config.json"), "invalid");
|
||||
await expect(migration().migrate()).rejects.toThrow("Legacy Cloud runtime configuration");
|
||||
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
|
||||
await adapterSettingsService(db, { runtimeEnv: env() }).seedIfAbsent({ disabledTypes: ["codex_local"] });
|
||||
await cloudRuntimeConfigService(db).seedIfAbsent(null);
|
||||
await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid");
|
||||
const receipt = await migration(peer).migrate();
|
||||
expect(receipt.migration.adapter.authority).toBe("database");
|
||||
expect(receipt.migration.config.authority).toBe("database");
|
||||
expect((await cloudRuntimeConfigService(peer).inspectStored())?.config).toBeNull();
|
||||
});
|
||||
|
||||
it("returns database errors instead of falling back to files or a completed receipt", async () => {
|
||||
await legacy();
|
||||
await migration().migrate();
|
||||
const broken = createDb(database.connectionString);
|
||||
await broken.$client.end({ timeout: 0 });
|
||||
await expect(migration(broken).migrate()).rejects.toThrow();
|
||||
const image = fixture();
|
||||
await expect(qualify(image, broken)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("keeps external registration blockers durable when B has no legacy home and requires deliberate original-source resolution", async () => {
|
||||
await legacy();
|
||||
const sentinel = path.join(homeA, "external-code-executed");
|
||||
await fs.writeFile(path.join(homeA, "external.js"), `require('node:fs').writeFileSync(${JSON.stringify(sentinel)}, 'executed');`);
|
||||
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), JSON.stringify([{ type: "legacy_external", packageName: "legacy-adapter", localPath: path.join(homeA, "external.js") }]));
|
||||
const image = fixture();
|
||||
await install(image);
|
||||
await migration().migrate();
|
||||
expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "unsupported_external_adapters" }));
|
||||
await expect(fs.stat(sentinel)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await migration(peer, homeB).migrate();
|
||||
const replacement = await qualify(image, peer, homeB);
|
||||
expect(replacement.configurationPackagesEligible).toBe(false);
|
||||
await expect(migration(peer, homeB).recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("original legacy source");
|
||||
await fs.unlink(path.join(homeA, "adapter-plugins.json"));
|
||||
await expect(migration().recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("present, valid registration file");
|
||||
expect((await migration().inspectStored())?.migration.externalRegistrations.blocker).toBe("unsupported_external_adapters");
|
||||
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]");
|
||||
await expect(migration().recheckLegacyExternalRegistrations(0)).rejects.toThrow("revision conflict");
|
||||
await migration().recheckLegacyExternalRegistrations(replacement.receiptRevision);
|
||||
expect((await qualify(image)).configurationPackagesEligible).toBe(true);
|
||||
});
|
||||
|
||||
it("retains a metadata blocker for malformed or oversized registration files", async () => {
|
||||
await legacy();
|
||||
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), Buffer.alloc(65_537));
|
||||
const receipt = await migration().migrate();
|
||||
expect(receipt.migration.externalRegistrations.blocker).toBe("invalid_external_registration_metadata");
|
||||
await fs.rm(homeA, { recursive: true });
|
||||
expect((await migration(peer, homeB).migrate()).migration.externalRegistrations).toEqual(receipt.migration.externalRegistrations);
|
||||
});
|
||||
|
||||
it("records image mismatches before reconciliation, permits maintenance upgrades, and requires explicit fresh overlap evidence", async () => {
|
||||
await legacy();
|
||||
const oldImage = fixture();
|
||||
const nextImage = fixture("1.1.0");
|
||||
await install(oldImage);
|
||||
await migration().migrate();
|
||||
const old = await qualify(oldImage);
|
||||
const mismatch = await qualify(nextImage, peer, homeB, old.packageEvidenceDigest);
|
||||
expect(mismatch.configurationPackagesEligible).toBe(false);
|
||||
expect(mismatch.samePackageOverlapEligible).toBe(false);
|
||||
expect(mismatch.blockers).toContainEqual(expect.objectContaining({ code: "incompatible_bundled_package", registeredVersion: "1.0.0", imageVersion: "1.1.0" }));
|
||||
const registry = pluginRegistryService(db);
|
||||
await ensureBundledPlugins(nextImage.installs, {
|
||||
registry, loader: loaderFor(nextImage), lifecycle: { load: vi.fn() },
|
||||
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
|
||||
}, { reinstallUninstalled: true, preservePackagePath: true });
|
||||
const current = await qualify(nextImage);
|
||||
expect(current.configurationPackagesEligible).toBe(true);
|
||||
expect(current.samePackageOverlapEligible).toBe(false);
|
||||
expect((await migration().inspectStored())?.lastPackageMismatch?.packageEvidenceDigest).toBe(mismatch.packageEvidenceDigest);
|
||||
expect((await qualify(nextImage, peer, homeB, current.packageEvidenceDigest)).samePackageOverlapEligible).toBe(true);
|
||||
});
|
||||
|
||||
it("fails qualification for missing package or manifest bytes without loading a home fallback", async () => {
|
||||
await legacy();
|
||||
const image = fixture();
|
||||
await install(image);
|
||||
await migration().migrate();
|
||||
const good = await qualify(image);
|
||||
await fs.rm(path.join(image.packageRoot, "dist", "manifest.js"));
|
||||
const broken = await qualify(image, peer, homeB, good.packageEvidenceDigest);
|
||||
expect(broken.configurationPackagesEligible).toBe(false);
|
||||
expect(broken.samePackageOverlapEligible).toBe(false);
|
||||
expect(broken.blockers.length).toBeGreaterThan(0);
|
||||
await fs.rm(image.packageRoot, { recursive: true });
|
||||
expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "missing_bundled_package" }));
|
||||
});
|
||||
|
||||
it("rejects changed image bytes before manifest import and never uses stale completed qualification", async () => {
|
||||
await legacy();
|
||||
const image = fixture();
|
||||
await install(image);
|
||||
await migration().migrate();
|
||||
const original = await qualify(image);
|
||||
await fs.appendFile(path.join(image.packageRoot, "dist", "worker.js"), "\nprocess.exit(99);\n");
|
||||
const loadManifest = vi.fn();
|
||||
const current = await migration(peer, homeB).qualify({ installs: image.installs, loadManifest, expectedPackageEvidenceDigest: original.packageEvidenceDigest });
|
||||
expect(current.configurationPackagesEligible).toBe(false);
|
||||
expect(current.samePackageOverlapEligible).toBe(false);
|
||||
expect(current.blockers).toContainEqual(expect.objectContaining({ code: "package_integrity_mismatch" }));
|
||||
expect(loadManifest).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows an administrator update while manifest inspection waits and refuses to certify the stale snapshot", async () => {
|
||||
await legacy();
|
||||
const image = fixture();
|
||||
await install(image);
|
||||
await migration().migrate();
|
||||
let started!: () => void;
|
||||
let resume!: () => void;
|
||||
const entered = new Promise<void>((resolve) => { started = resolve; });
|
||||
const release = new Promise<void>((resolve) => { resume = resolve; });
|
||||
const loader = loaderFor(image);
|
||||
const pending = migration().qualify({ installs: image.installs, loadManifest: async (root) => { started(); await release; return loader.loadManifest(root); } });
|
||||
await entered;
|
||||
try {
|
||||
const change = adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true);
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
expect(await Promise.race([change, new Promise<never>((_resolve, reject) => { timer = setTimeout(() => reject(new Error("Admin update was blocked by image inspection")), 2_000); })])).toBe(true);
|
||||
} finally { clearTimeout(timer); }
|
||||
} finally { resume(); }
|
||||
const result = await pending;
|
||||
expect(result).toMatchObject({ configurationPackagesEligible: false, adapterRevision: 1 });
|
||||
expect(result.blockers).toContainEqual(expect.objectContaining({ code: "configuration_changed_during_inspection" }));
|
||||
expect((await qualify(image)).configurationPackagesEligible).toBe(true);
|
||||
});
|
||||
|
||||
it("uses the current managed catalog through the operator command and durably records catalog failure", async () => {
|
||||
await legacy();
|
||||
const image = fixture();
|
||||
await install(image);
|
||||
await migration().migrate();
|
||||
const runtimeEnv = {
|
||||
...env(homeB), DATABASE_URL: database.connectionString, PAPERCLIP_BUNDLED_PLUGIN_ROOT: image.root,
|
||||
PAPERCLIP_MANAGED_CONFIG: JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "fixture", features: {}, plugins: { autoInstall: ["fixture"] } }),
|
||||
};
|
||||
const initial = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv });
|
||||
expect(initial?.configurationPackagesEligible).toBe(true);
|
||||
const stdout = vi.spyOn(process.stdout, "write").mockImplementation(() => true);
|
||||
try {
|
||||
expect(await runCloudConfigurationQualification(["--expected-package-evidence", initial!.packageEvidenceDigest], runtimeEnv)).toBe(0);
|
||||
const report = stdout.mock.calls.map((call) => String(call[0])).find((text) => text.startsWith('{\n "receiptRevision"'));
|
||||
expect(report && JSON.parse(report).samePackageOverlapEligible).toBe(true);
|
||||
} finally { stdout.mockRestore(); }
|
||||
const missing = await qualifyCloudConfigurationForCurrentImage(peer, { runtimeEnv: { ...runtimeEnv, PAPERCLIP_MANAGED_CONFIG: runtimeEnv.PAPERCLIP_MANAGED_CONFIG.replace('"fixture"]', '"absent"]') } });
|
||||
expect(missing?.configurationPackagesEligible).toBe(false);
|
||||
expect(missing?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" }));
|
||||
expect((await migration().inspectStored())?.lastPackageMismatch?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" }));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,48 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { readLegacyAdapterSettings } from "../services/adapter-settings.js";
|
||||
import { inspectLegacyExternalRegistrations } from "../services/cloud-configuration-evidence.js";
|
||||
import { cloudBundledPackageDigest } from "../services/cloud-bundled-package-digest.js";
|
||||
|
||||
vi.mock("../adapters/plugin-loader.js", () => ({ buildExternalAdapters: async () => [] }));
|
||||
|
||||
describe("bounded Cloud legacy readers and image inventory", () => {
|
||||
let root: string;
|
||||
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-config-reader-")); });
|
||||
afterEach(async () => { await fs.rm(root, { recursive: true, force: true }); });
|
||||
|
||||
it("distinguishes an absent adapter file from a saved empty preference", async () => {
|
||||
expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toBeNull();
|
||||
await fs.writeFile(path.join(root, "adapter-settings.json"), '{"disabledTypes":[]}');
|
||||
expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toEqual({ disabledTypes: [] });
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")("rejects symlinks and FIFOs without waiting for a writer", async () => {
|
||||
const target = path.join(root, "target.json");
|
||||
const settings = path.join(root, "adapter-settings.json");
|
||||
await fs.writeFile(target, '{"disabledTypes":[]}');
|
||||
await fs.symlink(target, settings);
|
||||
await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toMatchObject({ code: "ELOOP" });
|
||||
expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("unreadable_external_registration_metadata");
|
||||
await fs.unlink(settings);
|
||||
execFileSync("mkfifo", [settings]);
|
||||
await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toThrow("regular file");
|
||||
expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("invalid_external_registration_metadata");
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")("hashes pnpm-style contained links and the physical dependency bytes", async () => {
|
||||
const dependency = path.join(root, "node_modules", ".pnpm", "dep@1.0.0", "node_modules", "dep");
|
||||
await fs.mkdir(dependency, { recursive: true });
|
||||
await fs.writeFile(path.join(dependency, "index.js"), "export const value = 1;");
|
||||
await fs.symlink(path.relative(path.join(root, "node_modules"), dependency), path.join(root, "node_modules", "dep"));
|
||||
const first = cloudBundledPackageDigest(root);
|
||||
await fs.writeFile(path.join(dependency, "index.js"), "export const value = 2;");
|
||||
expect(cloudBundledPackageDigest(root)).not.toBe(first);
|
||||
await fs.unlink(path.join(root, "node_modules", "dep"));
|
||||
await fs.symlink("/", path.join(root, "node_modules", "dep"));
|
||||
expect(() => cloudBundledPackageDigest(root)).toThrow("escapes its package");
|
||||
});
|
||||
});
|
||||
@@ -31,6 +31,9 @@ vi.mock("../services/adapter-settings.js", async () => ({
|
||||
initializeAdapterSettings: vi.fn(async () => null),
|
||||
}));
|
||||
vi.mock("../services/cloud-runtime-config.js", () => ({ initializeCloudRuntimeConfig: cloudRuntimeStartupMocks.initialize }));
|
||||
const cloudMigrationMocks = vi.hoisted(() => ({ migrate: vi.fn(async () => null), qualify: vi.fn(async () => null) }));
|
||||
vi.mock("../services/cloud-configuration-migration.js", () => ({ migrateCloudConfiguration: cloudMigrationMocks.migrate }));
|
||||
vi.mock("../services/cloud-configuration-qualification.js", () => ({ qualifyCloudConfigurationForCurrentImage: cloudMigrationMocks.qualify }));
|
||||
vi.mock("../telemetry.js", () => ({ initTelemetry: cloudRuntimeStartupMocks.telemetry, getTelemetryClient: vi.fn(() => null) }));
|
||||
|
||||
const ORIGINAL_PAPERCLIP_API_URL = process.env.PAPERCLIP_API_URL;
|
||||
@@ -627,6 +630,7 @@ describe("startServer feedback export wiring", () => {
|
||||
const starting = startServer();
|
||||
for (let index = 0; index < 30 && !release; index += 1) await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(release).toBeTypeOf("function");
|
||||
expect(cloudMigrationMocks.migrate.mock.invocationCallOrder[0]).toBeLessThan(cloudRuntimeStartupMocks.initialize.mock.invocationCallOrder[0]!);
|
||||
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
|
||||
expect(createBetterAuthInstanceMock).not.toHaveBeenCalled();
|
||||
expect(createAppMock).not.toHaveBeenCalled();
|
||||
@@ -656,6 +660,14 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("stops before configuration consumers when the atomic Cloud migration fails", async () => {
|
||||
cloudMigrationMocks.migrate.mockRejectedValueOnce(new Error("migration did not commit"));
|
||||
await expect(startServer()).rejects.toThrow("migration did not commit");
|
||||
expect(cloudRuntimeStartupMocks.initialize).not.toHaveBeenCalled();
|
||||
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
|
||||
expect(createAppMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
|
||||
@@ -188,6 +188,8 @@ import {
|
||||
} from "./shutdown.js";
|
||||
import { initializeCloudRuntimeIdentity } from "./services/cloud-runtime-identity.js";
|
||||
import { initializeAdapterSettings } from "./services/adapter-settings.js";
|
||||
import { migrateCloudConfiguration } from "./services/cloud-configuration-migration.js";
|
||||
import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js";
|
||||
import { systemdNotify } from "./services/systemd-notify.js";
|
||||
import { flushInFlightRunLogMirrors } from "./services/run-log-store.js";
|
||||
import {
|
||||
@@ -719,6 +721,7 @@ async function startServerWithDatabaseTeardown(
|
||||
};
|
||||
startupDatabase.close = closeDatabaseClients;
|
||||
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
|
||||
await migrateCloudConfiguration(db as any);
|
||||
await initializeAdapterSettings(db as any);
|
||||
|
||||
await initializeCloudRuntimeConfig(db as any);
|
||||
@@ -977,6 +980,10 @@ async function startServerWithDatabaseTeardown(
|
||||
// document parsed fail-closed above (`plugins.autoInstall`). Absent env means
|
||||
// self-hosted: createApp falls back to its built-in kubernetes-only default.
|
||||
const managedPluginAutoInstall = managedConfig?.plugins.autoInstall ?? null;
|
||||
const cloudQualification = await qualifyCloudConfigurationForCurrentImage(db as any);
|
||||
if (cloudQualification && !cloudQualification.configurationPackagesEligible) {
|
||||
logger.warn({ scope: cloudQualification.scope, blockerCodes: cloudQualification.blockers.map((blocker) => blocker.code) }, "Cloud configuration and package qualification has blockers");
|
||||
}
|
||||
const app = await createApp(db as any, {
|
||||
prepareIdleDatabaseBackup: idleDatabaseBackupEnabled
|
||||
? async () => (await runServerDatabaseBackup("idle")) !== null
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createDb } from "@paperclipai/db";
|
||||
import { cloudConfigurationMigrationService } from "./services/cloud-configuration-migration.js";
|
||||
import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js";
|
||||
|
||||
export async function runCloudConfigurationQualification(args: string[], env: NodeJS.ProcessEnv = process.env): Promise<number> {
|
||||
const allowed = new Set(["--migrate", "--expected-package-evidence", "--recheck-external-revision"]);
|
||||
let migrate = false;
|
||||
let expectedPackageEvidenceDigest: string | undefined;
|
||||
let recheckRevision: number | undefined;
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const argument = args[index];
|
||||
if (!allowed.has(argument)) throw new Error(`Unknown qualification argument: ${argument}`);
|
||||
if (argument === "--migrate") migrate = true;
|
||||
if (argument === "--expected-package-evidence") {
|
||||
expectedPackageEvidenceDigest = args[++index];
|
||||
if (!expectedPackageEvidenceDigest || !/^sha256:[a-f0-9]{64}$/.test(expectedPackageEvidenceDigest)) throw new Error("A SHA-256 package evidence digest is required");
|
||||
}
|
||||
if (argument === "--recheck-external-revision") {
|
||||
const value = args[++index];
|
||||
if (!value || !/^\d+$/.test(value) || !Number.isSafeInteger(Number(value))) throw new Error("A nonnegative receipt revision is required");
|
||||
recheckRevision = Number(value);
|
||||
}
|
||||
}
|
||||
const url = env.DATABASE_URL?.trim();
|
||||
if (!url) throw new Error("DATABASE_URL is required for Cloud configuration qualification");
|
||||
const db = createDb(url);
|
||||
try {
|
||||
const service = cloudConfigurationMigrationService(db, { runtimeEnv: env });
|
||||
if (migrate) await service.migrate();
|
||||
if (recheckRevision !== undefined) await service.recheckLegacyExternalRegistrations(recheckRevision);
|
||||
const report = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv: env, expectedPackageEvidenceDigest });
|
||||
if (!report) throw new Error("Cloud configuration qualification requires a managed instance");
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
return report.configurationPackagesEligible && (!expectedPackageEvidenceDigest || report.samePackageOverlapEligible) ? 0 : 2;
|
||||
} finally {
|
||||
await db.$client.end({ timeout: 5 });
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
||||
runCloudConfigurationQualification(process.argv.slice(2)).then((code) => { process.exitCode = code; }).catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : "Cloud configuration qualification failed"}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -45,8 +45,8 @@ interface AdapterSettings {
|
||||
// Paths
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function adapterPluginPaths() {
|
||||
const paperclipDir = resolvePaperclipHomeDir();
|
||||
function adapterPluginPaths(homeOverride?: string) {
|
||||
const paperclipDir = resolvePaperclipHomeDir(homeOverride);
|
||||
return {
|
||||
adapterPluginsDir: path.join(paperclipDir, "adapter-plugins"),
|
||||
adapterPluginsStorePath: path.join(paperclipDir, "adapter-plugins.json"),
|
||||
@@ -133,8 +133,8 @@ function writeSettings(settings: AdapterSettings): void {
|
||||
// Public API
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function getAdapterPluginStorePath(): string {
|
||||
return adapterPluginPaths().adapterPluginsStorePath;
|
||||
export function getAdapterPluginStorePath(homeOverride?: string): string {
|
||||
return adapterPluginPaths(homeOverride).adapterPluginsStorePath;
|
||||
}
|
||||
|
||||
export function listAdapterPlugins(): AdapterPluginRecord[] {
|
||||
|
||||
@@ -2,6 +2,7 @@ import type { Db } from "@paperclipai/db";
|
||||
import { instanceSettings } from "@paperclipai/db";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { open } from "node:fs/promises";
|
||||
import { constants } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { z } from "zod";
|
||||
import { listServerAdapters } from "../adapters/registry.js";
|
||||
@@ -118,23 +119,31 @@ export async function initializeAdapterSettings(db: AdapterSettingsDb, options:
|
||||
const service = adapterSettingsService(db, options);
|
||||
const current = await service.inspectStored();
|
||||
if (current) return { seeded: false, current };
|
||||
return service.seedIfAbsent(await readLegacyAdapterSettings(env) ?? { disabledTypes: [] });
|
||||
}
|
||||
|
||||
export async function readLegacyAdapterSettings(env: Record<string, string | undefined> = process.env): Promise<{ disabledTypes: string[] } | null> {
|
||||
const filePath = path.join(resolvePaperclipHomeDir(env.PAPERCLIP_HOME), "adapter-settings.json");
|
||||
let disabledTypes: string[] = [];
|
||||
try {
|
||||
const file = await open(filePath, "r");
|
||||
const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
|
||||
try {
|
||||
const stat = await file.stat();
|
||||
if (!stat.isFile() || stat.size > 65_536) throw new Error("Legacy adapter settings exceed the size limit or are not a regular file");
|
||||
const buffer = Buffer.alloc(65_537);
|
||||
const { bytesRead } = await file.read(buffer, 0, buffer.length, 0);
|
||||
if (bytesRead > 65_536) throw new Error("Legacy adapter settings exceed the size limit");
|
||||
const value = JSON.parse(buffer.subarray(0, bytesRead).toString("utf8"));
|
||||
disabledTypes = parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes;
|
||||
let total = 0;
|
||||
while (total < buffer.length) {
|
||||
const { bytesRead } = await file.read(buffer, total, buffer.length - total, total);
|
||||
if (bytesRead === 0) break;
|
||||
total += bytesRead;
|
||||
}
|
||||
if (total > 65_536) throw new Error("Legacy adapter settings exceed the size limit");
|
||||
const value = JSON.parse(buffer.subarray(0, total).toString("utf8"));
|
||||
return { disabledTypes: parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes };
|
||||
} finally {
|
||||
await file.close();
|
||||
}
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
||||
return null;
|
||||
}
|
||||
return service.seedIfAbsent({ disabledTypes });
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js";
|
||||
|
||||
export function cloudBundledPackageDigest(packageRoot: string): string {
|
||||
const root = fs.realpathSync(packageRoot);
|
||||
const inventory: Array<[string, string, string, number]> = [];
|
||||
let bytes = 0;
|
||||
let entries = 0;
|
||||
function walk(directory: string, relative = "") {
|
||||
if (relative.split("/").length > 32) throw new Error("Image package exceeds the directory depth limit");
|
||||
for (const name of fs.readdirSync(directory).sort()) {
|
||||
if (++entries > 10_000) throw new Error("Image package exceeds the entry limit");
|
||||
const local = relative ? `${relative}/${name}` : name;
|
||||
const file = path.join(directory, name);
|
||||
const stat = fs.lstatSync(file);
|
||||
if (stat.isSymbolicLink()) {
|
||||
const target = fs.realpathSync(file);
|
||||
const contained = path.relative(root, target);
|
||||
if (!contained || contained.startsWith("..") || path.isAbsolute(contained)) throw new Error("Image package symlink escapes its package");
|
||||
const targetStat = fs.statSync(target);
|
||||
if (!targetStat.isFile() && !targetStat.isDirectory()) throw new Error("Image package symlink has an unsupported target");
|
||||
inventory.push([local, "link", fs.readlinkSync(file), stat.mode & 0o777]);
|
||||
} else if (stat.isDirectory()) walk(file, local);
|
||||
else if (stat.isFile()) {
|
||||
if (bytes + stat.size > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit");
|
||||
const handle = fs.openSync(file, "r");
|
||||
const hash = createHash("sha256");
|
||||
const buffer = Buffer.alloc(65_536);
|
||||
try {
|
||||
while (true) {
|
||||
const count = fs.readSync(handle, buffer);
|
||||
if (!count) break;
|
||||
bytes += count;
|
||||
if (bytes > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit");
|
||||
hash.update(buffer.subarray(0, count));
|
||||
}
|
||||
} finally { fs.closeSync(handle); }
|
||||
inventory.push([local, "file", `sha256:${hash.digest("hex")}`, stat.mode & 0o777]);
|
||||
} else throw new Error("Image package contains a special file");
|
||||
}
|
||||
}
|
||||
walk(root);
|
||||
if (!inventory.length) throw new Error("Image package is empty");
|
||||
return cloudConfigurationDigest(inventory);
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { constants } from "node:fs";
|
||||
import { open } from "node:fs/promises";
|
||||
import { z } from "zod";
|
||||
|
||||
function canonical(value: unknown): unknown {
|
||||
if (Array.isArray(value)) return value.map(canonical);
|
||||
if (value && typeof value === "object") {
|
||||
return Object.fromEntries(Object.entries(value).sort(([left], [right]) => left < right ? -1 : left > right ? 1 : 0).map(([key, child]) => [key, canonical(child)]));
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function cloudConfigurationDigest(value: unknown): string {
|
||||
return `sha256:${createHash("sha256").update(JSON.stringify(canonical(value))).digest("hex")}`;
|
||||
}
|
||||
|
||||
export const externalRegistrationEvidenceSchema = z.object({
|
||||
sourceLocatorDigest: z.string(),
|
||||
sourceDigest: z.string(),
|
||||
state: z.enum(["missing", "parsed", "invalid", "unreadable"]),
|
||||
adapterTypes: z.array(z.string()),
|
||||
blocker: z.enum(["unsupported_external_adapters", "invalid_external_registration_metadata", "unreadable_external_registration_metadata"]).nullable(),
|
||||
}).strict();
|
||||
export type ExternalRegistrationEvidence = z.infer<typeof externalRegistrationEvidenceSchema>;
|
||||
|
||||
const legacyRegistrationSchema = z.array(z.object({
|
||||
type: z.string().regex(/^[a-zA-Z0-9_.-]{1,128}$/),
|
||||
packageName: z.string().min(1).max(1024),
|
||||
version: z.string().max(128).optional(),
|
||||
localPath: z.string().max(4096).optional(),
|
||||
disabled: z.boolean().optional(),
|
||||
}).passthrough()).max(100);
|
||||
|
||||
export async function inspectLegacyExternalRegistrations(filePath: string): Promise<ExternalRegistrationEvidence> {
|
||||
const sourceLocatorDigest = cloudConfigurationDigest(filePath);
|
||||
try {
|
||||
const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
|
||||
let raw: unknown;
|
||||
try {
|
||||
const stat = await file.stat();
|
||||
if (!stat.isFile() || stat.size > 65_536) throw new Error("invalid legacy registration file");
|
||||
const buffer = Buffer.alloc(65_537);
|
||||
let total = 0;
|
||||
while (total < buffer.length) {
|
||||
const { bytesRead } = await file.read(buffer, total, buffer.length - total, total);
|
||||
if (!bytesRead) break;
|
||||
total += bytesRead;
|
||||
}
|
||||
if (total > 65_536) throw new Error("oversized legacy registration file");
|
||||
raw = JSON.parse(buffer.subarray(0, total).toString("utf8"));
|
||||
} finally {
|
||||
await file.close();
|
||||
}
|
||||
const records = legacyRegistrationSchema.safeParse(raw);
|
||||
if (!records.success) {
|
||||
return { sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state: "invalid" }), state: "invalid", adapterTypes: [], blocker: "invalid_external_registration_metadata" };
|
||||
}
|
||||
const metadata = records.data.map((record) => ({ type: record.type, packageName: record.packageName, version: record.version, disabled: record.disabled, hasLocalPath: record.localPath !== undefined }));
|
||||
return {
|
||||
sourceLocatorDigest, sourceDigest: cloudConfigurationDigest(metadata), state: "parsed",
|
||||
adapterTypes: [...new Set(metadata.map((record) => record.type))].sort(),
|
||||
blocker: metadata.length ? "unsupported_external_adapters" : null,
|
||||
};
|
||||
} catch (error) {
|
||||
const absent = (error as NodeJS.ErrnoException).code === "ENOENT";
|
||||
const invalid = error instanceof SyntaxError || (error instanceof Error && /legacy registration file/.test(error.message));
|
||||
const state = absent ? "missing" : invalid ? "invalid" : "unreadable";
|
||||
return {
|
||||
sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state }), state, adapterTypes: [],
|
||||
blocker: absent ? null : invalid ? "invalid_external_registration_metadata" : "unreadable_external_registration_metadata",
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
import { instanceSettings, plugins, type Db } from "@paperclipai/db";
|
||||
import { eq, inArray, ne } from "drizzle-orm";
|
||||
import { z } from "zod";
|
||||
import { adapterSettingsService, readLegacyAdapterSettings } from "./adapter-settings.js";
|
||||
import { getAdapterPluginStorePath } from "./adapter-plugin-store.js";
|
||||
import { cloudRuntimeConfigService, readLegacyCloudRuntimeConfig } from "./cloud-runtime-config.js";
|
||||
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
|
||||
import { cloudConfigurationDigest, externalRegistrationEvidenceSchema, inspectLegacyExternalRegistrations } from "./cloud-configuration-evidence.js";
|
||||
import { cloudPackageEvidenceSchema, inspectCloudPackages, qualificationBlockerSchema } from "./cloud-package-qualification.js";
|
||||
import type { ResolvedBundledPlugin } from "./bundled-plugins.js";
|
||||
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
|
||||
const RECEIPT_KEY = "cloud-configuration-migration:v1";
|
||||
const CONFIGURATION_KEYS = ["adapter-settings:v1", "cloud-runtime-config:v1"];
|
||||
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
export type CloudConfigurationDb = Pick<Db | Transaction, "select" | "insert" | "update" | "transaction">;
|
||||
const sourceSchema = z.object({
|
||||
authority: z.enum(["database", "legacy_file", "default"]), sourceDigest: z.string(),
|
||||
importedRevision: z.number().int().nonnegative(),
|
||||
}).strict();
|
||||
const qualificationSchema = z.object({
|
||||
scope: z.literal("configuration_and_packages"), checkedAt: z.string(),
|
||||
adapterRevision: z.number().int().nonnegative(), configRevision: z.number().int().nonnegative(),
|
||||
configurationDigest: z.string(), packageEvidenceDigest: z.string(),
|
||||
packageInspectionDurationMs: z.number().nonnegative(),
|
||||
expectedPackageEvidenceDigest: z.string().nullable(),
|
||||
configurationPackagesEligible: z.boolean(), samePackageOverlapEligible: z.boolean(),
|
||||
packages: z.array(cloudPackageEvidenceSchema), blockers: z.array(qualificationBlockerSchema),
|
||||
overlapBlockers: z.array(qualificationBlockerSchema),
|
||||
}).strict();
|
||||
const receiptSchema = z.object({
|
||||
version: z.literal(1), revision: z.number().int().nonnegative(),
|
||||
migration: z.object({ completedAt: z.string(), adapter: sourceSchema, config: sourceSchema, externalRegistrations: externalRegistrationEvidenceSchema }).strict(),
|
||||
qualification: qualificationSchema.nullable(), lastPackageMismatch: qualificationSchema.nullable(),
|
||||
}).strict();
|
||||
export type CloudConfigurationReceipt = z.infer<typeof receiptSchema>;
|
||||
|
||||
function parseReceipt(value: unknown): CloudConfigurationReceipt {
|
||||
const result = receiptSchema.safeParse(value);
|
||||
if (!result.success) throw new Error("Persisted Cloud configuration migration receipt is invalid");
|
||||
return result.data;
|
||||
}
|
||||
|
||||
export function cloudConfigurationMigrationService(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv; now?: () => Date } = {}) {
|
||||
const env = options.runtimeEnv ?? process.env;
|
||||
const now = () => (options.now?.() ?? new Date()).toISOString();
|
||||
const adapters = adapterSettingsService(db, { runtimeEnv: env });
|
||||
const config = cloudRuntimeConfigService(db);
|
||||
|
||||
function assertManaged() {
|
||||
if (!isCloudManagedInstance(env)) throw new Error("Cloud configuration qualification requires a managed instance");
|
||||
}
|
||||
async function readRow(handle: CloudConfigurationDb, lock = false) {
|
||||
const query = handle.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, RECEIPT_KEY));
|
||||
const [row] = lock ? await query.for("update") : await query;
|
||||
return row ?? null;
|
||||
}
|
||||
async function currentConfiguration(handle: CloudConfigurationDb) {
|
||||
await handle.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS))
|
||||
.orderBy(instanceSettings.singletonKey).for("update");
|
||||
const adapter = await adapters.inspectStored(handle);
|
||||
const runtime = await config.inspectStored(handle);
|
||||
if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete");
|
||||
return { adapter, runtime };
|
||||
}
|
||||
async function registrationSnapshot(handle: CloudConfigurationDb, lock = false) {
|
||||
const query = handle.select({
|
||||
pluginKey: plugins.pluginKey, packageName: plugins.packageName, version: plugins.version,
|
||||
manifestJson: plugins.manifestJson, status: plugins.status,
|
||||
}).from(plugins).where(ne(plugins.status, "uninstalled")).orderBy(plugins.pluginKey);
|
||||
return lock ? query.for("share") : query;
|
||||
}
|
||||
async function save(handle: CloudConfigurationDb, id: string, receipt: CloudConfigurationReceipt) {
|
||||
const validated = parseReceipt(receipt);
|
||||
const updated = await handle.update(instanceSettings).set({ general: validated, updatedAt: new Date() }).where(eq(instanceSettings.id, id)).returning({ id: instanceSettings.id });
|
||||
if (updated.length !== 1) throw new Error("Cloud configuration migration receipt was not persisted");
|
||||
return validated;
|
||||
}
|
||||
|
||||
return {
|
||||
async inspectStored(): Promise<CloudConfigurationReceipt | null> {
|
||||
const row = await readRow(db);
|
||||
return row ? parseReceipt(row.general) : null;
|
||||
},
|
||||
async migrate(): Promise<CloudConfigurationReceipt> {
|
||||
assertManaged();
|
||||
return db.transaction(async (tx) => {
|
||||
const inserted = await tx.insert(instanceSettings).values({ singletonKey: RECEIPT_KEY, general: { version: 1 } })
|
||||
.onConflictDoNothing({ target: instanceSettings.singletonKey }).returning({ id: instanceSettings.id });
|
||||
const row = await readRow(tx, true);
|
||||
if (!row) throw new Error("Cloud configuration migration receipt is unavailable");
|
||||
if (!inserted.length) {
|
||||
const receipt = parseReceipt(row.general);
|
||||
await currentConfiguration(tx);
|
||||
return receipt;
|
||||
}
|
||||
await tx.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS))
|
||||
.orderBy(instanceSettings.singletonKey).for("update");
|
||||
const storedAdapter = await adapters.inspectStored(tx);
|
||||
const storedConfig = await config.inspectStored(tx);
|
||||
const adapterCandidate = storedAdapter ? null : await readLegacyAdapterSettings(env);
|
||||
const configCandidate = storedConfig ? null : await readLegacyCloudRuntimeConfig(env);
|
||||
const adapterSeed = storedAdapter ? null : await adapters.seedIfAbsent({ disabledTypes: adapterCandidate?.disabledTypes ?? [] }, tx);
|
||||
const configSeed = storedConfig ? null : await config.seedIfAbsent(configCandidate, tx);
|
||||
const adapter = storedAdapter ?? adapterSeed!.current;
|
||||
const runtime = storedConfig ?? configSeed!.current;
|
||||
const adapterFromDatabase = Boolean(storedAdapter || !adapterSeed?.seeded);
|
||||
const configFromDatabase = Boolean(storedConfig || !configSeed?.seeded);
|
||||
const receipt: CloudConfigurationReceipt = {
|
||||
version: 1, revision: 0,
|
||||
migration: {
|
||||
completedAt: now(),
|
||||
adapter: { authority: adapterFromDatabase ? "database" : adapterCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(adapterFromDatabase ? { disabledTypes: adapter.disabledTypes } : adapterCandidate), importedRevision: adapter.revision },
|
||||
config: { authority: configFromDatabase ? "database" : configCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(configFromDatabase ? runtime.config : configCandidate), importedRevision: runtime.revision },
|
||||
externalRegistrations: await inspectLegacyExternalRegistrations(getAdapterPluginStorePath(env.PAPERCLIP_HOME)),
|
||||
},
|
||||
qualification: null, lastPackageMismatch: null,
|
||||
};
|
||||
return save(tx, row.id, receipt);
|
||||
});
|
||||
},
|
||||
async recheckLegacyExternalRegistrations(expectedRevision: number): Promise<CloudConfigurationReceipt> {
|
||||
assertManaged();
|
||||
return db.transaction(async (tx) => {
|
||||
const row = await readRow(tx, true);
|
||||
if (!row) throw new Error("Cloud configuration migration has not completed");
|
||||
const receipt = parseReceipt(row.general);
|
||||
if (receipt.revision !== expectedRevision) throw new Error("Cloud configuration migration receipt revision conflict");
|
||||
const filePath = getAdapterPluginStorePath(env.PAPERCLIP_HOME);
|
||||
if (cloudConfigurationDigest(filePath) !== receipt.migration.externalRegistrations.sourceLocatorDigest) {
|
||||
throw new Error("External registration recheck requires the original legacy source location");
|
||||
}
|
||||
const evidence = await inspectLegacyExternalRegistrations(filePath);
|
||||
if (evidence.state !== "parsed") throw new Error("External registration recheck requires a present, valid registration file");
|
||||
receipt.migration.externalRegistrations = evidence;
|
||||
receipt.revision += 1;
|
||||
receipt.qualification = null;
|
||||
return save(tx, row.id, receipt);
|
||||
});
|
||||
},
|
||||
async qualify(input: {
|
||||
installs: readonly ResolvedBundledPlugin[];
|
||||
loadManifest: (packageRoot: string) => Promise<PaperclipPluginManifestV1 | null>;
|
||||
expectedPackageEvidenceDigest?: string;
|
||||
imageCatalogUnavailable?: boolean;
|
||||
}) {
|
||||
assertManaged();
|
||||
const adapter = await adapters.inspectStored();
|
||||
const runtime = await config.inspectStored();
|
||||
if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete");
|
||||
const configurationBefore = cloudConfigurationDigest({ adapter, runtime });
|
||||
const registrationsBefore = await registrationSnapshot(db);
|
||||
const evidence = await inspectCloudPackages({ ...input, registrations: registrationsBefore });
|
||||
return db.transaction(async (tx) => {
|
||||
const row = await readRow(tx, true);
|
||||
if (!row) throw new Error("Cloud configuration migration has not completed");
|
||||
const receipt = parseReceipt(row.general);
|
||||
const current = await currentConfiguration(tx);
|
||||
const registrations = await registrationSnapshot(tx, true);
|
||||
const blockers = [...evidence.blockers];
|
||||
if (configurationBefore !== cloudConfigurationDigest(current) || cloudConfigurationDigest(registrationsBefore) !== cloudConfigurationDigest(registrations)) {
|
||||
blockers.push({ code: "configuration_changed_during_inspection", reason: "Configuration or plugin registration changed during image inspection; run qualification again" });
|
||||
}
|
||||
if (input.imageCatalogUnavailable) blockers.push({ code: "current_image_catalog_unavailable", reason: "The current image package catalog could not be verified" });
|
||||
const external = receipt.migration.externalRegistrations;
|
||||
if (external.blocker) blockers.push({ code: external.blocker, reason: "Legacy external adapter registration metadata requires an explicit operator resolution" });
|
||||
const overlapBlockers = input.expectedPackageEvidenceDigest && input.expectedPackageEvidenceDigest !== evidence.packageEvidenceDigest
|
||||
? [{ code: "package_evidence_changed", reason: "Current packages do not match the expected package evidence digest" }] : [];
|
||||
const qualification: z.infer<typeof qualificationSchema> = {
|
||||
scope: "configuration_and_packages", checkedAt: now(),
|
||||
adapterRevision: current.adapter.revision, configRevision: current.runtime.revision,
|
||||
configurationDigest: cloudConfigurationDigest(current), packageEvidenceDigest: evidence.packageEvidenceDigest,
|
||||
packageInspectionDurationMs: evidence.packageInspectionDurationMs,
|
||||
expectedPackageEvidenceDigest: input.expectedPackageEvidenceDigest ?? null,
|
||||
configurationPackagesEligible: blockers.length === 0,
|
||||
samePackageOverlapEligible: Boolean(input.expectedPackageEvidenceDigest && blockers.length === 0 && overlapBlockers.length === 0),
|
||||
packages: evidence.packages, blockers, overlapBlockers,
|
||||
};
|
||||
receipt.revision += 1;
|
||||
receipt.qualification = qualification;
|
||||
if (evidence.blockers.length || input.imageCatalogUnavailable || overlapBlockers.length) receipt.lastPackageMismatch = qualification;
|
||||
await save(tx, row.id, receipt);
|
||||
return { receiptRevision: receipt.revision, ...qualification };
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function migrateCloudConfiguration(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv } = {}) {
|
||||
if (!isCloudManagedInstance(options.runtimeEnv ?? process.env)) return null;
|
||||
return cloudConfigurationMigrationService(db, options).migrate();
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { BUNDLED_PLUGIN_CATALOG, resolveBundledCatalogRoot, resolveBundledPluginInstalls, type ResolvedBundledPlugin } from "./bundled-plugins.js";
|
||||
import { readDistributionPluginCatalog } from "./distribution-plugin-catalog.js";
|
||||
import { createBundledPluginPackageResolver } from "./bundled-plugin-package.js";
|
||||
import { cloudConfigurationMigrationService } from "./cloud-configuration-migration.js";
|
||||
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
|
||||
import { getManagedInstanceConfig } from "./managed-config.js";
|
||||
import { pluginLoader } from "./plugin-loader.js";
|
||||
|
||||
export async function qualifyCloudConfigurationForCurrentImage(db: Db, options: {
|
||||
runtimeEnv?: CloudInstanceEnv;
|
||||
expectedPackageEvidenceDigest?: string;
|
||||
} = {}) {
|
||||
const env = options.runtimeEnv ?? process.env;
|
||||
if (!isCloudManagedInstance(env)) return null;
|
||||
const service = cloudConfigurationMigrationService(db, { runtimeEnv: env });
|
||||
const catalogRoot = resolveBundledCatalogRoot(env);
|
||||
let installs: ResolvedBundledPlugin[];
|
||||
try {
|
||||
const distribution = readDistributionPluginCatalog(catalogRoot, BUNDLED_PLUGIN_CATALOG);
|
||||
installs = resolveBundledPluginInstalls(getManagedInstanceConfig(env)?.plugins.autoInstall ?? [], {
|
||||
catalogRoot, env, enforceCatalogRoot: true, distributionPlugins: distribution,
|
||||
});
|
||||
} catch {
|
||||
return service.qualify({ installs: [], loadManifest: async () => null, imageCatalogUnavailable: true, expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest });
|
||||
}
|
||||
const resolver = createBundledPluginPackageResolver(installs);
|
||||
const loader = pluginLoader(db, {
|
||||
prebuiltOnly: true, enableLocalFilesystem: false, enableNpmDiscovery: false,
|
||||
assertPackageActivation: resolver.assertActivation, resolvePackageRoot: resolver.resolve,
|
||||
});
|
||||
return service.qualify({ installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest });
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import { z } from "zod";
|
||||
import { createBundledPluginPackageResolver, type BundledPluginRegistration } from "./bundled-plugin-package.js";
|
||||
import type { ResolvedBundledPlugin } from "./bundled-plugins.js";
|
||||
import { distributionBundleDigest } from "./distribution-plugin-catalog.js";
|
||||
import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js";
|
||||
import { cloudBundledPackageDigest } from "./cloud-bundled-package-digest.js";
|
||||
|
||||
export const qualificationBlockerSchema = z.object({
|
||||
code: z.string(), pluginKey: z.string().optional(), reason: z.string(),
|
||||
registeredVersion: z.string().optional(), imageVersion: z.string().optional(),
|
||||
}).strict();
|
||||
export type QualificationBlocker = z.infer<typeof qualificationBlockerSchema>;
|
||||
export const cloudPackageEvidenceSchema = z.object({
|
||||
catalogKey: z.string(), pluginKey: z.string(), packageDigest: z.string().nullable(),
|
||||
imageManifestDigest: z.string().nullable(), approvedManifestDigest: z.string().nullable(),
|
||||
}).strict();
|
||||
export type CloudPackageEvidence = z.infer<typeof cloudPackageEvidenceSchema>;
|
||||
|
||||
export async function inspectCloudPackages(input: {
|
||||
installs: readonly ResolvedBundledPlugin[];
|
||||
registrations: readonly BundledPluginRegistration[];
|
||||
loadManifest: (packageRoot: string) => Promise<PaperclipPluginManifestV1 | null>;
|
||||
}) {
|
||||
const started = performance.now();
|
||||
const resolver = createBundledPluginPackageResolver(input.installs);
|
||||
const registrations = new Map(input.registrations.map((record) => [record.pluginKey, record]));
|
||||
const selected = new Set(input.installs.map((install) => install.pluginKey));
|
||||
const blockers: QualificationBlocker[] = input.registrations.filter((record) => !selected.has(record.pluginKey)).map((record) => ({
|
||||
code: "unsupported_plugin_registration", pluginKey: record.pluginKey, reason: "An installed plugin is not selected from the current image", registeredVersion: record.version,
|
||||
}));
|
||||
const packages: CloudPackageEvidence[] = [];
|
||||
for (const install of [...input.installs].sort((left, right) => left.pluginKey < right.pluginKey ? -1 : left.pluginKey > right.pluginKey ? 1 : 0)) {
|
||||
const registration = registrations.get(install.pluginKey);
|
||||
const evidence: CloudPackageEvidence = {
|
||||
catalogKey: install.key, pluginKey: install.pluginKey, packageDigest: null,
|
||||
imageManifestDigest: null, approvedManifestDigest: registration ? cloudConfigurationDigest(registration.manifestJson) : null,
|
||||
};
|
||||
packages.push(evidence);
|
||||
let stage = "missing_bundled_package";
|
||||
try {
|
||||
resolver.assertActivation({ packageRoot: install.localPath });
|
||||
stage = "unverified_package_integrity";
|
||||
evidence.packageDigest = install.distribution ? distributionBundleDigest(install.localPath) : cloudBundledPackageDigest(install.localPath);
|
||||
if (install.distribution && evidence.packageDigest !== install.distribution.digest) {
|
||||
blockers.push({ code: "package_integrity_mismatch", pluginKey: install.pluginKey, reason: "Current package bytes do not match the verified distribution digest" });
|
||||
continue;
|
||||
}
|
||||
stage = "missing_manifest_evidence";
|
||||
const manifest = await input.loadManifest(install.localPath);
|
||||
if (!manifest) {
|
||||
blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image has no verified plugin manifest" });
|
||||
continue;
|
||||
}
|
||||
evidence.imageManifestDigest = cloudConfigurationDigest(manifest);
|
||||
if (!registration) {
|
||||
blockers.push({ code: "missing_plugin_registration", pluginKey: install.pluginKey, reason: "The current image package is not registered in the database", imageVersion: manifest.version });
|
||||
continue;
|
||||
}
|
||||
const inspected = resolver.inspect(registration, manifest);
|
||||
if (!inspected.compatible) blockers.push({
|
||||
code: inspected.code, pluginKey: install.pluginKey, reason: inspected.reason,
|
||||
registeredVersion: registration.version, imageVersion: manifest.version,
|
||||
});
|
||||
} catch {
|
||||
blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image package could not be verified" });
|
||||
}
|
||||
}
|
||||
const packageEvidenceDigest = cloudConfigurationDigest(packages.map(({ approvedManifestDigest: _approved, ...evidence }) => evidence));
|
||||
return { packages, blockers, packageEvidenceDigest, packageInspectionDurationMs: Math.round(performance.now() - started) };
|
||||
}
|
||||
Reference in new issue
Block a user