feat(cloud): record configuration migration and package qualification

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-30 11:37:04 -05:00
1 parent 9dcf3f9904
commit c190103824
16 files changed
+1029 -11

No files matched your search

+21
View File
@@ -755,6 +755,27 @@ The focused PostgreSQL suite can use an owned disposable PostgreSQL database via
unavailable. It applies migrations and inserts test data, so never point this
variable at an existing application database.
## Cloud configuration migration receipt
The private `instance_settings` singleton `cloud-configuration-migration:v1`
records atomic import completion, semantic source digests, imported revisions,
and metadata-only external adapter blockers. It composes the adapter and
startup-config owners without replacing an existing authoritative row. Receipt
locking and the import transaction handle concurrent attempts and retries; no
local completion marker participates in conflict resolution.
Qualification records current configuration revisions and selected image package
evidence before normal plugin reconciliation. Image inspection does not hold
admin write locks; configuration and registration snapshots are rechecked in a
short transaction before the report is saved. A historical successful report
never grants current eligibility. External metadata blockers survive loss of the
legacy home and require a deliberate revision-checked recheck of a present valid
original source file. Package mismatches remain recorded after maintenance
reconciliation. See the [bounded migration and qualification procedure](cloud-configuration-migration.md)
for the stopped-writer prerequisite, commands, retry semantics, integrity limits,
and the explicit exclusion of secret bytes, whole-server zero downtime, and
volume removal.
## Cloud runtime identity singleton
The private `instance_settings` row whose singleton key is
+131
View File
@@ -0,0 +1,131 @@
# Cloud configuration migration and package qualification
This procedure covers saved adapter preferences, the allowlisted nonsecret
startup configuration, and selected bundled plugin packages. The result is a
point-in-time report. It does not authorize a deployment, guarantee zero
downtime, establish general Railway admission, or prove that a volume can be
removed. Secret and key bytes, database credentials, artifacts, logs, local
storage, workspaces, and plugin state outside PostgreSQL need their own storage
and replacement checks. This includes the decision signing key and local
encryption keys. Their byte authority is unchanged by this procedure.
## Migration prerequisite
Stop every legacy file writer before the first import. This includes old
application replicas and operator commands that can write configuration or
adapter settings. One maintenance window is sufficient. The importer cannot
make an old file writer participate in its database transaction. Keep those
writers stopped after import; later file edits do not update database authority.
Keep the normal database backup and recovery procedure for the instance. Use the
new release with the same managed environment and database connection. The
server runs migration before constructing configuration-dependent services.
For an explicit source-checkout invocation, use:
```sh
pnpm exec tsx server/src/qualify-cloud-configuration.ts --migrate
```
The command uses `DATABASE_URL` and the normal Cloud environment. It does not
accept credentials as arguments. It writes the receipt to PostgreSQL and prints
a JSON report; no report file or local completion marker is required. Exit code
0 means the requested configuration/package check passed, 2 means the report has
blockers, and 1 means the operation failed. A fresh installation can report a
missing plugin registration before normal boot installation completes. Run the
check again after that installation settles.
## Database authority and retry behavior
The private `instance_settings` singleton `cloud-configuration-migration:v1`
contains the versioned receipt. The transaction locks that row and composes the
adapter and startup-config owners' atomic seed methods. Completion and both
seeds commit together. A failed write or crash before commit leaves no completed
receipt. A retry uses committed database state and never replaces a later
settings revision. Earlier independently seeded owner records remain authoritative.
The receipt records source kind, a digest of the nonsecret semantic candidate,
the imported revision, and bounded legacy external-registration evidence.
An absent startup configuration remains an authoritative `config: null`; it is
distinct from an uninitialized database record. No raw secret values are copied.
Once the receipt exists, ordinary boots do not reread legacy source files.
A replacement home can be empty or absent without changing saved authority.
All legacy readers have a 64 KiB limit. They require regular files, reject
symlinks and FIFOs, and use bounded reads. External registration inspection reads
metadata only. It never loads a registered module or follows a registered package
path. A nonempty, invalid, or unreadable registration file creates a durable
blocker. Removing the old home does not clear that blocker.
To resolve an external-registration blocker, stop its writer, deliberately
remove the unsupported registration metadata at the original source location,
and leave a present valid empty JSON array file. Read the current receipt revision
from the report, then run:
```sh
pnpm exec tsx server/src/qualify-cloud-configuration.ts --recheck-external-revision 3
```
Use the actual current revision in place of `3`. A changed revision, a different
source location, or a missing/invalid source file rejects the recheck. Reusing
the same container path with an empty home is insufficient. This is an explicit
operator resolution; qualification never clears the blocker automatically.
## Current packages and overlap comparison
Every qualification reads current settings revisions and plugin registrations.
It resolves selected packages from this image with the same bundled package
identity rules as normal loading. It verifies package bytes before importing a
trusted image manifest, then compares that manifest with the approved database
registration. Missing packages, missing evidence, incompatible versions,
unselected registrations, and integrity failures prevent qualification. Legacy
package paths and npm discovery cannot supply missing image bytes.
Image inspection happens outside database write locks. A short transaction then
locks and rereads the current configuration and registration records. A change
during inspection produces a blocker and requires a new check. The report gives
`adapterRevision`, `configRevision`, `configurationDigest`, and
`packageEvidenceDigest`. A historical completed receipt is evidence, not current
authorization. Call qualification again for a current result.
The server records this report before ordinary bundled-plugin reconciliation.
It retains `lastPackageMismatch` after later successful checks, so reconciliation
cannot erase the evidence seen before an upgrade. The report does not block
normal maintenance upgrades or replace capability approval. After an upgrade
settles, its current packages can qualify normally.
For a same-package comparison, capture the current release's
`packageEvidenceDigest` and pass it explicitly on the replacement:
```sh
pnpm exec tsx server/src/qualify-cloud-configuration.ts --expected-package-evidence sha256:REPLACE_WITH_THE_64_HEX_DIGEST
```
`configurationPackagesEligible` describes current database/configuration/package
compatibility. `samePackageOverlapEligible` is true only when an explicit expected
digest also matches. There is no permanent first-release baseline that would
prevent future maintenance releases from establishing their own evidence.
The package evidence digest covers selected packages, not container image bytes.
Distribution packages reuse and reverify their catalog digest. Builtin packages
have a bounded inventory of physical file bytes, modes, and contained symlink
text, including standalone pnpm dependency trees. Dangling or escaping links,
special files, more than 10,000 entries, depth above 32, or more than 256 MiB per
package fail verification. Inventory work is synchronous and linear in selected
package bytes. It runs only during startup before the application is constructed
or in the explicit operator process, with no idle polling. The report includes
`packageInspectionDurationMs`. Packages must remain immutable throughout a check.
## Verification scope
The focused PostgreSQL tests use two independent clients and separate home/image
roots. They import legacy settings, change saved revisions, remove the old roots,
reconstruct from the database, and start a real plugin worker from the replacement
image fixture. They also cover concurrent migration, commit failure and retry,
database failure, stale sources, durable external blockers, package mismatch,
maintenance reconciliation, and changes during manifest inspection.
Set `PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL` only to an owned disposable
PostgreSQL database when embedded PostgreSQL is unavailable. The suite applies
migrations and writes/deletes test records. A complete release image, hosted
replacement, overlapping production processes, secret-byte migration, and volume
removal are not established by these local fixtures.
@@ -112,6 +112,17 @@ exercises persistence, concurrent writers, admission freshness, activity audit,
and failed writes; [authorization tests](../../server/src/__tests__/adapter-routes-authz.test.ts)
cover admin and hidden-setting gates. These are local tests, not deployed proof.
Cloud configuration migration and package qualification are operator commands,
not agent-accessible REST capabilities. They require the managed environment and
database operator access. The [command](../../server/src/qualify-cloud-configuration.ts)
composes transaction-safe owner seeds and writes a private receipt. Every
qualification rereads current revisions and image package evidence; a completed
historical receipt grants no deployment or execution authority. [PostgreSQL coverage](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts)
uses independent clients, lost-home reconstruction, a real replacement fixture
worker, rollback/retry, and durable blockers. [Reader coverage](../../server/src/__tests__/cloud-configuration-readers.test.ts)
checks bounded sources and pnpm-style link integrity. These are local fixtures;
see the [scope and operator procedure](../../doc/cloud-configuration-migration.md).
Cloud stack Admins may update experimental settings through the trusted Cloud
actor without receiving other instance-admin powers. This uses the current
stack role on each request; company membership alone does not grant it.
+10
View File
@@ -75,6 +75,16 @@ These controls do not qualify a deployed Cloud rollout or durable file storage.
Automated: [experimental settings](../../ui/src/pages/InstanceExperimentalSettings.test.tsx) and [database backup routes](../../server/src/__tests__/instance-database-backups-routes.test.ts) cover their UI/API contracts.
Cloud migration: in one maintenance window, stop legacy file writers and run the
[configuration/package qualification command](../../doc/cloud-configuration-migration.md).
Inspect its database receipt, update a saved preference, then repeat against a
second owned application/image root with the first home unavailable. Use an
explicit package evidence digest for same-package comparison. Confirm stale
qualification and missing legacy files cannot clear blockers. This recipe covers
configuration/packages and does not qualify whole-server zero downtime, secret
bytes, or volume removal. [Local PostgreSQL evidence path](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts)
includes a real fixture worker; it does not establish a hosted replacement.
Managed adapters: change a visible adapter preference as an instance admin,
then use a second application process against the same disposable database to
confirm catalog and new-agent selection reflect it. Verify saved agents remain
@@ -0,0 +1,305 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { applyPendingMigrations, closeRegisteredClients, createDb, instanceSettings, plugins, type Db } from "@paperclipai/db";
import { eq, inArray, sql } from "drizzle-orm";
import { paperclipConfigSchema } from "@paperclipai/shared";
import { adapterSettingsService, initializeAdapterSettings } from "../services/adapter-settings.js";
import { cloudRuntimeConfigService, initializeCloudRuntimeConfig } from "../services/cloud-runtime-config.js";
import { cloudConfigurationMigrationService } from "../services/cloud-configuration-migration.js";
import { qualifyCloudConfigurationForCurrentImage } from "../services/cloud-configuration-qualification.js";
import { runCloudConfigurationQualification } from "../qualify-cloud-configuration.js";
import { pluginLoader, type PluginRuntimeServices } from "../services/plugin-loader.js";
import { pluginRegistryService } from "../services/plugin-registry.js";
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
import { ensureBundledPlugins } from "../services/bundled-plugins.js";
import { bundledImageFixture } from "./helpers/bundled-image-fixture.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const externalDatabaseUrl = process.env.PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL?.trim();
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configuration-migration:v1"];
(support.supported ? describe : describe.skip)("Cloud configuration migration and current-image qualification", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
let peer: Db;
let homeA: string;
let homeB: string;
const roots: string[] = [];
const fixture = (version = "1.0.0") => { const image = bundledImageFixture({ version }); roots.push(image.root); return image; };
const env = (home = homeA) => ({ PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "test-cloud-token", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: path.join(home, "config.json") });
const migration = (client = db, home = homeA) => cloudConfigurationMigrationService(client, { runtimeEnv: env(home) });
beforeAll(async () => {
if (externalDatabaseUrl) {
await applyPendingMigrations(externalDatabaseUrl);
database = { connectionString: externalDatabaseUrl, cleanup: () => closeRegisteredClients(externalDatabaseUrl) };
} else database = await startEmbeddedPostgresTestDatabase("cloud-configuration-migration-");
db = createDb(database.connectionString);
peer = createDb(database.connectionString);
}, 90_000);
beforeEach(async () => {
homeA = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-legacy-A-"));
homeB = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-replacement-B-"));
roots.push(homeA, homeB);
await db.delete(instanceSettings).where(inArray(instanceSettings.singletonKey, keys));
await db.delete(plugins);
});
afterEach(async () => {
vi.restoreAllMocks();
for (const root of roots.splice(0)) await fs.rm(root, { recursive: true, force: true });
});
afterAll(async () => { await database?.cleanup(); });
async function legacy() {
const config = paperclipConfigSchema.parse({
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
database: { mode: "postgres" }, logging: { mode: "cloud" },
server: { port: 3199 }, telemetry: { enabled: false },
});
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
await fs.writeFile(path.join(homeA, "adapter-settings.json"), JSON.stringify({ disabledTypes: ["claude_local", "future_adapter"] }));
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]");
return config;
}
function loaderFor(image: ReturnType<typeof fixture>, client = db, services?: PluginRuntimeServices) {
return pluginLoader(client, {
localPluginDir: path.join(homeB, "plugins"), prebuiltOnly: true,
resolvePackageRoot: image.resolver.resolve, assertPackageActivation: image.resolver.assertActivation,
enableLocalFilesystem: false, enableNpmDiscovery: false,
}, services);
}
async function install(image: ReturnType<typeof fixture>) {
await loaderFor(image).installPlugin({ localPath: image.packageRoot });
const registry = pluginRegistryService(db);
const row = (await registry.getByKey(image.manifest.id))!;
await registry.updateStatus(row.id, { status: "ready" });
return row;
}
function qualify(image: ReturnType<typeof fixture>, client = db, home = homeA, expectedPackageEvidenceDigest?: string) {
const loader = loaderFor(image, client);
return migration(client, home).qualify({ installs: image.installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest });
}
it("imports A atomically, retains later DB revisions, and starts B's real image worker after A disappears", async () => {
await legacy();
const oldImage = fixture();
const newImage = fixture();
const row = await install(oldImage);
const first = await migration().migrate();
expect(first.migration.adapter.authority).toBe("legacy_file");
expect(first.migration.config.authority).toBe("legacy_file");
const original = await qualify(oldImage);
expect(original.configurationPackagesEligible).toBe(true);
expect(original.samePackageOverlapEligible).toBe(false);
await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true);
const config = (await cloudRuntimeConfigService(peer).inspectStored())!;
expect(config.config).not.toBeNull();
await cloudRuntimeConfigService(peer).replace({ ...config.config!, server: { ...config.config!.server, port: 3299 } }, config.revision);
await fs.rm(homeA, { recursive: true });
await fs.rm(oldImage.root, { recursive: true });
expect((await migration(peer, homeB).migrate()).migration).toEqual(first.migration);
await initializeAdapterSettings(peer, { runtimeEnv: env(homeB) });
const restored = await initializeCloudRuntimeConfig(peer, { runtimeEnv: env(homeB) });
expect(restored?.current.config?.server.port).toBe(3299);
expect(await adapterSettingsService(peer, { runtimeEnv: env(homeB) }).getDisabledTypes()).toEqual(["claude_local", "codex_local", "future_adapter"]);
const replacement = await qualify(newImage, peer, homeB, original.packageEvidenceDigest);
expect(replacement).toMatchObject({ adapterRevision: 1, configRevision: 1, configurationPackagesEligible: true, samePackageOverlapEligible: true });
expect(replacement.configurationDigest).not.toBe(original.configurationDigest);
const manager = createPluginWorkerManager();
try {
const services = {
workerManager: manager, lifecycleManager: { markError: vi.fn() },
jobStore: { syncJobDeclarations: vi.fn() }, jobScheduler: { registerPlugin: vi.fn() },
toolDispatcher: {}, eventBus: { forPlugin: vi.fn(), subscriptionCount: vi.fn(() => 0) },
buildHostHandlers: vi.fn(() => ({})), instanceInfo: { instanceId: "qualification-fixture", hostVersion: "1.0.0" },
} as unknown as PluginRuntimeServices;
expect((await loaderFor(newImage, peer, services).loadSingle(row.id)).success).toBe(true);
expect(await manager.call(row.id, "health", {})).toMatchObject({ status: "ok", details: { source: expect.stringContaining(newImage.packageRoot) } });
} finally { await manager.stopAll(); }
});
it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => {
await legacy();
const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]);
expect(first).toEqual(second);
await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("claude_local", false);
await fs.writeFile(path.join(homeA, "config.json"), "invalid stale file");
await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid stale file");
expect(await migration().migrate()).toEqual(first);
expect(await adapterSettingsService(db, { runtimeEnv: env() }).getDisabledTypes()).toEqual(["future_adapter"]);
});
it("rolls back both seeds and the receipt if completion fails, then retries after the crash boundary", async () => {
await legacy();
await db.execute(sql`CREATE FUNCTION reject_cloud_migration_receipt() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.singleton_key = 'cloud-configuration-migration:v1' THEN RAISE EXCEPTION 'test receipt failure'; END IF; RETURN NEW; END $$`);
await db.execute(sql`CREATE TRIGGER reject_cloud_migration_receipt BEFORE UPDATE ON instance_settings FOR EACH ROW EXECUTE FUNCTION reject_cloud_migration_receipt()`);
try {
await expect(migration().migrate()).rejects.toMatchObject({ cause: { message: expect.stringContaining("test receipt failure") } });
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
} finally {
await db.execute(sql`DROP TRIGGER reject_cloud_migration_receipt ON instance_settings`);
await db.execute(sql`DROP FUNCTION reject_cloud_migration_receipt()`);
}
expect((await migration(peer).migrate()).revision).toBe(0);
expect((await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).length).toBe(3);
});
it("fails without partial authority for malformed legacy config and preserves already-authoritative rows", async () => {
await legacy();
await fs.writeFile(path.join(homeA, "config.json"), "invalid");
await expect(migration().migrate()).rejects.toThrow("Legacy Cloud runtime configuration");
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
await adapterSettingsService(db, { runtimeEnv: env() }).seedIfAbsent({ disabledTypes: ["codex_local"] });
await cloudRuntimeConfigService(db).seedIfAbsent(null);
await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid");
const receipt = await migration(peer).migrate();
expect(receipt.migration.adapter.authority).toBe("database");
expect(receipt.migration.config.authority).toBe("database");
expect((await cloudRuntimeConfigService(peer).inspectStored())?.config).toBeNull();
});
it("returns database errors instead of falling back to files or a completed receipt", async () => {
await legacy();
await migration().migrate();
const broken = createDb(database.connectionString);
await broken.$client.end({ timeout: 0 });
await expect(migration(broken).migrate()).rejects.toThrow();
const image = fixture();
await expect(qualify(image, broken)).rejects.toThrow();
});
it("keeps external registration blockers durable when B has no legacy home and requires deliberate original-source resolution", async () => {
await legacy();
const sentinel = path.join(homeA, "external-code-executed");
await fs.writeFile(path.join(homeA, "external.js"), `require('node:fs').writeFileSync(${JSON.stringify(sentinel)}, 'executed');`);
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), JSON.stringify([{ type: "legacy_external", packageName: "legacy-adapter", localPath: path.join(homeA, "external.js") }]));
const image = fixture();
await install(image);
await migration().migrate();
expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "unsupported_external_adapters" }));
await expect(fs.stat(sentinel)).rejects.toMatchObject({ code: "ENOENT" });
await migration(peer, homeB).migrate();
const replacement = await qualify(image, peer, homeB);
expect(replacement.configurationPackagesEligible).toBe(false);
await expect(migration(peer, homeB).recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("original legacy source");
await fs.unlink(path.join(homeA, "adapter-plugins.json"));
await expect(migration().recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("present, valid registration file");
expect((await migration().inspectStored())?.migration.externalRegistrations.blocker).toBe("unsupported_external_adapters");
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]");
await expect(migration().recheckLegacyExternalRegistrations(0)).rejects.toThrow("revision conflict");
await migration().recheckLegacyExternalRegistrations(replacement.receiptRevision);
expect((await qualify(image)).configurationPackagesEligible).toBe(true);
});
it("retains a metadata blocker for malformed or oversized registration files", async () => {
await legacy();
await fs.writeFile(path.join(homeA, "adapter-plugins.json"), Buffer.alloc(65_537));
const receipt = await migration().migrate();
expect(receipt.migration.externalRegistrations.blocker).toBe("invalid_external_registration_metadata");
await fs.rm(homeA, { recursive: true });
expect((await migration(peer, homeB).migrate()).migration.externalRegistrations).toEqual(receipt.migration.externalRegistrations);
});
it("records image mismatches before reconciliation, permits maintenance upgrades, and requires explicit fresh overlap evidence", async () => {
await legacy();
const oldImage = fixture();
const nextImage = fixture("1.1.0");
await install(oldImage);
await migration().migrate();
const old = await qualify(oldImage);
const mismatch = await qualify(nextImage, peer, homeB, old.packageEvidenceDigest);
expect(mismatch.configurationPackagesEligible).toBe(false);
expect(mismatch.samePackageOverlapEligible).toBe(false);
expect(mismatch.blockers).toContainEqual(expect.objectContaining({ code: "incompatible_bundled_package", registeredVersion: "1.0.0", imageVersion: "1.1.0" }));
const registry = pluginRegistryService(db);
await ensureBundledPlugins(nextImage.installs, {
registry, loader: loaderFor(nextImage), lifecycle: { load: vi.fn() },
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
}, { reinstallUninstalled: true, preservePackagePath: true });
const current = await qualify(nextImage);
expect(current.configurationPackagesEligible).toBe(true);
expect(current.samePackageOverlapEligible).toBe(false);
expect((await migration().inspectStored())?.lastPackageMismatch?.packageEvidenceDigest).toBe(mismatch.packageEvidenceDigest);
expect((await qualify(nextImage, peer, homeB, current.packageEvidenceDigest)).samePackageOverlapEligible).toBe(true);
});
it("fails qualification for missing package or manifest bytes without loading a home fallback", async () => {
await legacy();
const image = fixture();
await install(image);
await migration().migrate();
const good = await qualify(image);
await fs.rm(path.join(image.packageRoot, "dist", "manifest.js"));
const broken = await qualify(image, peer, homeB, good.packageEvidenceDigest);
expect(broken.configurationPackagesEligible).toBe(false);
expect(broken.samePackageOverlapEligible).toBe(false);
expect(broken.blockers.length).toBeGreaterThan(0);
await fs.rm(image.packageRoot, { recursive: true });
expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "missing_bundled_package" }));
});
it("rejects changed image bytes before manifest import and never uses stale completed qualification", async () => {
await legacy();
const image = fixture();
await install(image);
await migration().migrate();
const original = await qualify(image);
await fs.appendFile(path.join(image.packageRoot, "dist", "worker.js"), "\nprocess.exit(99);\n");
const loadManifest = vi.fn();
const current = await migration(peer, homeB).qualify({ installs: image.installs, loadManifest, expectedPackageEvidenceDigest: original.packageEvidenceDigest });
expect(current.configurationPackagesEligible).toBe(false);
expect(current.samePackageOverlapEligible).toBe(false);
expect(current.blockers).toContainEqual(expect.objectContaining({ code: "package_integrity_mismatch" }));
expect(loadManifest).not.toHaveBeenCalled();
});
it("allows an administrator update while manifest inspection waits and refuses to certify the stale snapshot", async () => {
await legacy();
const image = fixture();
await install(image);
await migration().migrate();
let started!: () => void;
let resume!: () => void;
const entered = new Promise<void>((resolve) => { started = resolve; });
const release = new Promise<void>((resolve) => { resume = resolve; });
const loader = loaderFor(image);
const pending = migration().qualify({ installs: image.installs, loadManifest: async (root) => { started(); await release; return loader.loadManifest(root); } });
await entered;
try {
const change = adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
expect(await Promise.race([change, new Promise<never>((_resolve, reject) => { timer = setTimeout(() => reject(new Error("Admin update was blocked by image inspection")), 2_000); })])).toBe(true);
} finally { clearTimeout(timer); }
} finally { resume(); }
const result = await pending;
expect(result).toMatchObject({ configurationPackagesEligible: false, adapterRevision: 1 });
expect(result.blockers).toContainEqual(expect.objectContaining({ code: "configuration_changed_during_inspection" }));
expect((await qualify(image)).configurationPackagesEligible).toBe(true);
});
it("uses the current managed catalog through the operator command and durably records catalog failure", async () => {
await legacy();
const image = fixture();
await install(image);
await migration().migrate();
const runtimeEnv = {
...env(homeB), DATABASE_URL: database.connectionString, PAPERCLIP_BUNDLED_PLUGIN_ROOT: image.root,
PAPERCLIP_MANAGED_CONFIG: JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "fixture", features: {}, plugins: { autoInstall: ["fixture"] } }),
};
const initial = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv });
expect(initial?.configurationPackagesEligible).toBe(true);
const stdout = vi.spyOn(process.stdout, "write").mockImplementation(() => true);
try {
expect(await runCloudConfigurationQualification(["--expected-package-evidence", initial!.packageEvidenceDigest], runtimeEnv)).toBe(0);
const report = stdout.mock.calls.map((call) => String(call[0])).find((text) => text.startsWith('{\n "receiptRevision"'));
expect(report && JSON.parse(report).samePackageOverlapEligible).toBe(true);
} finally { stdout.mockRestore(); }
const missing = await qualifyCloudConfigurationForCurrentImage(peer, { runtimeEnv: { ...runtimeEnv, PAPERCLIP_MANAGED_CONFIG: runtimeEnv.PAPERCLIP_MANAGED_CONFIG.replace('"fixture"]', '"absent"]') } });
expect(missing?.configurationPackagesEligible).toBe(false);
expect(missing?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" }));
expect((await migration().inspectStored())?.lastPackageMismatch?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" }));
});
});
@@ -0,0 +1,48 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { execFileSync } from "node:child_process";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { readLegacyAdapterSettings } from "../services/adapter-settings.js";
import { inspectLegacyExternalRegistrations } from "../services/cloud-configuration-evidence.js";
import { cloudBundledPackageDigest } from "../services/cloud-bundled-package-digest.js";
vi.mock("../adapters/plugin-loader.js", () => ({ buildExternalAdapters: async () => [] }));
describe("bounded Cloud legacy readers and image inventory", () => {
let root: string;
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-config-reader-")); });
afterEach(async () => { await fs.rm(root, { recursive: true, force: true }); });
it("distinguishes an absent adapter file from a saved empty preference", async () => {
expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toBeNull();
await fs.writeFile(path.join(root, "adapter-settings.json"), '{"disabledTypes":[]}');
expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toEqual({ disabledTypes: [] });
});
it.skipIf(process.platform === "win32")("rejects symlinks and FIFOs without waiting for a writer", async () => {
const target = path.join(root, "target.json");
const settings = path.join(root, "adapter-settings.json");
await fs.writeFile(target, '{"disabledTypes":[]}');
await fs.symlink(target, settings);
await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toMatchObject({ code: "ELOOP" });
expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("unreadable_external_registration_metadata");
await fs.unlink(settings);
execFileSync("mkfifo", [settings]);
await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toThrow("regular file");
expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("invalid_external_registration_metadata");
});
it.skipIf(process.platform === "win32")("hashes pnpm-style contained links and the physical dependency bytes", async () => {
const dependency = path.join(root, "node_modules", ".pnpm", "dep@1.0.0", "node_modules", "dep");
await fs.mkdir(dependency, { recursive: true });
await fs.writeFile(path.join(dependency, "index.js"), "export const value = 1;");
await fs.symlink(path.relative(path.join(root, "node_modules"), dependency), path.join(root, "node_modules", "dep"));
const first = cloudBundledPackageDigest(root);
await fs.writeFile(path.join(dependency, "index.js"), "export const value = 2;");
expect(cloudBundledPackageDigest(root)).not.toBe(first);
await fs.unlink(path.join(root, "node_modules", "dep"));
await fs.symlink("/", path.join(root, "node_modules", "dep"));
expect(() => cloudBundledPackageDigest(root)).toThrow("escapes its package");
});
});
@@ -31,6 +31,9 @@ vi.mock("../services/adapter-settings.js", async () => ({
initializeAdapterSettings: vi.fn(async () => null),
}));
vi.mock("../services/cloud-runtime-config.js", () => ({ initializeCloudRuntimeConfig: cloudRuntimeStartupMocks.initialize }));
const cloudMigrationMocks = vi.hoisted(() => ({ migrate: vi.fn(async () => null), qualify: vi.fn(async () => null) }));
vi.mock("../services/cloud-configuration-migration.js", () => ({ migrateCloudConfiguration: cloudMigrationMocks.migrate }));
vi.mock("../services/cloud-configuration-qualification.js", () => ({ qualifyCloudConfigurationForCurrentImage: cloudMigrationMocks.qualify }));
vi.mock("../telemetry.js", () => ({ initTelemetry: cloudRuntimeStartupMocks.telemetry, getTelemetryClient: vi.fn(() => null) }));
const ORIGINAL_PAPERCLIP_API_URL = process.env.PAPERCLIP_API_URL;
@@ -627,6 +630,7 @@ describe("startServer feedback export wiring", () => {
const starting = startServer();
for (let index = 0; index < 30 && !release; index += 1) await new Promise<void>(resolve => setImmediate(resolve));
expect(release).toBeTypeOf("function");
expect(cloudMigrationMocks.migrate.mock.invocationCallOrder[0]).toBeLessThan(cloudRuntimeStartupMocks.initialize.mock.invocationCallOrder[0]!);
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
expect(createBetterAuthInstanceMock).not.toHaveBeenCalled();
expect(createAppMock).not.toHaveBeenCalled();
@@ -656,6 +660,14 @@ describe("startServer feedback export wiring", () => {
}
});
it("stops before configuration consumers when the atomic Cloud migration fails", async () => {
cloudMigrationMocks.migrate.mockRejectedValueOnce(new Error("migration did not commit"));
await expect(startServer()).rejects.toThrow("migration did not commit");
expect(cloudRuntimeStartupMocks.initialize).not.toHaveBeenCalled();
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
expect(createAppMock).not.toHaveBeenCalled();
});
it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
+7
View File
@@ -188,6 +188,8 @@ import {
} from "./shutdown.js";
import { initializeCloudRuntimeIdentity } from "./services/cloud-runtime-identity.js";
import { initializeAdapterSettings } from "./services/adapter-settings.js";
import { migrateCloudConfiguration } from "./services/cloud-configuration-migration.js";
import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js";
import { systemdNotify } from "./services/systemd-notify.js";
import { flushInFlightRunLogMirrors } from "./services/run-log-store.js";
import {
@@ -719,6 +721,7 @@ async function startServerWithDatabaseTeardown(
};
startupDatabase.close = closeDatabaseClients;
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
await migrateCloudConfiguration(db as any);
await initializeAdapterSettings(db as any);
await initializeCloudRuntimeConfig(db as any);
@@ -977,6 +980,10 @@ async function startServerWithDatabaseTeardown(
// document parsed fail-closed above (`plugins.autoInstall`). Absent env means
// self-hosted: createApp falls back to its built-in kubernetes-only default.
const managedPluginAutoInstall = managedConfig?.plugins.autoInstall ?? null;
const cloudQualification = await qualifyCloudConfigurationForCurrentImage(db as any);
if (cloudQualification && !cloudQualification.configurationPackagesEligible) {
logger.warn({ scope: cloudQualification.scope, blockerCodes: cloudQualification.blockers.map((blocker) => blocker.code) }, "Cloud configuration and package qualification has blockers");
}
const app = await createApp(db as any, {
prepareIdleDatabaseBackup: idleDatabaseBackupEnabled
? async () => (await runServerDatabaseBackup("idle")) !== null
+47
View File
@@ -0,0 +1,47 @@
import path from "node:path";
import { pathToFileURL } from "node:url";
import { createDb } from "@paperclipai/db";
import { cloudConfigurationMigrationService } from "./services/cloud-configuration-migration.js";
import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js";
export async function runCloudConfigurationQualification(args: string[], env: NodeJS.ProcessEnv = process.env): Promise<number> {
const allowed = new Set(["--migrate", "--expected-package-evidence", "--recheck-external-revision"]);
let migrate = false;
let expectedPackageEvidenceDigest: string | undefined;
let recheckRevision: number | undefined;
for (let index = 0; index < args.length; index += 1) {
const argument = args[index];
if (!allowed.has(argument)) throw new Error(`Unknown qualification argument: ${argument}`);
if (argument === "--migrate") migrate = true;
if (argument === "--expected-package-evidence") {
expectedPackageEvidenceDigest = args[++index];
if (!expectedPackageEvidenceDigest || !/^sha256:[a-f0-9]{64}$/.test(expectedPackageEvidenceDigest)) throw new Error("A SHA-256 package evidence digest is required");
}
if (argument === "--recheck-external-revision") {
const value = args[++index];
if (!value || !/^\d+$/.test(value) || !Number.isSafeInteger(Number(value))) throw new Error("A nonnegative receipt revision is required");
recheckRevision = Number(value);
}
}
const url = env.DATABASE_URL?.trim();
if (!url) throw new Error("DATABASE_URL is required for Cloud configuration qualification");
const db = createDb(url);
try {
const service = cloudConfigurationMigrationService(db, { runtimeEnv: env });
if (migrate) await service.migrate();
if (recheckRevision !== undefined) await service.recheckLegacyExternalRegistrations(recheckRevision);
const report = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv: env, expectedPackageEvidenceDigest });
if (!report) throw new Error("Cloud configuration qualification requires a managed instance");
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
return report.configurationPackagesEligible && (!expectedPackageEvidenceDigest || report.samePackageOverlapEligible) ? 0 : 2;
} finally {
await db.$client.end({ timeout: 5 });
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
runCloudConfigurationQualification(process.argv.slice(2)).then((code) => { process.exitCode = code; }).catch((error) => {
process.stderr.write(`${error instanceof Error ? error.message : "Cloud configuration qualification failed"}\n`);
process.exitCode = 1;
});
}
+4 -4
View File
@@ -45,8 +45,8 @@ interface AdapterSettings {
// Paths
// ---------------------------------------------------------------------------
function adapterPluginPaths() {
const paperclipDir = resolvePaperclipHomeDir();
function adapterPluginPaths(homeOverride?: string) {
const paperclipDir = resolvePaperclipHomeDir(homeOverride);
return {
adapterPluginsDir: path.join(paperclipDir, "adapter-plugins"),
adapterPluginsStorePath: path.join(paperclipDir, "adapter-plugins.json"),
@@ -133,8 +133,8 @@ function writeSettings(settings: AdapterSettings): void {
// Public API
// ---------------------------------------------------------------------------
export function getAdapterPluginStorePath(): string {
return adapterPluginPaths().adapterPluginsStorePath;
export function getAdapterPluginStorePath(homeOverride?: string): string {
return adapterPluginPaths(homeOverride).adapterPluginsStorePath;
}
export function listAdapterPlugins(): AdapterPluginRecord[] {
+16 -7
View File
@@ -2,6 +2,7 @@ import type { Db } from "@paperclipai/db";
import { instanceSettings } from "@paperclipai/db";
import { eq } from "drizzle-orm";
import { open } from "node:fs/promises";
import { constants } from "node:fs";
import path from "node:path";
import { z } from "zod";
import { listServerAdapters } from "../adapters/registry.js";
@@ -118,23 +119,31 @@ export async function initializeAdapterSettings(db: AdapterSettingsDb, options:
const service = adapterSettingsService(db, options);
const current = await service.inspectStored();
if (current) return { seeded: false, current };
return service.seedIfAbsent(await readLegacyAdapterSettings(env) ?? { disabledTypes: [] });
}
export async function readLegacyAdapterSettings(env: Record<string, string | undefined> = process.env): Promise<{ disabledTypes: string[] } | null> {
const filePath = path.join(resolvePaperclipHomeDir(env.PAPERCLIP_HOME), "adapter-settings.json");
let disabledTypes: string[] = [];
try {
const file = await open(filePath, "r");
const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await file.stat();
if (!stat.isFile() || stat.size > 65_536) throw new Error("Legacy adapter settings exceed the size limit or are not a regular file");
const buffer = Buffer.alloc(65_537);
const { bytesRead } = await file.read(buffer, 0, buffer.length, 0);
if (bytesRead > 65_536) throw new Error("Legacy adapter settings exceed the size limit");
const value = JSON.parse(buffer.subarray(0, bytesRead).toString("utf8"));
disabledTypes = parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes;
let total = 0;
while (total < buffer.length) {
const { bytesRead } = await file.read(buffer, total, buffer.length - total, total);
if (bytesRead === 0) break;
total += bytesRead;
}
if (total > 65_536) throw new Error("Legacy adapter settings exceed the size limit");
const value = JSON.parse(buffer.subarray(0, total).toString("utf8"));
return { disabledTypes: parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes };
} finally {
await file.close();
}
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
return null;
}
return service.seedIfAbsent({ disabledTypes });
}
@@ -0,0 +1,47 @@
import fs from "node:fs";
import path from "node:path";
import { createHash } from "node:crypto";
import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js";
export function cloudBundledPackageDigest(packageRoot: string): string {
const root = fs.realpathSync(packageRoot);
const inventory: Array<[string, string, string, number]> = [];
let bytes = 0;
let entries = 0;
function walk(directory: string, relative = "") {
if (relative.split("/").length > 32) throw new Error("Image package exceeds the directory depth limit");
for (const name of fs.readdirSync(directory).sort()) {
if (++entries > 10_000) throw new Error("Image package exceeds the entry limit");
const local = relative ? `${relative}/${name}` : name;
const file = path.join(directory, name);
const stat = fs.lstatSync(file);
if (stat.isSymbolicLink()) {
const target = fs.realpathSync(file);
const contained = path.relative(root, target);
if (!contained || contained.startsWith("..") || path.isAbsolute(contained)) throw new Error("Image package symlink escapes its package");
const targetStat = fs.statSync(target);
if (!targetStat.isFile() && !targetStat.isDirectory()) throw new Error("Image package symlink has an unsupported target");
inventory.push([local, "link", fs.readlinkSync(file), stat.mode & 0o777]);
} else if (stat.isDirectory()) walk(file, local);
else if (stat.isFile()) {
if (bytes + stat.size > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit");
const handle = fs.openSync(file, "r");
const hash = createHash("sha256");
const buffer = Buffer.alloc(65_536);
try {
while (true) {
const count = fs.readSync(handle, buffer);
if (!count) break;
bytes += count;
if (bytes > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit");
hash.update(buffer.subarray(0, count));
}
} finally { fs.closeSync(handle); }
inventory.push([local, "file", `sha256:${hash.digest("hex")}`, stat.mode & 0o777]);
} else throw new Error("Image package contains a special file");
}
}
walk(root);
if (!inventory.length) throw new Error("Image package is empty");
return cloudConfigurationDigest(inventory);
}
@@ -0,0 +1,74 @@
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { open } from "node:fs/promises";
import { z } from "zod";
function canonical(value: unknown): unknown {
if (Array.isArray(value)) return value.map(canonical);
if (value && typeof value === "object") {
return Object.fromEntries(Object.entries(value).sort(([left], [right]) => left < right ? -1 : left > right ? 1 : 0).map(([key, child]) => [key, canonical(child)]));
}
return value;
}
export function cloudConfigurationDigest(value: unknown): string {
return `sha256:${createHash("sha256").update(JSON.stringify(canonical(value))).digest("hex")}`;
}
export const externalRegistrationEvidenceSchema = z.object({
sourceLocatorDigest: z.string(),
sourceDigest: z.string(),
state: z.enum(["missing", "parsed", "invalid", "unreadable"]),
adapterTypes: z.array(z.string()),
blocker: z.enum(["unsupported_external_adapters", "invalid_external_registration_metadata", "unreadable_external_registration_metadata"]).nullable(),
}).strict();
export type ExternalRegistrationEvidence = z.infer<typeof externalRegistrationEvidenceSchema>;
const legacyRegistrationSchema = z.array(z.object({
type: z.string().regex(/^[a-zA-Z0-9_.-]{1,128}$/),
packageName: z.string().min(1).max(1024),
version: z.string().max(128).optional(),
localPath: z.string().max(4096).optional(),
disabled: z.boolean().optional(),
}).passthrough()).max(100);
export async function inspectLegacyExternalRegistrations(filePath: string): Promise<ExternalRegistrationEvidence> {
const sourceLocatorDigest = cloudConfigurationDigest(filePath);
try {
const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
let raw: unknown;
try {
const stat = await file.stat();
if (!stat.isFile() || stat.size > 65_536) throw new Error("invalid legacy registration file");
const buffer = Buffer.alloc(65_537);
let total = 0;
while (total < buffer.length) {
const { bytesRead } = await file.read(buffer, total, buffer.length - total, total);
if (!bytesRead) break;
total += bytesRead;
}
if (total > 65_536) throw new Error("oversized legacy registration file");
raw = JSON.parse(buffer.subarray(0, total).toString("utf8"));
} finally {
await file.close();
}
const records = legacyRegistrationSchema.safeParse(raw);
if (!records.success) {
return { sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state: "invalid" }), state: "invalid", adapterTypes: [], blocker: "invalid_external_registration_metadata" };
}
const metadata = records.data.map((record) => ({ type: record.type, packageName: record.packageName, version: record.version, disabled: record.disabled, hasLocalPath: record.localPath !== undefined }));
return {
sourceLocatorDigest, sourceDigest: cloudConfigurationDigest(metadata), state: "parsed",
adapterTypes: [...new Set(metadata.map((record) => record.type))].sort(),
blocker: metadata.length ? "unsupported_external_adapters" : null,
};
} catch (error) {
const absent = (error as NodeJS.ErrnoException).code === "ENOENT";
const invalid = error instanceof SyntaxError || (error instanceof Error && /legacy registration file/.test(error.message));
const state = absent ? "missing" : invalid ? "invalid" : "unreadable";
return {
sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state }), state, adapterTypes: [],
blocker: absent ? null : invalid ? "invalid_external_registration_metadata" : "unreadable_external_registration_metadata",
};
}
}
@@ -0,0 +1,192 @@
import { instanceSettings, plugins, type Db } from "@paperclipai/db";
import { eq, inArray, ne } from "drizzle-orm";
import { z } from "zod";
import { adapterSettingsService, readLegacyAdapterSettings } from "./adapter-settings.js";
import { getAdapterPluginStorePath } from "./adapter-plugin-store.js";
import { cloudRuntimeConfigService, readLegacyCloudRuntimeConfig } from "./cloud-runtime-config.js";
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
import { cloudConfigurationDigest, externalRegistrationEvidenceSchema, inspectLegacyExternalRegistrations } from "./cloud-configuration-evidence.js";
import { cloudPackageEvidenceSchema, inspectCloudPackages, qualificationBlockerSchema } from "./cloud-package-qualification.js";
import type { ResolvedBundledPlugin } from "./bundled-plugins.js";
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
const RECEIPT_KEY = "cloud-configuration-migration:v1";
const CONFIGURATION_KEYS = ["adapter-settings:v1", "cloud-runtime-config:v1"];
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export type CloudConfigurationDb = Pick<Db | Transaction, "select" | "insert" | "update" | "transaction">;
const sourceSchema = z.object({
authority: z.enum(["database", "legacy_file", "default"]), sourceDigest: z.string(),
importedRevision: z.number().int().nonnegative(),
}).strict();
const qualificationSchema = z.object({
scope: z.literal("configuration_and_packages"), checkedAt: z.string(),
adapterRevision: z.number().int().nonnegative(), configRevision: z.number().int().nonnegative(),
configurationDigest: z.string(), packageEvidenceDigest: z.string(),
packageInspectionDurationMs: z.number().nonnegative(),
expectedPackageEvidenceDigest: z.string().nullable(),
configurationPackagesEligible: z.boolean(), samePackageOverlapEligible: z.boolean(),
packages: z.array(cloudPackageEvidenceSchema), blockers: z.array(qualificationBlockerSchema),
overlapBlockers: z.array(qualificationBlockerSchema),
}).strict();
const receiptSchema = z.object({
version: z.literal(1), revision: z.number().int().nonnegative(),
migration: z.object({ completedAt: z.string(), adapter: sourceSchema, config: sourceSchema, externalRegistrations: externalRegistrationEvidenceSchema }).strict(),
qualification: qualificationSchema.nullable(), lastPackageMismatch: qualificationSchema.nullable(),
}).strict();
export type CloudConfigurationReceipt = z.infer<typeof receiptSchema>;
function parseReceipt(value: unknown): CloudConfigurationReceipt {
const result = receiptSchema.safeParse(value);
if (!result.success) throw new Error("Persisted Cloud configuration migration receipt is invalid");
return result.data;
}
export function cloudConfigurationMigrationService(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv; now?: () => Date } = {}) {
const env = options.runtimeEnv ?? process.env;
const now = () => (options.now?.() ?? new Date()).toISOString();
const adapters = adapterSettingsService(db, { runtimeEnv: env });
const config = cloudRuntimeConfigService(db);
function assertManaged() {
if (!isCloudManagedInstance(env)) throw new Error("Cloud configuration qualification requires a managed instance");
}
async function readRow(handle: CloudConfigurationDb, lock = false) {
const query = handle.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, RECEIPT_KEY));
const [row] = lock ? await query.for("update") : await query;
return row ?? null;
}
async function currentConfiguration(handle: CloudConfigurationDb) {
await handle.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS))
.orderBy(instanceSettings.singletonKey).for("update");
const adapter = await adapters.inspectStored(handle);
const runtime = await config.inspectStored(handle);
if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete");
return { adapter, runtime };
}
async function registrationSnapshot(handle: CloudConfigurationDb, lock = false) {
const query = handle.select({
pluginKey: plugins.pluginKey, packageName: plugins.packageName, version: plugins.version,
manifestJson: plugins.manifestJson, status: plugins.status,
}).from(plugins).where(ne(plugins.status, "uninstalled")).orderBy(plugins.pluginKey);
return lock ? query.for("share") : query;
}
async function save(handle: CloudConfigurationDb, id: string, receipt: CloudConfigurationReceipt) {
const validated = parseReceipt(receipt);
const updated = await handle.update(instanceSettings).set({ general: validated, updatedAt: new Date() }).where(eq(instanceSettings.id, id)).returning({ id: instanceSettings.id });
if (updated.length !== 1) throw new Error("Cloud configuration migration receipt was not persisted");
return validated;
}
return {
async inspectStored(): Promise<CloudConfigurationReceipt | null> {
const row = await readRow(db);
return row ? parseReceipt(row.general) : null;
},
async migrate(): Promise<CloudConfigurationReceipt> {
assertManaged();
return db.transaction(async (tx) => {
const inserted = await tx.insert(instanceSettings).values({ singletonKey: RECEIPT_KEY, general: { version: 1 } })
.onConflictDoNothing({ target: instanceSettings.singletonKey }).returning({ id: instanceSettings.id });
const row = await readRow(tx, true);
if (!row) throw new Error("Cloud configuration migration receipt is unavailable");
if (!inserted.length) {
const receipt = parseReceipt(row.general);
await currentConfiguration(tx);
return receipt;
}
await tx.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS))
.orderBy(instanceSettings.singletonKey).for("update");
const storedAdapter = await adapters.inspectStored(tx);
const storedConfig = await config.inspectStored(tx);
const adapterCandidate = storedAdapter ? null : await readLegacyAdapterSettings(env);
const configCandidate = storedConfig ? null : await readLegacyCloudRuntimeConfig(env);
const adapterSeed = storedAdapter ? null : await adapters.seedIfAbsent({ disabledTypes: adapterCandidate?.disabledTypes ?? [] }, tx);
const configSeed = storedConfig ? null : await config.seedIfAbsent(configCandidate, tx);
const adapter = storedAdapter ?? adapterSeed!.current;
const runtime = storedConfig ?? configSeed!.current;
const adapterFromDatabase = Boolean(storedAdapter || !adapterSeed?.seeded);
const configFromDatabase = Boolean(storedConfig || !configSeed?.seeded);
const receipt: CloudConfigurationReceipt = {
version: 1, revision: 0,
migration: {
completedAt: now(),
adapter: { authority: adapterFromDatabase ? "database" : adapterCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(adapterFromDatabase ? { disabledTypes: adapter.disabledTypes } : adapterCandidate), importedRevision: adapter.revision },
config: { authority: configFromDatabase ? "database" : configCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(configFromDatabase ? runtime.config : configCandidate), importedRevision: runtime.revision },
externalRegistrations: await inspectLegacyExternalRegistrations(getAdapterPluginStorePath(env.PAPERCLIP_HOME)),
},
qualification: null, lastPackageMismatch: null,
};
return save(tx, row.id, receipt);
});
},
async recheckLegacyExternalRegistrations(expectedRevision: number): Promise<CloudConfigurationReceipt> {
assertManaged();
return db.transaction(async (tx) => {
const row = await readRow(tx, true);
if (!row) throw new Error("Cloud configuration migration has not completed");
const receipt = parseReceipt(row.general);
if (receipt.revision !== expectedRevision) throw new Error("Cloud configuration migration receipt revision conflict");
const filePath = getAdapterPluginStorePath(env.PAPERCLIP_HOME);
if (cloudConfigurationDigest(filePath) !== receipt.migration.externalRegistrations.sourceLocatorDigest) {
throw new Error("External registration recheck requires the original legacy source location");
}
const evidence = await inspectLegacyExternalRegistrations(filePath);
if (evidence.state !== "parsed") throw new Error("External registration recheck requires a present, valid registration file");
receipt.migration.externalRegistrations = evidence;
receipt.revision += 1;
receipt.qualification = null;
return save(tx, row.id, receipt);
});
},
async qualify(input: {
installs: readonly ResolvedBundledPlugin[];
loadManifest: (packageRoot: string) => Promise<PaperclipPluginManifestV1 | null>;
expectedPackageEvidenceDigest?: string;
imageCatalogUnavailable?: boolean;
}) {
assertManaged();
const adapter = await adapters.inspectStored();
const runtime = await config.inspectStored();
if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete");
const configurationBefore = cloudConfigurationDigest({ adapter, runtime });
const registrationsBefore = await registrationSnapshot(db);
const evidence = await inspectCloudPackages({ ...input, registrations: registrationsBefore });
return db.transaction(async (tx) => {
const row = await readRow(tx, true);
if (!row) throw new Error("Cloud configuration migration has not completed");
const receipt = parseReceipt(row.general);
const current = await currentConfiguration(tx);
const registrations = await registrationSnapshot(tx, true);
const blockers = [...evidence.blockers];
if (configurationBefore !== cloudConfigurationDigest(current) || cloudConfigurationDigest(registrationsBefore) !== cloudConfigurationDigest(registrations)) {
blockers.push({ code: "configuration_changed_during_inspection", reason: "Configuration or plugin registration changed during image inspection; run qualification again" });
}
if (input.imageCatalogUnavailable) blockers.push({ code: "current_image_catalog_unavailable", reason: "The current image package catalog could not be verified" });
const external = receipt.migration.externalRegistrations;
if (external.blocker) blockers.push({ code: external.blocker, reason: "Legacy external adapter registration metadata requires an explicit operator resolution" });
const overlapBlockers = input.expectedPackageEvidenceDigest && input.expectedPackageEvidenceDigest !== evidence.packageEvidenceDigest
? [{ code: "package_evidence_changed", reason: "Current packages do not match the expected package evidence digest" }] : [];
const qualification: z.infer<typeof qualificationSchema> = {
scope: "configuration_and_packages", checkedAt: now(),
adapterRevision: current.adapter.revision, configRevision: current.runtime.revision,
configurationDigest: cloudConfigurationDigest(current), packageEvidenceDigest: evidence.packageEvidenceDigest,
packageInspectionDurationMs: evidence.packageInspectionDurationMs,
expectedPackageEvidenceDigest: input.expectedPackageEvidenceDigest ?? null,
configurationPackagesEligible: blockers.length === 0,
samePackageOverlapEligible: Boolean(input.expectedPackageEvidenceDigest && blockers.length === 0 && overlapBlockers.length === 0),
packages: evidence.packages, blockers, overlapBlockers,
};
receipt.revision += 1;
receipt.qualification = qualification;
if (evidence.blockers.length || input.imageCatalogUnavailable || overlapBlockers.length) receipt.lastPackageMismatch = qualification;
await save(tx, row.id, receipt);
return { receiptRevision: receipt.revision, ...qualification };
});
},
};
}
export async function migrateCloudConfiguration(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv } = {}) {
if (!isCloudManagedInstance(options.runtimeEnv ?? process.env)) return null;
return cloudConfigurationMigrationService(db, options).migrate();
}
@@ -0,0 +1,33 @@
import type { Db } from "@paperclipai/db";
import { BUNDLED_PLUGIN_CATALOG, resolveBundledCatalogRoot, resolveBundledPluginInstalls, type ResolvedBundledPlugin } from "./bundled-plugins.js";
import { readDistributionPluginCatalog } from "./distribution-plugin-catalog.js";
import { createBundledPluginPackageResolver } from "./bundled-plugin-package.js";
import { cloudConfigurationMigrationService } from "./cloud-configuration-migration.js";
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
import { getManagedInstanceConfig } from "./managed-config.js";
import { pluginLoader } from "./plugin-loader.js";
export async function qualifyCloudConfigurationForCurrentImage(db: Db, options: {
runtimeEnv?: CloudInstanceEnv;
expectedPackageEvidenceDigest?: string;
} = {}) {
const env = options.runtimeEnv ?? process.env;
if (!isCloudManagedInstance(env)) return null;
const service = cloudConfigurationMigrationService(db, { runtimeEnv: env });
const catalogRoot = resolveBundledCatalogRoot(env);
let installs: ResolvedBundledPlugin[];
try {
const distribution = readDistributionPluginCatalog(catalogRoot, BUNDLED_PLUGIN_CATALOG);
installs = resolveBundledPluginInstalls(getManagedInstanceConfig(env)?.plugins.autoInstall ?? [], {
catalogRoot, env, enforceCatalogRoot: true, distributionPlugins: distribution,
});
} catch {
return service.qualify({ installs: [], loadManifest: async () => null, imageCatalogUnavailable: true, expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest });
}
const resolver = createBundledPluginPackageResolver(installs);
const loader = pluginLoader(db, {
prebuiltOnly: true, enableLocalFilesystem: false, enableNpmDiscovery: false,
assertPackageActivation: resolver.assertActivation, resolvePackageRoot: resolver.resolve,
});
return service.qualify({ installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest });
}
@@ -0,0 +1,71 @@
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
import { z } from "zod";
import { createBundledPluginPackageResolver, type BundledPluginRegistration } from "./bundled-plugin-package.js";
import type { ResolvedBundledPlugin } from "./bundled-plugins.js";
import { distributionBundleDigest } from "./distribution-plugin-catalog.js";
import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js";
import { cloudBundledPackageDigest } from "./cloud-bundled-package-digest.js";
export const qualificationBlockerSchema = z.object({
code: z.string(), pluginKey: z.string().optional(), reason: z.string(),
registeredVersion: z.string().optional(), imageVersion: z.string().optional(),
}).strict();
export type QualificationBlocker = z.infer<typeof qualificationBlockerSchema>;
export const cloudPackageEvidenceSchema = z.object({
catalogKey: z.string(), pluginKey: z.string(), packageDigest: z.string().nullable(),
imageManifestDigest: z.string().nullable(), approvedManifestDigest: z.string().nullable(),
}).strict();
export type CloudPackageEvidence = z.infer<typeof cloudPackageEvidenceSchema>;
export async function inspectCloudPackages(input: {
installs: readonly ResolvedBundledPlugin[];
registrations: readonly BundledPluginRegistration[];
loadManifest: (packageRoot: string) => Promise<PaperclipPluginManifestV1 | null>;
}) {
const started = performance.now();
const resolver = createBundledPluginPackageResolver(input.installs);
const registrations = new Map(input.registrations.map((record) => [record.pluginKey, record]));
const selected = new Set(input.installs.map((install) => install.pluginKey));
const blockers: QualificationBlocker[] = input.registrations.filter((record) => !selected.has(record.pluginKey)).map((record) => ({
code: "unsupported_plugin_registration", pluginKey: record.pluginKey, reason: "An installed plugin is not selected from the current image", registeredVersion: record.version,
}));
const packages: CloudPackageEvidence[] = [];
for (const install of [...input.installs].sort((left, right) => left.pluginKey < right.pluginKey ? -1 : left.pluginKey > right.pluginKey ? 1 : 0)) {
const registration = registrations.get(install.pluginKey);
const evidence: CloudPackageEvidence = {
catalogKey: install.key, pluginKey: install.pluginKey, packageDigest: null,
imageManifestDigest: null, approvedManifestDigest: registration ? cloudConfigurationDigest(registration.manifestJson) : null,
};
packages.push(evidence);
let stage = "missing_bundled_package";
try {
resolver.assertActivation({ packageRoot: install.localPath });
stage = "unverified_package_integrity";
evidence.packageDigest = install.distribution ? distributionBundleDigest(install.localPath) : cloudBundledPackageDigest(install.localPath);
if (install.distribution && evidence.packageDigest !== install.distribution.digest) {
blockers.push({ code: "package_integrity_mismatch", pluginKey: install.pluginKey, reason: "Current package bytes do not match the verified distribution digest" });
continue;
}
stage = "missing_manifest_evidence";
const manifest = await input.loadManifest(install.localPath);
if (!manifest) {
blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image has no verified plugin manifest" });
continue;
}
evidence.imageManifestDigest = cloudConfigurationDigest(manifest);
if (!registration) {
blockers.push({ code: "missing_plugin_registration", pluginKey: install.pluginKey, reason: "The current image package is not registered in the database", imageVersion: manifest.version });
continue;
}
const inspected = resolver.inspect(registration, manifest);
if (!inspected.compatible) blockers.push({
code: inspected.code, pluginKey: install.pluginKey, reason: inspected.reason,
registeredVersion: registration.version, imageVersion: manifest.version,
});
} catch {
blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image package could not be verified" });
}
}
const packageEvidenceDigest = cloudConfigurationDigest(packages.map(({ approvedManifestDigest: _approved, ...evidence }) => evidence));
return { packages, blockers, packageEvidenceDigest, packageInspectionDurationMs: Math.round(performance.now() - started) };
}