diff --git a/doc/DATABASE.md b/doc/DATABASE.md index b77d312c735..b794f1e51ee 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -755,6 +755,27 @@ The focused PostgreSQL suite can use an owned disposable PostgreSQL database via unavailable. It applies migrations and inserts test data, so never point this variable at an existing application database. +## Cloud configuration migration receipt + +The private `instance_settings` singleton `cloud-configuration-migration:v1` +records atomic import completion, semantic source digests, imported revisions, +and metadata-only external adapter blockers. It composes the adapter and +startup-config owners without replacing an existing authoritative row. Receipt +locking and the import transaction handle concurrent attempts and retries; no +local completion marker participates in conflict resolution. + +Qualification records current configuration revisions and selected image package +evidence before normal plugin reconciliation. Image inspection does not hold +admin write locks; configuration and registration snapshots are rechecked in a +short transaction before the report is saved. A historical successful report +never grants current eligibility. External metadata blockers survive loss of the +legacy home and require a deliberate revision-checked recheck of a present valid +original source file. Package mismatches remain recorded after maintenance +reconciliation. See the [bounded migration and qualification procedure](cloud-configuration-migration.md) +for the stopped-writer prerequisite, commands, retry semantics, integrity limits, +and the explicit exclusion of secret bytes, whole-server zero downtime, and +volume removal. + ## Cloud runtime identity singleton The private `instance_settings` row whose singleton key is diff --git a/doc/cloud-configuration-migration.md b/doc/cloud-configuration-migration.md new file mode 100644 index 00000000000..92c16843aeb --- /dev/null +++ b/doc/cloud-configuration-migration.md @@ -0,0 +1,131 @@ +# Cloud configuration migration and package qualification + +This procedure covers saved adapter preferences, the allowlisted nonsecret +startup configuration, and selected bundled plugin packages. The result is a +point-in-time report. It does not authorize a deployment, guarantee zero +downtime, establish general Railway admission, or prove that a volume can be +removed. Secret and key bytes, database credentials, artifacts, logs, local +storage, workspaces, and plugin state outside PostgreSQL need their own storage +and replacement checks. This includes the decision signing key and local +encryption keys. Their byte authority is unchanged by this procedure. + +## Migration prerequisite + +Stop every legacy file writer before the first import. This includes old +application replicas and operator commands that can write configuration or +adapter settings. One maintenance window is sufficient. The importer cannot +make an old file writer participate in its database transaction. Keep those +writers stopped after import; later file edits do not update database authority. + +Keep the normal database backup and recovery procedure for the instance. Use the +new release with the same managed environment and database connection. The +server runs migration before constructing configuration-dependent services. +For an explicit source-checkout invocation, use: + +```sh +pnpm exec tsx server/src/qualify-cloud-configuration.ts --migrate +``` + +The command uses `DATABASE_URL` and the normal Cloud environment. It does not +accept credentials as arguments. It writes the receipt to PostgreSQL and prints +a JSON report; no report file or local completion marker is required. Exit code +0 means the requested configuration/package check passed, 2 means the report has +blockers, and 1 means the operation failed. A fresh installation can report a +missing plugin registration before normal boot installation completes. Run the +check again after that installation settles. + +## Database authority and retry behavior + +The private `instance_settings` singleton `cloud-configuration-migration:v1` +contains the versioned receipt. The transaction locks that row and composes the +adapter and startup-config owners' atomic seed methods. Completion and both +seeds commit together. A failed write or crash before commit leaves no completed +receipt. A retry uses committed database state and never replaces a later +settings revision. Earlier independently seeded owner records remain authoritative. + +The receipt records source kind, a digest of the nonsecret semantic candidate, +the imported revision, and bounded legacy external-registration evidence. +An absent startup configuration remains an authoritative `config: null`; it is +distinct from an uninitialized database record. No raw secret values are copied. +Once the receipt exists, ordinary boots do not reread legacy source files. +A replacement home can be empty or absent without changing saved authority. + +All legacy readers have a 64 KiB limit. They require regular files, reject +symlinks and FIFOs, and use bounded reads. External registration inspection reads +metadata only. It never loads a registered module or follows a registered package +path. A nonempty, invalid, or unreadable registration file creates a durable +blocker. Removing the old home does not clear that blocker. + +To resolve an external-registration blocker, stop its writer, deliberately +remove the unsupported registration metadata at the original source location, +and leave a present valid empty JSON array file. Read the current receipt revision +from the report, then run: + +```sh +pnpm exec tsx server/src/qualify-cloud-configuration.ts --recheck-external-revision 3 +``` + +Use the actual current revision in place of `3`. A changed revision, a different +source location, or a missing/invalid source file rejects the recheck. Reusing +the same container path with an empty home is insufficient. This is an explicit +operator resolution; qualification never clears the blocker automatically. + +## Current packages and overlap comparison + +Every qualification reads current settings revisions and plugin registrations. +It resolves selected packages from this image with the same bundled package +identity rules as normal loading. It verifies package bytes before importing a +trusted image manifest, then compares that manifest with the approved database +registration. Missing packages, missing evidence, incompatible versions, +unselected registrations, and integrity failures prevent qualification. Legacy +package paths and npm discovery cannot supply missing image bytes. + +Image inspection happens outside database write locks. A short transaction then +locks and rereads the current configuration and registration records. A change +during inspection produces a blocker and requires a new check. The report gives +`adapterRevision`, `configRevision`, `configurationDigest`, and +`packageEvidenceDigest`. A historical completed receipt is evidence, not current +authorization. Call qualification again for a current result. + +The server records this report before ordinary bundled-plugin reconciliation. +It retains `lastPackageMismatch` after later successful checks, so reconciliation +cannot erase the evidence seen before an upgrade. The report does not block +normal maintenance upgrades or replace capability approval. After an upgrade +settles, its current packages can qualify normally. + +For a same-package comparison, capture the current release's +`packageEvidenceDigest` and pass it explicitly on the replacement: + +```sh +pnpm exec tsx server/src/qualify-cloud-configuration.ts --expected-package-evidence sha256:REPLACE_WITH_THE_64_HEX_DIGEST +``` + +`configurationPackagesEligible` describes current database/configuration/package +compatibility. `samePackageOverlapEligible` is true only when an explicit expected +digest also matches. There is no permanent first-release baseline that would +prevent future maintenance releases from establishing their own evidence. + +The package evidence digest covers selected packages, not container image bytes. +Distribution packages reuse and reverify their catalog digest. Builtin packages +have a bounded inventory of physical file bytes, modes, and contained symlink +text, including standalone pnpm dependency trees. Dangling or escaping links, +special files, more than 10,000 entries, depth above 32, or more than 256 MiB per +package fail verification. Inventory work is synchronous and linear in selected +package bytes. It runs only during startup before the application is constructed +or in the explicit operator process, with no idle polling. The report includes +`packageInspectionDurationMs`. Packages must remain immutable throughout a check. + +## Verification scope + +The focused PostgreSQL tests use two independent clients and separate home/image +roots. They import legacy settings, change saved revisions, remove the old roots, +reconstruct from the database, and start a real plugin worker from the replacement +image fixture. They also cover concurrent migration, commit failure and retry, +database failure, stale sources, durable external blockers, package mismatch, +maintenance reconciliation, and changes during manifest inspection. + +Set `PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL` only to an owned disposable +PostgreSQL database when embedded PostgreSQL is unavailable. The suite applies +migrations and writes/deletes test records. A complete release image, hosted +replacement, overlapping production processes, secret-byte migration, and volume +removal are not established by these local fixtures. diff --git a/maps/agent-access-map/instance-operations.md b/maps/agent-access-map/instance-operations.md index b58eb870b6e..3e21ef72901 100644 --- a/maps/agent-access-map/instance-operations.md +++ b/maps/agent-access-map/instance-operations.md @@ -112,6 +112,17 @@ exercises persistence, concurrent writers, admission freshness, activity audit, and failed writes; [authorization tests](../../server/src/__tests__/adapter-routes-authz.test.ts) cover admin and hidden-setting gates. These are local tests, not deployed proof. +Cloud configuration migration and package qualification are operator commands, +not agent-accessible REST capabilities. They require the managed environment and +database operator access. The [command](../../server/src/qualify-cloud-configuration.ts) +composes transaction-safe owner seeds and writes a private receipt. Every +qualification rereads current revisions and image package evidence; a completed +historical receipt grants no deployment or execution authority. [PostgreSQL coverage](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts) +uses independent clients, lost-home reconstruction, a real replacement fixture +worker, rollback/retry, and durable blockers. [Reader coverage](../../server/src/__tests__/cloud-configuration-readers.test.ts) +checks bounded sources and pnpm-style link integrity. These are local fixtures; +see the [scope and operator procedure](../../doc/cloud-configuration-migration.md). + Cloud stack Admins may update experimental settings through the trusted Cloud actor without receiving other instance-admin powers. This uses the current stack role on each request; company membership alone does not grant it. diff --git a/maps/feature-map/instance-operations.md b/maps/feature-map/instance-operations.md index daae7ef5955..7a01c24ebf6 100644 --- a/maps/feature-map/instance-operations.md +++ b/maps/feature-map/instance-operations.md @@ -75,6 +75,16 @@ These controls do not qualify a deployed Cloud rollout or durable file storage. Automated: [experimental settings](../../ui/src/pages/InstanceExperimentalSettings.test.tsx) and [database backup routes](../../server/src/__tests__/instance-database-backups-routes.test.ts) cover their UI/API contracts. +Cloud migration: in one maintenance window, stop legacy file writers and run the +[configuration/package qualification command](../../doc/cloud-configuration-migration.md). +Inspect its database receipt, update a saved preference, then repeat against a +second owned application/image root with the first home unavailable. Use an +explicit package evidence digest for same-package comparison. Confirm stale +qualification and missing legacy files cannot clear blockers. This recipe covers +configuration/packages and does not qualify whole-server zero downtime, secret +bytes, or volume removal. [Local PostgreSQL evidence path](../../server/src/__tests__/cloud-configuration-migration-postgres.test.ts) +includes a real fixture worker; it does not establish a hosted replacement. + Managed adapters: change a visible adapter preference as an instance admin, then use a second application process against the same disposable database to confirm catalog and new-agent selection reflect it. Verify saved agents remain diff --git a/server/src/__tests__/cloud-configuration-migration-postgres.test.ts b/server/src/__tests__/cloud-configuration-migration-postgres.test.ts new file mode 100644 index 00000000000..43582478155 --- /dev/null +++ b/server/src/__tests__/cloud-configuration-migration-postgres.test.ts @@ -0,0 +1,305 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { applyPendingMigrations, closeRegisteredClients, createDb, instanceSettings, plugins, type Db } from "@paperclipai/db"; +import { eq, inArray, sql } from "drizzle-orm"; +import { paperclipConfigSchema } from "@paperclipai/shared"; +import { adapterSettingsService, initializeAdapterSettings } from "../services/adapter-settings.js"; +import { cloudRuntimeConfigService, initializeCloudRuntimeConfig } from "../services/cloud-runtime-config.js"; +import { cloudConfigurationMigrationService } from "../services/cloud-configuration-migration.js"; +import { qualifyCloudConfigurationForCurrentImage } from "../services/cloud-configuration-qualification.js"; +import { runCloudConfigurationQualification } from "../qualify-cloud-configuration.js"; +import { pluginLoader, type PluginRuntimeServices } from "../services/plugin-loader.js"; +import { pluginRegistryService } from "../services/plugin-registry.js"; +import { createPluginWorkerManager } from "../services/plugin-worker-manager.js"; +import { ensureBundledPlugins } from "../services/bundled-plugins.js"; +import { bundledImageFixture } from "./helpers/bundled-image-fixture.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const externalDatabaseUrl = process.env.PAPERCLIP_CLOUD_CONFIGURATION_TEST_DATABASE_URL?.trim(); +const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configuration-migration:v1"]; +(support.supported ? describe : describe.skip)("Cloud configuration migration and current-image qualification", () => { + let database: Awaited>; + let db: Db; + let peer: Db; + let homeA: string; + let homeB: string; + const roots: string[] = []; + const fixture = (version = "1.0.0") => { const image = bundledImageFixture({ version }); roots.push(image.root); return image; }; + const env = (home = homeA) => ({ PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "test-cloud-token", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: path.join(home, "config.json") }); + const migration = (client = db, home = homeA) => cloudConfigurationMigrationService(client, { runtimeEnv: env(home) }); + beforeAll(async () => { + if (externalDatabaseUrl) { + await applyPendingMigrations(externalDatabaseUrl); + database = { connectionString: externalDatabaseUrl, cleanup: () => closeRegisteredClients(externalDatabaseUrl) }; + } else database = await startEmbeddedPostgresTestDatabase("cloud-configuration-migration-"); + db = createDb(database.connectionString); + peer = createDb(database.connectionString); + }, 90_000); + beforeEach(async () => { + homeA = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-legacy-A-")); + homeB = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-replacement-B-")); + roots.push(homeA, homeB); + await db.delete(instanceSettings).where(inArray(instanceSettings.singletonKey, keys)); + await db.delete(plugins); + }); + afterEach(async () => { + vi.restoreAllMocks(); + for (const root of roots.splice(0)) await fs.rm(root, { recursive: true, force: true }); + }); + afterAll(async () => { await database?.cleanup(); }); + + async function legacy() { + const config = paperclipConfigSchema.parse({ + $meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" }, + database: { mode: "postgres" }, logging: { mode: "cloud" }, + server: { port: 3199 }, telemetry: { enabled: false }, + }); + await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config)); + await fs.writeFile(path.join(homeA, "adapter-settings.json"), JSON.stringify({ disabledTypes: ["claude_local", "future_adapter"] })); + await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]"); + return config; + } + + function loaderFor(image: ReturnType, client = db, services?: PluginRuntimeServices) { + return pluginLoader(client, { + localPluginDir: path.join(homeB, "plugins"), prebuiltOnly: true, + resolvePackageRoot: image.resolver.resolve, assertPackageActivation: image.resolver.assertActivation, + enableLocalFilesystem: false, enableNpmDiscovery: false, + }, services); + } + async function install(image: ReturnType) { + await loaderFor(image).installPlugin({ localPath: image.packageRoot }); + const registry = pluginRegistryService(db); + const row = (await registry.getByKey(image.manifest.id))!; + await registry.updateStatus(row.id, { status: "ready" }); + return row; + } + function qualify(image: ReturnType, client = db, home = homeA, expectedPackageEvidenceDigest?: string) { + const loader = loaderFor(image, client); + return migration(client, home).qualify({ installs: image.installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest }); + } + + it("imports A atomically, retains later DB revisions, and starts B's real image worker after A disappears", async () => { + await legacy(); + const oldImage = fixture(); + const newImage = fixture(); + const row = await install(oldImage); + const first = await migration().migrate(); + expect(first.migration.adapter.authority).toBe("legacy_file"); + expect(first.migration.config.authority).toBe("legacy_file"); + const original = await qualify(oldImage); + expect(original.configurationPackagesEligible).toBe(true); + expect(original.samePackageOverlapEligible).toBe(false); + await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true); + const config = (await cloudRuntimeConfigService(peer).inspectStored())!; + expect(config.config).not.toBeNull(); + await cloudRuntimeConfigService(peer).replace({ ...config.config!, server: { ...config.config!.server, port: 3299 } }, config.revision); + await fs.rm(homeA, { recursive: true }); + await fs.rm(oldImage.root, { recursive: true }); + expect((await migration(peer, homeB).migrate()).migration).toEqual(first.migration); + await initializeAdapterSettings(peer, { runtimeEnv: env(homeB) }); + const restored = await initializeCloudRuntimeConfig(peer, { runtimeEnv: env(homeB) }); + expect(restored?.current.config?.server.port).toBe(3299); + expect(await adapterSettingsService(peer, { runtimeEnv: env(homeB) }).getDisabledTypes()).toEqual(["claude_local", "codex_local", "future_adapter"]); + const replacement = await qualify(newImage, peer, homeB, original.packageEvidenceDigest); + expect(replacement).toMatchObject({ adapterRevision: 1, configRevision: 1, configurationPackagesEligible: true, samePackageOverlapEligible: true }); + expect(replacement.configurationDigest).not.toBe(original.configurationDigest); + const manager = createPluginWorkerManager(); + try { + const services = { + workerManager: manager, lifecycleManager: { markError: vi.fn() }, + jobStore: { syncJobDeclarations: vi.fn() }, jobScheduler: { registerPlugin: vi.fn() }, + toolDispatcher: {}, eventBus: { forPlugin: vi.fn(), subscriptionCount: vi.fn(() => 0) }, + buildHostHandlers: vi.fn(() => ({})), instanceInfo: { instanceId: "qualification-fixture", hostVersion: "1.0.0" }, + } as unknown as PluginRuntimeServices; + expect((await loaderFor(newImage, peer, services).loadSingle(row.id)).success).toBe(true); + expect(await manager.call(row.id, "health", {})).toMatchObject({ status: "ok", details: { source: expect.stringContaining(newImage.packageRoot) } }); + } finally { await manager.stopAll(); } + }); + + it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => { + await legacy(); + const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]); + expect(first).toEqual(second); + await adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("claude_local", false); + await fs.writeFile(path.join(homeA, "config.json"), "invalid stale file"); + await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid stale file"); + expect(await migration().migrate()).toEqual(first); + expect(await adapterSettingsService(db, { runtimeEnv: env() }).getDisabledTypes()).toEqual(["future_adapter"]); + }); + + it("rolls back both seeds and the receipt if completion fails, then retries after the crash boundary", async () => { + await legacy(); + await db.execute(sql`CREATE FUNCTION reject_cloud_migration_receipt() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.singleton_key = 'cloud-configuration-migration:v1' THEN RAISE EXCEPTION 'test receipt failure'; END IF; RETURN NEW; END $$`); + await db.execute(sql`CREATE TRIGGER reject_cloud_migration_receipt BEFORE UPDATE ON instance_settings FOR EACH ROW EXECUTE FUNCTION reject_cloud_migration_receipt()`); + try { + await expect(migration().migrate()).rejects.toMatchObject({ cause: { message: expect.stringContaining("test receipt failure") } }); + expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]); + } finally { + await db.execute(sql`DROP TRIGGER reject_cloud_migration_receipt ON instance_settings`); + await db.execute(sql`DROP FUNCTION reject_cloud_migration_receipt()`); + } + expect((await migration(peer).migrate()).revision).toBe(0); + expect((await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).length).toBe(3); + }); + + it("fails without partial authority for malformed legacy config and preserves already-authoritative rows", async () => { + await legacy(); + await fs.writeFile(path.join(homeA, "config.json"), "invalid"); + await expect(migration().migrate()).rejects.toThrow("Legacy Cloud runtime configuration"); + expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]); + await adapterSettingsService(db, { runtimeEnv: env() }).seedIfAbsent({ disabledTypes: ["codex_local"] }); + await cloudRuntimeConfigService(db).seedIfAbsent(null); + await fs.writeFile(path.join(homeA, "adapter-settings.json"), "invalid"); + const receipt = await migration(peer).migrate(); + expect(receipt.migration.adapter.authority).toBe("database"); + expect(receipt.migration.config.authority).toBe("database"); + expect((await cloudRuntimeConfigService(peer).inspectStored())?.config).toBeNull(); + }); + + it("returns database errors instead of falling back to files or a completed receipt", async () => { + await legacy(); + await migration().migrate(); + const broken = createDb(database.connectionString); + await broken.$client.end({ timeout: 0 }); + await expect(migration(broken).migrate()).rejects.toThrow(); + const image = fixture(); + await expect(qualify(image, broken)).rejects.toThrow(); + }); + + it("keeps external registration blockers durable when B has no legacy home and requires deliberate original-source resolution", async () => { + await legacy(); + const sentinel = path.join(homeA, "external-code-executed"); + await fs.writeFile(path.join(homeA, "external.js"), `require('node:fs').writeFileSync(${JSON.stringify(sentinel)}, 'executed');`); + await fs.writeFile(path.join(homeA, "adapter-plugins.json"), JSON.stringify([{ type: "legacy_external", packageName: "legacy-adapter", localPath: path.join(homeA, "external.js") }])); + const image = fixture(); + await install(image); + await migration().migrate(); + expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "unsupported_external_adapters" })); + await expect(fs.stat(sentinel)).rejects.toMatchObject({ code: "ENOENT" }); + await migration(peer, homeB).migrate(); + const replacement = await qualify(image, peer, homeB); + expect(replacement.configurationPackagesEligible).toBe(false); + await expect(migration(peer, homeB).recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("original legacy source"); + await fs.unlink(path.join(homeA, "adapter-plugins.json")); + await expect(migration().recheckLegacyExternalRegistrations(replacement.receiptRevision)).rejects.toThrow("present, valid registration file"); + expect((await migration().inspectStored())?.migration.externalRegistrations.blocker).toBe("unsupported_external_adapters"); + await fs.writeFile(path.join(homeA, "adapter-plugins.json"), "[]"); + await expect(migration().recheckLegacyExternalRegistrations(0)).rejects.toThrow("revision conflict"); + await migration().recheckLegacyExternalRegistrations(replacement.receiptRevision); + expect((await qualify(image)).configurationPackagesEligible).toBe(true); + }); + + it("retains a metadata blocker for malformed or oversized registration files", async () => { + await legacy(); + await fs.writeFile(path.join(homeA, "adapter-plugins.json"), Buffer.alloc(65_537)); + const receipt = await migration().migrate(); + expect(receipt.migration.externalRegistrations.blocker).toBe("invalid_external_registration_metadata"); + await fs.rm(homeA, { recursive: true }); + expect((await migration(peer, homeB).migrate()).migration.externalRegistrations).toEqual(receipt.migration.externalRegistrations); + }); + + it("records image mismatches before reconciliation, permits maintenance upgrades, and requires explicit fresh overlap evidence", async () => { + await legacy(); + const oldImage = fixture(); + const nextImage = fixture("1.1.0"); + await install(oldImage); + await migration().migrate(); + const old = await qualify(oldImage); + const mismatch = await qualify(nextImage, peer, homeB, old.packageEvidenceDigest); + expect(mismatch.configurationPackagesEligible).toBe(false); + expect(mismatch.samePackageOverlapEligible).toBe(false); + expect(mismatch.blockers).toContainEqual(expect.objectContaining({ code: "incompatible_bundled_package", registeredVersion: "1.0.0", imageVersion: "1.1.0" })); + const registry = pluginRegistryService(db); + await ensureBundledPlugins(nextImage.installs, { + registry, loader: loaderFor(nextImage), lifecycle: { load: vi.fn() }, + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, + }, { reinstallUninstalled: true, preservePackagePath: true }); + const current = await qualify(nextImage); + expect(current.configurationPackagesEligible).toBe(true); + expect(current.samePackageOverlapEligible).toBe(false); + expect((await migration().inspectStored())?.lastPackageMismatch?.packageEvidenceDigest).toBe(mismatch.packageEvidenceDigest); + expect((await qualify(nextImage, peer, homeB, current.packageEvidenceDigest)).samePackageOverlapEligible).toBe(true); + }); + + it("fails qualification for missing package or manifest bytes without loading a home fallback", async () => { + await legacy(); + const image = fixture(); + await install(image); + await migration().migrate(); + const good = await qualify(image); + await fs.rm(path.join(image.packageRoot, "dist", "manifest.js")); + const broken = await qualify(image, peer, homeB, good.packageEvidenceDigest); + expect(broken.configurationPackagesEligible).toBe(false); + expect(broken.samePackageOverlapEligible).toBe(false); + expect(broken.blockers.length).toBeGreaterThan(0); + await fs.rm(image.packageRoot, { recursive: true }); + expect((await qualify(image)).blockers).toContainEqual(expect.objectContaining({ code: "missing_bundled_package" })); + }); + + it("rejects changed image bytes before manifest import and never uses stale completed qualification", async () => { + await legacy(); + const image = fixture(); + await install(image); + await migration().migrate(); + const original = await qualify(image); + await fs.appendFile(path.join(image.packageRoot, "dist", "worker.js"), "\nprocess.exit(99);\n"); + const loadManifest = vi.fn(); + const current = await migration(peer, homeB).qualify({ installs: image.installs, loadManifest, expectedPackageEvidenceDigest: original.packageEvidenceDigest }); + expect(current.configurationPackagesEligible).toBe(false); + expect(current.samePackageOverlapEligible).toBe(false); + expect(current.blockers).toContainEqual(expect.objectContaining({ code: "package_integrity_mismatch" })); + expect(loadManifest).not.toHaveBeenCalled(); + }); + + it("allows an administrator update while manifest inspection waits and refuses to certify the stale snapshot", async () => { + await legacy(); + const image = fixture(); + await install(image); + await migration().migrate(); + let started!: () => void; + let resume!: () => void; + const entered = new Promise((resolve) => { started = resolve; }); + const release = new Promise((resolve) => { resume = resolve; }); + const loader = loaderFor(image); + const pending = migration().qualify({ installs: image.installs, loadManifest: async (root) => { started(); await release; return loader.loadManifest(root); } }); + await entered; + try { + const change = adapterSettingsService(peer, { runtimeEnv: env() }).setDisabled("codex_local", true); + let timer: ReturnType | undefined; + try { + expect(await Promise.race([change, new Promise((_resolve, reject) => { timer = setTimeout(() => reject(new Error("Admin update was blocked by image inspection")), 2_000); })])).toBe(true); + } finally { clearTimeout(timer); } + } finally { resume(); } + const result = await pending; + expect(result).toMatchObject({ configurationPackagesEligible: false, adapterRevision: 1 }); + expect(result.blockers).toContainEqual(expect.objectContaining({ code: "configuration_changed_during_inspection" })); + expect((await qualify(image)).configurationPackagesEligible).toBe(true); + }); + + it("uses the current managed catalog through the operator command and durably records catalog failure", async () => { + await legacy(); + const image = fixture(); + await install(image); + await migration().migrate(); + const runtimeEnv = { + ...env(homeB), DATABASE_URL: database.connectionString, PAPERCLIP_BUNDLED_PLUGIN_ROOT: image.root, + PAPERCLIP_MANAGED_CONFIG: JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "fixture", features: {}, plugins: { autoInstall: ["fixture"] } }), + }; + const initial = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv }); + expect(initial?.configurationPackagesEligible).toBe(true); + const stdout = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + try { + expect(await runCloudConfigurationQualification(["--expected-package-evidence", initial!.packageEvidenceDigest], runtimeEnv)).toBe(0); + const report = stdout.mock.calls.map((call) => String(call[0])).find((text) => text.startsWith('{\n "receiptRevision"')); + expect(report && JSON.parse(report).samePackageOverlapEligible).toBe(true); + } finally { stdout.mockRestore(); } + const missing = await qualifyCloudConfigurationForCurrentImage(peer, { runtimeEnv: { ...runtimeEnv, PAPERCLIP_MANAGED_CONFIG: runtimeEnv.PAPERCLIP_MANAGED_CONFIG.replace('"fixture"]', '"absent"]') } }); + expect(missing?.configurationPackagesEligible).toBe(false); + expect(missing?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" })); + expect((await migration().inspectStored())?.lastPackageMismatch?.blockers).toContainEqual(expect.objectContaining({ code: "current_image_catalog_unavailable" })); + }); +}); diff --git a/server/src/__tests__/cloud-configuration-readers.test.ts b/server/src/__tests__/cloud-configuration-readers.test.ts new file mode 100644 index 00000000000..9abfda97b7d --- /dev/null +++ b/server/src/__tests__/cloud-configuration-readers.test.ts @@ -0,0 +1,48 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { readLegacyAdapterSettings } from "../services/adapter-settings.js"; +import { inspectLegacyExternalRegistrations } from "../services/cloud-configuration-evidence.js"; +import { cloudBundledPackageDigest } from "../services/cloud-bundled-package-digest.js"; + +vi.mock("../adapters/plugin-loader.js", () => ({ buildExternalAdapters: async () => [] })); + +describe("bounded Cloud legacy readers and image inventory", () => { + let root: string; + beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "cloud-config-reader-")); }); + afterEach(async () => { await fs.rm(root, { recursive: true, force: true }); }); + + it("distinguishes an absent adapter file from a saved empty preference", async () => { + expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toBeNull(); + await fs.writeFile(path.join(root, "adapter-settings.json"), '{"disabledTypes":[]}'); + expect(await readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).toEqual({ disabledTypes: [] }); + }); + + it.skipIf(process.platform === "win32")("rejects symlinks and FIFOs without waiting for a writer", async () => { + const target = path.join(root, "target.json"); + const settings = path.join(root, "adapter-settings.json"); + await fs.writeFile(target, '{"disabledTypes":[]}'); + await fs.symlink(target, settings); + await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toMatchObject({ code: "ELOOP" }); + expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("unreadable_external_registration_metadata"); + await fs.unlink(settings); + execFileSync("mkfifo", [settings]); + await expect(readLegacyAdapterSettings({ PAPERCLIP_HOME: root })).rejects.toThrow("regular file"); + expect((await inspectLegacyExternalRegistrations(settings)).blocker).toBe("invalid_external_registration_metadata"); + }); + + it.skipIf(process.platform === "win32")("hashes pnpm-style contained links and the physical dependency bytes", async () => { + const dependency = path.join(root, "node_modules", ".pnpm", "dep@1.0.0", "node_modules", "dep"); + await fs.mkdir(dependency, { recursive: true }); + await fs.writeFile(path.join(dependency, "index.js"), "export const value = 1;"); + await fs.symlink(path.relative(path.join(root, "node_modules"), dependency), path.join(root, "node_modules", "dep")); + const first = cloudBundledPackageDigest(root); + await fs.writeFile(path.join(dependency, "index.js"), "export const value = 2;"); + expect(cloudBundledPackageDigest(root)).not.toBe(first); + await fs.unlink(path.join(root, "node_modules", "dep")); + await fs.symlink("/", path.join(root, "node_modules", "dep")); + expect(() => cloudBundledPackageDigest(root)).toThrow("escapes its package"); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index cc276572807..53360904b82 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -31,6 +31,9 @@ vi.mock("../services/adapter-settings.js", async () => ({ initializeAdapterSettings: vi.fn(async () => null), })); vi.mock("../services/cloud-runtime-config.js", () => ({ initializeCloudRuntimeConfig: cloudRuntimeStartupMocks.initialize })); +const cloudMigrationMocks = vi.hoisted(() => ({ migrate: vi.fn(async () => null), qualify: vi.fn(async () => null) })); +vi.mock("../services/cloud-configuration-migration.js", () => ({ migrateCloudConfiguration: cloudMigrationMocks.migrate })); +vi.mock("../services/cloud-configuration-qualification.js", () => ({ qualifyCloudConfigurationForCurrentImage: cloudMigrationMocks.qualify })); vi.mock("../telemetry.js", () => ({ initTelemetry: cloudRuntimeStartupMocks.telemetry, getTelemetryClient: vi.fn(() => null) })); const ORIGINAL_PAPERCLIP_API_URL = process.env.PAPERCLIP_API_URL; @@ -627,6 +630,7 @@ describe("startServer feedback export wiring", () => { const starting = startServer(); for (let index = 0; index < 30 && !release; index += 1) await new Promise(resolve => setImmediate(resolve)); expect(release).toBeTypeOf("function"); + expect(cloudMigrationMocks.migrate.mock.invocationCallOrder[0]).toBeLessThan(cloudRuntimeStartupMocks.initialize.mock.invocationCallOrder[0]!); expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled(); expect(createBetterAuthInstanceMock).not.toHaveBeenCalled(); expect(createAppMock).not.toHaveBeenCalled(); @@ -656,6 +660,14 @@ describe("startServer feedback export wiring", () => { } }); + it("stops before configuration consumers when the atomic Cloud migration fails", async () => { + cloudMigrationMocks.migrate.mockRejectedValueOnce(new Error("migration did not commit")); + await expect(startServer()).rejects.toThrow("migration did not commit"); + expect(cloudRuntimeStartupMocks.initialize).not.toHaveBeenCalled(); + expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled(); + expect(createAppMock).not.toHaveBeenCalled(); + }); + it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); diff --git a/server/src/index.ts b/server/src/index.ts index 44e4a77ca1b..9ba0e2c02d2 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -188,6 +188,8 @@ import { } from "./shutdown.js"; import { initializeCloudRuntimeIdentity } from "./services/cloud-runtime-identity.js"; import { initializeAdapterSettings } from "./services/adapter-settings.js"; +import { migrateCloudConfiguration } from "./services/cloud-configuration-migration.js"; +import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js"; import { systemdNotify } from "./services/systemd-notify.js"; import { flushInFlightRunLogMirrors } from "./services/run-log-store.js"; import { @@ -719,6 +721,7 @@ async function startServerWithDatabaseTeardown( }; startupDatabase.close = closeDatabaseClients; setSentryRunLinkResolver(createSentryRunLinkResolver(db)); + await migrateCloudConfiguration(db as any); await initializeAdapterSettings(db as any); await initializeCloudRuntimeConfig(db as any); @@ -977,6 +980,10 @@ async function startServerWithDatabaseTeardown( // document parsed fail-closed above (`plugins.autoInstall`). Absent env means // self-hosted: createApp falls back to its built-in kubernetes-only default. const managedPluginAutoInstall = managedConfig?.plugins.autoInstall ?? null; + const cloudQualification = await qualifyCloudConfigurationForCurrentImage(db as any); + if (cloudQualification && !cloudQualification.configurationPackagesEligible) { + logger.warn({ scope: cloudQualification.scope, blockerCodes: cloudQualification.blockers.map((blocker) => blocker.code) }, "Cloud configuration and package qualification has blockers"); + } const app = await createApp(db as any, { prepareIdleDatabaseBackup: idleDatabaseBackupEnabled ? async () => (await runServerDatabaseBackup("idle")) !== null diff --git a/server/src/qualify-cloud-configuration.ts b/server/src/qualify-cloud-configuration.ts new file mode 100644 index 00000000000..f2e50830a3f --- /dev/null +++ b/server/src/qualify-cloud-configuration.ts @@ -0,0 +1,47 @@ +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { createDb } from "@paperclipai/db"; +import { cloudConfigurationMigrationService } from "./services/cloud-configuration-migration.js"; +import { qualifyCloudConfigurationForCurrentImage } from "./services/cloud-configuration-qualification.js"; + +export async function runCloudConfigurationQualification(args: string[], env: NodeJS.ProcessEnv = process.env): Promise { + const allowed = new Set(["--migrate", "--expected-package-evidence", "--recheck-external-revision"]); + let migrate = false; + let expectedPackageEvidenceDigest: string | undefined; + let recheckRevision: number | undefined; + for (let index = 0; index < args.length; index += 1) { + const argument = args[index]; + if (!allowed.has(argument)) throw new Error(`Unknown qualification argument: ${argument}`); + if (argument === "--migrate") migrate = true; + if (argument === "--expected-package-evidence") { + expectedPackageEvidenceDigest = args[++index]; + if (!expectedPackageEvidenceDigest || !/^sha256:[a-f0-9]{64}$/.test(expectedPackageEvidenceDigest)) throw new Error("A SHA-256 package evidence digest is required"); + } + if (argument === "--recheck-external-revision") { + const value = args[++index]; + if (!value || !/^\d+$/.test(value) || !Number.isSafeInteger(Number(value))) throw new Error("A nonnegative receipt revision is required"); + recheckRevision = Number(value); + } + } + const url = env.DATABASE_URL?.trim(); + if (!url) throw new Error("DATABASE_URL is required for Cloud configuration qualification"); + const db = createDb(url); + try { + const service = cloudConfigurationMigrationService(db, { runtimeEnv: env }); + if (migrate) await service.migrate(); + if (recheckRevision !== undefined) await service.recheckLegacyExternalRegistrations(recheckRevision); + const report = await qualifyCloudConfigurationForCurrentImage(db, { runtimeEnv: env, expectedPackageEvidenceDigest }); + if (!report) throw new Error("Cloud configuration qualification requires a managed instance"); + process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); + return report.configurationPackagesEligible && (!expectedPackageEvidenceDigest || report.samePackageOverlapEligible) ? 0 : 2; + } finally { + await db.$client.end({ timeout: 5 }); + } +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + runCloudConfigurationQualification(process.argv.slice(2)).then((code) => { process.exitCode = code; }).catch((error) => { + process.stderr.write(`${error instanceof Error ? error.message : "Cloud configuration qualification failed"}\n`); + process.exitCode = 1; + }); +} diff --git a/server/src/services/adapter-plugin-store.ts b/server/src/services/adapter-plugin-store.ts index 3559b4f0036..8ccdf3f6049 100644 --- a/server/src/services/adapter-plugin-store.ts +++ b/server/src/services/adapter-plugin-store.ts @@ -45,8 +45,8 @@ interface AdapterSettings { // Paths // --------------------------------------------------------------------------- -function adapterPluginPaths() { - const paperclipDir = resolvePaperclipHomeDir(); +function adapterPluginPaths(homeOverride?: string) { + const paperclipDir = resolvePaperclipHomeDir(homeOverride); return { adapterPluginsDir: path.join(paperclipDir, "adapter-plugins"), adapterPluginsStorePath: path.join(paperclipDir, "adapter-plugins.json"), @@ -133,8 +133,8 @@ function writeSettings(settings: AdapterSettings): void { // Public API // --------------------------------------------------------------------------- -export function getAdapterPluginStorePath(): string { - return adapterPluginPaths().adapterPluginsStorePath; +export function getAdapterPluginStorePath(homeOverride?: string): string { + return adapterPluginPaths(homeOverride).adapterPluginsStorePath; } export function listAdapterPlugins(): AdapterPluginRecord[] { diff --git a/server/src/services/adapter-settings.ts b/server/src/services/adapter-settings.ts index 4c2081c5578..4b9459b7165 100644 --- a/server/src/services/adapter-settings.ts +++ b/server/src/services/adapter-settings.ts @@ -2,6 +2,7 @@ import type { Db } from "@paperclipai/db"; import { instanceSettings } from "@paperclipai/db"; import { eq } from "drizzle-orm"; import { open } from "node:fs/promises"; +import { constants } from "node:fs"; import path from "node:path"; import { z } from "zod"; import { listServerAdapters } from "../adapters/registry.js"; @@ -118,23 +119,31 @@ export async function initializeAdapterSettings(db: AdapterSettingsDb, options: const service = adapterSettingsService(db, options); const current = await service.inspectStored(); if (current) return { seeded: false, current }; + return service.seedIfAbsent(await readLegacyAdapterSettings(env) ?? { disabledTypes: [] }); +} + +export async function readLegacyAdapterSettings(env: Record = process.env): Promise<{ disabledTypes: string[] } | null> { const filePath = path.join(resolvePaperclipHomeDir(env.PAPERCLIP_HOME), "adapter-settings.json"); - let disabledTypes: string[] = []; try { - const file = await open(filePath, "r"); + const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); try { const stat = await file.stat(); if (!stat.isFile() || stat.size > 65_536) throw new Error("Legacy adapter settings exceed the size limit or are not a regular file"); const buffer = Buffer.alloc(65_537); - const { bytesRead } = await file.read(buffer, 0, buffer.length, 0); - if (bytesRead > 65_536) throw new Error("Legacy adapter settings exceed the size limit"); - const value = JSON.parse(buffer.subarray(0, bytesRead).toString("utf8")); - disabledTypes = parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes; + let total = 0; + while (total < buffer.length) { + const { bytesRead } = await file.read(buffer, total, buffer.length - total, total); + if (bytesRead === 0) break; + total += bytesRead; + } + if (total > 65_536) throw new Error("Legacy adapter settings exceed the size limit"); + const value = JSON.parse(buffer.subarray(0, total).toString("utf8")); + return { disabledTypes: parseStored({ version: 1, revision: 0, disabledTypes: value?.disabledTypes }).disabledTypes }; } finally { await file.close(); } } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + return null; } - return service.seedIfAbsent({ disabledTypes }); } diff --git a/server/src/services/cloud-bundled-package-digest.ts b/server/src/services/cloud-bundled-package-digest.ts new file mode 100644 index 00000000000..a7695fd31f0 --- /dev/null +++ b/server/src/services/cloud-bundled-package-digest.ts @@ -0,0 +1,47 @@ +import fs from "node:fs"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js"; + +export function cloudBundledPackageDigest(packageRoot: string): string { + const root = fs.realpathSync(packageRoot); + const inventory: Array<[string, string, string, number]> = []; + let bytes = 0; + let entries = 0; + function walk(directory: string, relative = "") { + if (relative.split("/").length > 32) throw new Error("Image package exceeds the directory depth limit"); + for (const name of fs.readdirSync(directory).sort()) { + if (++entries > 10_000) throw new Error("Image package exceeds the entry limit"); + const local = relative ? `${relative}/${name}` : name; + const file = path.join(directory, name); + const stat = fs.lstatSync(file); + if (stat.isSymbolicLink()) { + const target = fs.realpathSync(file); + const contained = path.relative(root, target); + if (!contained || contained.startsWith("..") || path.isAbsolute(contained)) throw new Error("Image package symlink escapes its package"); + const targetStat = fs.statSync(target); + if (!targetStat.isFile() && !targetStat.isDirectory()) throw new Error("Image package symlink has an unsupported target"); + inventory.push([local, "link", fs.readlinkSync(file), stat.mode & 0o777]); + } else if (stat.isDirectory()) walk(file, local); + else if (stat.isFile()) { + if (bytes + stat.size > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit"); + const handle = fs.openSync(file, "r"); + const hash = createHash("sha256"); + const buffer = Buffer.alloc(65_536); + try { + while (true) { + const count = fs.readSync(handle, buffer); + if (!count) break; + bytes += count; + if (bytes > 256 * 1024 * 1024) throw new Error("Image package exceeds the byte limit"); + hash.update(buffer.subarray(0, count)); + } + } finally { fs.closeSync(handle); } + inventory.push([local, "file", `sha256:${hash.digest("hex")}`, stat.mode & 0o777]); + } else throw new Error("Image package contains a special file"); + } + } + walk(root); + if (!inventory.length) throw new Error("Image package is empty"); + return cloudConfigurationDigest(inventory); +} diff --git a/server/src/services/cloud-configuration-evidence.ts b/server/src/services/cloud-configuration-evidence.ts new file mode 100644 index 00000000000..63303fbf055 --- /dev/null +++ b/server/src/services/cloud-configuration-evidence.ts @@ -0,0 +1,74 @@ +import { createHash } from "node:crypto"; +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; +import { z } from "zod"; + +function canonical(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonical); + if (value && typeof value === "object") { + return Object.fromEntries(Object.entries(value).sort(([left], [right]) => left < right ? -1 : left > right ? 1 : 0).map(([key, child]) => [key, canonical(child)])); + } + return value; +} + +export function cloudConfigurationDigest(value: unknown): string { + return `sha256:${createHash("sha256").update(JSON.stringify(canonical(value))).digest("hex")}`; +} + +export const externalRegistrationEvidenceSchema = z.object({ + sourceLocatorDigest: z.string(), + sourceDigest: z.string(), + state: z.enum(["missing", "parsed", "invalid", "unreadable"]), + adapterTypes: z.array(z.string()), + blocker: z.enum(["unsupported_external_adapters", "invalid_external_registration_metadata", "unreadable_external_registration_metadata"]).nullable(), +}).strict(); +export type ExternalRegistrationEvidence = z.infer; + +const legacyRegistrationSchema = z.array(z.object({ + type: z.string().regex(/^[a-zA-Z0-9_.-]{1,128}$/), + packageName: z.string().min(1).max(1024), + version: z.string().max(128).optional(), + localPath: z.string().max(4096).optional(), + disabled: z.boolean().optional(), +}).passthrough()).max(100); + +export async function inspectLegacyExternalRegistrations(filePath: string): Promise { + const sourceLocatorDigest = cloudConfigurationDigest(filePath); + try { + const file = await open(filePath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + let raw: unknown; + try { + const stat = await file.stat(); + if (!stat.isFile() || stat.size > 65_536) throw new Error("invalid legacy registration file"); + const buffer = Buffer.alloc(65_537); + let total = 0; + while (total < buffer.length) { + const { bytesRead } = await file.read(buffer, total, buffer.length - total, total); + if (!bytesRead) break; + total += bytesRead; + } + if (total > 65_536) throw new Error("oversized legacy registration file"); + raw = JSON.parse(buffer.subarray(0, total).toString("utf8")); + } finally { + await file.close(); + } + const records = legacyRegistrationSchema.safeParse(raw); + if (!records.success) { + return { sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state: "invalid" }), state: "invalid", adapterTypes: [], blocker: "invalid_external_registration_metadata" }; + } + const metadata = records.data.map((record) => ({ type: record.type, packageName: record.packageName, version: record.version, disabled: record.disabled, hasLocalPath: record.localPath !== undefined })); + return { + sourceLocatorDigest, sourceDigest: cloudConfigurationDigest(metadata), state: "parsed", + adapterTypes: [...new Set(metadata.map((record) => record.type))].sort(), + blocker: metadata.length ? "unsupported_external_adapters" : null, + }; + } catch (error) { + const absent = (error as NodeJS.ErrnoException).code === "ENOENT"; + const invalid = error instanceof SyntaxError || (error instanceof Error && /legacy registration file/.test(error.message)); + const state = absent ? "missing" : invalid ? "invalid" : "unreadable"; + return { + sourceLocatorDigest, sourceDigest: cloudConfigurationDigest({ state }), state, adapterTypes: [], + blocker: absent ? null : invalid ? "invalid_external_registration_metadata" : "unreadable_external_registration_metadata", + }; + } +} diff --git a/server/src/services/cloud-configuration-migration.ts b/server/src/services/cloud-configuration-migration.ts new file mode 100644 index 00000000000..b5c8e3450f0 --- /dev/null +++ b/server/src/services/cloud-configuration-migration.ts @@ -0,0 +1,192 @@ +import { instanceSettings, plugins, type Db } from "@paperclipai/db"; +import { eq, inArray, ne } from "drizzle-orm"; +import { z } from "zod"; +import { adapterSettingsService, readLegacyAdapterSettings } from "./adapter-settings.js"; +import { getAdapterPluginStorePath } from "./adapter-plugin-store.js"; +import { cloudRuntimeConfigService, readLegacyCloudRuntimeConfig } from "./cloud-runtime-config.js"; +import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js"; +import { cloudConfigurationDigest, externalRegistrationEvidenceSchema, inspectLegacyExternalRegistrations } from "./cloud-configuration-evidence.js"; +import { cloudPackageEvidenceSchema, inspectCloudPackages, qualificationBlockerSchema } from "./cloud-package-qualification.js"; +import type { ResolvedBundledPlugin } from "./bundled-plugins.js"; +import type { PaperclipPluginManifestV1 } from "@paperclipai/shared"; + +const RECEIPT_KEY = "cloud-configuration-migration:v1"; +const CONFIGURATION_KEYS = ["adapter-settings:v1", "cloud-runtime-config:v1"]; +type Transaction = Parameters[0]>[0]; +export type CloudConfigurationDb = Pick; +const sourceSchema = z.object({ + authority: z.enum(["database", "legacy_file", "default"]), sourceDigest: z.string(), + importedRevision: z.number().int().nonnegative(), +}).strict(); +const qualificationSchema = z.object({ + scope: z.literal("configuration_and_packages"), checkedAt: z.string(), + adapterRevision: z.number().int().nonnegative(), configRevision: z.number().int().nonnegative(), + configurationDigest: z.string(), packageEvidenceDigest: z.string(), + packageInspectionDurationMs: z.number().nonnegative(), + expectedPackageEvidenceDigest: z.string().nullable(), + configurationPackagesEligible: z.boolean(), samePackageOverlapEligible: z.boolean(), + packages: z.array(cloudPackageEvidenceSchema), blockers: z.array(qualificationBlockerSchema), + overlapBlockers: z.array(qualificationBlockerSchema), +}).strict(); +const receiptSchema = z.object({ + version: z.literal(1), revision: z.number().int().nonnegative(), + migration: z.object({ completedAt: z.string(), adapter: sourceSchema, config: sourceSchema, externalRegistrations: externalRegistrationEvidenceSchema }).strict(), + qualification: qualificationSchema.nullable(), lastPackageMismatch: qualificationSchema.nullable(), +}).strict(); +export type CloudConfigurationReceipt = z.infer; + +function parseReceipt(value: unknown): CloudConfigurationReceipt { + const result = receiptSchema.safeParse(value); + if (!result.success) throw new Error("Persisted Cloud configuration migration receipt is invalid"); + return result.data; +} + +export function cloudConfigurationMigrationService(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv; now?: () => Date } = {}) { + const env = options.runtimeEnv ?? process.env; + const now = () => (options.now?.() ?? new Date()).toISOString(); + const adapters = adapterSettingsService(db, { runtimeEnv: env }); + const config = cloudRuntimeConfigService(db); + + function assertManaged() { + if (!isCloudManagedInstance(env)) throw new Error("Cloud configuration qualification requires a managed instance"); + } + async function readRow(handle: CloudConfigurationDb, lock = false) { + const query = handle.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, RECEIPT_KEY)); + const [row] = lock ? await query.for("update") : await query; + return row ?? null; + } + async function currentConfiguration(handle: CloudConfigurationDb) { + await handle.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS)) + .orderBy(instanceSettings.singletonKey).for("update"); + const adapter = await adapters.inspectStored(handle); + const runtime = await config.inspectStored(handle); + if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete"); + return { adapter, runtime }; + } + async function registrationSnapshot(handle: CloudConfigurationDb, lock = false) { + const query = handle.select({ + pluginKey: plugins.pluginKey, packageName: plugins.packageName, version: plugins.version, + manifestJson: plugins.manifestJson, status: plugins.status, + }).from(plugins).where(ne(plugins.status, "uninstalled")).orderBy(plugins.pluginKey); + return lock ? query.for("share") : query; + } + async function save(handle: CloudConfigurationDb, id: string, receipt: CloudConfigurationReceipt) { + const validated = parseReceipt(receipt); + const updated = await handle.update(instanceSettings).set({ general: validated, updatedAt: new Date() }).where(eq(instanceSettings.id, id)).returning({ id: instanceSettings.id }); + if (updated.length !== 1) throw new Error("Cloud configuration migration receipt was not persisted"); + return validated; + } + + return { + async inspectStored(): Promise { + const row = await readRow(db); + return row ? parseReceipt(row.general) : null; + }, + async migrate(): Promise { + assertManaged(); + return db.transaction(async (tx) => { + const inserted = await tx.insert(instanceSettings).values({ singletonKey: RECEIPT_KEY, general: { version: 1 } }) + .onConflictDoNothing({ target: instanceSettings.singletonKey }).returning({ id: instanceSettings.id }); + const row = await readRow(tx, true); + if (!row) throw new Error("Cloud configuration migration receipt is unavailable"); + if (!inserted.length) { + const receipt = parseReceipt(row.general); + await currentConfiguration(tx); + return receipt; + } + await tx.select({ id: instanceSettings.id }).from(instanceSettings).where(inArray(instanceSettings.singletonKey, CONFIGURATION_KEYS)) + .orderBy(instanceSettings.singletonKey).for("update"); + const storedAdapter = await adapters.inspectStored(tx); + const storedConfig = await config.inspectStored(tx); + const adapterCandidate = storedAdapter ? null : await readLegacyAdapterSettings(env); + const configCandidate = storedConfig ? null : await readLegacyCloudRuntimeConfig(env); + const adapterSeed = storedAdapter ? null : await adapters.seedIfAbsent({ disabledTypes: adapterCandidate?.disabledTypes ?? [] }, tx); + const configSeed = storedConfig ? null : await config.seedIfAbsent(configCandidate, tx); + const adapter = storedAdapter ?? adapterSeed!.current; + const runtime = storedConfig ?? configSeed!.current; + const adapterFromDatabase = Boolean(storedAdapter || !adapterSeed?.seeded); + const configFromDatabase = Boolean(storedConfig || !configSeed?.seeded); + const receipt: CloudConfigurationReceipt = { + version: 1, revision: 0, + migration: { + completedAt: now(), + adapter: { authority: adapterFromDatabase ? "database" : adapterCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(adapterFromDatabase ? { disabledTypes: adapter.disabledTypes } : adapterCandidate), importedRevision: adapter.revision }, + config: { authority: configFromDatabase ? "database" : configCandidate ? "legacy_file" : "default", sourceDigest: cloudConfigurationDigest(configFromDatabase ? runtime.config : configCandidate), importedRevision: runtime.revision }, + externalRegistrations: await inspectLegacyExternalRegistrations(getAdapterPluginStorePath(env.PAPERCLIP_HOME)), + }, + qualification: null, lastPackageMismatch: null, + }; + return save(tx, row.id, receipt); + }); + }, + async recheckLegacyExternalRegistrations(expectedRevision: number): Promise { + assertManaged(); + return db.transaction(async (tx) => { + const row = await readRow(tx, true); + if (!row) throw new Error("Cloud configuration migration has not completed"); + const receipt = parseReceipt(row.general); + if (receipt.revision !== expectedRevision) throw new Error("Cloud configuration migration receipt revision conflict"); + const filePath = getAdapterPluginStorePath(env.PAPERCLIP_HOME); + if (cloudConfigurationDigest(filePath) !== receipt.migration.externalRegistrations.sourceLocatorDigest) { + throw new Error("External registration recheck requires the original legacy source location"); + } + const evidence = await inspectLegacyExternalRegistrations(filePath); + if (evidence.state !== "parsed") throw new Error("External registration recheck requires a present, valid registration file"); + receipt.migration.externalRegistrations = evidence; + receipt.revision += 1; + receipt.qualification = null; + return save(tx, row.id, receipt); + }); + }, + async qualify(input: { + installs: readonly ResolvedBundledPlugin[]; + loadManifest: (packageRoot: string) => Promise; + expectedPackageEvidenceDigest?: string; + imageCatalogUnavailable?: boolean; + }) { + assertManaged(); + const adapter = await adapters.inspectStored(); + const runtime = await config.inspectStored(); + if (!adapter || !runtime) throw new Error("Cloud configuration authority is incomplete"); + const configurationBefore = cloudConfigurationDigest({ adapter, runtime }); + const registrationsBefore = await registrationSnapshot(db); + const evidence = await inspectCloudPackages({ ...input, registrations: registrationsBefore }); + return db.transaction(async (tx) => { + const row = await readRow(tx, true); + if (!row) throw new Error("Cloud configuration migration has not completed"); + const receipt = parseReceipt(row.general); + const current = await currentConfiguration(tx); + const registrations = await registrationSnapshot(tx, true); + const blockers = [...evidence.blockers]; + if (configurationBefore !== cloudConfigurationDigest(current) || cloudConfigurationDigest(registrationsBefore) !== cloudConfigurationDigest(registrations)) { + blockers.push({ code: "configuration_changed_during_inspection", reason: "Configuration or plugin registration changed during image inspection; run qualification again" }); + } + if (input.imageCatalogUnavailable) blockers.push({ code: "current_image_catalog_unavailable", reason: "The current image package catalog could not be verified" }); + const external = receipt.migration.externalRegistrations; + if (external.blocker) blockers.push({ code: external.blocker, reason: "Legacy external adapter registration metadata requires an explicit operator resolution" }); + const overlapBlockers = input.expectedPackageEvidenceDigest && input.expectedPackageEvidenceDigest !== evidence.packageEvidenceDigest + ? [{ code: "package_evidence_changed", reason: "Current packages do not match the expected package evidence digest" }] : []; + const qualification: z.infer = { + scope: "configuration_and_packages", checkedAt: now(), + adapterRevision: current.adapter.revision, configRevision: current.runtime.revision, + configurationDigest: cloudConfigurationDigest(current), packageEvidenceDigest: evidence.packageEvidenceDigest, + packageInspectionDurationMs: evidence.packageInspectionDurationMs, + expectedPackageEvidenceDigest: input.expectedPackageEvidenceDigest ?? null, + configurationPackagesEligible: blockers.length === 0, + samePackageOverlapEligible: Boolean(input.expectedPackageEvidenceDigest && blockers.length === 0 && overlapBlockers.length === 0), + packages: evidence.packages, blockers, overlapBlockers, + }; + receipt.revision += 1; + receipt.qualification = qualification; + if (evidence.blockers.length || input.imageCatalogUnavailable || overlapBlockers.length) receipt.lastPackageMismatch = qualification; + await save(tx, row.id, receipt); + return { receiptRevision: receipt.revision, ...qualification }; + }); + }, + }; +} + +export async function migrateCloudConfiguration(db: CloudConfigurationDb, options: { runtimeEnv?: CloudInstanceEnv } = {}) { + if (!isCloudManagedInstance(options.runtimeEnv ?? process.env)) return null; + return cloudConfigurationMigrationService(db, options).migrate(); +} diff --git a/server/src/services/cloud-configuration-qualification.ts b/server/src/services/cloud-configuration-qualification.ts new file mode 100644 index 00000000000..b069b253340 --- /dev/null +++ b/server/src/services/cloud-configuration-qualification.ts @@ -0,0 +1,33 @@ +import type { Db } from "@paperclipai/db"; +import { BUNDLED_PLUGIN_CATALOG, resolveBundledCatalogRoot, resolveBundledPluginInstalls, type ResolvedBundledPlugin } from "./bundled-plugins.js"; +import { readDistributionPluginCatalog } from "./distribution-plugin-catalog.js"; +import { createBundledPluginPackageResolver } from "./bundled-plugin-package.js"; +import { cloudConfigurationMigrationService } from "./cloud-configuration-migration.js"; +import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js"; +import { getManagedInstanceConfig } from "./managed-config.js"; +import { pluginLoader } from "./plugin-loader.js"; + +export async function qualifyCloudConfigurationForCurrentImage(db: Db, options: { + runtimeEnv?: CloudInstanceEnv; + expectedPackageEvidenceDigest?: string; +} = {}) { + const env = options.runtimeEnv ?? process.env; + if (!isCloudManagedInstance(env)) return null; + const service = cloudConfigurationMigrationService(db, { runtimeEnv: env }); + const catalogRoot = resolveBundledCatalogRoot(env); + let installs: ResolvedBundledPlugin[]; + try { + const distribution = readDistributionPluginCatalog(catalogRoot, BUNDLED_PLUGIN_CATALOG); + installs = resolveBundledPluginInstalls(getManagedInstanceConfig(env)?.plugins.autoInstall ?? [], { + catalogRoot, env, enforceCatalogRoot: true, distributionPlugins: distribution, + }); + } catch { + return service.qualify({ installs: [], loadManifest: async () => null, imageCatalogUnavailable: true, expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest }); + } + const resolver = createBundledPluginPackageResolver(installs); + const loader = pluginLoader(db, { + prebuiltOnly: true, enableLocalFilesystem: false, enableNpmDiscovery: false, + assertPackageActivation: resolver.assertActivation, resolvePackageRoot: resolver.resolve, + }); + return service.qualify({ installs, loadManifest: (root) => loader.loadManifest(root), expectedPackageEvidenceDigest: options.expectedPackageEvidenceDigest }); +} diff --git a/server/src/services/cloud-package-qualification.ts b/server/src/services/cloud-package-qualification.ts new file mode 100644 index 00000000000..ffdf05da861 --- /dev/null +++ b/server/src/services/cloud-package-qualification.ts @@ -0,0 +1,71 @@ +import type { PaperclipPluginManifestV1 } from "@paperclipai/shared"; +import { z } from "zod"; +import { createBundledPluginPackageResolver, type BundledPluginRegistration } from "./bundled-plugin-package.js"; +import type { ResolvedBundledPlugin } from "./bundled-plugins.js"; +import { distributionBundleDigest } from "./distribution-plugin-catalog.js"; +import { cloudConfigurationDigest } from "./cloud-configuration-evidence.js"; +import { cloudBundledPackageDigest } from "./cloud-bundled-package-digest.js"; + +export const qualificationBlockerSchema = z.object({ + code: z.string(), pluginKey: z.string().optional(), reason: z.string(), + registeredVersion: z.string().optional(), imageVersion: z.string().optional(), +}).strict(); +export type QualificationBlocker = z.infer; +export const cloudPackageEvidenceSchema = z.object({ + catalogKey: z.string(), pluginKey: z.string(), packageDigest: z.string().nullable(), + imageManifestDigest: z.string().nullable(), approvedManifestDigest: z.string().nullable(), +}).strict(); +export type CloudPackageEvidence = z.infer; + +export async function inspectCloudPackages(input: { + installs: readonly ResolvedBundledPlugin[]; + registrations: readonly BundledPluginRegistration[]; + loadManifest: (packageRoot: string) => Promise; +}) { + const started = performance.now(); + const resolver = createBundledPluginPackageResolver(input.installs); + const registrations = new Map(input.registrations.map((record) => [record.pluginKey, record])); + const selected = new Set(input.installs.map((install) => install.pluginKey)); + const blockers: QualificationBlocker[] = input.registrations.filter((record) => !selected.has(record.pluginKey)).map((record) => ({ + code: "unsupported_plugin_registration", pluginKey: record.pluginKey, reason: "An installed plugin is not selected from the current image", registeredVersion: record.version, + })); + const packages: CloudPackageEvidence[] = []; + for (const install of [...input.installs].sort((left, right) => left.pluginKey < right.pluginKey ? -1 : left.pluginKey > right.pluginKey ? 1 : 0)) { + const registration = registrations.get(install.pluginKey); + const evidence: CloudPackageEvidence = { + catalogKey: install.key, pluginKey: install.pluginKey, packageDigest: null, + imageManifestDigest: null, approvedManifestDigest: registration ? cloudConfigurationDigest(registration.manifestJson) : null, + }; + packages.push(evidence); + let stage = "missing_bundled_package"; + try { + resolver.assertActivation({ packageRoot: install.localPath }); + stage = "unverified_package_integrity"; + evidence.packageDigest = install.distribution ? distributionBundleDigest(install.localPath) : cloudBundledPackageDigest(install.localPath); + if (install.distribution && evidence.packageDigest !== install.distribution.digest) { + blockers.push({ code: "package_integrity_mismatch", pluginKey: install.pluginKey, reason: "Current package bytes do not match the verified distribution digest" }); + continue; + } + stage = "missing_manifest_evidence"; + const manifest = await input.loadManifest(install.localPath); + if (!manifest) { + blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image has no verified plugin manifest" }); + continue; + } + evidence.imageManifestDigest = cloudConfigurationDigest(manifest); + if (!registration) { + blockers.push({ code: "missing_plugin_registration", pluginKey: install.pluginKey, reason: "The current image package is not registered in the database", imageVersion: manifest.version }); + continue; + } + const inspected = resolver.inspect(registration, manifest); + if (!inspected.compatible) blockers.push({ + code: inspected.code, pluginKey: install.pluginKey, reason: inspected.reason, + registeredVersion: registration.version, imageVersion: manifest.version, + }); + } catch { + blockers.push({ code: stage, pluginKey: install.pluginKey, reason: "The current image package could not be verified" }); + } + } + const packageEvidenceDigest = cloudConfigurationDigest(packages.map(({ approvedManifestDigest: _approved, ...evidence }) => evidence)); + return { packages, blockers, packageEvidenceDigest, packageInspectionDurationMs: Math.round(performance.now() - started) }; +}