From 198218bf554877fe3b7a83742eefb3b8f12392c6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 30 Oct 2026 16:21:46 -0500 Subject: [PATCH] fix(config): require shared storage for managed cloud (#18350) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed Cloud instances must retain shared objects and secrets across server replacement. > - Implicit storage defaults can select local disk or an unintended bucket. > - Cloud runtime settings now come from a canonical database snapshot. > - This pull request validates bootstrap prerequisites before database setup and effective S3 settings after snapshot resolution. > - The benefit is an actionable refusal before local signing state or HTTP setup, while valid canonical Cloud settings work. ## Linked Issues or Issue Description **What happened?** A managed Cloud instance could start with local disk storage, an implicit bucket, or a local key file. After canonical Cloud configuration landed, checking only environment or file values would also reject a valid database bucket. Shared schema defaults could make an omitted raw bucket appear explicit. **Expected behavior** Managed Cloud requires external PostgreSQL, a stable environment master key, and effective S3 settings with a nonblank bucket. The canonical snapshot remains the configuration authority. Invalid configuration refuses startup before signing or HTTP effects. Self-hosted defaults remain compatible. **Steps to reproduce** 1. Select managed Cloud with its tenant token or managed configuration document. 2. Leave storage as local disk, or omit the S3 bucket from a raw canonical snapshot. 3. Start the server. Startup must refuse. An explicit valid bucket in the canonical snapshot must work without a file or environment bucket. **Paperclip version or commit** Current base: `4edd10804e2b42941aa2038e90c362c13a0b98dc`. **Deployment mode** Managed Cloud. Ordinary self-hosted defaults remain unchanged. Related PRs: #18335 owns canonical Cloud runtime configuration. #18327 owns S3 run logs. #18352 owns credential persistence. #18347 owns cutover checks. This PR adds configuration prerequisites to those existing boundaries. ## What Changed - Validate external PostgreSQL and a stable environment master key at the Cloud bootstrap boundary, before database setup. - Validate effective S3 settings after canonical snapshot publication and before Cloud signing, runtime consumers, or HTTP setup. - Require a nonblank bucket in the Cloud snapshot schema. Reject omitted or blank buckets in every legacy Cloud file before one-time import normalization, even when an environment bucket is supplied. Keep shared self-hosted schema defaults. - Preserve typed bucket provenance at the existing file-read boundary. Accept explicit canonical bucket names, including `paperclip`, and environment settings when the snapshot is null. - Validate effective legacy storage with the existing resolver before seeding. A rejected candidate leaves no canonical rows, so a corrected source file can be imported on retry. Environment overlays are checked but never persisted. - Add real PostgreSQL snapshot and migration-retry tests, startup refusal tests, and environment/file compatibility controls. Document that Cloud always requires its stable environment key, even with an external default secrets provider. ## Verification - Latest focused acceptance passed 199/199 in six files (54.29 seconds), using the command below plus `server/src/__tests__/cloud-configuration-migration-postgres.test.ts`. Real PostgreSQL proves rejected storage leaves all three migration authority rows absent, then a corrected legacy file imports on retry. An external-default-secrets control proves the deliberate unconditional Cloud master-key contract. - Before the peer correction, the frozen product diff passed 178 tests in five files: managed-cloud-prerequisites, cloud-runtime-config, server-startup-feedback-export, config-file, and cloud-configuration-readers. Command: `pnpm exec vitest run server/src/__tests__/{managed-cloud-prerequisites,cloud-runtime-config,server-startup-feedback-export,config-file,cloud-configuration-readers}.test.ts --maxWorkers=1 --no-file-parallelism`. Real PostgreSQL used a qualified POSIX test-only PG18.6 override because the host cannot supply the stock runtime's SysV semaphore allocation. - Coverage includes raw database and legacy omitted/blank bucket rejection, valid database and environment buckets, null snapshots, explicit default-named buckets, self-hosted compatibility, and refusal before local signing mutation. - The real AWS SDK configured-provider control uses a simulated loopback S3 service. It writes, reads from a fresh provider, and deletes an object. It also reads an encrypted credential with the stable environment key. This does not qualify production cloud permissions. - `git diff --check` passed. Final merges add unrelated master changes from #18241, #18363, and #18370; the focused storage product diff is unchanged. - Peer review reproduced a fresh legacy local-disk import that gained the default bucket before an S3 environment overlay. The importer now requires a raw explicit bucket for every legacy Cloud file. Six controls cover omission/blank values with and without an environment bucket; the full focused rerun passed 184/184 across all five files in 194.89 seconds. - Initial revised startup testing passed 167/168: one fixture omitted the database URL and failed before its intended key assertion. The corrected final run passed 178/178. An initial direct server typecheck failed because Runner declarations were not built; this is a setup failure, not a compile pass. Integration verification rebuilds dependencies before full checks. - Prior head `fae1e4aa` passed complete CI with 53 successes, two intended skips, and Greptile 5/5. `de71345f379cecb71dbfb219f35e14f942c10a3f` then passed complete CI (55 successes, three workflow skips, and a neutral external review), with Greptile 5/5 and zero unresolved threads. The external review identified the preseed rejection problem; its correction is committed at `75e04d0d77e19dcef0a8cb070b6b518f0f173b14`. The final merge head is `19b809ea2823a6a4e5ceb48ef2c2bac4721733d6`; [Fresh complete CI](https://github.com/paperclipai/paperclip/actions/runs/41127309762) passed on that head: 55 successful checks, three workflow skips, and one neutral external review. Greptile reviewed this exact head at 5/5 with no actionable issues. Both earlier external review threads are resolved with the tested fixes. Earlier infrastructure retries remain documented in [the first diagnosis](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6529964690) and [the runner shutdown evidence](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6530364769). Earlier missing SDK build and fixture metadata failures were repaired before those passes. - Current frozen-head CI first attempt failed one unchanged Antigravity authentication success case in Runner Vitest 1/2 (`outcome=failure`, 1994 tests passed). The same file passed 11/11 locally in 2.33 seconds, and identical Runner source passed the prior complete CI run. Attempt 2 retried only failed jobs (the Runner shard and dependent aggregate). Both passed without a source change. No second retry was used. The exact timing cause is not established. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/41127309762/job/124309181778). - No production data, buckets, or live credentials were used. Complete local repository suites are not duplicated in this worktree; current-head required CI supplies the full suite gate. ## Risks - Managed deployments with local disk, omitted buckets, file-only keys, or missing external database URLs now refuse startup. Operators must supply valid prerequisites before rollout. - Required deployment environment overlays must remain stable across replacements. For example, a saved local-disk provider with an explicit bucket can use an environment S3 provider; removing that required overlay correctly refuses startup until the environment or canonical configuration is corrected. Environment settings are never silently copied into the canonical row. - Effective legacy storage must also pass before first seeding. A bad candidate does not create authority rows. Existing saved rows remain authoritative and can require environment overrides or a canonical configuration change if they are invalid. - A present legacy Cloud file needs an explicit raw bucket even if its saved provider is local disk or its environment supplies a bucket. Correct the file before one-time import, or use the existing absent-file/null-snapshot environment-only setup. Environment overlays are not persisted as operator choices. - A previously stored canonical nonblank `paperclip` bucket is authoritative. Prior normalization erased whether that value was originally explicit. This change cannot recover that provenance and does not reject existing nonblank canonical values. - Validation proves configuration shape, not cloud permissions, database reachability, or key compatibility with existing encrypted data. Keep the master key stable across replacements. - Cloud snapshot JSON parsing is stricter; omitted or blank raw S3 bucket values need correction. No database migration or provider operation changes are added. These prerequisites do not prove that every server persistence path is disposable. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Context window: unknown. The model used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 39 ++++ doc/cloud-runtime-configuration.md | 29 ++- ...d-configuration-migration-postgres.test.ts | 13 ++ .../__tests__/cloud-runtime-config.test.ts | 27 +++ server/src/__tests__/config-file.test.ts | 14 ++ .../managed-cloud-prerequisites.test.ts | 201 ++++++++++++++++++ .../server-startup-feedback-export.test.ts | 68 +++++- server/src/config-file.ts | 24 ++- server/src/config.ts | 12 +- server/src/index.ts | 9 +- server/src/managed-cloud-prerequisites.ts | 77 +++++++ .../src/secrets/local-encrypted-provider.ts | 5 + .../services/cloud-runtime-config-snapshot.ts | 2 +- server/src/services/cloud-runtime-config.ts | 9 + 14 files changed, 519 insertions(+), 10 deletions(-) create mode 100644 server/src/__tests__/managed-cloud-prerequisites.test.ts create mode 100644 server/src/managed-cloud-prerequisites.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index c84cd452b4e..d8652f0eaa3 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1511,6 +1511,45 @@ Configure storage provider/settings: pnpm paperclipai configure --section storage ``` +### Managed cloud prerequisites + +A nonblank Cloud tenant token or the presence of `PAPERCLIP_MANAGED_CONFIG` +selects the managed-cloud profile. A supplied blank or invalid managed feature +document refuses startup. Without either signal, the instance remains self-hosted. +The managed profile also requires: + +- `PAPERCLIP_STORAGE_PROVIDER=s3` or `storage.provider: "s3"` in the canonical + database snapshot. An invalid explicit environment value refuses startup even + if the snapshot selects S3. +- An explicit nonblank `PAPERCLIP_STORAGE_S3_BUCKET` or `storage.s3.bucket`. + The self-hosted `paperclip` bucket default does not qualify. The region defaults + to `us-east-1`; an explicit blank region refuses startup. An optional endpoint + must be an HTTP(S) URL without embedded credentials. +- An external PostgreSQL URL from the environment `DATABASE_URL`. + The managed profile never falls back to embedded PostgreSQL. +- A stable `PAPERCLIP_SECRETS_MASTER_KEY` supplied by the deployment secret store. + Use the same value across replacements. The existing provider accepts 32-byte + base64, 64-character hex, or a raw 32-byte UTF-8 string. A key file alone does + not qualify because a replacement web server may not have that file. + +Bootstrap validates the environment database URL and encryption master key before +opening the database or creating local state. Effective storage validation follows +canonical snapshot resolution, before signing keys, first-party Telemetry or app +construction. Raw canonical rows and legacy S3 imports cannot obtain a missing +bucket from schema defaults. Already-stored nonblank bucket values remain +canonical, including the deliberate name `paperclip`; prior normalization cannot +reliably establish their original provenance. +It does not contact S3 or verify credentials, bucket access, database connectivity, +or that a key matches existing encrypted data. Operators must qualify ordinary +S3 put/read/delete with deployment credentials and preserve the existing key. +AWS credentials continue to use the provider's normal SDK credential chain. +Local and self-hosted storage and key-file defaults remain unchanged. + +These are prerequisites for shared storage and secrets. They do not establish +that the whole web server is disposable or that all execution and recovery paths +survive web-server replacement. Run-log/trace storage and producer retention are +separate changes. + ## Agent Artifact Uploads When an agent generates a file that a board user or reviewer should inspect as diff --git a/doc/cloud-runtime-configuration.md b/doc/cloud-runtime-configuration.md index 62f2eabb114..577976c5236 100644 --- a/doc/cloud-runtime-configuration.md +++ b/doc/cloud-runtime-configuration.md @@ -18,7 +18,8 @@ existing validation at application startup. to embedded PostgreSQL. Credentials, trust keys, provider configuration and deployment wiring needed before the database connection remain supplied by the environment or the existing secret system. This phase parses only the - database connection record. Runtime mode, host binding and authentication + database connection record and requires the stable environment encryption + master key. Runtime mode, host binding and authentication checks wait until the saved snapshot has loaded. 2. After schema migration and database connection, startup reads the private runtime configuration row. An existing invalid row or a database error stops @@ -38,6 +39,27 @@ existing validation at application startup. authentication and application services. A saved telemetry opt-out therefore takes effect before a telemetry client can be created. +Managed Cloud always requires a stable environment master key, including when +an external default secrets provider is selected. Internal local-encrypted +credential paths still need that key. + +Managed Cloud validates effective S3 storage after publishing the canonical +snapshot, before signing-key initialization, telemetry, authentication or app +construction. Cloud requires S3 and a nonblank bucket; an authoritative absent +configuration needs explicit environment storage settings. Raw saved snapshots +cannot obtain a bucket from a schema default. Every legacy Cloud file must name its +bucket before its one-time import, even if its saved provider is local disk and +an environment bucket is present. Omission or blank values fail before seeding; +The effective storage candidate is validated before seeding, so a rejected +legacy file leaves no canonical row and can be corrected before retry. +Environment overlays are not saved as operator choices. Correct that file before +import, or use the existing absent-file/null-snapshot environment-only setup. +Ordinary self-hosted file defaults remain unchanged. +Already-canonical nonblank values, including `paperclip`, remain authoritative. +Prior normalization cannot reliably distinguish an operator's deliberate name +from a historical default, so this release does not invent retrospective +provenance or replace those saved values. + Existing operator environment overlays keep precedence over saved values. Reading an overlaid value does not save it back to the database. Credentials and connection URLs are never copied into this JSONB row. @@ -78,8 +100,9 @@ bytes. Restart tests prove configuration restoration under a different home; they do not prove durable file, secret, or backup storage. The decision signing service still uses `PAPERCLIP_DECISION_SIGNING_SECRET` when -supplied and otherwise reads or creates its instance key file before the database -connection. The local encrypted secret provider still uses its environment key +supplied and otherwise reads or creates its instance key file. Managed Cloud +performs that step after effective storage validation; self-hosted startup keeps +its pre-connection ordering. The local encrypted secret provider still uses its environment key or existing key-file mechanism. Those byte-storage fallbacks are outside this configuration migration. Qualification must account for them separately; this PR does not establish file-free secret storage on Cloud. diff --git a/server/src/__tests__/cloud-configuration-migration-postgres.test.ts b/server/src/__tests__/cloud-configuration-migration-postgres.test.ts index dd747d06ac6..719db156137 100644 --- a/server/src/__tests__/cloud-configuration-migration-postgres.test.ts +++ b/server/src/__tests__/cloud-configuration-migration-postgres.test.ts @@ -55,6 +55,7 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura const config = paperclipConfigSchema.parse({ $meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" }, database: { mode: "postgres" }, logging: { mode: "cloud" }, + storage: { provider: "s3", s3: { bucket: "migration-bucket" } }, server: { port: 3199 }, telemetry: { enabled: false }, }); await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config)); @@ -120,6 +121,18 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura } finally { await manager.stopAll(); } }); + it("does not seed rejected storage and imports the corrected legacy file on retry", async () => { + const config = await legacy(); + config.storage.provider = "local_disk"; + await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config)); + await expect(migration().migrate()).rejects.toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/); + expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]); + config.storage.provider = "s3"; + await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config)); + expect((await migration(peer).migrate()).migration.config.authority).toBe("legacy_file"); + expect((await cloudRuntimeConfigService(peer).inspectStored())?.config?.storage.provider).toBe("s3"); + }); + it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => { await legacy(); const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]); diff --git a/server/src/__tests__/cloud-runtime-config.test.ts b/server/src/__tests__/cloud-runtime-config.test.ts index c7078f7cddc..a959fee8f64 100644 --- a/server/src/__tests__/cloud-runtime-config.test.ts +++ b/server/src/__tests__/cloud-runtime-config.test.ts @@ -66,6 +66,9 @@ beforeEach(() => { vi.stubEnv("PAPERCLIP_ALLOWED_HOSTNAMES", undefined); vi.stubEnv("PAPERCLIP_TAILNET_BIND_HOST", "100.100.0.1"); vi.stubEnv("OPENAI_API_KEY", "environment-api-secret"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012"); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined); + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); setCloudRuntimeConfigSnapshot(null); }); @@ -170,6 +173,10 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => { expect(fs.existsSync(configPath)).toBe(false); vi.stubEnv("PAPERCLIP_DEPLOYMENT_MODE", "authenticated"); vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://environment.example.test"); + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined); + vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "test-token"); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3"); + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "environment-bucket"); expect(loadConfig()).toMatchObject({ databaseMode: "postgres", authBaseUrlMode: "explicit", secretsStrictMode: true, telemetryEnabled: true }); }); @@ -284,6 +291,26 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => { expect(await second.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "runtime-config-test-receipt"))).toEqual([]); }); + it.each([undefined, "", " "])("rejects raw canonical S3 rows with missing or blank bucket %j", async bucket => { + const config = structuredClone(projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig()))); + const raw = config; + if (bucket === undefined) Reflect.deleteProperty(raw.storage.s3, "bucket"); + else raw.storage.s3.bucket = bucket; + await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 1, revision: 0, config: raw } }); + await expect(initializeCloudRuntimeConfig(second)).rejects.toThrow(/Persisted Cloud runtime configuration is invalid/); + expect(() => getCloudRuntimeConfigSnapshot()).toThrow(/not been initialized/); + }); + + it.each(["database-bucket", "paperclip"])("accepts canonical DB bucket %s without local configuration provenance", async bucket => { + const config = projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig())); + config.storage.s3.bucket = bucket; + await cloudRuntimeConfigService(first).seedIfAbsent(config); + fs.writeFileSync(configPath, "invalid local file must not be consulted"); + await initializeCloudRuntimeConfig(second); + expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: bucket }); + expect(readConfigFile({ includeProvenance: true })).toMatchObject({ explicitStorageS3Bucket: true }); + }); + it("fails closed on invalid database settings without consulting a valid legacy file", async () => { fs.writeFileSync(configPath, JSON.stringify(legacyConfig())); await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 99, config: { secret: "hidden" } } }); diff --git a/server/src/__tests__/config-file.test.ts b/server/src/__tests__/config-file.test.ts index 9cc9451138a..75065561848 100644 --- a/server/src/__tests__/config-file.test.ts +++ b/server/src/__tests__/config-file.test.ts @@ -91,6 +91,20 @@ describe("readConfigFile", () => { }); }); + it("preserves explicit bucket provenance without changing defaults for existing readers", () => { + for (const bucket of [undefined, "paperclip"]) { + writeConfig(configPath, { + ...minimalConfig() as object, + storage: { provider: "s3", s3: bucket === undefined ? {} : { bucket } }, + }); + expect(readConfigFile()?.storage.s3.bucket).toBe("paperclip"); + expect(readConfigFile({ includeProvenance: true })).toMatchObject({ + config: { storage: { s3: { bucket: "paperclip" } } }, + explicitStorageS3Bucket: bucket !== undefined, + }); + } + }); + it("warns about likely misspellings without stripping them", () => { const config = { ...(minimalConfig() as Record), diff --git a/server/src/__tests__/managed-cloud-prerequisites.test.ts b/server/src/__tests__/managed-cloud-prerequisites.test.ts new file mode 100644 index 00000000000..7f0e80ac907 --- /dev/null +++ b/server/src/__tests__/managed-cloud-prerequisites.test.ts @@ -0,0 +1,201 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { createServer } from "node:http"; +import type { AddressInfo } from "node:net"; +import { createStorageProviderFromConfig } from "../storage/provider-registry.js"; +import { createLocalEncryptedCipher } from "../secrets/local-encrypted-provider.js"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { loadConfig } from "../config.js"; +import { readConfigFile } from "../config-file.js"; +import { readLegacyCloudRuntimeConfig } from "../services/cloud-runtime-config.js"; +import { setCloudRuntimeConfigSnapshot } from "../services/cloud-runtime-config-snapshot.js"; +const managed = JSON.stringify({ + v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] }, +}); + +let configDir: string; + +beforeEach(() => { + setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null }); + configDir = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-config-")); + vi.stubEnv("PAPERCLIP_CONFIG", path.join(configDir, "config.json")); + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3"); + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "test-bucket"); + vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", undefined); + vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", undefined); + vi.stubEnv("DATABASE_URL", "postgresql://user:private@db.example.test/paperclip"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012"); +}); +afterEach(() => { + setCloudRuntimeConfigSnapshot(null); + vi.unstubAllEnvs(); + rmSync(configDir, { recursive: true, force: true }); +}); + +function writeConfig(storage: Record) { + writeFileSync(path.join(configDir, "config.json"), JSON.stringify({ + $meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" }, + logging: { mode: "file" }, + server: {}, + storage, + database: { mode: "postgres", connectionString: "postgres://db.example.test/paperclip" }, + })); +} + +describe("managed cloud configuration boundary", () => { + it("accepts configured S3, external PostgreSQL, and the existing master-key formats", () => { + for (const key of ["12345678901234567890123456789012", Buffer.alloc(32, 7).toString("hex"), Buffer.alloc(32, 7).toString("base64")]) { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", key); + expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "test-bucket", storageS3Region: "us-east-1" }); + } + }); + + it.each([undefined, "", "local_disk", "typo"])("rejects storage provider %j without silent fallback", (value) => { + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", value); + expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/); + }); + + it.each([undefined, "", " "])("requires an explicit bucket (%j)", (value) => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", value); + expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/); + }); + + it.each(["", " "])("rejects blank region %j", (value) => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", value); + expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_REGION/); + }); + + it.each(["", "ftp://store.test", "https://user:private@store.test"])("rejects invalid endpoint without exposing it", (value) => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", value); + expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_ENDPOINT/); + try { loadConfig(); } catch (error) { expect(String(error)).not.toContain("private"); } + }); + + it.each([undefined, "", "sqlite://local/data", "postgresql://"])("requires an external database URL (%j)", (value) => { + vi.stubEnv("DATABASE_URL", value); + expect(() => loadConfig()).toThrow(/external PostgreSQL URL/); + }); + + it.each([undefined, "", " ", "short"])("rejects missing or invalid env key (%j), even with a configured key file", (value) => { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", value); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", "/tmp/unused-master-key"); + expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/); + }); + + it("requires the Cloud bootstrap key even when an external secret provider is selected", () => { + vi.stubEnv("PAPERCLIP_SECRETS_PROVIDER", "aws_secrets_manager"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined); + expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012"); + expect(loadConfig().secretsProvider).toBe("aws_secrets_manager"); + }); + + it.each(["", "{bad", JSON.stringify({ v: 1, mode: "cloud" })])("rejects malformed managed config before startup (%j)", (value) => { + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", value); + expect(() => loadConfig()).toThrow(/PAPERCLIP_MANAGED_CONFIG/); + }); + + it("accepts imported explicit file storage with an environment-owned database", async () => { + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined); + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); + writeConfig({ provider: "s3", s3: { bucket: "file-bucket", region: "eu-west-1" } }); + setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() }); + expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "file-bucket", storageS3Region: "eu-west-1" }); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo"); + expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/); + }); + + it.each([{ provider: "s3" }, { provider: "s3", s3: { region: "us-east-1" } }])("rejects a real config file with an omitted bucket before import (%j)", async storage => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); + writeConfig(storage); + // Self-hosted readers still expose their default and raw-file provenance. + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined); + expect(readConfigFile({ includeProvenance: true })).toMatchObject({ config: { storage: { s3: { bucket: "paperclip" } } }, explicitStorageS3Bucket: false }); + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed); + await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/); + }); + + it.each(["", " "])("rejects an explicitly blank bucket in a real file (%j)", async bucket => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); + writeConfig({ provider: "s3", s3: { bucket } }); + await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/); + }); + + it.each([undefined, "", " "].flatMap(bucket => [undefined, "environment-bucket"].map(environmentBucket => ({ bucket, environmentBucket }))))("rejects a legacy local-disk file with no explicit bucket before an S3 environment overlay (%j)", async ({ bucket, environmentBucket }) => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", environmentBucket); + writeConfig({ provider: "local_disk", s3: { bucket } }); + await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/); + }); + + it("accepts an explicitly supplied default bucket name in a real file", async () => { + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); + writeConfig({ provider: "s3", s3: { bucket: "paperclip" } }); + setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() }); + expect(loadConfig().storageS3Bucket).toBe("paperclip"); + }); + + it("accepts an environment bucket with an authoritative absent-file snapshot", () => { + writeConfig({ provider: "s3" }); + expect(loadConfig().storageS3Bucket).toBe("test-bucket"); + }); + + it("preserves ordinary self-hosted defaults, including key-file use", () => { + for (const name of ["PAPERCLIP_MANAGED_CONFIG", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_STORAGE_S3_BUCKET", "DATABASE_URL", "PAPERCLIP_SECRETS_MASTER_KEY"]) vi.stubEnv(name, undefined); + expect(loadConfig()).toMatchObject({ storageProvider: "local_disk", storageS3Bucket: "paperclip", databaseUrl: undefined, databaseMode: "embedded-postgres" }); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo"); + expect(loadConfig().storageProvider).toBe("local_disk"); + }); +}); + +// Real provider + AWS SDK over loopback. The remote S3 service and credentials +// are simulated; this does not establish deployment bucket permissions. +it("uses the validated config for normal S3 put/read/delete and stable env-key encryption", async () => { + const objects = new Map(); + const operations: string[] = []; + const server = createServer(async (req, res) => { + const key = new URL(req.url!, "http://localhost").pathname; + operations.push(`${req.method} ${key}`); + if (req.method === "PUT") { + const chunks: Buffer[] = []; + for await (const chunk of req) chunks.push(Buffer.from(chunk)); + objects.set(key, Buffer.concat(chunks)); + res.end(); + } else if (req.method === "GET" && objects.has(key)) { + const body = objects.get(key)!; + res.writeHead(200, { "content-type": "text/plain", "content-length": body.length }); + res.end(body); + } else if (req.method === "DELETE") { + objects.delete(key); + res.writeHead(204).end(); + } else { + res.writeHead(404).end(); + } + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture-access-key"); + vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture-secret-key"); + vi.stubEnv("AWS_SESSION_TOKEN", undefined); + vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", `http://127.0.0.1:${(server.address() as AddressInfo).port}`); + vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true"); + vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "disposable-qualification"); + try { + const config = loadConfig(); + const first = createStorageProviderFromConfig(config); + await first.putObject({ objectKey: "company/object.txt", body: Buffer.from("retained"), contentType: "text/plain", contentLength: 8 }); + const replacement = createStorageProviderFromConfig(loadConfig()); + const result = await replacement.getObject({ objectKey: "company/object.txt" }); + const chunks: Buffer[] = []; + for await (const chunk of result.stream) chunks.push(Buffer.from(chunk)); + expect(Buffer.concat(chunks).toString()).toBe("retained"); + await replacement.deleteObject({ objectKey: "company/object.txt" }); + expect(objects.size).toBe(0); + expect(operations).toEqual(["PUT", "GET", "DELETE"].map(method => `${method} /test-bucket/disposable-qualification/company/object.txt`)); + const encrypted = createLocalEncryptedCipher("write").encrypt("saved credential"); + expect(createLocalEncryptedCipher("read").decrypt(encrypted)).toBe("saved credential"); + } finally { + server.closeAllConnections(); + await new Promise((resolve) => server.close(() => resolve())); + } +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index b645d7437c8..bbb020c865d 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -614,6 +614,70 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + + it("refuses invalid cloud bootstrap prerequisites before signing, DB creation, or HTTP setup", async () => { + const realConfig = await vi.importActual("../config.js"); + const originalHome = process.env.PAPERCLIP_HOME; + const originalManaged = process.env.PAPERCLIP_MANAGED_CONFIG; + const originalStorage = process.env.PAPERCLIP_STORAGE_PROVIDER; + const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-refusal-")); + process.env.PAPERCLIP_HOME = tempHome; + process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({ + v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] }, + }); + process.env.PAPERCLIP_STORAGE_PROVIDER = "s3"; + delete process.env.PAPERCLIP_DECISION_SIGNING_SECRET; + cloudRuntimeStartupMocks.bootstrap.mockImplementationOnce(realConfig.loadCloudBootstrapConfig); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined); + vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl); + try { + await expect(startServer()).rejects.toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/); + expect(createDbMock).not.toHaveBeenCalled(); + expect(createAppMock).not.toHaveBeenCalled(); + expect(existsSync(path.join(tempHome, "instances"))).toBe(false); + } finally { + if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = originalHome; + if (originalManaged === undefined) delete process.env.PAPERCLIP_MANAGED_CONFIG; + else process.env.PAPERCLIP_MANAGED_CONFIG = originalManaged; + if (originalStorage === undefined) delete process.env.PAPERCLIP_STORAGE_PROVIDER; + else process.env.PAPERCLIP_STORAGE_PROVIDER = originalStorage; + vi.unstubAllEnvs(); + rmSync(tempHome, { recursive: true, force: true }); + } + }); + + + it("rejects missing effective Cloud storage after the snapshot and before local signing mutations", async () => { + const realConfig = await vi.importActual("../config.js"); + const { setCloudRuntimeConfigSnapshot } = await import("../services/cloud-runtime-config-snapshot.js"); + const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-storage-refusal-")); + vi.stubEnv("PAPERCLIP_HOME", tempHome); + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] } })); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012"); + vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3"); + vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined); + vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", undefined); + cloudRuntimeStartupMocks.initialize.mockImplementationOnce(async () => { + setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null }); + return null; + }); + loadConfigMock.mockImplementationOnce(realConfig.loadConfig); + try { + await expect(startServer()).rejects.toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/); + expect(createDbMock).toHaveBeenCalled(); + expect(cloudRuntimeStartupMocks.initialize).toHaveBeenCalledOnce(); + expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled(); + expect(createAppMock).not.toHaveBeenCalled(); + expect(existsSync(path.join(tempHome, "instances"))).toBe(false); + } finally { + setCloudRuntimeConfigSnapshot(null); + vi.unstubAllEnvs(); + rmSync(tempHome, { recursive: true, force: true }); + } + }); + it("awaits saved Cloud configuration before telemetry, authentication and app construction", async () => { vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "startup-test-token"); vi.stubEnv("HOST", "0.0.0.0"); @@ -672,6 +736,7 @@ describe("startServer feedback export wiring", () => { } finally { vi.unstubAllEnvs(); } }); + it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1175,7 +1240,8 @@ describe("startServer feedback export wiring", () => { it("refuses startup when an explicit decision signing secret is too short", async () => { process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "too-short"; await expect(startServer()).rejects.toThrow("PAPERCLIP_DECISION_SIGNING_SECRET must be at least 32 characters"); - expect(loadConfigMock).not.toHaveBeenCalled(); + expect(createDbMock).not.toHaveBeenCalled(); + expect(createAppMock).not.toHaveBeenCalled(); }); it("passes the feedback export service into createApp so pending traces flush in runtime", async () => { diff --git a/server/src/config-file.ts b/server/src/config-file.ts index d750793c204..fe85ae2feee 100644 --- a/server/src/config-file.ts +++ b/server/src/config-file.ts @@ -20,8 +20,22 @@ function formatConfigValidationError(error: ZodError): string { .join("; "); } -export function readConfigFile(): PaperclipConfig | null { - if (isCloudManagedInstance()) return toPaperclipConfig(getCloudRuntimeConfigSnapshot().config); +interface ConfigFileWithProvenance { + config: PaperclipConfig; + explicitStorageS3Bucket: boolean; +} + +export function readConfigFile(options: { includeProvenance: true }): ConfigFileWithProvenance | null; +export function readConfigFile(): PaperclipConfig | null; +export function readConfigFile(options?: { includeProvenance: true }): PaperclipConfig | ConfigFileWithProvenance | null { + if (isCloudManagedInstance()) { + const saved = getCloudRuntimeConfigSnapshot().config; + const config = toPaperclipConfig(saved); + // A stored snapshot is canonical configuration, including its bucket. + // It must not inherit local-file provenance or inspect an old home. + return config && options?.includeProvenance ? { config, explicitStorageS3Bucket: true } : config; + } + const configPath = resolvePaperclipConfigPath(); if (!fs.existsSync(configPath)) return null; @@ -41,6 +55,12 @@ export function readConfigFile(): PaperclipConfig | null { `Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`, ); } + if (options?.includeProvenance) { + // Schema validation above establishes the input shape, but fills omitted + // buckets with a default. Keep only that provenance at the file boundary. + const input = raw as Partial; + return { config, explicitStorageS3Bucket: input.storage?.s3?.bucket !== undefined }; + } return config; } catch (error) { if (error instanceof ZodError) { diff --git a/server/src/config.ts b/server/src/config.ts index c1156068fa8..a30c7335142 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -1,3 +1,4 @@ +import { validateManagedCloudBootstrapPrerequisites, validateManagedCloudPrerequisites } from "./managed-cloud-prerequisites.js"; import { readConfigFile, resolveDeploymentMode } from "./config-file.js"; import { isCloudManagedInstance } from "./services/cloud-instance.js"; import { StartupRefusalError } from "./startup-refusals.js"; @@ -143,7 +144,10 @@ function detectTailnetBindHost(): string | undefined { } } -export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()): Config { +export function loadConfig(override?: PaperclipConfig | null): Config { + const configFile = override === undefined ? readConfigFile({ includeProvenance: true }) + : override ? { config: override, explicitStorageS3Bucket: true } : null; + const fileConfig = configFile?.config ?? null; const fileDatabaseMode = (isCloudManagedInstance() || fileConfig?.database.mode === "postgres" ? "postgres" : "embedded-postgres") as DatabaseMode; @@ -163,6 +167,11 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile() const secretsProvider: SecretProvider = providerFromEnv ?? providerFromFile ?? "local_encrypted"; const storage = loadStorageConfig(fileConfig); + validateManagedCloudPrerequisites({ ...storage, + storageS3Bucket: process.env.PAPERCLIP_STORAGE_S3_BUCKET + ?? (configFile?.explicitStorageS3Bucket ? fileConfig?.storage.s3.bucket : undefined), + databaseUrl: process.env.DATABASE_URL ?? fileDbUrl, + }, process.env); const feedbackExportBackendUrl = process.env.PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL?.trim() || process.env.PAPERCLIP_TELEMETRY_BACKEND_URL?.trim() || @@ -448,5 +457,6 @@ export function loadCloudBootstrapConfig(): CloudBootstrapConfig { try { parsed = new URL(value); } catch { throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`); } if (!["postgres:", "postgresql:"].includes(parsed.protocol)) throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`); } + validateManagedCloudBootstrapPrerequisites({ databaseUrl: url }, process.env); return { databaseMode: "postgres", databaseUrl: url, databaseMigrationUrl: migrationUrl }; } diff --git a/server/src/index.ts b/server/src/index.ts index ae229f03518..c127753889c 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -296,9 +296,11 @@ async function startServerWithDatabaseTeardown( // Error monitoring must be ready before the first request can fail — see // sentry.ts. await sentryReady; - ensureDecisionSigningSecret(); const startupConfig = isCloudManagedInstance() ? loadCloudBootstrapConfig() : loadConfig(); - if ("deploymentMode" in startupConfig) initializeRuntimeConfigConsumers(startupConfig); + if ("deploymentMode" in startupConfig) { + ensureDecisionSigningSecret(); + initializeRuntimeConfigConsumers(startupConfig); + } type MigrationSummary = | "skipped" @@ -728,6 +730,9 @@ async function startServerWithDatabaseTeardown( await initializeCloudRuntimeConfig(db as any); let config = "deploymentMode" in startupConfig ? startupConfig : loadConfig(); + // Cloud storage depends on the canonical DB snapshot. Validate it before + // signing keys or other configuration consumers can mutate local state. + if (isCloudManagedInstance()) ensureDecisionSigningSecret(); // A claimed warm-pool stack may restart while its provider environment still // names the pool host. Restore the signed, durable identity before Better diff --git a/server/src/managed-cloud-prerequisites.ts b/server/src/managed-cloud-prerequisites.ts new file mode 100644 index 00000000000..84305467af8 --- /dev/null +++ b/server/src/managed-cloud-prerequisites.ts @@ -0,0 +1,77 @@ +import { isCloudManagedInstance } from "./services/cloud-instance.js"; +import { parseManagedConfigEnv, type ManagedConfigEnv } from "./services/managed-config.js"; +import { isValidLocalEncryptedMasterKey } from "./secrets/local-encrypted-provider.js"; +import type { StorageProvider } from "@paperclipai/shared"; + +interface ManagedCloudPrerequisites { + storageProvider: StorageProvider; + /** The canonical DB/config value or explicit env/file value, before defaults. */ + storageS3Bucket: string | undefined; + storageS3Region: string; + storageS3Endpoint: string | undefined; + databaseUrl: string | undefined; +} + +/** Pure startup policy for the existing managed-cloud profile. No I/O or probes. */ +export function validateManagedCloudPrerequisites( + config: ManagedCloudPrerequisites, + env: ManagedConfigEnv, +): void { + validateManagedCloudStoragePrerequisites(config, env); + validateManagedCloudBootstrapPrerequisites(config, env); +} + +/** Validate a resolved candidate before its first Cloud import or startup use. */ +export function validateManagedCloudStoragePrerequisites( + config: Pick, + env: ManagedConfigEnv, +): void { + if (!isCloudManagedInstance(env)) return; + parseManagedConfigEnv(env); + + const refuse = (message: string): never => { + throw new Error(`Managed cloud configuration: ${message}; refusing to start`); + }; + // An invalid explicit env value must not fall through to a file/default value. + if ((env.PAPERCLIP_STORAGE_PROVIDER !== undefined && env.PAPERCLIP_STORAGE_PROVIDER !== "s3") + || config.storageProvider !== "s3") { + refuse("set PAPERCLIP_STORAGE_PROVIDER=s3 or storage.provider=s3 in the config file"); + } + if (!config.storageS3Bucket?.trim()) { + refuse("set PAPERCLIP_STORAGE_S3_BUCKET or storage.s3.bucket explicitly"); + } + if (!config.storageS3Region.trim()) { + refuse("set a nonblank PAPERCLIP_STORAGE_S3_REGION or storage.s3.region"); + } + if (config.storageS3Endpoint !== undefined) { + let validEndpoint = false; + try { + const url = new URL(config.storageS3Endpoint); + validEndpoint = (url.protocol === "https:" || url.protocol === "http:") + && !!url.hostname && !url.username && !url.password; + } catch { /* The diagnostic must not include a possibly secret URL. */ } + if (!validEndpoint) refuse("PAPERCLIP_STORAGE_S3_ENDPOINT must be an HTTP(S) URL without credentials"); + } +} + +/** Environment-owned prerequisites needed before opening the Cloud database. */ +export function validateManagedCloudBootstrapPrerequisites( + config: { databaseUrl: string | undefined }, env: ManagedConfigEnv, +): void { + if (!isCloudManagedInstance(env)) return; + parseManagedConfigEnv(env); + const refuse = (message: string): never => { + throw new Error(`Managed cloud configuration: ${message}; refusing to start`); + }; + let validDatabase = false; + try { + const url = new URL(config.databaseUrl ?? ""); + validDatabase = (url.protocol === "postgres:" || url.protocol === "postgresql:") && !!url.hostname; + } catch { /* The diagnostic must not include database credentials. */ } + if (!validDatabase) { + refuse("set DATABASE_URL to an external PostgreSQL URL"); + } + if (!isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY ?? "")) { + refuse("set a stable PAPERCLIP_SECRETS_MASTER_KEY (32-byte base64, 64-char hex, or raw 32-char string); key files alone are not supported"); + } +} diff --git a/server/src/secrets/local-encrypted-provider.ts b/server/src/secrets/local-encrypted-provider.ts index f0162fb56e8..bc4820a3d38 100644 --- a/server/src/secrets/local-encrypted-provider.ts +++ b/server/src/secrets/local-encrypted-provider.ts @@ -50,6 +50,11 @@ function decodeMasterKey(raw: string): Buffer | null { return null; } +/** Check the configured key without reading, creating, or changing key files. */ +export function isValidLocalEncryptedMasterKey(raw: string): boolean { + return decodeMasterKey(raw) !== null; +} + function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = process.env): Buffer { const envKeyRaw = env.PAPERCLIP_SECRETS_MASTER_KEY; if (envKeyRaw && envKeyRaw.trim().length > 0) { diff --git a/server/src/services/cloud-runtime-config-snapshot.ts b/server/src/services/cloud-runtime-config-snapshot.ts index e97cd534b24..b63f7f0563d 100644 --- a/server/src/services/cloud-runtime-config-snapshot.ts +++ b/server/src/services/cloud-runtime-config-snapshot.ts @@ -25,7 +25,7 @@ export const cloudRuntimeConfigSchema = z.object({ backup: databaseBackupConfigSchema.strict(), storage: storageConfigSchema.extend({ localDisk: storageLocalDiskConfigSchema.strict(), - s3: storageS3ConfigSchema.extend({ endpoint: publicUrl.optional() }).strict(), + s3: storageS3ConfigSchema.extend({ bucket: z.string().trim().min(1), endpoint: publicUrl.optional() }).strict(), }).strict(), secrets: secretsConfigSchema.extend({ localEncrypted: secretsLocalEncryptedConfigSchema.strict(), diff --git a/server/src/services/cloud-runtime-config.ts b/server/src/services/cloud-runtime-config.ts index 61082ae07c7..6a396cc3849 100644 --- a/server/src/services/cloud-runtime-config.ts +++ b/server/src/services/cloud-runtime-config.ts @@ -5,6 +5,8 @@ import { eq } from "drizzle-orm"; import { instanceSettings, type Db } from "@paperclipai/db"; import { paperclipConfigSchema } from "@paperclipai/shared"; import { resolvePaperclipConfigPathForInstance } from "@paperclipai/shared/home-paths"; +import { validateManagedCloudStoragePrerequisites } from "../managed-cloud-prerequisites.js"; +import { loadStorageConfig } from "../storage/config.js"; import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js"; import { parseStoredCloudRuntimeConfig, @@ -83,11 +85,18 @@ export async function readLegacyCloudRuntimeConfig(env: CloudInstanceEnv = proce if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Legacy Cloud runtime configuration could not be read safely"); return null; } + // Refuse an unconfigured legacy bucket before shared-schema defaults can + // turn its omission into an apparently explicit canonical database value. + const legacyStorage = (raw as { storage?: { provider?: unknown; s3?: { bucket?: unknown } } } | null)?.storage; + if (typeof legacyStorage?.s3?.bucket !== "string" || !legacyStorage.s3.bucket.trim()) { + throw new Error("Legacy Cloud runtime configuration requires an explicit nonblank storage.s3.bucket"); + } const parsed = paperclipConfigSchema.safeParse(raw); if (!parsed.success) throw new Error("Legacy Cloud runtime configuration is invalid"); if (parsed.data.llm?.provider === "openai" && parsed.data.llm.apiKey?.trim() && !env.OPENAI_API_KEY?.trim()) { throw new Error("Move the legacy OpenAI API key to OPENAI_API_KEY before initializing Cloud runtime configuration"); } + validateManagedCloudStoragePrerequisites(loadStorageConfig(parsed.data, env), env); return projectCloudRuntimeConfig(parsed.data); }