mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
fix(config): require shared storage for managed cloud (#18350)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed Cloud instances must retain shared objects and secrets across server replacement. > - Implicit storage defaults can select local disk or an unintended bucket. > - Cloud runtime settings now come from a canonical database snapshot. > - This pull request validates bootstrap prerequisites before database setup and effective S3 settings after snapshot resolution. > - The benefit is an actionable refusal before local signing state or HTTP setup, while valid canonical Cloud settings work. ## Linked Issues or Issue Description **What happened?** A managed Cloud instance could start with local disk storage, an implicit bucket, or a local key file. After canonical Cloud configuration landed, checking only environment or file values would also reject a valid database bucket. Shared schema defaults could make an omitted raw bucket appear explicit. **Expected behavior** Managed Cloud requires external PostgreSQL, a stable environment master key, and effective S3 settings with a nonblank bucket. The canonical snapshot remains the configuration authority. Invalid configuration refuses startup before signing or HTTP effects. Self-hosted defaults remain compatible. **Steps to reproduce** 1. Select managed Cloud with its tenant token or managed configuration document. 2. Leave storage as local disk, or omit the S3 bucket from a raw canonical snapshot. 3. Start the server. Startup must refuse. An explicit valid bucket in the canonical snapshot must work without a file or environment bucket. **Paperclip version or commit** Current base: `4edd10804e2b42941aa2038e90c362c13a0b98dc`. **Deployment mode** Managed Cloud. Ordinary self-hosted defaults remain unchanged. Related PRs: #18335 owns canonical Cloud runtime configuration. #18327 owns S3 run logs. #18352 owns credential persistence. #18347 owns cutover checks. This PR adds configuration prerequisites to those existing boundaries. ## What Changed - Validate external PostgreSQL and a stable environment master key at the Cloud bootstrap boundary, before database setup. - Validate effective S3 settings after canonical snapshot publication and before Cloud signing, runtime consumers, or HTTP setup. - Require a nonblank bucket in the Cloud snapshot schema. Reject omitted or blank buckets in every legacy Cloud file before one-time import normalization, even when an environment bucket is supplied. Keep shared self-hosted schema defaults. - Preserve typed bucket provenance at the existing file-read boundary. Accept explicit canonical bucket names, including `paperclip`, and environment settings when the snapshot is null. - Validate effective legacy storage with the existing resolver before seeding. A rejected candidate leaves no canonical rows, so a corrected source file can be imported on retry. Environment overlays are checked but never persisted. - Add real PostgreSQL snapshot and migration-retry tests, startup refusal tests, and environment/file compatibility controls. Document that Cloud always requires its stable environment key, even with an external default secrets provider. ## Verification - Latest focused acceptance passed 199/199 in six files (54.29 seconds), using the command below plus `server/src/__tests__/cloud-configuration-migration-postgres.test.ts`. Real PostgreSQL proves rejected storage leaves all three migration authority rows absent, then a corrected legacy file imports on retry. An external-default-secrets control proves the deliberate unconditional Cloud master-key contract. - Before the peer correction, the frozen product diff passed 178 tests in five files: managed-cloud-prerequisites, cloud-runtime-config, server-startup-feedback-export, config-file, and cloud-configuration-readers. Command: `pnpm exec vitest run server/src/__tests__/{managed-cloud-prerequisites,cloud-runtime-config,server-startup-feedback-export,config-file,cloud-configuration-readers}.test.ts --maxWorkers=1 --no-file-parallelism`. Real PostgreSQL used a qualified POSIX test-only PG18.6 override because the host cannot supply the stock runtime's SysV semaphore allocation. - Coverage includes raw database and legacy omitted/blank bucket rejection, valid database and environment buckets, null snapshots, explicit default-named buckets, self-hosted compatibility, and refusal before local signing mutation. - The real AWS SDK configured-provider control uses a simulated loopback S3 service. It writes, reads from a fresh provider, and deletes an object. It also reads an encrypted credential with the stable environment key. This does not qualify production cloud permissions. - `git diff --check` passed. Final merges add unrelated master changes from #18241, #18363, and #18370; the focused storage product diff is unchanged. - Peer review reproduced a fresh legacy local-disk import that gained the default bucket before an S3 environment overlay. The importer now requires a raw explicit bucket for every legacy Cloud file. Six controls cover omission/blank values with and without an environment bucket; the full focused rerun passed 184/184 across all five files in 194.89 seconds. - Initial revised startup testing passed 167/168: one fixture omitted the database URL and failed before its intended key assertion. The corrected final run passed 178/178. An initial direct server typecheck failed because Runner declarations were not built; this is a setup failure, not a compile pass. Integration verification rebuilds dependencies before full checks. - Prior head `fae1e4aa` passed complete CI with 53 successes, two intended skips, and Greptile 5/5. `de71345f379cecb71dbfb219f35e14f942c10a3f` then passed complete CI (55 successes, three workflow skips, and a neutral external review), with Greptile 5/5 and zero unresolved threads. The external review identified the preseed rejection problem; its correction is committed at `75e04d0d77e19dcef0a8cb070b6b518f0f173b14`. The final merge head is `19b809ea2823a6a4e5ceb48ef2c2bac4721733d6`; [Fresh complete CI](https://github.com/paperclipai/paperclip/actions/runs/41127309762) passed on that head: 55 successful checks, three workflow skips, and one neutral external review. Greptile reviewed this exact head at 5/5 with no actionable issues. Both earlier external review threads are resolved with the tested fixes. Earlier infrastructure retries remain documented in [the first diagnosis](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6529964690) and [the runner shutdown evidence](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6530364769). Earlier missing SDK build and fixture metadata failures were repaired before those passes. - Current frozen-head CI first attempt failed one unchanged Antigravity authentication success case in Runner Vitest 1/2 (`outcome=failure`, 1994 tests passed). The same file passed 11/11 locally in 2.33 seconds, and identical Runner source passed the prior complete CI run. Attempt 2 retried only failed jobs (the Runner shard and dependent aggregate). Both passed without a source change. No second retry was used. The exact timing cause is not established. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/41127309762/job/124309181778). - No production data, buckets, or live credentials were used. Complete local repository suites are not duplicated in this worktree; current-head required CI supplies the full suite gate. ## Risks - Managed deployments with local disk, omitted buckets, file-only keys, or missing external database URLs now refuse startup. Operators must supply valid prerequisites before rollout. - Required deployment environment overlays must remain stable across replacements. For example, a saved local-disk provider with an explicit bucket can use an environment S3 provider; removing that required overlay correctly refuses startup until the environment or canonical configuration is corrected. Environment settings are never silently copied into the canonical row. - Effective legacy storage must also pass before first seeding. A bad candidate does not create authority rows. Existing saved rows remain authoritative and can require environment overrides or a canonical configuration change if they are invalid. - A present legacy Cloud file needs an explicit raw bucket even if its saved provider is local disk or its environment supplies a bucket. Correct the file before one-time import, or use the existing absent-file/null-snapshot environment-only setup. Environment overlays are not persisted as operator choices. - A previously stored canonical nonblank `paperclip` bucket is authoritative. Prior normalization erased whether that value was originally explicit. This change cannot recover that provenance and does not reject existing nonblank canonical values. - Validation proves configuration shape, not cloud permissions, database reachability, or key compatibility with existing encrypted data. Keep the master key stable across replacements. - Cloud snapshot JSON parsing is stricter; omitted or blank raw S3 bucket values need correction. No database migration or provider operation changes are added. These prerequisites do not prove that every server persistence path is disposable. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Context window: unknown. The model used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
0e15c92073
commit
198218bf55
14 files changed
+519
-10
No files matched your search
@@ -1511,6 +1511,45 @@ Configure storage provider/settings:
|
||||
pnpm paperclipai configure --section storage
|
||||
```
|
||||
|
||||
### Managed cloud prerequisites
|
||||
|
||||
A nonblank Cloud tenant token or the presence of `PAPERCLIP_MANAGED_CONFIG`
|
||||
selects the managed-cloud profile. A supplied blank or invalid managed feature
|
||||
document refuses startup. Without either signal, the instance remains self-hosted.
|
||||
The managed profile also requires:
|
||||
|
||||
- `PAPERCLIP_STORAGE_PROVIDER=s3` or `storage.provider: "s3"` in the canonical
|
||||
database snapshot. An invalid explicit environment value refuses startup even
|
||||
if the snapshot selects S3.
|
||||
- An explicit nonblank `PAPERCLIP_STORAGE_S3_BUCKET` or `storage.s3.bucket`.
|
||||
The self-hosted `paperclip` bucket default does not qualify. The region defaults
|
||||
to `us-east-1`; an explicit blank region refuses startup. An optional endpoint
|
||||
must be an HTTP(S) URL without embedded credentials.
|
||||
- An external PostgreSQL URL from the environment `DATABASE_URL`.
|
||||
The managed profile never falls back to embedded PostgreSQL.
|
||||
- A stable `PAPERCLIP_SECRETS_MASTER_KEY` supplied by the deployment secret store.
|
||||
Use the same value across replacements. The existing provider accepts 32-byte
|
||||
base64, 64-character hex, or a raw 32-byte UTF-8 string. A key file alone does
|
||||
not qualify because a replacement web server may not have that file.
|
||||
|
||||
Bootstrap validates the environment database URL and encryption master key before
|
||||
opening the database or creating local state. Effective storage validation follows
|
||||
canonical snapshot resolution, before signing keys, first-party Telemetry or app
|
||||
construction. Raw canonical rows and legacy S3 imports cannot obtain a missing
|
||||
bucket from schema defaults. Already-stored nonblank bucket values remain
|
||||
canonical, including the deliberate name `paperclip`; prior normalization cannot
|
||||
reliably establish their original provenance.
|
||||
It does not contact S3 or verify credentials, bucket access, database connectivity,
|
||||
or that a key matches existing encrypted data. Operators must qualify ordinary
|
||||
S3 put/read/delete with deployment credentials and preserve the existing key.
|
||||
AWS credentials continue to use the provider's normal SDK credential chain.
|
||||
Local and self-hosted storage and key-file defaults remain unchanged.
|
||||
|
||||
These are prerequisites for shared storage and secrets. They do not establish
|
||||
that the whole web server is disposable or that all execution and recovery paths
|
||||
survive web-server replacement. Run-log/trace storage and producer retention are
|
||||
separate changes.
|
||||
|
||||
## Agent Artifact Uploads
|
||||
|
||||
When an agent generates a file that a board user or reviewer should inspect as
|
||||
|
||||
@@ -18,7 +18,8 @@ existing validation at application startup.
|
||||
to embedded PostgreSQL. Credentials, trust keys, provider configuration and
|
||||
deployment wiring needed before the database connection remain supplied by
|
||||
the environment or the existing secret system. This phase parses only the
|
||||
database connection record. Runtime mode, host binding and authentication
|
||||
database connection record and requires the stable environment encryption
|
||||
master key. Runtime mode, host binding and authentication
|
||||
checks wait until the saved snapshot has loaded.
|
||||
2. After schema migration and database connection, startup reads the private
|
||||
runtime configuration row. An existing invalid row or a database error stops
|
||||
@@ -38,6 +39,27 @@ existing validation at application startup.
|
||||
authentication and application services. A saved telemetry opt-out therefore
|
||||
takes effect before a telemetry client can be created.
|
||||
|
||||
Managed Cloud always requires a stable environment master key, including when
|
||||
an external default secrets provider is selected. Internal local-encrypted
|
||||
credential paths still need that key.
|
||||
|
||||
Managed Cloud validates effective S3 storage after publishing the canonical
|
||||
snapshot, before signing-key initialization, telemetry, authentication or app
|
||||
construction. Cloud requires S3 and a nonblank bucket; an authoritative absent
|
||||
configuration needs explicit environment storage settings. Raw saved snapshots
|
||||
cannot obtain a bucket from a schema default. Every legacy Cloud file must name its
|
||||
bucket before its one-time import, even if its saved provider is local disk and
|
||||
an environment bucket is present. Omission or blank values fail before seeding;
|
||||
The effective storage candidate is validated before seeding, so a rejected
|
||||
legacy file leaves no canonical row and can be corrected before retry.
|
||||
Environment overlays are not saved as operator choices. Correct that file before
|
||||
import, or use the existing absent-file/null-snapshot environment-only setup.
|
||||
Ordinary self-hosted file defaults remain unchanged.
|
||||
Already-canonical nonblank values, including `paperclip`, remain authoritative.
|
||||
Prior normalization cannot reliably distinguish an operator's deliberate name
|
||||
from a historical default, so this release does not invent retrospective
|
||||
provenance or replace those saved values.
|
||||
|
||||
Existing operator environment overlays keep precedence over saved values.
|
||||
Reading an overlaid value does not save it back to the database. Credentials and
|
||||
connection URLs are never copied into this JSONB row.
|
||||
@@ -78,8 +100,9 @@ bytes. Restart tests prove configuration restoration under a different home;
|
||||
they do not prove durable file, secret, or backup storage.
|
||||
|
||||
The decision signing service still uses `PAPERCLIP_DECISION_SIGNING_SECRET` when
|
||||
supplied and otherwise reads or creates its instance key file before the database
|
||||
connection. The local encrypted secret provider still uses its environment key
|
||||
supplied and otherwise reads or creates its instance key file. Managed Cloud
|
||||
performs that step after effective storage validation; self-hosted startup keeps
|
||||
its pre-connection ordering. The local encrypted secret provider still uses its environment key
|
||||
or existing key-file mechanism. Those byte-storage fallbacks are outside this
|
||||
configuration migration. Qualification must account for them separately; this
|
||||
PR does not establish file-free secret storage on Cloud.
|
||||
|
||||
@@ -55,6 +55,7 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura
|
||||
const config = paperclipConfigSchema.parse({
|
||||
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
|
||||
database: { mode: "postgres" }, logging: { mode: "cloud" },
|
||||
storage: { provider: "s3", s3: { bucket: "migration-bucket" } },
|
||||
server: { port: 3199 }, telemetry: { enabled: false },
|
||||
});
|
||||
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
|
||||
@@ -120,6 +121,18 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura
|
||||
} finally { await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("does not seed rejected storage and imports the corrected legacy file on retry", async () => {
|
||||
const config = await legacy();
|
||||
config.storage.provider = "local_disk";
|
||||
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
|
||||
await expect(migration().migrate()).rejects.toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
|
||||
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
|
||||
config.storage.provider = "s3";
|
||||
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
|
||||
expect((await migration(peer).migrate()).migration.config.authority).toBe("legacy_file");
|
||||
expect((await cloudRuntimeConfigService(peer).inspectStored())?.config?.storage.provider).toBe("s3");
|
||||
});
|
||||
|
||||
it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => {
|
||||
await legacy();
|
||||
const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]);
|
||||
|
||||
@@ -66,6 +66,9 @@ beforeEach(() => {
|
||||
vi.stubEnv("PAPERCLIP_ALLOWED_HOSTNAMES", undefined);
|
||||
vi.stubEnv("PAPERCLIP_TAILNET_BIND_HOST", "100.100.0.1");
|
||||
vi.stubEnv("OPENAI_API_KEY", "environment-api-secret");
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
setCloudRuntimeConfigSnapshot(null);
|
||||
});
|
||||
|
||||
@@ -170,6 +173,10 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => {
|
||||
expect(fs.existsSync(configPath)).toBe(false);
|
||||
vi.stubEnv("PAPERCLIP_DEPLOYMENT_MODE", "authenticated");
|
||||
vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://environment.example.test");
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "test-token");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "environment-bucket");
|
||||
expect(loadConfig()).toMatchObject({ databaseMode: "postgres", authBaseUrlMode: "explicit", secretsStrictMode: true, telemetryEnabled: true });
|
||||
});
|
||||
|
||||
@@ -284,6 +291,26 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => {
|
||||
expect(await second.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "runtime-config-test-receipt"))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([undefined, "", " "])("rejects raw canonical S3 rows with missing or blank bucket %j", async bucket => {
|
||||
const config = structuredClone(projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig())));
|
||||
const raw = config;
|
||||
if (bucket === undefined) Reflect.deleteProperty(raw.storage.s3, "bucket");
|
||||
else raw.storage.s3.bucket = bucket;
|
||||
await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 1, revision: 0, config: raw } });
|
||||
await expect(initializeCloudRuntimeConfig(second)).rejects.toThrow(/Persisted Cloud runtime configuration is invalid/);
|
||||
expect(() => getCloudRuntimeConfigSnapshot()).toThrow(/not been initialized/);
|
||||
});
|
||||
|
||||
it.each(["database-bucket", "paperclip"])("accepts canonical DB bucket %s without local configuration provenance", async bucket => {
|
||||
const config = projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig()));
|
||||
config.storage.s3.bucket = bucket;
|
||||
await cloudRuntimeConfigService(first).seedIfAbsent(config);
|
||||
fs.writeFileSync(configPath, "invalid local file must not be consulted");
|
||||
await initializeCloudRuntimeConfig(second);
|
||||
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: bucket });
|
||||
expect(readConfigFile({ includeProvenance: true })).toMatchObject({ explicitStorageS3Bucket: true });
|
||||
});
|
||||
|
||||
it("fails closed on invalid database settings without consulting a valid legacy file", async () => {
|
||||
fs.writeFileSync(configPath, JSON.stringify(legacyConfig()));
|
||||
await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 99, config: { secret: "hidden" } } });
|
||||
|
||||
@@ -91,6 +91,20 @@ describe("readConfigFile", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves explicit bucket provenance without changing defaults for existing readers", () => {
|
||||
for (const bucket of [undefined, "paperclip"]) {
|
||||
writeConfig(configPath, {
|
||||
...minimalConfig() as object,
|
||||
storage: { provider: "s3", s3: bucket === undefined ? {} : { bucket } },
|
||||
});
|
||||
expect(readConfigFile()?.storage.s3.bucket).toBe("paperclip");
|
||||
expect(readConfigFile({ includeProvenance: true })).toMatchObject({
|
||||
config: { storage: { s3: { bucket: "paperclip" } } },
|
||||
explicitStorageS3Bucket: bucket !== undefined,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("warns about likely misspellings without stripping them", () => {
|
||||
const config = {
|
||||
...(minimalConfig() as Record<string, unknown>),
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { createStorageProviderFromConfig } from "../storage/provider-registry.js";
|
||||
import { createLocalEncryptedCipher } from "../secrets/local-encrypted-provider.js";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { loadConfig } from "../config.js";
|
||||
import { readConfigFile } from "../config-file.js";
|
||||
import { readLegacyCloudRuntimeConfig } from "../services/cloud-runtime-config.js";
|
||||
import { setCloudRuntimeConfigSnapshot } from "../services/cloud-runtime-config-snapshot.js";
|
||||
const managed = JSON.stringify({
|
||||
v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] },
|
||||
});
|
||||
|
||||
let configDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null });
|
||||
configDir = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-config-"));
|
||||
vi.stubEnv("PAPERCLIP_CONFIG", path.join(configDir, "config.json"));
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "test-bucket");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", undefined);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", undefined);
|
||||
vi.stubEnv("DATABASE_URL", "postgresql://user:private@db.example.test/paperclip");
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
|
||||
});
|
||||
afterEach(() => {
|
||||
setCloudRuntimeConfigSnapshot(null);
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(configDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeConfig(storage: Record<string, unknown>) {
|
||||
writeFileSync(path.join(configDir, "config.json"), JSON.stringify({
|
||||
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
|
||||
logging: { mode: "file" },
|
||||
server: {},
|
||||
storage,
|
||||
database: { mode: "postgres", connectionString: "postgres://db.example.test/paperclip" },
|
||||
}));
|
||||
}
|
||||
|
||||
describe("managed cloud configuration boundary", () => {
|
||||
it("accepts configured S3, external PostgreSQL, and the existing master-key formats", () => {
|
||||
for (const key of ["12345678901234567890123456789012", Buffer.alloc(32, 7).toString("hex"), Buffer.alloc(32, 7).toString("base64")]) {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", key);
|
||||
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "test-bucket", storageS3Region: "us-east-1" });
|
||||
}
|
||||
});
|
||||
|
||||
it.each([undefined, "", "local_disk", "typo"])("rejects storage provider %j without silent fallback", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", value);
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
|
||||
});
|
||||
|
||||
it.each([undefined, "", " "])("requires an explicit bucket (%j)", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", value);
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/);
|
||||
});
|
||||
|
||||
it.each(["", " "])("rejects blank region %j", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", value);
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_REGION/);
|
||||
});
|
||||
|
||||
it.each(["", "ftp://store.test", "https://user:private@store.test"])("rejects invalid endpoint without exposing it", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", value);
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_ENDPOINT/);
|
||||
try { loadConfig(); } catch (error) { expect(String(error)).not.toContain("private"); }
|
||||
});
|
||||
|
||||
it.each([undefined, "", "sqlite://local/data", "postgresql://"])("requires an external database URL (%j)", (value) => {
|
||||
vi.stubEnv("DATABASE_URL", value);
|
||||
expect(() => loadConfig()).toThrow(/external PostgreSQL URL/);
|
||||
});
|
||||
|
||||
it.each([undefined, "", " ", "short"])("rejects missing or invalid env key (%j), even with a configured key file", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", value);
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", "/tmp/unused-master-key");
|
||||
expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
|
||||
});
|
||||
|
||||
it("requires the Cloud bootstrap key even when an external secret provider is selected", () => {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_PROVIDER", "aws_secrets_manager");
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
|
||||
expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
|
||||
expect(loadConfig().secretsProvider).toBe("aws_secrets_manager");
|
||||
});
|
||||
|
||||
it.each(["", "{bad", JSON.stringify({ v: 1, mode: "cloud" })])("rejects malformed managed config before startup (%j)", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", value);
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_MANAGED_CONFIG/);
|
||||
});
|
||||
|
||||
it("accepts imported explicit file storage with an environment-owned database", async () => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
writeConfig({ provider: "s3", s3: { bucket: "file-bucket", region: "eu-west-1" } });
|
||||
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() });
|
||||
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "file-bucket", storageS3Region: "eu-west-1" });
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo");
|
||||
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
|
||||
});
|
||||
|
||||
it.each([{ provider: "s3" }, { provider: "s3", s3: { region: "us-east-1" } }])("rejects a real config file with an omitted bucket before import (%j)", async storage => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
writeConfig(storage);
|
||||
// Self-hosted readers still expose their default and raw-file provenance.
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined);
|
||||
expect(readConfigFile({ includeProvenance: true })).toMatchObject({ config: { storage: { s3: { bucket: "paperclip" } } }, explicitStorageS3Bucket: false });
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed);
|
||||
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
|
||||
});
|
||||
|
||||
it.each(["", " "])("rejects an explicitly blank bucket in a real file (%j)", async bucket => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
writeConfig({ provider: "s3", s3: { bucket } });
|
||||
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
|
||||
});
|
||||
|
||||
it.each([undefined, "", " "].flatMap(bucket => [undefined, "environment-bucket"].map(environmentBucket => ({ bucket, environmentBucket }))))("rejects a legacy local-disk file with no explicit bucket before an S3 environment overlay (%j)", async ({ bucket, environmentBucket }) => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", environmentBucket);
|
||||
writeConfig({ provider: "local_disk", s3: { bucket } });
|
||||
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
|
||||
});
|
||||
|
||||
it("accepts an explicitly supplied default bucket name in a real file", async () => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
writeConfig({ provider: "s3", s3: { bucket: "paperclip" } });
|
||||
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() });
|
||||
expect(loadConfig().storageS3Bucket).toBe("paperclip");
|
||||
});
|
||||
|
||||
it("accepts an environment bucket with an authoritative absent-file snapshot", () => {
|
||||
writeConfig({ provider: "s3" });
|
||||
expect(loadConfig().storageS3Bucket).toBe("test-bucket");
|
||||
});
|
||||
|
||||
it("preserves ordinary self-hosted defaults, including key-file use", () => {
|
||||
for (const name of ["PAPERCLIP_MANAGED_CONFIG", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_STORAGE_S3_BUCKET", "DATABASE_URL", "PAPERCLIP_SECRETS_MASTER_KEY"]) vi.stubEnv(name, undefined);
|
||||
expect(loadConfig()).toMatchObject({ storageProvider: "local_disk", storageS3Bucket: "paperclip", databaseUrl: undefined, databaseMode: "embedded-postgres" });
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo");
|
||||
expect(loadConfig().storageProvider).toBe("local_disk");
|
||||
});
|
||||
});
|
||||
|
||||
// Real provider + AWS SDK over loopback. The remote S3 service and credentials
|
||||
// are simulated; this does not establish deployment bucket permissions.
|
||||
it("uses the validated config for normal S3 put/read/delete and stable env-key encryption", async () => {
|
||||
const objects = new Map<string, Buffer>();
|
||||
const operations: string[] = [];
|
||||
const server = createServer(async (req, res) => {
|
||||
const key = new URL(req.url!, "http://localhost").pathname;
|
||||
operations.push(`${req.method} ${key}`);
|
||||
if (req.method === "PUT") {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of req) chunks.push(Buffer.from(chunk));
|
||||
objects.set(key, Buffer.concat(chunks));
|
||||
res.end();
|
||||
} else if (req.method === "GET" && objects.has(key)) {
|
||||
const body = objects.get(key)!;
|
||||
res.writeHead(200, { "content-type": "text/plain", "content-length": body.length });
|
||||
res.end(body);
|
||||
} else if (req.method === "DELETE") {
|
||||
objects.delete(key);
|
||||
res.writeHead(204).end();
|
||||
} else {
|
||||
res.writeHead(404).end();
|
||||
}
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture-access-key");
|
||||
vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture-secret-key");
|
||||
vi.stubEnv("AWS_SESSION_TOKEN", undefined);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", `http://127.0.0.1:${(server.address() as AddressInfo).port}`);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "disposable-qualification");
|
||||
try {
|
||||
const config = loadConfig();
|
||||
const first = createStorageProviderFromConfig(config);
|
||||
await first.putObject({ objectKey: "company/object.txt", body: Buffer.from("retained"), contentType: "text/plain", contentLength: 8 });
|
||||
const replacement = createStorageProviderFromConfig(loadConfig());
|
||||
const result = await replacement.getObject({ objectKey: "company/object.txt" });
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of result.stream) chunks.push(Buffer.from(chunk));
|
||||
expect(Buffer.concat(chunks).toString()).toBe("retained");
|
||||
await replacement.deleteObject({ objectKey: "company/object.txt" });
|
||||
expect(objects.size).toBe(0);
|
||||
expect(operations).toEqual(["PUT", "GET", "DELETE"].map(method => `${method} /test-bucket/disposable-qualification/company/object.txt`));
|
||||
const encrypted = createLocalEncryptedCipher("write").encrypt("saved credential");
|
||||
expect(createLocalEncryptedCipher("read").decrypt(encrypted)).toBe("saved credential");
|
||||
} finally {
|
||||
server.closeAllConnections();
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
@@ -614,6 +614,70 @@ describe("startServer feedback export wiring", () => {
|
||||
process.env.BETTER_AUTH_SECRET = "test-secret";
|
||||
});
|
||||
|
||||
|
||||
it("refuses invalid cloud bootstrap prerequisites before signing, DB creation, or HTTP setup", async () => {
|
||||
const realConfig = await vi.importActual<typeof import("../config.js")>("../config.js");
|
||||
const originalHome = process.env.PAPERCLIP_HOME;
|
||||
const originalManaged = process.env.PAPERCLIP_MANAGED_CONFIG;
|
||||
const originalStorage = process.env.PAPERCLIP_STORAGE_PROVIDER;
|
||||
const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-refusal-"));
|
||||
process.env.PAPERCLIP_HOME = tempHome;
|
||||
process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({
|
||||
v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] },
|
||||
});
|
||||
process.env.PAPERCLIP_STORAGE_PROVIDER = "s3";
|
||||
delete process.env.PAPERCLIP_DECISION_SIGNING_SECRET;
|
||||
cloudRuntimeStartupMocks.bootstrap.mockImplementationOnce(realConfig.loadCloudBootstrapConfig);
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
|
||||
vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl);
|
||||
try {
|
||||
await expect(startServer()).rejects.toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
|
||||
expect(createDbMock).not.toHaveBeenCalled();
|
||||
expect(createAppMock).not.toHaveBeenCalled();
|
||||
expect(existsSync(path.join(tempHome, "instances"))).toBe(false);
|
||||
} finally {
|
||||
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = originalHome;
|
||||
if (originalManaged === undefined) delete process.env.PAPERCLIP_MANAGED_CONFIG;
|
||||
else process.env.PAPERCLIP_MANAGED_CONFIG = originalManaged;
|
||||
if (originalStorage === undefined) delete process.env.PAPERCLIP_STORAGE_PROVIDER;
|
||||
else process.env.PAPERCLIP_STORAGE_PROVIDER = originalStorage;
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(tempHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
it("rejects missing effective Cloud storage after the snapshot and before local signing mutations", async () => {
|
||||
const realConfig = await vi.importActual<typeof import("../config.js")>("../config.js");
|
||||
const { setCloudRuntimeConfigSnapshot } = await import("../services/cloud-runtime-config-snapshot.js");
|
||||
const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-storage-refusal-"));
|
||||
vi.stubEnv("PAPERCLIP_HOME", tempHome);
|
||||
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] } }));
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
|
||||
vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
|
||||
vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", undefined);
|
||||
cloudRuntimeStartupMocks.initialize.mockImplementationOnce(async () => {
|
||||
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null });
|
||||
return null;
|
||||
});
|
||||
loadConfigMock.mockImplementationOnce(realConfig.loadConfig);
|
||||
try {
|
||||
await expect(startServer()).rejects.toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/);
|
||||
expect(createDbMock).toHaveBeenCalled();
|
||||
expect(cloudRuntimeStartupMocks.initialize).toHaveBeenCalledOnce();
|
||||
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
|
||||
expect(createAppMock).not.toHaveBeenCalled();
|
||||
expect(existsSync(path.join(tempHome, "instances"))).toBe(false);
|
||||
} finally {
|
||||
setCloudRuntimeConfigSnapshot(null);
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(tempHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("awaits saved Cloud configuration before telemetry, authentication and app construction", async () => {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "startup-test-token");
|
||||
vi.stubEnv("HOST", "0.0.0.0");
|
||||
@@ -672,6 +736,7 @@ describe("startServer feedback export wiring", () => {
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
});
|
||||
|
||||
|
||||
it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1175,7 +1240,8 @@ describe("startServer feedback export wiring", () => {
|
||||
it("refuses startup when an explicit decision signing secret is too short", async () => {
|
||||
process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "too-short";
|
||||
await expect(startServer()).rejects.toThrow("PAPERCLIP_DECISION_SIGNING_SECRET must be at least 32 characters");
|
||||
expect(loadConfigMock).not.toHaveBeenCalled();
|
||||
expect(createDbMock).not.toHaveBeenCalled();
|
||||
expect(createAppMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("passes the feedback export service into createApp so pending traces flush in runtime", async () => {
|
||||
|
||||
@@ -20,8 +20,22 @@ function formatConfigValidationError(error: ZodError): string {
|
||||
.join("; ");
|
||||
}
|
||||
|
||||
export function readConfigFile(): PaperclipConfig | null {
|
||||
if (isCloudManagedInstance()) return toPaperclipConfig(getCloudRuntimeConfigSnapshot().config);
|
||||
interface ConfigFileWithProvenance {
|
||||
config: PaperclipConfig;
|
||||
explicitStorageS3Bucket: boolean;
|
||||
}
|
||||
|
||||
export function readConfigFile(options: { includeProvenance: true }): ConfigFileWithProvenance | null;
|
||||
export function readConfigFile(): PaperclipConfig | null;
|
||||
export function readConfigFile(options?: { includeProvenance: true }): PaperclipConfig | ConfigFileWithProvenance | null {
|
||||
if (isCloudManagedInstance()) {
|
||||
const saved = getCloudRuntimeConfigSnapshot().config;
|
||||
const config = toPaperclipConfig(saved);
|
||||
// A stored snapshot is canonical configuration, including its bucket.
|
||||
// It must not inherit local-file provenance or inspect an old home.
|
||||
return config && options?.includeProvenance ? { config, explicitStorageS3Bucket: true } : config;
|
||||
}
|
||||
|
||||
const configPath = resolvePaperclipConfigPath();
|
||||
|
||||
if (!fs.existsSync(configPath)) return null;
|
||||
@@ -41,6 +55,12 @@ export function readConfigFile(): PaperclipConfig | null {
|
||||
`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`,
|
||||
);
|
||||
}
|
||||
if (options?.includeProvenance) {
|
||||
// Schema validation above establishes the input shape, but fills omitted
|
||||
// buckets with a default. Keep only that provenance at the file boundary.
|
||||
const input = raw as Partial<PaperclipConfig>;
|
||||
return { config, explicitStorageS3Bucket: input.storage?.s3?.bucket !== undefined };
|
||||
}
|
||||
return config;
|
||||
} catch (error) {
|
||||
if (error instanceof ZodError) {
|
||||
|
||||
+11
-1
@@ -1,3 +1,4 @@
|
||||
import { validateManagedCloudBootstrapPrerequisites, validateManagedCloudPrerequisites } from "./managed-cloud-prerequisites.js";
|
||||
import { readConfigFile, resolveDeploymentMode } from "./config-file.js";
|
||||
import { isCloudManagedInstance } from "./services/cloud-instance.js";
|
||||
import { StartupRefusalError } from "./startup-refusals.js";
|
||||
@@ -143,7 +144,10 @@ function detectTailnetBindHost(): string | undefined {
|
||||
}
|
||||
}
|
||||
|
||||
export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()): Config {
|
||||
export function loadConfig(override?: PaperclipConfig | null): Config {
|
||||
const configFile = override === undefined ? readConfigFile({ includeProvenance: true })
|
||||
: override ? { config: override, explicitStorageS3Bucket: true } : null;
|
||||
const fileConfig = configFile?.config ?? null;
|
||||
const fileDatabaseMode =
|
||||
(isCloudManagedInstance() || fileConfig?.database.mode === "postgres" ? "postgres" : "embedded-postgres") as DatabaseMode;
|
||||
|
||||
@@ -163,6 +167,11 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()
|
||||
const secretsProvider: SecretProvider = providerFromEnv ?? providerFromFile ?? "local_encrypted";
|
||||
|
||||
const storage = loadStorageConfig(fileConfig);
|
||||
validateManagedCloudPrerequisites({ ...storage,
|
||||
storageS3Bucket: process.env.PAPERCLIP_STORAGE_S3_BUCKET
|
||||
?? (configFile?.explicitStorageS3Bucket ? fileConfig?.storage.s3.bucket : undefined),
|
||||
databaseUrl: process.env.DATABASE_URL ?? fileDbUrl,
|
||||
}, process.env);
|
||||
const feedbackExportBackendUrl =
|
||||
process.env.PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL?.trim() ||
|
||||
process.env.PAPERCLIP_TELEMETRY_BACKEND_URL?.trim() ||
|
||||
@@ -448,5 +457,6 @@ export function loadCloudBootstrapConfig(): CloudBootstrapConfig {
|
||||
try { parsed = new URL(value); } catch { throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`); }
|
||||
if (!["postgres:", "postgresql:"].includes(parsed.protocol)) throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`);
|
||||
}
|
||||
validateManagedCloudBootstrapPrerequisites({ databaseUrl: url }, process.env);
|
||||
return { databaseMode: "postgres", databaseUrl: url, databaseMigrationUrl: migrationUrl };
|
||||
}
|
||||
+7
-2
@@ -296,9 +296,11 @@ async function startServerWithDatabaseTeardown(
|
||||
// Error monitoring must be ready before the first request can fail — see
|
||||
// sentry.ts.
|
||||
await sentryReady;
|
||||
ensureDecisionSigningSecret();
|
||||
const startupConfig = isCloudManagedInstance() ? loadCloudBootstrapConfig() : loadConfig();
|
||||
if ("deploymentMode" in startupConfig) initializeRuntimeConfigConsumers(startupConfig);
|
||||
if ("deploymentMode" in startupConfig) {
|
||||
ensureDecisionSigningSecret();
|
||||
initializeRuntimeConfigConsumers(startupConfig);
|
||||
}
|
||||
|
||||
type MigrationSummary =
|
||||
| "skipped"
|
||||
@@ -728,6 +730,9 @@ async function startServerWithDatabaseTeardown(
|
||||
|
||||
await initializeCloudRuntimeConfig(db as any);
|
||||
let config = "deploymentMode" in startupConfig ? startupConfig : loadConfig();
|
||||
// Cloud storage depends on the canonical DB snapshot. Validate it before
|
||||
// signing keys or other configuration consumers can mutate local state.
|
||||
if (isCloudManagedInstance()) ensureDecisionSigningSecret();
|
||||
|
||||
// A claimed warm-pool stack may restart while its provider environment still
|
||||
// names the pool host. Restore the signed, durable identity before Better
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import { isCloudManagedInstance } from "./services/cloud-instance.js";
|
||||
import { parseManagedConfigEnv, type ManagedConfigEnv } from "./services/managed-config.js";
|
||||
import { isValidLocalEncryptedMasterKey } from "./secrets/local-encrypted-provider.js";
|
||||
import type { StorageProvider } from "@paperclipai/shared";
|
||||
|
||||
interface ManagedCloudPrerequisites {
|
||||
storageProvider: StorageProvider;
|
||||
/** The canonical DB/config value or explicit env/file value, before defaults. */
|
||||
storageS3Bucket: string | undefined;
|
||||
storageS3Region: string;
|
||||
storageS3Endpoint: string | undefined;
|
||||
databaseUrl: string | undefined;
|
||||
}
|
||||
|
||||
/** Pure startup policy for the existing managed-cloud profile. No I/O or probes. */
|
||||
export function validateManagedCloudPrerequisites(
|
||||
config: ManagedCloudPrerequisites,
|
||||
env: ManagedConfigEnv,
|
||||
): void {
|
||||
validateManagedCloudStoragePrerequisites(config, env);
|
||||
validateManagedCloudBootstrapPrerequisites(config, env);
|
||||
}
|
||||
|
||||
/** Validate a resolved candidate before its first Cloud import or startup use. */
|
||||
export function validateManagedCloudStoragePrerequisites(
|
||||
config: Pick<ManagedCloudPrerequisites, "storageProvider" | "storageS3Bucket" | "storageS3Region" | "storageS3Endpoint">,
|
||||
env: ManagedConfigEnv,
|
||||
): void {
|
||||
if (!isCloudManagedInstance(env)) return;
|
||||
parseManagedConfigEnv(env);
|
||||
|
||||
const refuse = (message: string): never => {
|
||||
throw new Error(`Managed cloud configuration: ${message}; refusing to start`);
|
||||
};
|
||||
// An invalid explicit env value must not fall through to a file/default value.
|
||||
if ((env.PAPERCLIP_STORAGE_PROVIDER !== undefined && env.PAPERCLIP_STORAGE_PROVIDER !== "s3")
|
||||
|| config.storageProvider !== "s3") {
|
||||
refuse("set PAPERCLIP_STORAGE_PROVIDER=s3 or storage.provider=s3 in the config file");
|
||||
}
|
||||
if (!config.storageS3Bucket?.trim()) {
|
||||
refuse("set PAPERCLIP_STORAGE_S3_BUCKET or storage.s3.bucket explicitly");
|
||||
}
|
||||
if (!config.storageS3Region.trim()) {
|
||||
refuse("set a nonblank PAPERCLIP_STORAGE_S3_REGION or storage.s3.region");
|
||||
}
|
||||
if (config.storageS3Endpoint !== undefined) {
|
||||
let validEndpoint = false;
|
||||
try {
|
||||
const url = new URL(config.storageS3Endpoint);
|
||||
validEndpoint = (url.protocol === "https:" || url.protocol === "http:")
|
||||
&& !!url.hostname && !url.username && !url.password;
|
||||
} catch { /* The diagnostic must not include a possibly secret URL. */ }
|
||||
if (!validEndpoint) refuse("PAPERCLIP_STORAGE_S3_ENDPOINT must be an HTTP(S) URL without credentials");
|
||||
}
|
||||
}
|
||||
|
||||
/** Environment-owned prerequisites needed before opening the Cloud database. */
|
||||
export function validateManagedCloudBootstrapPrerequisites(
|
||||
config: { databaseUrl: string | undefined }, env: ManagedConfigEnv,
|
||||
): void {
|
||||
if (!isCloudManagedInstance(env)) return;
|
||||
parseManagedConfigEnv(env);
|
||||
const refuse = (message: string): never => {
|
||||
throw new Error(`Managed cloud configuration: ${message}; refusing to start`);
|
||||
};
|
||||
let validDatabase = false;
|
||||
try {
|
||||
const url = new URL(config.databaseUrl ?? "");
|
||||
validDatabase = (url.protocol === "postgres:" || url.protocol === "postgresql:") && !!url.hostname;
|
||||
} catch { /* The diagnostic must not include database credentials. */ }
|
||||
if (!validDatabase) {
|
||||
refuse("set DATABASE_URL to an external PostgreSQL URL");
|
||||
}
|
||||
if (!isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY ?? "")) {
|
||||
refuse("set a stable PAPERCLIP_SECRETS_MASTER_KEY (32-byte base64, 64-char hex, or raw 32-char string); key files alone are not supported");
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,11 @@ function decodeMasterKey(raw: string): Buffer | null {
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Check the configured key without reading, creating, or changing key files. */
|
||||
export function isValidLocalEncryptedMasterKey(raw: string): boolean {
|
||||
return decodeMasterKey(raw) !== null;
|
||||
}
|
||||
|
||||
function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = process.env): Buffer {
|
||||
const envKeyRaw = env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
if (envKeyRaw && envKeyRaw.trim().length > 0) {
|
||||
|
||||
@@ -25,7 +25,7 @@ export const cloudRuntimeConfigSchema = z.object({
|
||||
backup: databaseBackupConfigSchema.strict(),
|
||||
storage: storageConfigSchema.extend({
|
||||
localDisk: storageLocalDiskConfigSchema.strict(),
|
||||
s3: storageS3ConfigSchema.extend({ endpoint: publicUrl.optional() }).strict(),
|
||||
s3: storageS3ConfigSchema.extend({ bucket: z.string().trim().min(1), endpoint: publicUrl.optional() }).strict(),
|
||||
}).strict(),
|
||||
secrets: secretsConfigSchema.extend({
|
||||
localEncrypted: secretsLocalEncryptedConfigSchema.strict(),
|
||||
|
||||
@@ -5,6 +5,8 @@ import { eq } from "drizzle-orm";
|
||||
import { instanceSettings, type Db } from "@paperclipai/db";
|
||||
import { paperclipConfigSchema } from "@paperclipai/shared";
|
||||
import { resolvePaperclipConfigPathForInstance } from "@paperclipai/shared/home-paths";
|
||||
import { validateManagedCloudStoragePrerequisites } from "../managed-cloud-prerequisites.js";
|
||||
import { loadStorageConfig } from "../storage/config.js";
|
||||
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
|
||||
import {
|
||||
parseStoredCloudRuntimeConfig,
|
||||
@@ -83,11 +85,18 @@ export async function readLegacyCloudRuntimeConfig(env: CloudInstanceEnv = proce
|
||||
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Legacy Cloud runtime configuration could not be read safely");
|
||||
return null;
|
||||
}
|
||||
// Refuse an unconfigured legacy bucket before shared-schema defaults can
|
||||
// turn its omission into an apparently explicit canonical database value.
|
||||
const legacyStorage = (raw as { storage?: { provider?: unknown; s3?: { bucket?: unknown } } } | null)?.storage;
|
||||
if (typeof legacyStorage?.s3?.bucket !== "string" || !legacyStorage.s3.bucket.trim()) {
|
||||
throw new Error("Legacy Cloud runtime configuration requires an explicit nonblank storage.s3.bucket");
|
||||
}
|
||||
const parsed = paperclipConfigSchema.safeParse(raw);
|
||||
if (!parsed.success) throw new Error("Legacy Cloud runtime configuration is invalid");
|
||||
if (parsed.data.llm?.provider === "openai" && parsed.data.llm.apiKey?.trim() && !env.OPENAI_API_KEY?.trim()) {
|
||||
throw new Error("Move the legacy OpenAI API key to OPENAI_API_KEY before initializing Cloud runtime configuration");
|
||||
}
|
||||
validateManagedCloudStoragePrerequisites(loadStorageConfig(parsed.data, env), env);
|
||||
return projectCloudRuntimeConfig(parsed.data);
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user