Merge remote-tracking branch 'origin/master' into codex/sleep-05-merge-confirmations

* origin/master:
  fix(config): require shared storage for managed cloud (#18350)
This commit is contained in:
Dotta committed 2026-10-30 16:49:20 -05:00
commit 6e8fd558bc
14 files changed
+519 -10

No files matched your search

+39
View File
@@ -1511,6 +1511,45 @@ Configure storage provider/settings:
pnpm paperclipai configure --section storage
```
### Managed cloud prerequisites
A nonblank Cloud tenant token or the presence of `PAPERCLIP_MANAGED_CONFIG`
selects the managed-cloud profile. A supplied blank or invalid managed feature
document refuses startup. Without either signal, the instance remains self-hosted.
The managed profile also requires:
- `PAPERCLIP_STORAGE_PROVIDER=s3` or `storage.provider: "s3"` in the canonical
database snapshot. An invalid explicit environment value refuses startup even
if the snapshot selects S3.
- An explicit nonblank `PAPERCLIP_STORAGE_S3_BUCKET` or `storage.s3.bucket`.
The self-hosted `paperclip` bucket default does not qualify. The region defaults
to `us-east-1`; an explicit blank region refuses startup. An optional endpoint
must be an HTTP(S) URL without embedded credentials.
- An external PostgreSQL URL from the environment `DATABASE_URL`.
The managed profile never falls back to embedded PostgreSQL.
- A stable `PAPERCLIP_SECRETS_MASTER_KEY` supplied by the deployment secret store.
Use the same value across replacements. The existing provider accepts 32-byte
base64, 64-character hex, or a raw 32-byte UTF-8 string. A key file alone does
not qualify because a replacement web server may not have that file.
Bootstrap validates the environment database URL and encryption master key before
opening the database or creating local state. Effective storage validation follows
canonical snapshot resolution, before signing keys, first-party Telemetry or app
construction. Raw canonical rows and legacy S3 imports cannot obtain a missing
bucket from schema defaults. Already-stored nonblank bucket values remain
canonical, including the deliberate name `paperclip`; prior normalization cannot
reliably establish their original provenance.
It does not contact S3 or verify credentials, bucket access, database connectivity,
or that a key matches existing encrypted data. Operators must qualify ordinary
S3 put/read/delete with deployment credentials and preserve the existing key.
AWS credentials continue to use the provider's normal SDK credential chain.
Local and self-hosted storage and key-file defaults remain unchanged.
These are prerequisites for shared storage and secrets. They do not establish
that the whole web server is disposable or that all execution and recovery paths
survive web-server replacement. Run-log/trace storage and producer retention are
separate changes.
## Agent Artifact Uploads
When an agent generates a file that a board user or reviewer should inspect as
+26 -3
View File
@@ -18,7 +18,8 @@ existing validation at application startup.
to embedded PostgreSQL. Credentials, trust keys, provider configuration and
deployment wiring needed before the database connection remain supplied by
the environment or the existing secret system. This phase parses only the
database connection record. Runtime mode, host binding and authentication
database connection record and requires the stable environment encryption
master key. Runtime mode, host binding and authentication
checks wait until the saved snapshot has loaded.
2. After schema migration and database connection, startup reads the private
runtime configuration row. An existing invalid row or a database error stops
@@ -38,6 +39,27 @@ existing validation at application startup.
authentication and application services. A saved telemetry opt-out therefore
takes effect before a telemetry client can be created.
Managed Cloud always requires a stable environment master key, including when
an external default secrets provider is selected. Internal local-encrypted
credential paths still need that key.
Managed Cloud validates effective S3 storage after publishing the canonical
snapshot, before signing-key initialization, telemetry, authentication or app
construction. Cloud requires S3 and a nonblank bucket; an authoritative absent
configuration needs explicit environment storage settings. Raw saved snapshots
cannot obtain a bucket from a schema default. Every legacy Cloud file must name its
bucket before its one-time import, even if its saved provider is local disk and
an environment bucket is present. Omission or blank values fail before seeding;
The effective storage candidate is validated before seeding, so a rejected
legacy file leaves no canonical row and can be corrected before retry.
Environment overlays are not saved as operator choices. Correct that file before
import, or use the existing absent-file/null-snapshot environment-only setup.
Ordinary self-hosted file defaults remain unchanged.
Already-canonical nonblank values, including `paperclip`, remain authoritative.
Prior normalization cannot reliably distinguish an operator's deliberate name
from a historical default, so this release does not invent retrospective
provenance or replace those saved values.
Existing operator environment overlays keep precedence over saved values.
Reading an overlaid value does not save it back to the database. Credentials and
connection URLs are never copied into this JSONB row.
@@ -78,8 +100,9 @@ bytes. Restart tests prove configuration restoration under a different home;
they do not prove durable file, secret, or backup storage.
The decision signing service still uses `PAPERCLIP_DECISION_SIGNING_SECRET` when
supplied and otherwise reads or creates its instance key file before the database
connection. The local encrypted secret provider still uses its environment key
supplied and otherwise reads or creates its instance key file. Managed Cloud
performs that step after effective storage validation; self-hosted startup keeps
its pre-connection ordering. The local encrypted secret provider still uses its environment key
or existing key-file mechanism. Those byte-storage fallbacks are outside this
configuration migration. Qualification must account for them separately; this
PR does not establish file-free secret storage on Cloud.
@@ -55,6 +55,7 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura
const config = paperclipConfigSchema.parse({
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
database: { mode: "postgres" }, logging: { mode: "cloud" },
storage: { provider: "s3", s3: { bucket: "migration-bucket" } },
server: { port: 3199 }, telemetry: { enabled: false },
});
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
@@ -120,6 +121,18 @@ const keys = ["adapter-settings:v1", "cloud-runtime-config:v1", "cloud-configura
} finally { await manager.stopAll(); }
});
it("does not seed rejected storage and imports the corrected legacy file on retry", async () => {
const config = await legacy();
config.storage.provider = "local_disk";
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
await expect(migration().migrate()).rejects.toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
expect(await peer.select().from(instanceSettings).where(inArray(instanceSettings.singletonKey, keys))).toEqual([]);
config.storage.provider = "s3";
await fs.writeFile(path.join(homeA, "config.json"), JSON.stringify(config));
expect((await migration(peer).migrate()).migration.config.authority).toBe("legacy_file");
expect((await cloudRuntimeConfigService(peer).inspectStored())?.config?.storage.provider).toBe("s3");
});
it("serializes concurrent attempts and never imports stale inputs over later DB changes", async () => {
await legacy();
const [first, second] = await Promise.all([migration().migrate(), migration(peer).migrate()]);
@@ -66,6 +66,9 @@ beforeEach(() => {
vi.stubEnv("PAPERCLIP_ALLOWED_HOSTNAMES", undefined);
vi.stubEnv("PAPERCLIP_TAILNET_BIND_HOST", "100.100.0.1");
vi.stubEnv("OPENAI_API_KEY", "environment-api-secret");
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined);
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
setCloudRuntimeConfigSnapshot(null);
});
@@ -170,6 +173,10 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => {
expect(fs.existsSync(configPath)).toBe(false);
vi.stubEnv("PAPERCLIP_DEPLOYMENT_MODE", "authenticated");
vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://environment.example.test");
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined);
vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "test-token");
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "environment-bucket");
expect(loadConfig()).toMatchObject({ databaseMode: "postgres", authBaseUrlMode: "explicit", secretsStrictMode: true, telemetryEnabled: true });
});
@@ -284,6 +291,26 @@ describePostgres("Cloud runtime configuration in PostgreSQL", () => {
expect(await second.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "runtime-config-test-receipt"))).toEqual([]);
});
it.each([undefined, "", " "])("rejects raw canonical S3 rows with missing or blank bucket %j", async bucket => {
const config = structuredClone(projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig())));
const raw = config;
if (bucket === undefined) Reflect.deleteProperty(raw.storage.s3, "bucket");
else raw.storage.s3.bucket = bucket;
await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 1, revision: 0, config: raw } });
await expect(initializeCloudRuntimeConfig(second)).rejects.toThrow(/Persisted Cloud runtime configuration is invalid/);
expect(() => getCloudRuntimeConfigSnapshot()).toThrow(/not been initialized/);
});
it.each(["database-bucket", "paperclip"])("accepts canonical DB bucket %s without local configuration provenance", async bucket => {
const config = projectCloudRuntimeConfig(paperclipConfigSchema.parse(legacyConfig()));
config.storage.s3.bucket = bucket;
await cloudRuntimeConfigService(first).seedIfAbsent(config);
fs.writeFileSync(configPath, "invalid local file must not be consulted");
await initializeCloudRuntimeConfig(second);
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: bucket });
expect(readConfigFile({ includeProvenance: true })).toMatchObject({ explicitStorageS3Bucket: true });
});
it("fails closed on invalid database settings without consulting a valid legacy file", async () => {
fs.writeFileSync(configPath, JSON.stringify(legacyConfig()));
await first.insert(instanceSettings).values({ singletonKey: CLOUD_RUNTIME_CONFIG_KEY, general: { version: 99, config: { secret: "hidden" } } });
+14
View File
@@ -91,6 +91,20 @@ describe("readConfigFile", () => {
});
});
it("preserves explicit bucket provenance without changing defaults for existing readers", () => {
for (const bucket of [undefined, "paperclip"]) {
writeConfig(configPath, {
...minimalConfig() as object,
storage: { provider: "s3", s3: bucket === undefined ? {} : { bucket } },
});
expect(readConfigFile()?.storage.s3.bucket).toBe("paperclip");
expect(readConfigFile({ includeProvenance: true })).toMatchObject({
config: { storage: { s3: { bucket: "paperclip" } } },
explicitStorageS3Bucket: bucket !== undefined,
});
}
});
it("warns about likely misspellings without stripping them", () => {
const config = {
...(minimalConfig() as Record<string, unknown>),
@@ -0,0 +1,201 @@
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { createStorageProviderFromConfig } from "../storage/provider-registry.js";
import { createLocalEncryptedCipher } from "../secrets/local-encrypted-provider.js";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { loadConfig } from "../config.js";
import { readConfigFile } from "../config-file.js";
import { readLegacyCloudRuntimeConfig } from "../services/cloud-runtime-config.js";
import { setCloudRuntimeConfigSnapshot } from "../services/cloud-runtime-config-snapshot.js";
const managed = JSON.stringify({
v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] },
});
let configDir: string;
beforeEach(() => {
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null });
configDir = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-config-"));
vi.stubEnv("PAPERCLIP_CONFIG", path.join(configDir, "config.json"));
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed);
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", "test-bucket");
vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", undefined);
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", undefined);
vi.stubEnv("DATABASE_URL", "postgresql://user:private@db.example.test/paperclip");
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
});
afterEach(() => {
setCloudRuntimeConfigSnapshot(null);
vi.unstubAllEnvs();
rmSync(configDir, { recursive: true, force: true });
});
function writeConfig(storage: Record<string, unknown>) {
writeFileSync(path.join(configDir, "config.json"), JSON.stringify({
$meta: { version: 1, updatedAt: "2026-10-30T00:00:00.000Z", source: "configure" },
logging: { mode: "file" },
server: {},
storage,
database: { mode: "postgres", connectionString: "postgres://db.example.test/paperclip" },
}));
}
describe("managed cloud configuration boundary", () => {
it("accepts configured S3, external PostgreSQL, and the existing master-key formats", () => {
for (const key of ["12345678901234567890123456789012", Buffer.alloc(32, 7).toString("hex"), Buffer.alloc(32, 7).toString("base64")]) {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", key);
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "test-bucket", storageS3Region: "us-east-1" });
}
});
it.each([undefined, "", "local_disk", "typo"])("rejects storage provider %j without silent fallback", (value) => {
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", value);
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
});
it.each([undefined, "", " "])("requires an explicit bucket (%j)", (value) => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", value);
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/);
});
it.each(["", " "])("rejects blank region %j", (value) => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", value);
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_REGION/);
});
it.each(["", "ftp://store.test", "https://user:private@store.test"])("rejects invalid endpoint without exposing it", (value) => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", value);
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_S3_ENDPOINT/);
try { loadConfig(); } catch (error) { expect(String(error)).not.toContain("private"); }
});
it.each([undefined, "", "sqlite://local/data", "postgresql://"])("requires an external database URL (%j)", (value) => {
vi.stubEnv("DATABASE_URL", value);
expect(() => loadConfig()).toThrow(/external PostgreSQL URL/);
});
it.each([undefined, "", " ", "short"])("rejects missing or invalid env key (%j), even with a configured key file", (value) => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", value);
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", "/tmp/unused-master-key");
expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
});
it("requires the Cloud bootstrap key even when an external secret provider is selected", () => {
vi.stubEnv("PAPERCLIP_SECRETS_PROVIDER", "aws_secrets_manager");
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
expect(() => loadConfig()).toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
expect(loadConfig().secretsProvider).toBe("aws_secrets_manager");
});
it.each(["", "{bad", JSON.stringify({ v: 1, mode: "cloud" })])("rejects malformed managed config before startup (%j)", (value) => {
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", value);
expect(() => loadConfig()).toThrow(/PAPERCLIP_MANAGED_CONFIG/);
});
it("accepts imported explicit file storage with an environment-owned database", async () => {
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", undefined);
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
writeConfig({ provider: "s3", s3: { bucket: "file-bucket", region: "eu-west-1" } });
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() });
expect(loadConfig()).toMatchObject({ storageProvider: "s3", storageS3Bucket: "file-bucket", storageS3Region: "eu-west-1" });
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo");
expect(() => loadConfig()).toThrow(/PAPERCLIP_STORAGE_PROVIDER=s3/);
});
it.each([{ provider: "s3" }, { provider: "s3", s3: { region: "us-east-1" } }])("rejects a real config file with an omitted bucket before import (%j)", async storage => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
writeConfig(storage);
// Self-hosted readers still expose their default and raw-file provenance.
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined);
expect(readConfigFile({ includeProvenance: true })).toMatchObject({ config: { storage: { s3: { bucket: "paperclip" } } }, explicitStorageS3Bucket: false });
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", managed);
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
});
it.each(["", " "])("rejects an explicitly blank bucket in a real file (%j)", async bucket => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
writeConfig({ provider: "s3", s3: { bucket } });
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
});
it.each([undefined, "", " "].flatMap(bucket => [undefined, "environment-bucket"].map(environmentBucket => ({ bucket, environmentBucket }))))("rejects a legacy local-disk file with no explicit bucket before an S3 environment overlay (%j)", async ({ bucket, environmentBucket }) => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", environmentBucket);
writeConfig({ provider: "local_disk", s3: { bucket } });
await expect(readLegacyCloudRuntimeConfig()).rejects.toThrow(/storage.s3.bucket/);
});
it("accepts an explicitly supplied default bucket name in a real file", async () => {
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
writeConfig({ provider: "s3", s3: { bucket: "paperclip" } });
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: await readLegacyCloudRuntimeConfig() });
expect(loadConfig().storageS3Bucket).toBe("paperclip");
});
it("accepts an environment bucket with an authoritative absent-file snapshot", () => {
writeConfig({ provider: "s3" });
expect(loadConfig().storageS3Bucket).toBe("test-bucket");
});
it("preserves ordinary self-hosted defaults, including key-file use", () => {
for (const name of ["PAPERCLIP_MANAGED_CONFIG", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_STORAGE_S3_BUCKET", "DATABASE_URL", "PAPERCLIP_SECRETS_MASTER_KEY"]) vi.stubEnv(name, undefined);
expect(loadConfig()).toMatchObject({ storageProvider: "local_disk", storageS3Bucket: "paperclip", databaseUrl: undefined, databaseMode: "embedded-postgres" });
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "typo");
expect(loadConfig().storageProvider).toBe("local_disk");
});
});
// Real provider + AWS SDK over loopback. The remote S3 service and credentials
// are simulated; this does not establish deployment bucket permissions.
it("uses the validated config for normal S3 put/read/delete and stable env-key encryption", async () => {
const objects = new Map<string, Buffer>();
const operations: string[] = [];
const server = createServer(async (req, res) => {
const key = new URL(req.url!, "http://localhost").pathname;
operations.push(`${req.method} ${key}`);
if (req.method === "PUT") {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(Buffer.from(chunk));
objects.set(key, Buffer.concat(chunks));
res.end();
} else if (req.method === "GET" && objects.has(key)) {
const body = objects.get(key)!;
res.writeHead(200, { "content-type": "text/plain", "content-length": body.length });
res.end(body);
} else if (req.method === "DELETE") {
objects.delete(key);
res.writeHead(204).end();
} else {
res.writeHead(404).end();
}
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture-access-key");
vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture-secret-key");
vi.stubEnv("AWS_SESSION_TOKEN", undefined);
vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", `http://127.0.0.1:${(server.address() as AddressInfo).port}`);
vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "disposable-qualification");
try {
const config = loadConfig();
const first = createStorageProviderFromConfig(config);
await first.putObject({ objectKey: "company/object.txt", body: Buffer.from("retained"), contentType: "text/plain", contentLength: 8 });
const replacement = createStorageProviderFromConfig(loadConfig());
const result = await replacement.getObject({ objectKey: "company/object.txt" });
const chunks: Buffer[] = [];
for await (const chunk of result.stream) chunks.push(Buffer.from(chunk));
expect(Buffer.concat(chunks).toString()).toBe("retained");
await replacement.deleteObject({ objectKey: "company/object.txt" });
expect(objects.size).toBe(0);
expect(operations).toEqual(["PUT", "GET", "DELETE"].map(method => `${method} /test-bucket/disposable-qualification/company/object.txt`));
const encrypted = createLocalEncryptedCipher("write").encrypt("saved credential");
expect(createLocalEncryptedCipher("read").decrypt(encrypted)).toBe("saved credential");
} finally {
server.closeAllConnections();
await new Promise<void>((resolve) => server.close(() => resolve()));
}
});
@@ -616,6 +616,70 @@ describe("startServer feedback export wiring", () => {
process.env.BETTER_AUTH_SECRET = "test-secret";
});
it("refuses invalid cloud bootstrap prerequisites before signing, DB creation, or HTTP setup", async () => {
const realConfig = await vi.importActual<typeof import("../config.js")>("../config.js");
const originalHome = process.env.PAPERCLIP_HOME;
const originalManaged = process.env.PAPERCLIP_MANAGED_CONFIG;
const originalStorage = process.env.PAPERCLIP_STORAGE_PROVIDER;
const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-refusal-"));
process.env.PAPERCLIP_HOME = tempHome;
process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({
v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] },
});
process.env.PAPERCLIP_STORAGE_PROVIDER = "s3";
delete process.env.PAPERCLIP_DECISION_SIGNING_SECRET;
cloudRuntimeStartupMocks.bootstrap.mockImplementationOnce(realConfig.loadCloudBootstrapConfig);
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl);
try {
await expect(startServer()).rejects.toThrow(/stable PAPERCLIP_SECRETS_MASTER_KEY/);
expect(createDbMock).not.toHaveBeenCalled();
expect(createAppMock).not.toHaveBeenCalled();
expect(existsSync(path.join(tempHome, "instances"))).toBe(false);
} finally {
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = originalHome;
if (originalManaged === undefined) delete process.env.PAPERCLIP_MANAGED_CONFIG;
else process.env.PAPERCLIP_MANAGED_CONFIG = originalManaged;
if (originalStorage === undefined) delete process.env.PAPERCLIP_STORAGE_PROVIDER;
else process.env.PAPERCLIP_STORAGE_PROVIDER = originalStorage;
vi.unstubAllEnvs();
rmSync(tempHome, { recursive: true, force: true });
}
});
it("rejects missing effective Cloud storage after the snapshot and before local signing mutations", async () => {
const realConfig = await vi.importActual<typeof import("../config.js")>("../config.js");
const { setCloudRuntimeConfigSnapshot } = await import("../services/cloud-runtime-config-snapshot.js");
const tempHome = mkdtempSync(path.join(tmpdir(), "paperclip-cloud-storage-refusal-"));
vi.stubEnv("PAPERCLIP_HOME", tempHome);
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", JSON.stringify({ v: 1, mode: "cloud", catalogVersion: "2026.720.0", features: {}, plugins: { autoInstall: [] } }));
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "12345678901234567890123456789012");
vi.stubEnv("DATABASE_URL", buildTestConfig().databaseUrl);
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", undefined);
vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", undefined);
cloudRuntimeStartupMocks.initialize.mockImplementationOnce(async () => {
setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null });
return null;
});
loadConfigMock.mockImplementationOnce(realConfig.loadConfig);
try {
await expect(startServer()).rejects.toThrow(/PAPERCLIP_STORAGE_S3_BUCKET/);
expect(createDbMock).toHaveBeenCalled();
expect(cloudRuntimeStartupMocks.initialize).toHaveBeenCalledOnce();
expect(cloudRuntimeStartupMocks.telemetry).not.toHaveBeenCalled();
expect(createAppMock).not.toHaveBeenCalled();
expect(existsSync(path.join(tempHome, "instances"))).toBe(false);
} finally {
setCloudRuntimeConfigSnapshot(null);
vi.unstubAllEnvs();
rmSync(tempHome, { recursive: true, force: true });
}
});
it("awaits saved Cloud configuration before telemetry, authentication and app construction", async () => {
vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "startup-test-token");
vi.stubEnv("HOST", "0.0.0.0");
@@ -674,6 +738,7 @@ describe("startServer feedback export wiring", () => {
} finally { vi.unstubAllEnvs(); }
});
it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
@@ -1177,7 +1242,8 @@ describe("startServer feedback export wiring", () => {
it("refuses startup when an explicit decision signing secret is too short", async () => {
process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "too-short";
await expect(startServer()).rejects.toThrow("PAPERCLIP_DECISION_SIGNING_SECRET must be at least 32 characters");
expect(loadConfigMock).not.toHaveBeenCalled();
expect(createDbMock).not.toHaveBeenCalled();
expect(createAppMock).not.toHaveBeenCalled();
});
it("passes the feedback export service into createApp so pending traces flush in runtime", async () => {
+22 -2
View File
@@ -20,8 +20,22 @@ function formatConfigValidationError(error: ZodError): string {
.join("; ");
}
export function readConfigFile(): PaperclipConfig | null {
if (isCloudManagedInstance()) return toPaperclipConfig(getCloudRuntimeConfigSnapshot().config);
interface ConfigFileWithProvenance {
config: PaperclipConfig;
explicitStorageS3Bucket: boolean;
}
export function readConfigFile(options: { includeProvenance: true }): ConfigFileWithProvenance | null;
export function readConfigFile(): PaperclipConfig | null;
export function readConfigFile(options?: { includeProvenance: true }): PaperclipConfig | ConfigFileWithProvenance | null {
if (isCloudManagedInstance()) {
const saved = getCloudRuntimeConfigSnapshot().config;
const config = toPaperclipConfig(saved);
// A stored snapshot is canonical configuration, including its bucket.
// It must not inherit local-file provenance or inspect an old home.
return config && options?.includeProvenance ? { config, explicitStorageS3Bucket: true } : config;
}
const configPath = resolvePaperclipConfigPath();
if (!fs.existsSync(configPath)) return null;
@@ -41,6 +55,12 @@ export function readConfigFile(): PaperclipConfig | null {
`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`,
);
}
if (options?.includeProvenance) {
// Schema validation above establishes the input shape, but fills omitted
// buckets with a default. Keep only that provenance at the file boundary.
const input = raw as Partial<PaperclipConfig>;
return { config, explicitStorageS3Bucket: input.storage?.s3?.bucket !== undefined };
}
return config;
} catch (error) {
if (error instanceof ZodError) {
+11 -1
View File
@@ -1,3 +1,4 @@
import { validateManagedCloudBootstrapPrerequisites, validateManagedCloudPrerequisites } from "./managed-cloud-prerequisites.js";
import { readConfigFile, resolveDeploymentMode } from "./config-file.js";
import { isCloudManagedInstance } from "./services/cloud-instance.js";
import { StartupRefusalError } from "./startup-refusals.js";
@@ -143,7 +144,10 @@ function detectTailnetBindHost(): string | undefined {
}
}
export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()): Config {
export function loadConfig(override?: PaperclipConfig | null): Config {
const configFile = override === undefined ? readConfigFile({ includeProvenance: true })
: override ? { config: override, explicitStorageS3Bucket: true } : null;
const fileConfig = configFile?.config ?? null;
const fileDatabaseMode =
(isCloudManagedInstance() || fileConfig?.database.mode === "postgres" ? "postgres" : "embedded-postgres") as DatabaseMode;
@@ -163,6 +167,11 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()
const secretsProvider: SecretProvider = providerFromEnv ?? providerFromFile ?? "local_encrypted";
const storage = loadStorageConfig(fileConfig);
validateManagedCloudPrerequisites({ ...storage,
storageS3Bucket: process.env.PAPERCLIP_STORAGE_S3_BUCKET
?? (configFile?.explicitStorageS3Bucket ? fileConfig?.storage.s3.bucket : undefined),
databaseUrl: process.env.DATABASE_URL ?? fileDbUrl,
}, process.env);
const feedbackExportBackendUrl =
process.env.PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL?.trim() ||
process.env.PAPERCLIP_TELEMETRY_BACKEND_URL?.trim() ||
@@ -448,5 +457,6 @@ export function loadCloudBootstrapConfig(): CloudBootstrapConfig {
try { parsed = new URL(value); } catch { throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`); }
if (!["postgres:", "postgresql:"].includes(parsed.protocol)) throw new Error(`Cloud bootstrap ${name} must use PostgreSQL`);
}
validateManagedCloudBootstrapPrerequisites({ databaseUrl: url }, process.env);
return { databaseMode: "postgres", databaseUrl: url, databaseMigrationUrl: migrationUrl };
}
+7 -2
View File
@@ -297,9 +297,11 @@ async function startServerWithDatabaseTeardown(
// Error monitoring must be ready before the first request can fail — see
// sentry.ts.
await sentryReady;
ensureDecisionSigningSecret();
const startupConfig = isCloudManagedInstance() ? loadCloudBootstrapConfig() : loadConfig();
if ("deploymentMode" in startupConfig) initializeRuntimeConfigConsumers(startupConfig);
if ("deploymentMode" in startupConfig) {
ensureDecisionSigningSecret();
initializeRuntimeConfigConsumers(startupConfig);
}
type MigrationSummary =
| "skipped"
@@ -729,6 +731,9 @@ async function startServerWithDatabaseTeardown(
await initializeCloudRuntimeConfig(db as any);
let config = "deploymentMode" in startupConfig ? startupConfig : loadConfig();
// Cloud storage depends on the canonical DB snapshot. Validate it before
// signing keys or other configuration consumers can mutate local state.
if (isCloudManagedInstance()) ensureDecisionSigningSecret();
// A claimed warm-pool stack may restart while its provider environment still
// names the pool host. Restore the signed, durable identity before Better
+77
View File
@@ -0,0 +1,77 @@
import { isCloudManagedInstance } from "./services/cloud-instance.js";
import { parseManagedConfigEnv, type ManagedConfigEnv } from "./services/managed-config.js";
import { isValidLocalEncryptedMasterKey } from "./secrets/local-encrypted-provider.js";
import type { StorageProvider } from "@paperclipai/shared";
interface ManagedCloudPrerequisites {
storageProvider: StorageProvider;
/** The canonical DB/config value or explicit env/file value, before defaults. */
storageS3Bucket: string | undefined;
storageS3Region: string;
storageS3Endpoint: string | undefined;
databaseUrl: string | undefined;
}
/** Pure startup policy for the existing managed-cloud profile. No I/O or probes. */
export function validateManagedCloudPrerequisites(
config: ManagedCloudPrerequisites,
env: ManagedConfigEnv,
): void {
validateManagedCloudStoragePrerequisites(config, env);
validateManagedCloudBootstrapPrerequisites(config, env);
}
/** Validate a resolved candidate before its first Cloud import or startup use. */
export function validateManagedCloudStoragePrerequisites(
config: Pick<ManagedCloudPrerequisites, "storageProvider" | "storageS3Bucket" | "storageS3Region" | "storageS3Endpoint">,
env: ManagedConfigEnv,
): void {
if (!isCloudManagedInstance(env)) return;
parseManagedConfigEnv(env);
const refuse = (message: string): never => {
throw new Error(`Managed cloud configuration: ${message}; refusing to start`);
};
// An invalid explicit env value must not fall through to a file/default value.
if ((env.PAPERCLIP_STORAGE_PROVIDER !== undefined && env.PAPERCLIP_STORAGE_PROVIDER !== "s3")
|| config.storageProvider !== "s3") {
refuse("set PAPERCLIP_STORAGE_PROVIDER=s3 or storage.provider=s3 in the config file");
}
if (!config.storageS3Bucket?.trim()) {
refuse("set PAPERCLIP_STORAGE_S3_BUCKET or storage.s3.bucket explicitly");
}
if (!config.storageS3Region.trim()) {
refuse("set a nonblank PAPERCLIP_STORAGE_S3_REGION or storage.s3.region");
}
if (config.storageS3Endpoint !== undefined) {
let validEndpoint = false;
try {
const url = new URL(config.storageS3Endpoint);
validEndpoint = (url.protocol === "https:" || url.protocol === "http:")
&& !!url.hostname && !url.username && !url.password;
} catch { /* The diagnostic must not include a possibly secret URL. */ }
if (!validEndpoint) refuse("PAPERCLIP_STORAGE_S3_ENDPOINT must be an HTTP(S) URL without credentials");
}
}
/** Environment-owned prerequisites needed before opening the Cloud database. */
export function validateManagedCloudBootstrapPrerequisites(
config: { databaseUrl: string | undefined }, env: ManagedConfigEnv,
): void {
if (!isCloudManagedInstance(env)) return;
parseManagedConfigEnv(env);
const refuse = (message: string): never => {
throw new Error(`Managed cloud configuration: ${message}; refusing to start`);
};
let validDatabase = false;
try {
const url = new URL(config.databaseUrl ?? "");
validDatabase = (url.protocol === "postgres:" || url.protocol === "postgresql:") && !!url.hostname;
} catch { /* The diagnostic must not include database credentials. */ }
if (!validDatabase) {
refuse("set DATABASE_URL to an external PostgreSQL URL");
}
if (!isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY ?? "")) {
refuse("set a stable PAPERCLIP_SECRETS_MASTER_KEY (32-byte base64, 64-char hex, or raw 32-char string); key files alone are not supported");
}
}
@@ -50,6 +50,11 @@ function decodeMasterKey(raw: string): Buffer | null {
return null;
}
/** Check the configured key without reading, creating, or changing key files. */
export function isValidLocalEncryptedMasterKey(raw: string): boolean {
return decodeMasterKey(raw) !== null;
}
function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = process.env): Buffer {
const envKeyRaw = env.PAPERCLIP_SECRETS_MASTER_KEY;
if (envKeyRaw && envKeyRaw.trim().length > 0) {
@@ -25,7 +25,7 @@ export const cloudRuntimeConfigSchema = z.object({
backup: databaseBackupConfigSchema.strict(),
storage: storageConfigSchema.extend({
localDisk: storageLocalDiskConfigSchema.strict(),
s3: storageS3ConfigSchema.extend({ endpoint: publicUrl.optional() }).strict(),
s3: storageS3ConfigSchema.extend({ bucket: z.string().trim().min(1), endpoint: publicUrl.optional() }).strict(),
}).strict(),
secrets: secretsConfigSchema.extend({
localEncrypted: secretsLocalEncryptedConfigSchema.strict(),
@@ -5,6 +5,8 @@ import { eq } from "drizzle-orm";
import { instanceSettings, type Db } from "@paperclipai/db";
import { paperclipConfigSchema } from "@paperclipai/shared";
import { resolvePaperclipConfigPathForInstance } from "@paperclipai/shared/home-paths";
import { validateManagedCloudStoragePrerequisites } from "../managed-cloud-prerequisites.js";
import { loadStorageConfig } from "../storage/config.js";
import { isCloudManagedInstance, type CloudInstanceEnv } from "./cloud-instance.js";
import {
parseStoredCloudRuntimeConfig,
@@ -83,11 +85,18 @@ export async function readLegacyCloudRuntimeConfig(env: CloudInstanceEnv = proce
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Legacy Cloud runtime configuration could not be read safely");
return null;
}
// Refuse an unconfigured legacy bucket before shared-schema defaults can
// turn its omission into an apparently explicit canonical database value.
const legacyStorage = (raw as { storage?: { provider?: unknown; s3?: { bucket?: unknown } } } | null)?.storage;
if (typeof legacyStorage?.s3?.bucket !== "string" || !legacyStorage.s3.bucket.trim()) {
throw new Error("Legacy Cloud runtime configuration requires an explicit nonblank storage.s3.bucket");
}
const parsed = paperclipConfigSchema.safeParse(raw);
if (!parsed.success) throw new Error("Legacy Cloud runtime configuration is invalid");
if (parsed.data.llm?.provider === "openai" && parsed.data.llm.apiKey?.trim() && !env.OPENAI_API_KEY?.trim()) {
throw new Error("Move the legacy OpenAI API key to OPENAI_API_KEY before initializing Cloud runtime configuration");
}
validateManagedCloudStoragePrerequisites(loadStorageConfig(parsed.data, env), env);
return projectCloudRuntimeConfig(parsed.data);
}