mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 10:26:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The CLI and server share a JSON configuration contract for local installations and worktrees. > - Existing config writes removed extension keys because Zod stripped unknown object properties. > - Invalid config files could also be replaced with defaults before an operator preserved the original bytes. > - Configuration updates must preserve operator edits and must not rewrite files when the effective value is unchanged. > - This pull request adds extension-preserving merges, guarded invalid-config repair, atomic writes, and focused regression tests. > - The benefit is safe setup and configuration reruns without data loss or unnecessary mtime changes. ## Linked Issues or Issue Description **What happened?** Known-field updates through the CLI or server removed unknown top-level and nested config keys. Non-interactive configure and onboard paths could replace a present but invalid config with defaults. **Expected behavior** Writers preserve extension keys, skip semantic no-op writes, and require explicit interactive confirmation before an invalid config is replaced. Repair preserves an exact collision-safe backup first. **Steps to reproduce** 1. Add an unknown top-level key and an unknown nested provider key to `config.json`. 2. Update a known field through the CLI or worktree config writer. 3. Observe that the extension keys are removed on the base branch. 4. Write invalid JSON and run configure or onboard without an interactive terminal. 5. Observe that the original file can be replaced without a durable invalid-file backup on the base branch. **Paperclip version or commit** `master` at the pull request base commit. ## What Changed - Accept unknown properties at each extensible config object boundary while keeping every known field validated. - Merge known-field updates into the parsed source config and preserve only unknown extension data. - Warn about near-match key names without deleting or changing them. - Skip writes when the effective config is unchanged, which keeps file mtimes stable. - Write config changes through a temporary file, file sync, rename, and directory sync. - Distinguish a missing config from an invalid config in configure and onboard. - Back up invalid bytes as `config.json.invalid-N` and verify the source still matches that backup before repair. - Require interactive repair confirmation and reject non-interactive replacement with an actionable message. - Document the config preservation and repair behavior. ## Verification - `pnpm exec vitest run packages/shared/src/config-schema.test.ts cli/src/__tests__/config-store.test.ts cli/src/__tests__/configure-repair.test.ts cli/src/__tests__/configure.test.ts cli/src/__tests__/onboard.test.ts server/src/__tests__/config-file.test.ts server/src/__tests__/worktree-config.test.ts` - `pnpm -r typecheck` - `AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= VITEST_MAX_WORKERS=1 pnpm test:run` - `pnpm build` - Confirm all pull request checks are green on the latest commit. - Confirm Greptile reports 5/5 with no unresolved comments. ## Risks - Passthrough keeps misspelled keys. Near-match warnings make this visible without destructive cleanup. - Merge behavior must distinguish unknown extension keys from optional known keys. Schema-aware regression tests cover preservation and known-key deletion. - Repair must not overwrite bytes that changed after backup. The writer compares the current source with the selected backup before atomic replacement. - The change does not alter database schema, company scoping, or activity logging. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 model family. The exact deployment model ID and context window are not exposed. Agentic reasoning, tool use, and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
354 lines
12 KiB
TypeScript
354 lines
12 KiB
TypeScript
import { z } from "zod";
|
|
import {
|
|
AUTH_BASE_URL_MODES,
|
|
BIND_MODES,
|
|
DEPLOYMENT_EXPOSURES,
|
|
DEPLOYMENT_MODES,
|
|
SECRET_PROVIDERS,
|
|
STORAGE_PROVIDERS,
|
|
} from "./constants.js";
|
|
import { validateConfiguredBindMode } from "./network-bind.js";
|
|
|
|
export const configMetaSchema = z.object({
|
|
version: z.literal(1),
|
|
updatedAt: z.string(),
|
|
source: z.enum(["onboard", "configure", "doctor"]),
|
|
}).passthrough();
|
|
|
|
export const llmConfigSchema = z.object({
|
|
provider: z.enum(["claude", "openai"]),
|
|
apiKey: z.string().optional(),
|
|
}).passthrough();
|
|
|
|
export const databaseBackupConfigSchema = z.object({
|
|
enabled: z.boolean().default(true),
|
|
intervalMinutes: z.number().int().min(1).max(7 * 24 * 60).default(60),
|
|
retentionDays: z.number().int().min(1).max(3650).default(7),
|
|
dir: z.string().default("~/.paperclip/instances/default/data/backups"),
|
|
}).passthrough();
|
|
|
|
export const databaseConfigSchema = z.object({
|
|
mode: z.enum(["embedded-postgres", "postgres"]).default("embedded-postgres"),
|
|
connectionString: z.string().optional(),
|
|
embeddedPostgresDataDir: z.string().default("~/.paperclip/instances/default/db"),
|
|
embeddedPostgresPort: z.number().int().min(1).max(65535).default(54329),
|
|
backup: databaseBackupConfigSchema.default({
|
|
enabled: true,
|
|
intervalMinutes: 60,
|
|
retentionDays: 7,
|
|
dir: "~/.paperclip/instances/default/data/backups",
|
|
}),
|
|
}).passthrough();
|
|
|
|
export const loggingConfigSchema = z.object({
|
|
mode: z.enum(["file", "cloud"]),
|
|
logDir: z.string().default("~/.paperclip/instances/default/logs"),
|
|
}).passthrough();
|
|
|
|
export const serverConfigSchema = z.object({
|
|
deploymentMode: z.enum(DEPLOYMENT_MODES).default("local_trusted"),
|
|
exposure: z.enum(DEPLOYMENT_EXPOSURES).default("private"),
|
|
bind: z.enum(BIND_MODES).optional(),
|
|
customBindHost: z.string().optional(),
|
|
host: z.string().default("127.0.0.1"),
|
|
port: z.number().int().min(1).max(65535).default(3100),
|
|
allowedHostnames: z.array(z.string().min(1)).default([]),
|
|
serveUi: z.boolean().default(true),
|
|
}).passthrough();
|
|
|
|
export const authConfigSchema = z.object({
|
|
baseUrlMode: z.enum(AUTH_BASE_URL_MODES).default("auto"),
|
|
publicBaseUrl: z.string().url().optional(),
|
|
disableSignUp: z.boolean().default(false),
|
|
}).passthrough();
|
|
|
|
export const storageLocalDiskConfigSchema = z.object({
|
|
baseDir: z.string().default("~/.paperclip/instances/default/data/storage"),
|
|
}).passthrough();
|
|
|
|
export const storageS3ConfigSchema = z.object({
|
|
bucket: z.string().min(1).default("paperclip"),
|
|
region: z.string().min(1).default("us-east-1"),
|
|
endpoint: z.string().optional(),
|
|
prefix: z.string().default(""),
|
|
forcePathStyle: z.boolean().default(false),
|
|
}).passthrough();
|
|
|
|
export const storageConfigSchema = z.object({
|
|
provider: z.enum(STORAGE_PROVIDERS).default("local_disk"),
|
|
localDisk: storageLocalDiskConfigSchema.default({
|
|
baseDir: "~/.paperclip/instances/default/data/storage",
|
|
}),
|
|
s3: storageS3ConfigSchema.default({
|
|
bucket: "paperclip",
|
|
region: "us-east-1",
|
|
prefix: "",
|
|
forcePathStyle: false,
|
|
}),
|
|
}).passthrough();
|
|
|
|
export const secretsLocalEncryptedConfigSchema = z.object({
|
|
keyFilePath: z.string().default("~/.paperclip/instances/default/secrets/master.key"),
|
|
}).passthrough();
|
|
|
|
export const secretsConfigSchema = z.object({
|
|
provider: z.enum(SECRET_PROVIDERS).default("local_encrypted"),
|
|
strictMode: z.boolean().default(false),
|
|
localEncrypted: secretsLocalEncryptedConfigSchema.default({
|
|
keyFilePath: "~/.paperclip/instances/default/secrets/master.key",
|
|
}),
|
|
}).passthrough();
|
|
|
|
export const telemetryConfigSchema = z.object({
|
|
enabled: z.boolean().default(true),
|
|
}).passthrough().default({});
|
|
|
|
export const updatesConfigSchema = z.object({
|
|
checkEnabled: z.boolean().default(true),
|
|
}).passthrough().default({});
|
|
|
|
export const paperclipConfigSchema = z
|
|
.object({
|
|
$meta: configMetaSchema,
|
|
llm: llmConfigSchema.optional(),
|
|
database: databaseConfigSchema,
|
|
logging: loggingConfigSchema,
|
|
server: serverConfigSchema,
|
|
telemetry: telemetryConfigSchema,
|
|
updates: updatesConfigSchema.optional(),
|
|
auth: authConfigSchema.default({
|
|
baseUrlMode: "auto",
|
|
disableSignUp: false,
|
|
}),
|
|
storage: storageConfigSchema.default({
|
|
provider: "local_disk",
|
|
localDisk: {
|
|
baseDir: "~/.paperclip/instances/default/data/storage",
|
|
},
|
|
s3: {
|
|
bucket: "paperclip",
|
|
region: "us-east-1",
|
|
prefix: "",
|
|
forcePathStyle: false,
|
|
},
|
|
}),
|
|
secrets: secretsConfigSchema.default({
|
|
provider: "local_encrypted",
|
|
strictMode: false,
|
|
localEncrypted: {
|
|
keyFilePath: "~/.paperclip/instances/default/secrets/master.key",
|
|
},
|
|
}),
|
|
})
|
|
.passthrough()
|
|
.superRefine((value, ctx) => {
|
|
if (value.server.deploymentMode === "local_trusted" && value.server.exposure !== "private") {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
message: "server.exposure must be private when deploymentMode is local_trusted",
|
|
path: ["server", "exposure"],
|
|
});
|
|
}
|
|
|
|
for (const message of validateConfiguredBindMode({
|
|
deploymentMode: value.server.deploymentMode,
|
|
deploymentExposure: value.server.exposure,
|
|
bind: value.server.bind,
|
|
host: value.server.host,
|
|
customBindHost: value.server.customBindHost,
|
|
})) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
message,
|
|
path: message.includes("customBindHost") ? ["server", "customBindHost"] : ["server", "bind"],
|
|
});
|
|
}
|
|
|
|
if (value.auth.baseUrlMode === "explicit" && !value.auth.publicBaseUrl) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
message: "auth.publicBaseUrl is required when auth.baseUrlMode is explicit",
|
|
path: ["auth", "publicBaseUrl"],
|
|
});
|
|
}
|
|
|
|
if (value.server.exposure === "public" && value.auth.baseUrlMode !== "explicit") {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
message: "auth.baseUrlMode must be explicit when deploymentMode=authenticated and exposure=public",
|
|
path: ["auth", "baseUrlMode"],
|
|
});
|
|
}
|
|
|
|
if (value.server.exposure === "public" && !value.auth.publicBaseUrl) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
message: "auth.publicBaseUrl is required when deploymentMode=authenticated and exposure=public",
|
|
path: ["auth", "publicBaseUrl"],
|
|
});
|
|
}
|
|
});
|
|
|
|
export type PaperclipConfig = z.infer<typeof paperclipConfigSchema>;
|
|
export type LlmConfig = z.infer<typeof llmConfigSchema>;
|
|
export type DatabaseConfig = z.infer<typeof databaseConfigSchema>;
|
|
export type LoggingConfig = z.infer<typeof loggingConfigSchema>;
|
|
export type ServerConfig = z.infer<typeof serverConfigSchema>;
|
|
export type StorageConfig = z.infer<typeof storageConfigSchema>;
|
|
export type StorageLocalDiskConfig = z.infer<typeof storageLocalDiskConfigSchema>;
|
|
export type StorageS3Config = z.infer<typeof storageS3ConfigSchema>;
|
|
export type SecretsConfig = z.infer<typeof secretsConfigSchema>;
|
|
export type SecretsLocalEncryptedConfig = z.infer<typeof secretsLocalEncryptedConfigSchema>;
|
|
export type AuthConfig = z.infer<typeof authConfigSchema>;
|
|
export type TelemetryConfig = z.infer<typeof telemetryConfigSchema>;
|
|
export type UpdatesConfig = z.infer<typeof updatesConfigSchema>;
|
|
export type ConfigMeta = z.infer<typeof configMetaSchema>;
|
|
export type DatabaseBackupConfig = z.infer<typeof databaseBackupConfigSchema>;
|
|
|
|
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
}
|
|
|
|
function unwrapConfigSchema(schema: z.ZodTypeAny): z.ZodTypeAny {
|
|
let current = schema;
|
|
while (true) {
|
|
if (current instanceof z.ZodEffects) {
|
|
current = current.innerType();
|
|
continue;
|
|
}
|
|
if (current instanceof z.ZodOptional) {
|
|
current = current.unwrap();
|
|
continue;
|
|
}
|
|
if (current instanceof z.ZodDefault) {
|
|
current = current.removeDefault();
|
|
continue;
|
|
}
|
|
return current;
|
|
}
|
|
}
|
|
|
|
function mergeUnknownConfigKeys(
|
|
source: Record<string, unknown>,
|
|
update: Record<string, unknown>,
|
|
schema: z.ZodTypeAny,
|
|
): Record<string, unknown> {
|
|
const objectSchema = unwrapConfigSchema(schema);
|
|
if (!(objectSchema instanceof z.ZodObject)) return { ...update };
|
|
|
|
const shape = objectSchema.shape as Record<string, z.ZodTypeAny>;
|
|
const merged = { ...update };
|
|
|
|
for (const [key, sourceValue] of Object.entries(source)) {
|
|
const childSchema = shape[key];
|
|
if (childSchema === undefined) {
|
|
merged[key] = sourceValue;
|
|
continue;
|
|
}
|
|
|
|
const updateValue = update[key];
|
|
if (isRecord(sourceValue) && isRecord(updateValue)) {
|
|
merged[key] = mergeUnknownConfigKeys(sourceValue, updateValue, childSchema);
|
|
}
|
|
}
|
|
|
|
return merged;
|
|
}
|
|
|
|
/**
|
|
* Applies a config update while retaining extension keys from the parsed source.
|
|
* Known optional keys that are absent from the update stay absent, so callers can
|
|
* intentionally clear values such as llm.apiKey or auth.publicBaseUrl.
|
|
*/
|
|
export function mergePaperclipConfig(
|
|
source: PaperclipConfig,
|
|
update: PaperclipConfig,
|
|
): PaperclipConfig {
|
|
return mergeUnknownConfigKeys(
|
|
source as Record<string, unknown>,
|
|
update as Record<string, unknown>,
|
|
paperclipConfigSchema,
|
|
) as PaperclipConfig;
|
|
}
|
|
|
|
export type ConfigKeyWarning = {
|
|
path: string;
|
|
suggestion: string;
|
|
};
|
|
|
|
function editDistance(left: string, right: string): number {
|
|
const previous = Array.from({ length: right.length + 1 }, (_, index) => index);
|
|
|
|
for (let leftIndex = 1; leftIndex <= left.length; leftIndex += 1) {
|
|
const current = [leftIndex];
|
|
for (let rightIndex = 1; rightIndex <= right.length; rightIndex += 1) {
|
|
const substitutionCost = left[leftIndex - 1] === right[rightIndex - 1] ? 0 : 1;
|
|
current[rightIndex] = Math.min(
|
|
current[rightIndex - 1] + 1,
|
|
previous[rightIndex] + 1,
|
|
previous[rightIndex - 1] + substitutionCost,
|
|
);
|
|
}
|
|
previous.splice(0, previous.length, ...current);
|
|
}
|
|
|
|
return previous[right.length];
|
|
}
|
|
|
|
function nearMatch(key: string, candidates: string[]): string | null {
|
|
const normalizedKey = key.toLowerCase();
|
|
let best: { candidate: string; distance: number } | null = null;
|
|
|
|
for (const candidate of candidates) {
|
|
const normalizedCandidate = candidate.toLowerCase();
|
|
if (normalizedKey === normalizedCandidate && key !== candidate) return candidate;
|
|
|
|
const distance = editDistance(normalizedKey, normalizedCandidate);
|
|
const threshold = Math.max(normalizedKey.length, normalizedCandidate.length) >= 8 ? 2 : 1;
|
|
if (distance > threshold || (best && distance >= best.distance)) continue;
|
|
best = { candidate, distance };
|
|
}
|
|
|
|
return best?.candidate ?? null;
|
|
}
|
|
|
|
function collectConfigKeyWarnings(
|
|
value: Record<string, unknown>,
|
|
schema: z.ZodTypeAny,
|
|
prefix: string,
|
|
warnings: ConfigKeyWarning[],
|
|
): void {
|
|
const objectSchema = unwrapConfigSchema(schema);
|
|
if (!(objectSchema instanceof z.ZodObject)) return;
|
|
|
|
const shape = objectSchema.shape as Record<string, z.ZodTypeAny>;
|
|
const knownKeys = Object.keys(shape);
|
|
|
|
for (const [key, childValue] of Object.entries(value)) {
|
|
const childSchema = shape[key];
|
|
const childPath = prefix ? `${prefix}.${key}` : key;
|
|
if (childSchema === undefined) {
|
|
const suggestion = nearMatch(key, knownKeys);
|
|
if (suggestion) {
|
|
warnings.push({
|
|
path: childPath,
|
|
suggestion: prefix ? `${prefix}.${suggestion}` : suggestion,
|
|
});
|
|
}
|
|
continue;
|
|
}
|
|
|
|
if (isRecord(childValue)) {
|
|
collectConfigKeyWarnings(childValue, childSchema, childPath, warnings);
|
|
}
|
|
}
|
|
}
|
|
|
|
/** Returns likely misspellings among retained extension keys without modifying them. */
|
|
export function findPaperclipConfigKeyWarnings(config: unknown): ConfigKeyWarning[] {
|
|
if (!isRecord(config)) return [];
|
|
const warnings: ConfigKeyWarning[] = [];
|
|
collectConfigKeyWarnings(config, paperclipConfigSchema, "", warnings);
|
|
return warnings;
|
|
}
|