Files
PaperClipAI/server/src/services/instance-settings.ts
T
DottaandPaperclip e93732c3d4 fix(server): deliver voice replies from commits and saved deadlines (#17578)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Phone calls receive approved answers from the existing chat
publication queue.
> - The chat maintenance timer checks for voice replies every second,
including when no reply is ready.
> - A reply can be saved before its publication commits. An early wake
alone can miss that final transition.
> - Existing session, reply, publication, and action rows already record
delivery and recovery work.
> - This pull request uses committed changes and saved deadlines to
drive phone reply delivery.
> - Empty reply queues become quiet while authorization, ordering, and
uncertain-send protection remain in place.

## Linked Issues or Issue Description

Stacked on #17576. Related delivery infrastructure: #15625. Related
voice connection work: #11418.

**What existing behavior does this improve?**

Background phone reply delivery and recovery of abandoned dispatch
intents.

**Current behavior**

Chat maintenance calls voice reply delivery every second. It also scans
old dispatch intents. Publication transport starts a direct push before
the final publication commit. The timer masks that timing gap.

**Proposed behavior**

Committed publication, reply, session, and configuration changes wake
the shared coordinator. Ready replies drain in bounded passes. Saved
rate-limit and dispatch-expiry deadlines retain recovery. An empty queue
or a permanent delivery barrier has no recurring reply scan.

**Reason and benefit**

Remove recurring SQL from idle phone reply delivery without dropping
replies, replaying uncertain sends, or adding a queue table.

**Breaking changes**

No API or schema change. Explicit manual recovery remains available.
Inbound admission cleanup, history retention, and reports remain on
their current paths for separate migrations.

## What Changed

- Extract ordered reply state and multipart policy within the voice
owner. Retain current authorization and atomic dispatch claims.
- Register a scoped reply worker with startup reconstruction, fair work
batches, failure backoff, drain/resume handling and shutdown ownership.
- Extend the existing shared dispatcher with normalized partial-page
results to preserve the bounded parent recovery contract. Existing
complete-array consumers retain their calls.
- Signal before relevant writes in their existing transactions. Both
normal publication completion and operator resolution wake the actual
conversation after commit.
- Bound reply inventory to 100 total headers per refresh. Preserve
partial scans, concurrent changes and newer denied-call restoration
obligations across reconstruction.
- Track abandoned dispatch actions independently of live calls.
Quarantine expired intents without resending them.
- Use existing active-call authorization checks to reactivate replies
that were previously denied. Healthy call inspections do not rescan
empty reply queues.
- Remove the periodic reply scan and the early direct publication push.
Keep provider requests outside database transactions.

## Verification

Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice
recovery #17576 has landed at master
`e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles
squash ancestry while preserving the complete reviewed 31-file reply
patch byte for byte and all unrelated master changes. The PR now targets
master; fresh exact-head CI and Greptile are required. The prior
449-case/typecheck/build results below cover the identical owned source,
not a new full-suite run.

- **449 distinct targeted cases pass**: 111 complete voice-store/reply;
19 complete parent recovery; and 319 complete tool-gateway,
secret-proposal, shared dispatcher/coordinator/batching/startup/hostname
cases. Production server typecheck and build pass. The policy checker
and its Node tests pass. Later edits record evidence only.
- Two real normal-publication/operator-resolution tests seed 105
unrelated calls and 105 future dispatches. Five publications selected
**1,065/1,060 inventory rows before repair**; both paths now pass the
same **125-row total budget** and deliver every answer once. This counts
selected SQL results, not physical tuple visits or host savings.
- Publication commits now signal their actual conversation. The voice
reader returns at most 100 total session/action headers per refresh,
preserves phase/cursor state and retries, and resets on worker
registration. Current-state checks and dispatch claims still own
provider effects. The 105-action case measures the page cap, restarts
the same owner during reconstruction, waits at saved future deadlines
and quarantines every abandoned action without resending.
- A real 403 fixture proves that an older broad scan cannot erase a
denied call first observed behind its cursor. Its owner callback still
emits the committed restoration wake after scan completion. The existing
actual-service authority-restoration and final-publication timing tests
remain in the full suite. They are not replaced by direct helper calls.
- Guarded removal of conversation scoping, cursor reset or the denial
revision fence fails the intended budget, 100-versus-5 restart or
missing committed-wake assertion; source is restored. An initial
shadowed fixture name and an inadequate due-action reset probe are
retained as failed attempts in the [current
audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28).
- Parent recovery still passes paging, restart, behind-cursor insertion,
feature changes, actual disablement, quiet health observations, current
authority, exact-scope failure fences and wildcard-failure isolation
through the shared worker. Four existing array-based worker consumers
retain their API and pass their full suites.
- The two voice fixtures self-migrate separate empty databases. Legacy
tool-review/secrets suites use separate migrated clones. Earlier
full-local attempts remain limited by embedded PostgreSQL setup and are
not passing full-matrix claims. Older counts apply only to their stated
revisions. Fresh exact-head Greptile, Ponytail if present, security and
all CI remain merge gates. No live-provider, multi-writer,
external-wake, deployed-sleep or savings claim is made.

## Risks

- Missing a supported producer can strand work. Tests cover real
enqueue, retrieval, normal publication, and operator publication paths.
Startup reconstructs existing records.
- Signals cover the process-local database owner. Independent SQL
writers or other replicas need their own wake contract.
- Authority restoration relies on the existing live-call recovery check.
This dependency must be preserved if that owner changes later.
- A provider acceptance receipt does not prove speech. Unknown outcomes
remain blocked from automatic resend.
- This change does not extend Cloud alarms, allow sleep during live
calls, or prove deployed suspension or cost savings.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, reasoning, code
execution and test tools. The runtime does not expose a more specific
deployment identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with the available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run the selected tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:50:27 -05:00

639 lines
26 KiB
TypeScript

import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import type { Db } from "@paperclipai/db";
import { companies, instanceSettings } from "@paperclipai/db";
/**
* A `Db` or an open transaction handle — the subset of query builders the
* settings writes use. Lets `update` run inside a caller's transaction so
* it commits atomically with a sibling write.
*/
type InstanceSettingsTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export type InstanceSettingsWriteDb = Pick<
Db | InstanceSettingsTransaction,
"select" | "insert" | "update"
>;
import {
DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
DEFAULT_BACKUP_RETENTION,
PAPERCLIP_CLOUD_MANAGED_BY,
instanceGeneralSettingsSchema,
type InstanceGeneralSettings,
instanceExperimentalSettingsSchema,
type InstanceExperimentalSettings,
type InstanceExperimentalSettingsWithManaged,
type ManagedExperimentalFeatureKey,
type ManagedSettingMetadata,
type PatchInstanceGeneralSettings,
type InstanceSettings,
type PatchInstanceSettings,
type PatchInstanceExperimentalSettings,
} from "@paperclipai/shared";
import {
INSTANCE_FEATURE_CATALOG,
applyOperatorGeneralDefaults,
stripOperatorGeneralEchoes,
} from "@paperclipai/shared";
import { eq } from "drizzle-orm";
import { getManagedInstanceConfig, type ManagedInstanceConfig } from "./managed-config.js";
import { getOperatorSettingDefaults } from "./setting-defaults.js";
import { isCloudStagingExperimentalSetting, isCloudStagingInstance } from "./cloud-instance.js";
import { forbidden } from "../errors.js";
const DEFAULT_SINGLETON_KEY = "default";
const instanceGeneralSettingsStorageSchema = instanceGeneralSettingsSchema.strip();
const instanceExperimentalSettingsStorageSchema = instanceExperimentalSettingsSchema.strip();
const TRUTHY_RUNTIME_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
interface InstanceSettingsServiceOptions {
runtimeEnv?: Record<string, string | undefined>;
now?: () => Date;
}
type WorktreeRunExecutionSuppressedReason =
| "not_worktree_runtime"
| "flag_disabled"
| "missing_cutoff"
| "missing_instance_id"
| "instance_id_mismatch"
| "settings_read_error";
export type WorktreeRunExecutionActivationState =
| {
armed: true;
cutoff: string;
activationInstanceId: string;
reason: null;
}
| {
armed: false;
cutoff: null;
activationInstanceId: string | null;
reason: WorktreeRunExecutionSuppressedReason;
};
export function isTruthyRuntimeEnvValue(value: string | undefined) {
return typeof value === "string" && TRUTHY_RUNTIME_ENV_VALUES.has(value.trim().toLowerCase());
}
function getRuntimeInstanceId(env: Record<string, string | undefined>) {
const instanceId = env.PAPERCLIP_INSTANCE_ID?.trim();
return instanceId ? instanceId : null;
}
function stripServerManagedExperimentalPatchFields(
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
): PatchInstanceExperimentalSettings {
const {
worktreeRunExecutionActivatedAt: _ignoredActivatedAt,
worktreeRunExecutionActivationInstanceId: _ignoredActivationInstanceId,
...patchable
} = patch as Record<string, unknown>;
return patchable as PatchInstanceExperimentalSettings;
}
export function applyExperimentalSettingsPatch(
current: unknown,
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
options: InstanceSettingsServiceOptions = {},
): InstanceExperimentalSettings {
const previousExperimental = normalizeExperimentalSettings(current);
const patchable = stripServerManagedExperimentalPatchFields(patch);
const nextExperimental = normalizeExperimentalSettings({
...previousExperimental,
...patchable,
});
const hasWorktreeRunExecutionPatch = Object.prototype.hasOwnProperty.call(
patchable,
"enableWorktreeRunExecution",
);
if (!hasWorktreeRunExecutionPatch) {
return nextExperimental;
}
if (nextExperimental.enableWorktreeRunExecution !== true) {
return {
...nextExperimental,
worktreeRunExecutionActivatedAt: null,
worktreeRunExecutionActivationInstanceId: null,
};
}
if (previousExperimental.enableWorktreeRunExecution === true) {
return nextExperimental;
}
const runtimeEnv = options.runtimeEnv ?? process.env;
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
return nextExperimental;
}
return {
...nextExperimental,
worktreeRunExecutionActivatedAt: (options.now ?? (() => new Date()))().toISOString(),
worktreeRunExecutionActivationInstanceId: getRuntimeInstanceId(runtimeEnv),
};
}
function suppressWorktreeRunExecution(
reason: WorktreeRunExecutionSuppressedReason,
activationInstanceId: string | null = null,
): WorktreeRunExecutionActivationState {
return {
armed: false,
cutoff: null,
activationInstanceId,
reason,
};
}
export function resolveWorktreeRunExecutionActivation(
experimental: InstanceExperimentalSettings,
currentInstanceId: string | null | undefined,
): WorktreeRunExecutionActivationState {
if (experimental.enableWorktreeRunExecution !== true) {
return suppressWorktreeRunExecution(
"flag_disabled",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (!experimental.worktreeRunExecutionActivatedAt) {
return suppressWorktreeRunExecution(
"missing_cutoff",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (!currentInstanceId) {
return suppressWorktreeRunExecution(
"missing_instance_id",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (experimental.worktreeRunExecutionActivationInstanceId !== currentInstanceId) {
return suppressWorktreeRunExecution(
"instance_id_mismatch",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
return {
armed: true,
cutoff: experimental.worktreeRunExecutionActivatedAt,
activationInstanceId: currentInstanceId,
reason: null,
};
}
export async function resolveWorktreeRunExecutionActivationState(options: {
getExperimental: () => Promise<InstanceExperimentalSettings>;
runtimeEnv?: Record<string, string | undefined>;
}): Promise<WorktreeRunExecutionActivationState> {
const runtimeEnv = options.runtimeEnv ?? process.env;
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
return suppressWorktreeRunExecution("not_worktree_runtime");
}
try {
return resolveWorktreeRunExecutionActivation(
await options.getExperimental(),
getRuntimeInstanceId(runtimeEnv),
);
} catch {
return suppressWorktreeRunExecution("settings_read_error");
}
}
function normalizeGeneralSettings(raw: unknown): InstanceGeneralSettings {
const parsed = instanceGeneralSettingsStorageSchema.safeParse(raw ?? {});
if (parsed.success) {
return {
censorUsernameInLogs: parsed.data.censorUsernameInLogs ?? false,
feedbackDataSharingPreference:
parsed.data.feedbackDataSharingPreference ?? DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
backupRetention: parsed.data.backupRetention ?? DEFAULT_BACKUP_RETENTION,
// Absent => unrestricted; only carry through an explicit policy.
...(parsed.data.executionMode ? { executionMode: parsed.data.executionMode } : {}),
};
}
return {
censorUsernameInLogs: false,
feedbackDataSharingPreference: DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
backupRetention: DEFAULT_BACKUP_RETENTION,
};
}
export function normalizeExperimentalSettings(raw: unknown): InstanceExperimentalSettings {
const parsed = instanceExperimentalSettingsStorageSchema.safeParse(raw ?? {});
if (parsed.success) {
return {
enableEnvironments: parsed.data.enableEnvironments ?? false,
enableNativeRunner: parsed.data.enableNativeRunner ?? true,
enableAiConnectionRouters: parsed.data.enableAiConnectionRouters ?? false,
enableManagedSandboxOnly: parsed.data.enableManagedSandboxOnly ?? false,
enableIsolatedWorkspaces: parsed.data.enableIsolatedWorkspaces ?? false,
enableIsolatedWorkspacesByDefault: parsed.data.enableIsolatedWorkspacesByDefault ?? false,
enableStreamlinedLeftNavigation: parsed.data.enableStreamlinedLeftNavigation ?? true,
enableStreamlinedUi: parsed.data.enableStreamlinedUi ?? true,
// Apps graduated from Experimental. Ignore historical off values while
// continuing to accept the compatibility key in stored settings.
enableApps: true,
enableMcpAggregators: true,
enablePublicMcp: parsed.data.enablePublicMcp ?? false,
enableOpenAiDot: parsed.data.enableOpenAiDot ?? false,
enableMuse: parsed.data.enableMuse ?? false,
enableGrokBot: parsed.data.enableGrokBot ?? false,
enableChatConnectors: parsed.data.enableChatConnectors ?? false,
enableGitHubReviewBots: parsed.data.enableGitHubReviewBots ?? false,
enableMemoryConnectors: parsed.data.enableMemoryConnectors ?? false,
enablePaperclipNotebook: parsed.data.enablePaperclipNotebook ?? false,
enableCases: parsed.data.enableCases ?? false,
enableAgentChat: parsed.data.enableAgentChat ?? false,
enableCombinedInboxTasks: parsed.data.enableCombinedInboxTasks ?? false,
enableArtifactsV1: parsed.data.enableArtifactsV1 ?? false,
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
enableClassicTaskInterface: parsed.data.enableClassicTaskInterface ?? false,
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
enableExternalObjects: parsed.data.enableExternalObjects ?? false,
enableSmokeLab: parsed.data.enableSmokeLab ?? false,
enableBuiltInAgents: parsed.data.enableBuiltInAgents ?? false,
enableBetaSkills: parsed.data.enableBetaSkills ?? false,
enableSummaries: parsed.data.enableSummaries ?? false,
enableStatusCards: parsed.data.enableStatusCards ?? false,
enableDecisions: parsed.data.enableDecisions ?? false,
enableFastResponses: parsed.data.enableFastResponses ?? false,
enableGoalsSidebarLink: parsed.data.enableGoalsSidebarLink ?? false,
enableServerInfoDebugView: parsed.data.enableServerInfoDebugView ?? false,
enablePaperclipDeveloperMode: parsed.data.enablePaperclipDeveloperMode ?? false,
enableSimplifiedEnglishInteractions: parsed.data.enableSimplifiedEnglishInteractions ?? false,
enableFirstTaskPlanProposal: parsed.data.enableFirstTaskPlanProposal ?? false,
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
enableWorkspaceBranchReconcileForward: parsed.data.enableWorkspaceBranchReconcileForward ?? true,
enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true,
enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false,
enableSandboxDuplexBridge: parsed.data.enableSandboxDuplexBridge ?? false,
enableRunnerPreviewIngress: parsed.data.enableRunnerPreviewIngress ?? false,
enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false,
worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null,
worktreeRunExecutionActivationInstanceId:
parsed.data.worktreeRunExecutionActivationInstanceId ?? null,
};
}
return {
enableEnvironments: false,
enableNativeRunner: true,
enableAiConnectionRouters: false,
enableManagedSandboxOnly: false,
enableIsolatedWorkspaces: false,
enableIsolatedWorkspacesByDefault: false,
enableStreamlinedLeftNavigation: true,
enableStreamlinedUi: true,
enableApps: true,
enableMcpAggregators: true,
enablePublicMcp: false,
enableOpenAiDot: false,
enableMuse: false,
enableGrokBot: false,
enableChatConnectors: false,
enableGitHubReviewBots: false,
enableMemoryConnectors: false,
enablePaperclipNotebook: false,
enableCases: false,
enableAgentChat: false,
enableCombinedInboxTasks: false,
enableArtifactsV1: false,
enableConferenceRoomChat: false,
enableClassicTaskInterface: false,
enableIssuePlanDecompositions: false,
enableExperimentalFileViewer: false,
enableExternalObjects: false,
enableSmokeLab: false,
enableBuiltInAgents: false,
enableBetaSkills: false,
enableSummaries: false,
enableStatusCards: false,
enableDecisions: false,
enableFastResponses: false,
enableGoalsSidebarLink: false,
enableServerInfoDebugView: false,
enablePaperclipDeveloperMode: false,
enableSimplifiedEnglishInteractions: false,
enableFirstTaskPlanProposal: false,
autoRestartDevServerWhenIdle: false,
enableWorkspaceBranchReconcileForward: true,
enableWorkspaceDirtyQuarantineRepair: true,
enableOwnerInstanceAdmin: false,
enableSandboxDuplexBridge: false,
enableRunnerPreviewIngress: false,
enableWorktreeRunExecution: false,
worktreeRunExecutionActivatedAt: null,
worktreeRunExecutionActivationInstanceId: null,
};
}
export type ManagedExperimentalKeyMetadata = Partial<
Record<ManagedExperimentalFeatureKey, ManagedSettingMetadata>
>;
/**
* Overlay the cloud managed-config feature values over normalized settings.
*
* Read-time precedence: code floor (cloud) > managed overlay > tenant DB
* value > schema default. (No code floors are expressed as flags today —
* floors are enforced in code at the guarded routes, independent of any
* flag value.) The overlay is deliberately never persisted: it re-asserts on
* every read, so a DB restore or manual row edit cannot resurrect a
* capability the harness has disabled.
*/
export function applyManagedExperimentalOverlay(
experimental: InstanceExperimentalSettings,
managedConfig: ManagedInstanceConfig | null,
): { experimental: InstanceExperimentalSettings; managedKeys: ManagedExperimentalKeyMetadata } {
if (!managedConfig) return { experimental, managedKeys: {} };
const next: InstanceExperimentalSettings = { ...experimental };
const managedKeys: ManagedExperimentalKeyMetadata = {};
for (const [key, value] of Object.entries(managedConfig.features) as Array<
[ManagedExperimentalFeatureKey, boolean]
>) {
// Existing Cloud stack configs may still carry retired flags. Accept the
// document during rollout, but never let retired flags disable Apps or MCP aggregators.
if (key === "enableApps" || key === "enableMcpAggregators") continue;
next[key] = value;
managedKeys[key] = { managed: true, managedBy: PAPERCLIP_CLOUD_MANAGED_BY };
}
return { experimental: next, managedKeys };
}
/**
* Keep self-hosted-only defaults out of Cloud.
*
* The experimental schema carries one default per flag, and the feature
* catalog pins it to `selfHostedDefault`. A flag that is on by default for
* self-hosted but off by default for Cloud (`selfHostedDefault: true`,
* `cloudDefault: false`) would therefore normalize to "on" for a managed
* instance whose tenant row and managed overlay both leave it unset. Re-assert
* the declared Cloud default for exactly those flags. An explicit tenant value
* or a managed feature value still wins (the overlay is applied afterwards).
*/
export function applyCloudCatalogDefaults(
experimental: InstanceExperimentalSettings,
rawStored: unknown,
managedConfig: ManagedInstanceConfig | null,
): InstanceExperimentalSettings {
if (!managedConfig) return experimental;
const stored =
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
? (rawStored as Record<string, unknown>)
: {};
const next: InstanceExperimentalSettings = { ...experimental };
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
if (typeof stored[key] === "boolean") continue;
if (typeof managedConfig.features[key as ManagedExperimentalFeatureKey] === "boolean") continue;
(next as unknown as Record<string, unknown>)[key] = false;
}
return next;
}
/**
* Keep the write path from freezing a self-hosted default into a Cloud row.
*
* `updateExperimental` persists the whole normalized object, and the schema
* normalizes an omitted flag to its self-hosted default. Without this step an
* unrelated experimental write (say, turning on Cases) would store
* `enableNativeRunner: true` on a managed instance whose tenant row had never
* mentioned the flag; every later read would then treat the stored boolean as
* an explicit tenant choice and stop re-asserting the Cloud default.
*
* For each guarded flag (see `applyCloudCatalogDefaults`), the stored key is
* left absent unless the tenant already stored a boolean or this patch sets
* the flag to something other than the Cloud default. A patch value equal to
* the Cloud default is a full-GET echo of the read-time overlay, not a
* choice, and is stripped the same way `stripOperatorGeneralEchoes` treats
* operator defaults. Self-hosted rows are returned untouched.
*/
export function stripCloudCatalogDefaultEchoes(
rawStored: unknown,
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
next: InstanceExperimentalSettings,
managedConfig: ManagedInstanceConfig | null,
): Partial<InstanceExperimentalSettings> {
if (!managedConfig) return next;
const stored =
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
? (rawStored as Record<string, unknown>)
: {};
const patchRecord = patch as Record<string, unknown>;
const result: Record<string, unknown> = { ...next };
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
if (typeof stored[key] === "boolean") continue;
if (
Object.prototype.hasOwnProperty.call(patchRecord, key) &&
typeof patchRecord[key] === "boolean" &&
patchRecord[key] !== entry.cloudDefault
) {
continue;
}
delete result[key];
}
return result as Partial<InstanceExperimentalSettings>;
}
export function instanceSettingsService(db: Db, options: InstanceSettingsServiceOptions = {}) {
// Fail closed: a malformed PAPERCLIP_MANAGED_CONFIG throws here (and at
// boot in index.ts) rather than silently running without the overlay.
const managedConfig = getManagedInstanceConfig(options.runtimeEnv ?? process.env);
const staging = isCloudStagingInstance(options.runtimeEnv ?? process.env);
const enforcedManagedConfig = staging && managedConfig
? {
...managedConfig,
features: Object.fromEntries(Object.entries(managedConfig.features)
.filter(([key, enabled]) => enabled || !isCloudStagingExperimentalSetting(key))),
}
: managedConfig;
// Same posture for PAPERCLIP_SETTING_DEFAULTS: parsed once, applied per
// read, never persisted (see applyOperatorGeneralDefaults) — including on
// the write path, where a full-GET echo of the overlaid value is stripped
// back to the schema default (see stripOperatorGeneralEchoes).
const operatorDefaults = getOperatorSettingDefaults(options.runtimeEnv ?? process.env);
function toGeneralView(raw: unknown): InstanceGeneralSettings {
return applyOperatorGeneralDefaults(normalizeGeneralSettings(raw), operatorDefaults);
}
function toExperimentalView(raw: unknown): InstanceExperimentalSettingsWithManaged {
const stored = raw && typeof raw === "object" && !Array.isArray(raw) ? raw : {};
const defaults = staging ? { ...managedConfig?.features, ...stored } : raw;
const { experimental, managedKeys } = applyManagedExperimentalOverlay(
applyCloudCatalogDefaults(normalizeExperimentalSettings(defaults), defaults, managedConfig),
enforcedManagedConfig,
);
// Self-hosted responses stay byte-identical: no managedKeys field at all.
return managedConfig ? { ...experimental, managedKeys } : experimental;
}
function toInstanceSettings(row: typeof instanceSettings.$inferSelect): InstanceSettings {
return {
id: row.id,
defaultEnvironmentId: row.defaultEnvironmentId ?? null,
general: toGeneralView(row.general),
experimental: toExperimentalView(row.experimental),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
} as InstanceSettings;
}
async function getOrCreateRow(runner: InstanceSettingsWriteDb = db) {
const existing = await runner
.select()
.from(instanceSettings)
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
.then((rows) => rows[0] ?? null);
if (existing) return existing;
const now = new Date();
const [created] = await runner
.insert(instanceSettings)
.values({
singletonKey: DEFAULT_SINGLETON_KEY,
general: {},
experimental: {},
createdAt: now,
updatedAt: now,
})
.onConflictDoUpdate({
target: [instanceSettings.singletonKey],
set: {
updatedAt: now,
},
})
.returning();
if (created) return created;
const raced = await runner
.select()
.from(instanceSettings)
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
.then((rows) => rows[0] ?? null);
if (raced) return raced;
throw new Error("Failed to initialize instance settings row");
}
return {
get: async (): Promise<InstanceSettings> => toInstanceSettings(await getOrCreateRow()),
update: async (
patch: PatchInstanceSettings,
writeOptions?: { db?: InstanceSettingsWriteDb },
): Promise<InstanceSettings> => {
// The write may run inside a caller-supplied transaction so it commits
// atomically with a sibling write (e.g. clearing the managed-default
// stamp on the environment row alongside a defaultEnvironmentId
// change). Reads use the same runner so the row is visible to the tx.
const runner = writeOptions?.db ?? db;
const current = await getOrCreateRow(runner);
const now = new Date();
const [updated] = await runner
.update(instanceSettings)
.set({
...(Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId")
? { defaultEnvironmentId: patch.defaultEnvironmentId ?? null }
: {}),
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
return toInstanceSettings(updated ?? current);
},
getGeneral: async (
readOptions?: { db?: InstanceSettingsWriteDb },
): Promise<InstanceGeneralSettings> => {
const row = await getOrCreateRow(readOptions?.db);
return toGeneralView(row.general);
},
getExperimental: async (): Promise<InstanceExperimentalSettingsWithManaged> => {
const row = await getOrCreateRow();
return toExperimentalView(row.experimental);
},
updateGeneral: async (patch: PatchInstanceGeneralSettings): Promise<InstanceSettings> => {
const current = await getOrCreateRow();
const storedGeneral = normalizeGeneralSettings(current.general);
// A full-GET echo carries the overlaid operator value for a field the
// user never chose; stripping it keeps the overlay strictly read-time,
// so changing or unsetting the variable later still takes effect.
const nextGeneral = stripOperatorGeneralEchoes(
storedGeneral,
normalizeGeneralSettings({ ...storedGeneral, ...patch }),
operatorDefaults,
);
const now = new Date();
const [updated] = await db
.update(instanceSettings)
.set({
general: { ...nextGeneral },
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
return toInstanceSettings(updated ?? current);
},
updateExperimental: async (patch: PatchInstanceExperimentalSettings): Promise<InstanceSettings> => {
for (const [key, enabled] of Object.entries(managedConfig?.features ?? {})) {
if (enabled && patch[key as ManagedExperimentalFeatureKey] === false) {
throw forbidden(`${key} is enabled by Paperclip Cloud and cannot be disabled on this instance`, {
code: "settings_operator_managed",
});
}
}
const current = await getOrCreateRow();
// Guarded Cloud flags stay absent from the row unless chosen, so the
// read-time catalog default keeps applying (see stripCloudCatalogDefaultEchoes).
const nextExperimental = staging
? applyExperimentalSettingsPatch(toExperimentalView(current.experimental), patch, options)
: stripCloudCatalogDefaultEchoes(
current.experimental,
patch,
applyExperimentalSettingsPatch(current.experimental, patch, options),
managedConfig,
);
if (staging) {
const stored = current.experimental as Record<string, unknown> | null;
for (const key of Object.keys(INSTANCE_FEATURE_CATALOG) as ManagedExperimentalFeatureKey[]) {
if (typeof stored?.[key] !== "boolean" && !Object.prototype.hasOwnProperty.call(patch, key)) {
delete nextExperimental[key];
}
}
}
const now = new Date();
const [updated] = await db.transaction(async tx => {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*");
await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*");
await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch);
if (patch.enableChatConnectors !== undefined) await notifyVoiceChange(tx);
await notifyToolConnectionHealth(tx, "*");
return tx
.update(instanceSettings)
.set({
experimental: { ...nextExperimental },
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
});
return toInstanceSettings(updated ?? current);
},
listCompanyIds: async (): Promise<string[]> =>
db
.select({ id: companies.id })
.from(companies)
.then((rows) => rows.map((row) => row.id)),
};
}