mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 14:02:36 +01:00
## Thinking Path > - Paperclip manages agent work and should become idle when no work is owed. > - Grok routine dispatch already persists each outbound intent before delivery. > - Its dispatcher starts a one-second timer whenever public MCP is configured. > - That timer queries settings even when Grok is disabled, and scans dispatches when enabled but idle. > - The reservation transaction and feature-setting transaction already define when work can become available. > - This PR connects those writes to the shared coordinator and retains the Grok service's claim and delivery policy. > - The worker becomes quiet without adding a queue table or replaying uncertain sends. ## Linked Issues or Issue Description Stacked on #17486. This PR owns only Grok routine dispatch. It does not include decision retention or other remaining maintenance owners. Related open Grok PR #17278 concerns Runner credential-lease fixtures, not this dispatcher. The duplicate search found no overlapping dispatcher migration. **What existing behavior does this improve?** Grok routine dispatch recovery in the server. **Current behavior** With public MCP configured, the dispatcher reads instance settings every second, including when Grok is disabled. With Grok enabled, it scans for prepared dispatches every second. The baseline regression observed three queries in 3.2 seconds with Grok disabled. **Proposed behavior** Reconcile once at startup. Wake from committed dispatch reservations and experimental-setting changes. Schedule another pass only while prepared dispatches remain or reconciliation failed. Disabled and settled workers perform no recurring SQL or provider calls. **Reason and benefit** Remove unnecessary tenant database activity while preserving dispatch recovery, current authorization, provider outcome semantics, and shutdown ownership. This follows the existing sleep-aware maintenance direction; ROADMAP review found no competing dispatcher redesign. **Breaking changes** No public API or provider payload change. The internal dispatcher start method receives the shared coordinator; that coordinator now owns shutdown. Sending, accepted, unknown, and terminal dispatches remain non-replayable. This does not extend Cloud wake coverage. ## What Changed - Register the Grok worker with the existing coordinator; remove its independent timer. - Signal reservation and experimental-setting writes before commit. Nested writes follow their outer transaction. - Preserve ten-row batches, durable send claims, current authority checks, and existing cancellation/uncertainty policy. - Join admitted work during shutdown and leave later rows for restart. - Add a concurrent, replay-safe partial index for prepared dispatch lookup. Keep only the five newest generated snapshots. - Reuse the existing real OAuth/agent/routine fixture for lifecycle tests and add a validated external PostgreSQL test option. - Document producers, recovery, quiet conditions, and remaining host-sleep limitations. ## Verification Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by docs-only head `679f523d78`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed target `8dfed893d8`. Its entire runtime and acceptance-test sources match that reviewed/tested tree; only four independently merged Runner E2E accounting files and the ledger differ, preserved exactly. Migration 0391 and the parent journal prefix are unchanged. Local checks were not repeated on unchanged runtime; fresh final-head CI and reviews are required. - **273 cases pass** in nine complete suites: real-PostgreSQL dispatch lifecycle, Grok qualification, server startup, scoped deadlines and all five instance-settings suites. Server typecheck and production build pass with the updated Runner vendor. - The baseline disabled-feature regression failed with three SQL reads. Existing lifecycle cases cover first reservation, restart, non-replayable history, re-enable, outer commit/rollback, drain, failure/retry, concurrent enqueue, paging, foreground send race, expiry, settings commit/rollback, and shutdown. - Generated migration `0391_third_roulette.sql` adds only the concurrent/idempotent prepared-dispatch index. All 389 parent journal entries are unchanged; five generated snapshots remain (0387–0391). Migration safety/numbering, schema drift and actual fresh apply/replay on two isolated PostgreSQL databases pass. - The earlier 259-case selection and 40,000-row generic-plan probe remain historical evidence: both unchanged queries used the partial index, six shared buffers including the routine join, 0.032ms / 0.025ms local execution. No new deployed suspension or cost claim is made. - Earlier browser CI failed during agent-chat fixture restoration (lock timeout/HTTP 500), followed by a test seeing the experiment left enabled. Those failures remain recorded in the owning audit; they were not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks were terminal and green. Fresh CI and both 5/5 bot reviews are required for this head. Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`, `server-startup-feedback-export`, `scoped-deadline-work` and the five `instance-settings-*` suites; server typecheck/build; DB migration safety/numbering, snapshot drift and fresh migration apply/replay. ## Risks - Signals are local to the shared database owner and its child pools. Independent SQL writers and other replicas need explicit wake integration. Startup recovers committed prepared rows. - Disabled Grok retains prepared rows and resumes after a committed settings change. This does not relax Cloud's binding/assignment/hold guards. - Existing send-error cancellation policy is preserved. Sending or unknown outcomes are not reset or automatically replayed; this is not an exactly-once delivery change. - Shutdown awaits the current send within existing transport timeouts and stops before the next row. - The index adds write/storage cost but bounds queries independently of completed history. Existing migration execution runs concurrent index creation outside a transaction. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
635 lines
26 KiB
TypeScript
635 lines
26 KiB
TypeScript
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { companies, instanceSettings } from "@paperclipai/db";
|
|
|
|
/**
|
|
* A `Db` or an open transaction handle — the subset of query builders the
|
|
* settings writes use. Lets `update` run inside a caller's transaction so
|
|
* it commits atomically with a sibling write.
|
|
*/
|
|
type InstanceSettingsTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
export type InstanceSettingsWriteDb = Pick<
|
|
Db | InstanceSettingsTransaction,
|
|
"select" | "insert" | "update"
|
|
>;
|
|
import {
|
|
DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
DEFAULT_BACKUP_RETENTION,
|
|
PAPERCLIP_CLOUD_MANAGED_BY,
|
|
instanceGeneralSettingsSchema,
|
|
type InstanceGeneralSettings,
|
|
instanceExperimentalSettingsSchema,
|
|
type InstanceExperimentalSettings,
|
|
type InstanceExperimentalSettingsWithManaged,
|
|
type ManagedExperimentalFeatureKey,
|
|
type ManagedSettingMetadata,
|
|
type PatchInstanceGeneralSettings,
|
|
type InstanceSettings,
|
|
type PatchInstanceSettings,
|
|
type PatchInstanceExperimentalSettings,
|
|
} from "@paperclipai/shared";
|
|
import {
|
|
INSTANCE_FEATURE_CATALOG,
|
|
applyOperatorGeneralDefaults,
|
|
stripOperatorGeneralEchoes,
|
|
} from "@paperclipai/shared";
|
|
import { eq } from "drizzle-orm";
|
|
import { getManagedInstanceConfig, type ManagedInstanceConfig } from "./managed-config.js";
|
|
import { getOperatorSettingDefaults } from "./setting-defaults.js";
|
|
import { isCloudStagingExperimentalSetting, isCloudStagingInstance } from "./cloud-instance.js";
|
|
import { forbidden } from "../errors.js";
|
|
|
|
const DEFAULT_SINGLETON_KEY = "default";
|
|
const instanceGeneralSettingsStorageSchema = instanceGeneralSettingsSchema.strip();
|
|
const instanceExperimentalSettingsStorageSchema = instanceExperimentalSettingsSchema.strip();
|
|
const TRUTHY_RUNTIME_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
|
|
|
|
interface InstanceSettingsServiceOptions {
|
|
runtimeEnv?: Record<string, string | undefined>;
|
|
now?: () => Date;
|
|
}
|
|
|
|
type WorktreeRunExecutionSuppressedReason =
|
|
| "not_worktree_runtime"
|
|
| "flag_disabled"
|
|
| "missing_cutoff"
|
|
| "missing_instance_id"
|
|
| "instance_id_mismatch"
|
|
| "settings_read_error";
|
|
|
|
export type WorktreeRunExecutionActivationState =
|
|
| {
|
|
armed: true;
|
|
cutoff: string;
|
|
activationInstanceId: string;
|
|
reason: null;
|
|
}
|
|
| {
|
|
armed: false;
|
|
cutoff: null;
|
|
activationInstanceId: string | null;
|
|
reason: WorktreeRunExecutionSuppressedReason;
|
|
};
|
|
|
|
export function isTruthyRuntimeEnvValue(value: string | undefined) {
|
|
return typeof value === "string" && TRUTHY_RUNTIME_ENV_VALUES.has(value.trim().toLowerCase());
|
|
}
|
|
|
|
function getRuntimeInstanceId(env: Record<string, string | undefined>) {
|
|
const instanceId = env.PAPERCLIP_INSTANCE_ID?.trim();
|
|
return instanceId ? instanceId : null;
|
|
}
|
|
|
|
function stripServerManagedExperimentalPatchFields(
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
): PatchInstanceExperimentalSettings {
|
|
const {
|
|
worktreeRunExecutionActivatedAt: _ignoredActivatedAt,
|
|
worktreeRunExecutionActivationInstanceId: _ignoredActivationInstanceId,
|
|
...patchable
|
|
} = patch as Record<string, unknown>;
|
|
return patchable as PatchInstanceExperimentalSettings;
|
|
}
|
|
|
|
export function applyExperimentalSettingsPatch(
|
|
current: unknown,
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
options: InstanceSettingsServiceOptions = {},
|
|
): InstanceExperimentalSettings {
|
|
const previousExperimental = normalizeExperimentalSettings(current);
|
|
const patchable = stripServerManagedExperimentalPatchFields(patch);
|
|
const nextExperimental = normalizeExperimentalSettings({
|
|
...previousExperimental,
|
|
...patchable,
|
|
});
|
|
const hasWorktreeRunExecutionPatch = Object.prototype.hasOwnProperty.call(
|
|
patchable,
|
|
"enableWorktreeRunExecution",
|
|
);
|
|
|
|
if (!hasWorktreeRunExecutionPatch) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
if (nextExperimental.enableWorktreeRunExecution !== true) {
|
|
return {
|
|
...nextExperimental,
|
|
worktreeRunExecutionActivatedAt: null,
|
|
worktreeRunExecutionActivationInstanceId: null,
|
|
};
|
|
}
|
|
|
|
if (previousExperimental.enableWorktreeRunExecution === true) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
const runtimeEnv = options.runtimeEnv ?? process.env;
|
|
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
return {
|
|
...nextExperimental,
|
|
worktreeRunExecutionActivatedAt: (options.now ?? (() => new Date()))().toISOString(),
|
|
worktreeRunExecutionActivationInstanceId: getRuntimeInstanceId(runtimeEnv),
|
|
};
|
|
}
|
|
|
|
function suppressWorktreeRunExecution(
|
|
reason: WorktreeRunExecutionSuppressedReason,
|
|
activationInstanceId: string | null = null,
|
|
): WorktreeRunExecutionActivationState {
|
|
return {
|
|
armed: false,
|
|
cutoff: null,
|
|
activationInstanceId,
|
|
reason,
|
|
};
|
|
}
|
|
|
|
export function resolveWorktreeRunExecutionActivation(
|
|
experimental: InstanceExperimentalSettings,
|
|
currentInstanceId: string | null | undefined,
|
|
): WorktreeRunExecutionActivationState {
|
|
if (experimental.enableWorktreeRunExecution !== true) {
|
|
return suppressWorktreeRunExecution(
|
|
"flag_disabled",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (!experimental.worktreeRunExecutionActivatedAt) {
|
|
return suppressWorktreeRunExecution(
|
|
"missing_cutoff",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (!currentInstanceId) {
|
|
return suppressWorktreeRunExecution(
|
|
"missing_instance_id",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (experimental.worktreeRunExecutionActivationInstanceId !== currentInstanceId) {
|
|
return suppressWorktreeRunExecution(
|
|
"instance_id_mismatch",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
return {
|
|
armed: true,
|
|
cutoff: experimental.worktreeRunExecutionActivatedAt,
|
|
activationInstanceId: currentInstanceId,
|
|
reason: null,
|
|
};
|
|
}
|
|
|
|
export async function resolveWorktreeRunExecutionActivationState(options: {
|
|
getExperimental: () => Promise<InstanceExperimentalSettings>;
|
|
runtimeEnv?: Record<string, string | undefined>;
|
|
}): Promise<WorktreeRunExecutionActivationState> {
|
|
const runtimeEnv = options.runtimeEnv ?? process.env;
|
|
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
|
|
return suppressWorktreeRunExecution("not_worktree_runtime");
|
|
}
|
|
try {
|
|
return resolveWorktreeRunExecutionActivation(
|
|
await options.getExperimental(),
|
|
getRuntimeInstanceId(runtimeEnv),
|
|
);
|
|
} catch {
|
|
return suppressWorktreeRunExecution("settings_read_error");
|
|
}
|
|
}
|
|
|
|
function normalizeGeneralSettings(raw: unknown): InstanceGeneralSettings {
|
|
const parsed = instanceGeneralSettingsStorageSchema.safeParse(raw ?? {});
|
|
if (parsed.success) {
|
|
return {
|
|
censorUsernameInLogs: parsed.data.censorUsernameInLogs ?? false,
|
|
feedbackDataSharingPreference:
|
|
parsed.data.feedbackDataSharingPreference ?? DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
backupRetention: parsed.data.backupRetention ?? DEFAULT_BACKUP_RETENTION,
|
|
// Absent => unrestricted; only carry through an explicit policy.
|
|
...(parsed.data.executionMode ? { executionMode: parsed.data.executionMode } : {}),
|
|
};
|
|
}
|
|
return {
|
|
censorUsernameInLogs: false,
|
|
feedbackDataSharingPreference: DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
backupRetention: DEFAULT_BACKUP_RETENTION,
|
|
};
|
|
}
|
|
|
|
export function normalizeExperimentalSettings(raw: unknown): InstanceExperimentalSettings {
|
|
const parsed = instanceExperimentalSettingsStorageSchema.safeParse(raw ?? {});
|
|
if (parsed.success) {
|
|
return {
|
|
enableEnvironments: parsed.data.enableEnvironments ?? false,
|
|
enableNativeRunner: parsed.data.enableNativeRunner ?? true,
|
|
enableAiConnectionRouters: parsed.data.enableAiConnectionRouters ?? false,
|
|
enableManagedSandboxOnly: parsed.data.enableManagedSandboxOnly ?? false,
|
|
enableIsolatedWorkspaces: parsed.data.enableIsolatedWorkspaces ?? false,
|
|
enableIsolatedWorkspacesByDefault: parsed.data.enableIsolatedWorkspacesByDefault ?? false,
|
|
enableStreamlinedLeftNavigation: parsed.data.enableStreamlinedLeftNavigation ?? true,
|
|
enableStreamlinedUi: parsed.data.enableStreamlinedUi ?? true,
|
|
// Apps graduated from Experimental. Ignore historical off values while
|
|
// continuing to accept the compatibility key in stored settings.
|
|
enableApps: true,
|
|
enableMcpAggregators: true,
|
|
enablePublicMcp: parsed.data.enablePublicMcp ?? false,
|
|
enableOpenAiDot: parsed.data.enableOpenAiDot ?? false,
|
|
enableMuse: parsed.data.enableMuse ?? false,
|
|
enableGrokBot: parsed.data.enableGrokBot ?? false,
|
|
enableChatConnectors: parsed.data.enableChatConnectors ?? false,
|
|
enableGitHubReviewBots: parsed.data.enableGitHubReviewBots ?? false,
|
|
enableMemoryConnectors: parsed.data.enableMemoryConnectors ?? false,
|
|
enablePaperclipNotebook: parsed.data.enablePaperclipNotebook ?? false,
|
|
enableCases: parsed.data.enableCases ?? false,
|
|
enableAgentChat: parsed.data.enableAgentChat ?? false,
|
|
enableCombinedInboxTasks: parsed.data.enableCombinedInboxTasks ?? false,
|
|
enableArtifactsV1: parsed.data.enableArtifactsV1 ?? false,
|
|
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
|
|
enableClassicTaskInterface: parsed.data.enableClassicTaskInterface ?? false,
|
|
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
|
|
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
|
|
enableExternalObjects: parsed.data.enableExternalObjects ?? false,
|
|
enableSmokeLab: parsed.data.enableSmokeLab ?? false,
|
|
enableBuiltInAgents: parsed.data.enableBuiltInAgents ?? false,
|
|
enableBetaSkills: parsed.data.enableBetaSkills ?? false,
|
|
enableSummaries: parsed.data.enableSummaries ?? false,
|
|
enableStatusCards: parsed.data.enableStatusCards ?? false,
|
|
enableDecisions: parsed.data.enableDecisions ?? false,
|
|
enableFastResponses: parsed.data.enableFastResponses ?? false,
|
|
enableGoalsSidebarLink: parsed.data.enableGoalsSidebarLink ?? false,
|
|
enableServerInfoDebugView: parsed.data.enableServerInfoDebugView ?? false,
|
|
enablePaperclipDeveloperMode: parsed.data.enablePaperclipDeveloperMode ?? false,
|
|
enableSimplifiedEnglishInteractions: parsed.data.enableSimplifiedEnglishInteractions ?? false,
|
|
enableFirstTaskPlanProposal: parsed.data.enableFirstTaskPlanProposal ?? false,
|
|
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
|
|
enableWorkspaceBranchReconcileForward: parsed.data.enableWorkspaceBranchReconcileForward ?? true,
|
|
enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true,
|
|
enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false,
|
|
enableSandboxDuplexBridge: parsed.data.enableSandboxDuplexBridge ?? false,
|
|
enableRunnerPreviewIngress: parsed.data.enableRunnerPreviewIngress ?? false,
|
|
enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false,
|
|
worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null,
|
|
worktreeRunExecutionActivationInstanceId:
|
|
parsed.data.worktreeRunExecutionActivationInstanceId ?? null,
|
|
};
|
|
}
|
|
return {
|
|
enableEnvironments: false,
|
|
enableNativeRunner: true,
|
|
enableAiConnectionRouters: false,
|
|
enableManagedSandboxOnly: false,
|
|
enableIsolatedWorkspaces: false,
|
|
enableIsolatedWorkspacesByDefault: false,
|
|
enableStreamlinedLeftNavigation: true,
|
|
enableStreamlinedUi: true,
|
|
enableApps: true,
|
|
enableMcpAggregators: true,
|
|
enablePublicMcp: false,
|
|
enableOpenAiDot: false,
|
|
enableMuse: false,
|
|
enableGrokBot: false,
|
|
enableChatConnectors: false,
|
|
enableGitHubReviewBots: false,
|
|
enableMemoryConnectors: false,
|
|
enablePaperclipNotebook: false,
|
|
enableCases: false,
|
|
enableAgentChat: false,
|
|
enableCombinedInboxTasks: false,
|
|
enableArtifactsV1: false,
|
|
enableConferenceRoomChat: false,
|
|
enableClassicTaskInterface: false,
|
|
enableIssuePlanDecompositions: false,
|
|
enableExperimentalFileViewer: false,
|
|
enableExternalObjects: false,
|
|
enableSmokeLab: false,
|
|
enableBuiltInAgents: false,
|
|
enableBetaSkills: false,
|
|
enableSummaries: false,
|
|
enableStatusCards: false,
|
|
enableDecisions: false,
|
|
enableFastResponses: false,
|
|
enableGoalsSidebarLink: false,
|
|
enableServerInfoDebugView: false,
|
|
enablePaperclipDeveloperMode: false,
|
|
enableSimplifiedEnglishInteractions: false,
|
|
enableFirstTaskPlanProposal: false,
|
|
autoRestartDevServerWhenIdle: false,
|
|
enableWorkspaceBranchReconcileForward: true,
|
|
enableWorkspaceDirtyQuarantineRepair: true,
|
|
enableOwnerInstanceAdmin: false,
|
|
enableSandboxDuplexBridge: false,
|
|
enableRunnerPreviewIngress: false,
|
|
enableWorktreeRunExecution: false,
|
|
worktreeRunExecutionActivatedAt: null,
|
|
worktreeRunExecutionActivationInstanceId: null,
|
|
};
|
|
}
|
|
|
|
export type ManagedExperimentalKeyMetadata = Partial<
|
|
Record<ManagedExperimentalFeatureKey, ManagedSettingMetadata>
|
|
>;
|
|
|
|
/**
|
|
* Overlay the cloud managed-config feature values over normalized settings.
|
|
*
|
|
* Read-time precedence: code floor (cloud) > managed overlay > tenant DB
|
|
* value > schema default. (No code floors are expressed as flags today —
|
|
* floors are enforced in code at the guarded routes, independent of any
|
|
* flag value.) The overlay is deliberately never persisted: it re-asserts on
|
|
* every read, so a DB restore or manual row edit cannot resurrect a
|
|
* capability the harness has disabled.
|
|
*/
|
|
export function applyManagedExperimentalOverlay(
|
|
experimental: InstanceExperimentalSettings,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): { experimental: InstanceExperimentalSettings; managedKeys: ManagedExperimentalKeyMetadata } {
|
|
if (!managedConfig) return { experimental, managedKeys: {} };
|
|
const next: InstanceExperimentalSettings = { ...experimental };
|
|
const managedKeys: ManagedExperimentalKeyMetadata = {};
|
|
for (const [key, value] of Object.entries(managedConfig.features) as Array<
|
|
[ManagedExperimentalFeatureKey, boolean]
|
|
>) {
|
|
// Existing Cloud stack configs may still carry retired flags. Accept the
|
|
// document during rollout, but never let retired flags disable Apps or MCP aggregators.
|
|
if (key === "enableApps" || key === "enableMcpAggregators") continue;
|
|
next[key] = value;
|
|
managedKeys[key] = { managed: true, managedBy: PAPERCLIP_CLOUD_MANAGED_BY };
|
|
}
|
|
return { experimental: next, managedKeys };
|
|
}
|
|
|
|
/**
|
|
* Keep self-hosted-only defaults out of Cloud.
|
|
*
|
|
* The experimental schema carries one default per flag, and the feature
|
|
* catalog pins it to `selfHostedDefault`. A flag that is on by default for
|
|
* self-hosted but off by default for Cloud (`selfHostedDefault: true`,
|
|
* `cloudDefault: false`) would therefore normalize to "on" for a managed
|
|
* instance whose tenant row and managed overlay both leave it unset. Re-assert
|
|
* the declared Cloud default for exactly those flags. An explicit tenant value
|
|
* or a managed feature value still wins (the overlay is applied afterwards).
|
|
*/
|
|
export function applyCloudCatalogDefaults(
|
|
experimental: InstanceExperimentalSettings,
|
|
rawStored: unknown,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): InstanceExperimentalSettings {
|
|
if (!managedConfig) return experimental;
|
|
const stored =
|
|
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
|
|
? (rawStored as Record<string, unknown>)
|
|
: {};
|
|
const next: InstanceExperimentalSettings = { ...experimental };
|
|
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
|
|
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
|
|
if (typeof stored[key] === "boolean") continue;
|
|
if (typeof managedConfig.features[key as ManagedExperimentalFeatureKey] === "boolean") continue;
|
|
(next as unknown as Record<string, unknown>)[key] = false;
|
|
}
|
|
return next;
|
|
}
|
|
|
|
/**
|
|
* Keep the write path from freezing a self-hosted default into a Cloud row.
|
|
*
|
|
* `updateExperimental` persists the whole normalized object, and the schema
|
|
* normalizes an omitted flag to its self-hosted default. Without this step an
|
|
* unrelated experimental write (say, turning on Cases) would store
|
|
* `enableNativeRunner: true` on a managed instance whose tenant row had never
|
|
* mentioned the flag; every later read would then treat the stored boolean as
|
|
* an explicit tenant choice and stop re-asserting the Cloud default.
|
|
*
|
|
* For each guarded flag (see `applyCloudCatalogDefaults`), the stored key is
|
|
* left absent unless the tenant already stored a boolean or this patch sets
|
|
* the flag to something other than the Cloud default. A patch value equal to
|
|
* the Cloud default is a full-GET echo of the read-time overlay, not a
|
|
* choice, and is stripped the same way `stripOperatorGeneralEchoes` treats
|
|
* operator defaults. Self-hosted rows are returned untouched.
|
|
*/
|
|
export function stripCloudCatalogDefaultEchoes(
|
|
rawStored: unknown,
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
next: InstanceExperimentalSettings,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): Partial<InstanceExperimentalSettings> {
|
|
if (!managedConfig) return next;
|
|
const stored =
|
|
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
|
|
? (rawStored as Record<string, unknown>)
|
|
: {};
|
|
const patchRecord = patch as Record<string, unknown>;
|
|
const result: Record<string, unknown> = { ...next };
|
|
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
|
|
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
|
|
if (typeof stored[key] === "boolean") continue;
|
|
if (
|
|
Object.prototype.hasOwnProperty.call(patchRecord, key) &&
|
|
typeof patchRecord[key] === "boolean" &&
|
|
patchRecord[key] !== entry.cloudDefault
|
|
) {
|
|
continue;
|
|
}
|
|
delete result[key];
|
|
}
|
|
return result as Partial<InstanceExperimentalSettings>;
|
|
}
|
|
|
|
export function instanceSettingsService(db: Db, options: InstanceSettingsServiceOptions = {}) {
|
|
// Fail closed: a malformed PAPERCLIP_MANAGED_CONFIG throws here (and at
|
|
// boot in index.ts) rather than silently running without the overlay.
|
|
const managedConfig = getManagedInstanceConfig(options.runtimeEnv ?? process.env);
|
|
const staging = isCloudStagingInstance(options.runtimeEnv ?? process.env);
|
|
const enforcedManagedConfig = staging && managedConfig
|
|
? {
|
|
...managedConfig,
|
|
features: Object.fromEntries(Object.entries(managedConfig.features)
|
|
.filter(([key, enabled]) => enabled || !isCloudStagingExperimentalSetting(key))),
|
|
}
|
|
: managedConfig;
|
|
// Same posture for PAPERCLIP_SETTING_DEFAULTS: parsed once, applied per
|
|
// read, never persisted (see applyOperatorGeneralDefaults) — including on
|
|
// the write path, where a full-GET echo of the overlaid value is stripped
|
|
// back to the schema default (see stripOperatorGeneralEchoes).
|
|
const operatorDefaults = getOperatorSettingDefaults(options.runtimeEnv ?? process.env);
|
|
|
|
function toGeneralView(raw: unknown): InstanceGeneralSettings {
|
|
return applyOperatorGeneralDefaults(normalizeGeneralSettings(raw), operatorDefaults);
|
|
}
|
|
|
|
function toExperimentalView(raw: unknown): InstanceExperimentalSettingsWithManaged {
|
|
const stored = raw && typeof raw === "object" && !Array.isArray(raw) ? raw : {};
|
|
const defaults = staging ? { ...managedConfig?.features, ...stored } : raw;
|
|
const { experimental, managedKeys } = applyManagedExperimentalOverlay(
|
|
applyCloudCatalogDefaults(normalizeExperimentalSettings(defaults), defaults, managedConfig),
|
|
enforcedManagedConfig,
|
|
);
|
|
// Self-hosted responses stay byte-identical: no managedKeys field at all.
|
|
return managedConfig ? { ...experimental, managedKeys } : experimental;
|
|
}
|
|
|
|
function toInstanceSettings(row: typeof instanceSettings.$inferSelect): InstanceSettings {
|
|
return {
|
|
id: row.id,
|
|
defaultEnvironmentId: row.defaultEnvironmentId ?? null,
|
|
general: toGeneralView(row.general),
|
|
experimental: toExperimentalView(row.experimental),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
} as InstanceSettings;
|
|
}
|
|
async function getOrCreateRow(runner: InstanceSettingsWriteDb = db) {
|
|
const existing = await runner
|
|
.select()
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (existing) return existing;
|
|
|
|
const now = new Date();
|
|
const [created] = await runner
|
|
.insert(instanceSettings)
|
|
.values({
|
|
singletonKey: DEFAULT_SINGLETON_KEY,
|
|
general: {},
|
|
experimental: {},
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoUpdate({
|
|
target: [instanceSettings.singletonKey],
|
|
set: {
|
|
updatedAt: now,
|
|
},
|
|
})
|
|
.returning();
|
|
|
|
if (created) return created;
|
|
|
|
const raced = await runner
|
|
.select()
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (raced) return raced;
|
|
|
|
throw new Error("Failed to initialize instance settings row");
|
|
}
|
|
|
|
return {
|
|
get: async (): Promise<InstanceSettings> => toInstanceSettings(await getOrCreateRow()),
|
|
|
|
update: async (
|
|
patch: PatchInstanceSettings,
|
|
writeOptions?: { db?: InstanceSettingsWriteDb },
|
|
): Promise<InstanceSettings> => {
|
|
// The write may run inside a caller-supplied transaction so it commits
|
|
// atomically with a sibling write (e.g. clearing the managed-default
|
|
// stamp on the environment row alongside a defaultEnvironmentId
|
|
// change). Reads use the same runner so the row is visible to the tx.
|
|
const runner = writeOptions?.db ?? db;
|
|
const current = await getOrCreateRow(runner);
|
|
const now = new Date();
|
|
const [updated] = await runner
|
|
.update(instanceSettings)
|
|
.set({
|
|
...(Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId")
|
|
? { defaultEnvironmentId: patch.defaultEnvironmentId ?? null }
|
|
: {}),
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
getGeneral: async (
|
|
readOptions?: { db?: InstanceSettingsWriteDb },
|
|
): Promise<InstanceGeneralSettings> => {
|
|
const row = await getOrCreateRow(readOptions?.db);
|
|
return toGeneralView(row.general);
|
|
},
|
|
|
|
getExperimental: async (): Promise<InstanceExperimentalSettingsWithManaged> => {
|
|
const row = await getOrCreateRow();
|
|
return toExperimentalView(row.experimental);
|
|
},
|
|
|
|
updateGeneral: async (patch: PatchInstanceGeneralSettings): Promise<InstanceSettings> => {
|
|
const current = await getOrCreateRow();
|
|
const storedGeneral = normalizeGeneralSettings(current.general);
|
|
// A full-GET echo carries the overlaid operator value for a field the
|
|
// user never chose; stripping it keeps the overlay strictly read-time,
|
|
// so changing or unsetting the variable later still takes effect.
|
|
const nextGeneral = stripOperatorGeneralEchoes(
|
|
storedGeneral,
|
|
normalizeGeneralSettings({ ...storedGeneral, ...patch }),
|
|
operatorDefaults,
|
|
);
|
|
const now = new Date();
|
|
const [updated] = await db
|
|
.update(instanceSettings)
|
|
.set({
|
|
general: { ...nextGeneral },
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
updateExperimental: async (patch: PatchInstanceExperimentalSettings): Promise<InstanceSettings> => {
|
|
for (const [key, enabled] of Object.entries(managedConfig?.features ?? {})) {
|
|
if (enabled && patch[key as ManagedExperimentalFeatureKey] === false) {
|
|
throw forbidden(`${key} is enabled by Paperclip Cloud and cannot be disabled on this instance`, {
|
|
code: "settings_operator_managed",
|
|
});
|
|
}
|
|
}
|
|
const current = await getOrCreateRow();
|
|
// Guarded Cloud flags stay absent from the row unless chosen, so the
|
|
// read-time catalog default keeps applying (see stripCloudCatalogDefaultEchoes).
|
|
const nextExperimental = staging
|
|
? applyExperimentalSettingsPatch(toExperimentalView(current.experimental), patch, options)
|
|
: stripCloudCatalogDefaultEchoes(
|
|
current.experimental,
|
|
patch,
|
|
applyExperimentalSettingsPatch(current.experimental, patch, options),
|
|
managedConfig,
|
|
);
|
|
if (staging) {
|
|
const stored = current.experimental as Record<string, unknown> | null;
|
|
for (const key of Object.keys(INSTANCE_FEATURE_CATALOG) as ManagedExperimentalFeatureKey[]) {
|
|
if (typeof stored?.[key] !== "boolean" && !Object.prototype.hasOwnProperty.call(patch, key)) {
|
|
delete nextExperimental[key];
|
|
}
|
|
}
|
|
}
|
|
const now = new Date();
|
|
const [updated] = await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*");
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*");
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch);
|
|
return tx
|
|
.update(instanceSettings)
|
|
.set({
|
|
experimental: { ...nextExperimental },
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
});
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
listCompanyIds: async (): Promise<string[]> =>
|
|
db
|
|
.select({ id: companies.id })
|
|
.from(companies)
|
|
.then((rows) => rows.map((row) => row.id)),
|
|
};
|
|
}
|