mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server caches saved skill versions on disk so runs can restore their exact instructions. > - The old retention owner scans company rows every hour, even after all caches are protected or removed. > - Cache publication, version changes, and pin release already have local or transaction-owned boundaries. > - Grace windows and terminal controller leases already have saved timestamps. > - This pull request uses those changes and deadlines to drive the existing bounded prune service. > - Idle retention stops SQL and directory reads while pending cleanup remains recoverable. ## Linked Issues or Issue Description Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`). Related: #15625 establishes transaction-owned work signals. #17907 independently keeps an active cache-lock transaction alive; this PR preserves that lock and does not duplicate that repair. **What happened?** At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version retention has an hourly company scan, a startup timeout, and separate continuation timers. Empty or permanently protected caches still cause recurring SQL. **Expected behavior** Reconstruct once, inspect changed companies, and schedule saved access/staging grace or terminal controller deadlines. Stop recurring work when no timed or uncertain obligation remains. Keep current/newest/agent/run protection, restorable-byte validation, cache locking, final pin revalidation, atomic rename, and trash recovery. **Steps to reproduce** Start an enabled server with no cached versions, or only current/pinned versions. After the initial scan, advance an hour. The old interval selects companies again. The new worker has no deadline and performs no retention SQL or directory scan in that state. ## What Changed - Signal the missing-local-source reconciliation deletion in its transaction. Limit fresh scans of a successfully observed company to one per thirty seconds. Keep one-second page continuations and actual saved deadlines independent. Empty or uncertain passes do not delay first publication. - Restart a company scan if drain interrupts inspection or post-rename trash cleanup. Restore bounded candidate/byte logging with dry-run mode. Three controlled cases failed before repair and the complete 34-case retention suite passes afterward. - Register retention through the delivery coordinator and shared scheduler. Remove the hourly, startup, and continuation timers from `index.ts`. Track startup work without delaying readiness. Stop delivery work before closing retained iterators. - Keep one active company cursor and bounded 50-entry passes. Preserve observed deadlines across pages and bound company discovery to 51 rows per page. Retain concrete company failures and incomplete pins for retry. - Report saved grace and terminal controller deadlines from the existing cache owner. PostgreSQL remains the controller-lease authority. Current/newest/agent pins do not arm unrelated lease checks. - Wake on cache publication, rejected materialization, and access markers. Signal version creation/deletion and agent configuration/hire changes in their owning transactions. Extend existing instruction, finalization/environment, and retained-session fanouts with company hints. - Route reused provider-task prior-owner retirement through `writeEnvironmentCleanup`, so that supported pin-release write also settles a notification. - Add an optional scope callback to `registerDeadlineWork`. Existing consumers omit it and keep their behavior. The feature owns persistence predicates; startup only registers it. Add no table or migration. Full producer audit, shared-contract explanation, lifecycle evidence, and limitations: [`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). ## Verification - Reconciled all existing sleep changes with actual master `7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24 retention-owned and 95 unrelated upstream paths are preserved; package, UI, E2E and map trees equal master. - Combined-source qualification: **1,009 TypeScript cases across 42 complete files**, plus **28 Rust cases** in two targets. One intentional Linux-inotify-only adapter case is skipped on this macOS host. Coverage includes actual retention/cache/producer/runtime behavior, startup, shared deadline consumers, changed OpenCode/observability and native execution boundaries. - After the subsequent task-index/task-read upstream merges, **85 cases across six complete files** passed on final source `d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work, task reads/query behavior and concurrent migration repair. Counts overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413 journal entries through index 414. - Database/adapter/Runner builds, production server and Runner typechecking, module boundaries, target migration order and generated TypeScript/Rust release-profile consistency pass. Selected source hashes and temporary fixture imports restore exactly. - Reproduced and repaired Greptile's own-company due-cleanup P2 with identical old-worker-failing/new-worker-passing assertions. The 30-second ordinary scan floor remains; already-due cleanup receives its owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin acceptance remain in the audit. - **Inherited validation exception:** the separate Copilot source-identity check fails after upstream OpenCode reserved-environment changes; all 72 identity inputs equal current master. No provider release attestation is regenerated or new live qualification claimed here. Required exact-head CI/security and Greptile 5/5 still must pass. - Historical results, repair details and limitations remain in [the retention audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No full local repository-suite, live provider, production-load, external-alarm, measured suspend/wake or fleet-savings claim. - CI on the prior published head found four connection-review assertions racing the concealed conversation after reload. Its fresh Greptile/Ponytail reviews were both 5/5, but the failing browser gate was not waived or retried unchanged. The fixture now foregrounds the task tab and waits for visible history before retaining all existing saved-result assertions. The full file passes four journeys locally; the controlled-restart case is intentionally skipped by the default configuration. Original local journeys also passed unchanged, so no deterministic local red reproduction is claimed. See the audit for CI failure artifacts and scope. Runtime and prior sleep-test sources remain unchanged. ## Risks - Supported live writers are the existing service/API owners in one process. Independent SQL/FS writers need an ingress contract. Startup reconstruction covers restarts. - Notifications cause a bounded inspection; they do not authorize deletion. Durable pin predicates and the cache lock remain authoritative. - A database clock behind application time can require another one-second check of the concrete expired controller lease. Unknown pins, lock contention, and failed inspection retain retries until resolved. - The in-process scheduler cannot wake stopped compute. This PR preserves Cloud sleep-safety gates and external-alarm coverage. It does not prove deployed suspension or fleet savings. Cache reclamation can wait until a supported wake when compute is stopped. - Native-harness backup retention and the other remaining duties are separate stack slices. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL/filesystem integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
668 lines
28 KiB
TypeScript
668 lines
28 KiB
TypeScript
import { notifySkillVersionRetentionReconstruction } from "./skill-version-retention-notifications.js";
|
|
import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js";
|
|
import { notifyExternalAgentSettlement } from "../modules/external-agents/index.js";
|
|
import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js";
|
|
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
|
import { notifyNativeRunProjection } from "./native-run-projection-notifications.js";
|
|
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
|
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
|
|
import { notifyDirectoryCleanup } from "./directory-cleanup-notifications.js";
|
|
import { notifyInstructionCommit } from "./instruction-commit-notifications.js";
|
|
import { notifyScheduledRetryCancellation } from "./scheduled-retry-cancellation-notifications.js";
|
|
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
|
|
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { companies, instanceSettings } from "@paperclipai/db";
|
|
|
|
/**
|
|
* A `Db` or an open transaction handle — the subset of query builders the
|
|
* settings writes use. Lets `update` run inside a caller's transaction so
|
|
* it commits atomically with a sibling write.
|
|
*/
|
|
type InstanceSettingsTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
export type InstanceSettingsWriteDb = Pick<
|
|
Db | InstanceSettingsTransaction,
|
|
"select" | "insert" | "update"
|
|
>;
|
|
import {
|
|
DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
DEFAULT_BACKUP_RETENTION,
|
|
PAPERCLIP_CLOUD_MANAGED_BY,
|
|
instanceGeneralSettingsSchema,
|
|
type InstanceGeneralSettings,
|
|
instanceExperimentalSettingsSchema,
|
|
type InstanceExperimentalSettings,
|
|
type InstanceExperimentalSettingsWithManaged,
|
|
type ManagedExperimentalFeatureKey,
|
|
type ManagedSettingMetadata,
|
|
type PatchInstanceGeneralSettings,
|
|
type InstanceSettings,
|
|
type PatchInstanceSettings,
|
|
type PatchInstanceExperimentalSettings,
|
|
} from "@paperclipai/shared";
|
|
import {
|
|
INSTANCE_FEATURE_CATALOG,
|
|
applyOperatorGeneralDefaults,
|
|
stripOperatorGeneralEchoes,
|
|
} from "@paperclipai/shared";
|
|
import { eq } from "drizzle-orm";
|
|
import { getManagedInstanceConfig, type ManagedInstanceConfig } from "./managed-config.js";
|
|
import { getOperatorSettingDefaults } from "./setting-defaults.js";
|
|
import { isCloudStagingExperimentalSetting, isCloudStagingInstance } from "./cloud-instance.js";
|
|
import { forbidden } from "../errors.js";
|
|
|
|
const DEFAULT_SINGLETON_KEY = "default";
|
|
const instanceGeneralSettingsStorageSchema = instanceGeneralSettingsSchema.strip();
|
|
const instanceExperimentalSettingsStorageSchema = instanceExperimentalSettingsSchema.strip();
|
|
const TRUTHY_RUNTIME_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
|
|
|
|
interface InstanceSettingsServiceOptions {
|
|
runtimeEnv?: Record<string, string | undefined>;
|
|
now?: () => Date;
|
|
}
|
|
|
|
type WorktreeRunExecutionSuppressedReason =
|
|
| "not_worktree_runtime"
|
|
| "flag_disabled"
|
|
| "missing_cutoff"
|
|
| "missing_instance_id"
|
|
| "instance_id_mismatch"
|
|
| "settings_read_error";
|
|
|
|
export type WorktreeRunExecutionActivationState =
|
|
| {
|
|
armed: true;
|
|
cutoff: string;
|
|
activationInstanceId: string;
|
|
reason: null;
|
|
}
|
|
| {
|
|
armed: false;
|
|
cutoff: null;
|
|
activationInstanceId: string | null;
|
|
reason: WorktreeRunExecutionSuppressedReason;
|
|
};
|
|
|
|
export function isTruthyRuntimeEnvValue(value: string | undefined) {
|
|
return typeof value === "string" && TRUTHY_RUNTIME_ENV_VALUES.has(value.trim().toLowerCase());
|
|
}
|
|
|
|
function getRuntimeInstanceId(env: Record<string, string | undefined>) {
|
|
const instanceId = env.PAPERCLIP_INSTANCE_ID?.trim();
|
|
return instanceId ? instanceId : null;
|
|
}
|
|
|
|
function stripServerManagedExperimentalPatchFields(
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
): PatchInstanceExperimentalSettings {
|
|
const {
|
|
worktreeRunExecutionActivatedAt: _ignoredActivatedAt,
|
|
worktreeRunExecutionActivationInstanceId: _ignoredActivationInstanceId,
|
|
...patchable
|
|
} = patch as Record<string, unknown>;
|
|
return patchable as PatchInstanceExperimentalSettings;
|
|
}
|
|
|
|
export function applyExperimentalSettingsPatch(
|
|
current: unknown,
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
options: InstanceSettingsServiceOptions = {},
|
|
): InstanceExperimentalSettings {
|
|
const previousExperimental = normalizeExperimentalSettings(current);
|
|
const patchable = stripServerManagedExperimentalPatchFields(patch);
|
|
const nextExperimental = normalizeExperimentalSettings({
|
|
...previousExperimental,
|
|
...patchable,
|
|
});
|
|
const hasWorktreeRunExecutionPatch = Object.prototype.hasOwnProperty.call(
|
|
patchable,
|
|
"enableWorktreeRunExecution",
|
|
);
|
|
|
|
if (!hasWorktreeRunExecutionPatch) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
if (nextExperimental.enableWorktreeRunExecution !== true) {
|
|
return {
|
|
...nextExperimental,
|
|
worktreeRunExecutionActivatedAt: null,
|
|
worktreeRunExecutionActivationInstanceId: null,
|
|
};
|
|
}
|
|
|
|
if (previousExperimental.enableWorktreeRunExecution === true) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
const runtimeEnv = options.runtimeEnv ?? process.env;
|
|
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
|
|
return nextExperimental;
|
|
}
|
|
|
|
return {
|
|
...nextExperimental,
|
|
worktreeRunExecutionActivatedAt: (options.now ?? (() => new Date()))().toISOString(),
|
|
worktreeRunExecutionActivationInstanceId: getRuntimeInstanceId(runtimeEnv),
|
|
};
|
|
}
|
|
|
|
function suppressWorktreeRunExecution(
|
|
reason: WorktreeRunExecutionSuppressedReason,
|
|
activationInstanceId: string | null = null,
|
|
): WorktreeRunExecutionActivationState {
|
|
return {
|
|
armed: false,
|
|
cutoff: null,
|
|
activationInstanceId,
|
|
reason,
|
|
};
|
|
}
|
|
|
|
export function resolveWorktreeRunExecutionActivation(
|
|
experimental: InstanceExperimentalSettings,
|
|
currentInstanceId: string | null | undefined,
|
|
): WorktreeRunExecutionActivationState {
|
|
if (experimental.enableWorktreeRunExecution !== true) {
|
|
return suppressWorktreeRunExecution(
|
|
"flag_disabled",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (!experimental.worktreeRunExecutionActivatedAt) {
|
|
return suppressWorktreeRunExecution(
|
|
"missing_cutoff",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (!currentInstanceId) {
|
|
return suppressWorktreeRunExecution(
|
|
"missing_instance_id",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
if (experimental.worktreeRunExecutionActivationInstanceId !== currentInstanceId) {
|
|
return suppressWorktreeRunExecution(
|
|
"instance_id_mismatch",
|
|
experimental.worktreeRunExecutionActivationInstanceId,
|
|
);
|
|
}
|
|
return {
|
|
armed: true,
|
|
cutoff: experimental.worktreeRunExecutionActivatedAt,
|
|
activationInstanceId: currentInstanceId,
|
|
reason: null,
|
|
};
|
|
}
|
|
|
|
export async function resolveWorktreeRunExecutionActivationState(options: {
|
|
getExperimental: () => Promise<InstanceExperimentalSettings>;
|
|
runtimeEnv?: Record<string, string | undefined>;
|
|
}): Promise<WorktreeRunExecutionActivationState> {
|
|
const runtimeEnv = options.runtimeEnv ?? process.env;
|
|
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
|
|
return suppressWorktreeRunExecution("not_worktree_runtime");
|
|
}
|
|
try {
|
|
return resolveWorktreeRunExecutionActivation(
|
|
await options.getExperimental(),
|
|
getRuntimeInstanceId(runtimeEnv),
|
|
);
|
|
} catch {
|
|
return suppressWorktreeRunExecution("settings_read_error");
|
|
}
|
|
}
|
|
|
|
function normalizeGeneralSettings(raw: unknown): InstanceGeneralSettings {
|
|
const parsed = instanceGeneralSettingsStorageSchema.safeParse(raw ?? {});
|
|
if (parsed.success) {
|
|
return {
|
|
censorUsernameInLogs: parsed.data.censorUsernameInLogs ?? false,
|
|
feedbackDataSharingPreference:
|
|
parsed.data.feedbackDataSharingPreference ?? DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
backupRetention: parsed.data.backupRetention ?? DEFAULT_BACKUP_RETENTION,
|
|
// Absent => unrestricted; only carry through an explicit policy.
|
|
...(parsed.data.executionMode ? { executionMode: parsed.data.executionMode } : {}),
|
|
};
|
|
}
|
|
return {
|
|
censorUsernameInLogs: false,
|
|
feedbackDataSharingPreference: DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
|
|
backupRetention: DEFAULT_BACKUP_RETENTION,
|
|
};
|
|
}
|
|
|
|
export function normalizeExperimentalSettings(raw: unknown): InstanceExperimentalSettings {
|
|
const parsed = instanceExperimentalSettingsStorageSchema.safeParse(raw ?? {});
|
|
if (parsed.success) {
|
|
return {
|
|
enableEnvironments: parsed.data.enableEnvironments ?? false,
|
|
enableNativeRunner: parsed.data.enableNativeRunner ?? true,
|
|
enableAiConnectionRouters: parsed.data.enableAiConnectionRouters ?? false,
|
|
enableManagedSandboxOnly: parsed.data.enableManagedSandboxOnly ?? false,
|
|
enableIsolatedWorkspaces: parsed.data.enableIsolatedWorkspaces ?? false,
|
|
enableIsolatedWorkspacesByDefault: parsed.data.enableIsolatedWorkspacesByDefault ?? false,
|
|
enableStreamlinedLeftNavigation: parsed.data.enableStreamlinedLeftNavigation ?? true,
|
|
enableStreamlinedUi: parsed.data.enableStreamlinedUi ?? true,
|
|
// Apps graduated from Experimental. Ignore historical off values while
|
|
// continuing to accept the compatibility key in stored settings.
|
|
enableApps: true,
|
|
enableMcpAggregators: true,
|
|
enablePublicMcp: parsed.data.enablePublicMcp ?? false,
|
|
enableOpenAiDot: parsed.data.enableOpenAiDot ?? false,
|
|
enableMuse: parsed.data.enableMuse ?? false,
|
|
enableGrokBot: parsed.data.enableGrokBot ?? false,
|
|
enableChatConnectors: parsed.data.enableChatConnectors ?? false,
|
|
enableGitHubReviewBots: parsed.data.enableGitHubReviewBots ?? false,
|
|
enableMemoryConnectors: parsed.data.enableMemoryConnectors ?? false,
|
|
enablePaperclipNotebook: parsed.data.enablePaperclipNotebook ?? false,
|
|
enableCases: parsed.data.enableCases ?? false,
|
|
enableAgentChat: parsed.data.enableAgentChat ?? false,
|
|
enableCombinedInboxTasks: parsed.data.enableCombinedInboxTasks ?? false,
|
|
enableArtifactsV1: parsed.data.enableArtifactsV1 ?? false,
|
|
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
|
|
enableClassicTaskInterface: parsed.data.enableClassicTaskInterface ?? false,
|
|
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
|
|
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
|
|
enableExternalObjects: parsed.data.enableExternalObjects ?? false,
|
|
enableSmokeLab: parsed.data.enableSmokeLab ?? false,
|
|
enableBuiltInAgents: parsed.data.enableBuiltInAgents ?? false,
|
|
enableBetaSkills: parsed.data.enableBetaSkills ?? false,
|
|
enableSummaries: parsed.data.enableSummaries ?? false,
|
|
enableStatusCards: parsed.data.enableStatusCards ?? false,
|
|
enableDecisions: parsed.data.enableDecisions ?? false,
|
|
enableFastResponses: parsed.data.enableFastResponses ?? false,
|
|
enableGoalsSidebarLink: parsed.data.enableGoalsSidebarLink ?? false,
|
|
enableServerInfoDebugView: parsed.data.enableServerInfoDebugView ?? false,
|
|
enablePaperclipDeveloperMode: parsed.data.enablePaperclipDeveloperMode ?? false,
|
|
enableSimplifiedEnglishInteractions: parsed.data.enableSimplifiedEnglishInteractions ?? false,
|
|
enableFirstTaskPlanProposal: parsed.data.enableFirstTaskPlanProposal ?? false,
|
|
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
|
|
enableWorkspaceBranchReconcileForward: parsed.data.enableWorkspaceBranchReconcileForward ?? true,
|
|
enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true,
|
|
enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false,
|
|
enableSandboxDuplexBridge: parsed.data.enableSandboxDuplexBridge ?? false,
|
|
enableRunnerPreviewIngress: parsed.data.enableRunnerPreviewIngress ?? false,
|
|
enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false,
|
|
worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null,
|
|
worktreeRunExecutionActivationInstanceId:
|
|
parsed.data.worktreeRunExecutionActivationInstanceId ?? null,
|
|
};
|
|
}
|
|
return {
|
|
enableEnvironments: false,
|
|
enableNativeRunner: true,
|
|
enableAiConnectionRouters: false,
|
|
enableManagedSandboxOnly: false,
|
|
enableIsolatedWorkspaces: false,
|
|
enableIsolatedWorkspacesByDefault: false,
|
|
enableStreamlinedLeftNavigation: true,
|
|
enableStreamlinedUi: true,
|
|
enableApps: true,
|
|
enableMcpAggregators: true,
|
|
enablePublicMcp: false,
|
|
enableOpenAiDot: false,
|
|
enableMuse: false,
|
|
enableGrokBot: false,
|
|
enableChatConnectors: false,
|
|
enableGitHubReviewBots: false,
|
|
enableMemoryConnectors: false,
|
|
enablePaperclipNotebook: false,
|
|
enableCases: false,
|
|
enableAgentChat: false,
|
|
enableCombinedInboxTasks: false,
|
|
enableArtifactsV1: false,
|
|
enableConferenceRoomChat: false,
|
|
enableClassicTaskInterface: false,
|
|
enableIssuePlanDecompositions: false,
|
|
enableExperimentalFileViewer: false,
|
|
enableExternalObjects: false,
|
|
enableSmokeLab: false,
|
|
enableBuiltInAgents: false,
|
|
enableBetaSkills: false,
|
|
enableSummaries: false,
|
|
enableStatusCards: false,
|
|
enableDecisions: false,
|
|
enableFastResponses: false,
|
|
enableGoalsSidebarLink: false,
|
|
enableServerInfoDebugView: false,
|
|
enablePaperclipDeveloperMode: false,
|
|
enableSimplifiedEnglishInteractions: false,
|
|
enableFirstTaskPlanProposal: false,
|
|
autoRestartDevServerWhenIdle: false,
|
|
enableWorkspaceBranchReconcileForward: true,
|
|
enableWorkspaceDirtyQuarantineRepair: true,
|
|
enableOwnerInstanceAdmin: false,
|
|
enableSandboxDuplexBridge: false,
|
|
enableRunnerPreviewIngress: false,
|
|
enableWorktreeRunExecution: false,
|
|
worktreeRunExecutionActivatedAt: null,
|
|
worktreeRunExecutionActivationInstanceId: null,
|
|
};
|
|
}
|
|
|
|
export type ManagedExperimentalKeyMetadata = Partial<
|
|
Record<ManagedExperimentalFeatureKey, ManagedSettingMetadata>
|
|
>;
|
|
|
|
/**
|
|
* Overlay the cloud managed-config feature values over normalized settings.
|
|
*
|
|
* Read-time precedence: code floor (cloud) > managed overlay > tenant DB
|
|
* value > schema default. (No code floors are expressed as flags today —
|
|
* floors are enforced in code at the guarded routes, independent of any
|
|
* flag value.) The overlay is deliberately never persisted: it re-asserts on
|
|
* every read, so a DB restore or manual row edit cannot resurrect a
|
|
* capability the harness has disabled.
|
|
*/
|
|
export function applyManagedExperimentalOverlay(
|
|
experimental: InstanceExperimentalSettings,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): { experimental: InstanceExperimentalSettings; managedKeys: ManagedExperimentalKeyMetadata } {
|
|
if (!managedConfig) return { experimental, managedKeys: {} };
|
|
const next: InstanceExperimentalSettings = { ...experimental };
|
|
const managedKeys: ManagedExperimentalKeyMetadata = {};
|
|
for (const [key, value] of Object.entries(managedConfig.features) as Array<
|
|
[ManagedExperimentalFeatureKey, boolean]
|
|
>) {
|
|
// Existing Cloud stack configs may still carry retired flags. Accept the
|
|
// document during rollout, but never let retired flags disable Apps or MCP aggregators.
|
|
if (key === "enableApps" || key === "enableMcpAggregators") continue;
|
|
next[key] = value;
|
|
managedKeys[key] = { managed: true, managedBy: PAPERCLIP_CLOUD_MANAGED_BY };
|
|
}
|
|
return { experimental: next, managedKeys };
|
|
}
|
|
|
|
/**
|
|
* Keep self-hosted-only defaults out of Cloud.
|
|
*
|
|
* The experimental schema carries one default per flag, and the feature
|
|
* catalog pins it to `selfHostedDefault`. A flag that is on by default for
|
|
* self-hosted but off by default for Cloud (`selfHostedDefault: true`,
|
|
* `cloudDefault: false`) would therefore normalize to "on" for a managed
|
|
* instance whose tenant row and managed overlay both leave it unset. Re-assert
|
|
* the declared Cloud default for exactly those flags. An explicit tenant value
|
|
* or a managed feature value still wins (the overlay is applied afterwards).
|
|
*/
|
|
export function applyCloudCatalogDefaults(
|
|
experimental: InstanceExperimentalSettings,
|
|
rawStored: unknown,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): InstanceExperimentalSettings {
|
|
if (!managedConfig) return experimental;
|
|
const stored =
|
|
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
|
|
? (rawStored as Record<string, unknown>)
|
|
: {};
|
|
const next: InstanceExperimentalSettings = { ...experimental };
|
|
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
|
|
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
|
|
if (typeof stored[key] === "boolean") continue;
|
|
if (typeof managedConfig.features[key as ManagedExperimentalFeatureKey] === "boolean") continue;
|
|
(next as unknown as Record<string, unknown>)[key] = false;
|
|
}
|
|
return next;
|
|
}
|
|
|
|
/**
|
|
* Keep the write path from freezing a self-hosted default into a Cloud row.
|
|
*
|
|
* `updateExperimental` persists the whole normalized object, and the schema
|
|
* normalizes an omitted flag to its self-hosted default. Without this step an
|
|
* unrelated experimental write (say, turning on Cases) would store
|
|
* `enableNativeRunner: true` on a managed instance whose tenant row had never
|
|
* mentioned the flag; every later read would then treat the stored boolean as
|
|
* an explicit tenant choice and stop re-asserting the Cloud default.
|
|
*
|
|
* For each guarded flag (see `applyCloudCatalogDefaults`), the stored key is
|
|
* left absent unless the tenant already stored a boolean or this patch sets
|
|
* the flag to something other than the Cloud default. A patch value equal to
|
|
* the Cloud default is a full-GET echo of the read-time overlay, not a
|
|
* choice, and is stripped the same way `stripOperatorGeneralEchoes` treats
|
|
* operator defaults. Self-hosted rows are returned untouched.
|
|
*/
|
|
export function stripCloudCatalogDefaultEchoes(
|
|
rawStored: unknown,
|
|
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
|
|
next: InstanceExperimentalSettings,
|
|
managedConfig: ManagedInstanceConfig | null,
|
|
): Partial<InstanceExperimentalSettings> {
|
|
if (!managedConfig) return next;
|
|
const stored =
|
|
rawStored && typeof rawStored === "object" && !Array.isArray(rawStored)
|
|
? (rawStored as Record<string, unknown>)
|
|
: {};
|
|
const patchRecord = patch as Record<string, unknown>;
|
|
const result: Record<string, unknown> = { ...next };
|
|
for (const [key, entry] of Object.entries(INSTANCE_FEATURE_CATALOG)) {
|
|
if (entry.cloudDefault !== false || entry.selfHostedDefault !== true) continue;
|
|
if (typeof stored[key] === "boolean") continue;
|
|
if (
|
|
Object.prototype.hasOwnProperty.call(patchRecord, key) &&
|
|
typeof patchRecord[key] === "boolean" &&
|
|
patchRecord[key] !== entry.cloudDefault
|
|
) {
|
|
continue;
|
|
}
|
|
delete result[key];
|
|
}
|
|
return result as Partial<InstanceExperimentalSettings>;
|
|
}
|
|
|
|
export function instanceSettingsService(db: Db, options: InstanceSettingsServiceOptions = {}) {
|
|
// Fail closed: a malformed PAPERCLIP_MANAGED_CONFIG throws here (and at
|
|
// boot in index.ts) rather than silently running without the overlay.
|
|
const managedConfig = getManagedInstanceConfig(options.runtimeEnv ?? process.env);
|
|
const staging = isCloudStagingInstance(options.runtimeEnv ?? process.env);
|
|
const enforcedManagedConfig = staging && managedConfig
|
|
? {
|
|
...managedConfig,
|
|
features: Object.fromEntries(Object.entries(managedConfig.features)
|
|
.filter(([key, enabled]) => enabled || !isCloudStagingExperimentalSetting(key))),
|
|
}
|
|
: managedConfig;
|
|
// Same posture for PAPERCLIP_SETTING_DEFAULTS: parsed once, applied per
|
|
// read, never persisted (see applyOperatorGeneralDefaults) — including on
|
|
// the write path, where a full-GET echo of the overlaid value is stripped
|
|
// back to the schema default (see stripOperatorGeneralEchoes).
|
|
const operatorDefaults = getOperatorSettingDefaults(options.runtimeEnv ?? process.env);
|
|
|
|
function toGeneralView(raw: unknown): InstanceGeneralSettings {
|
|
return applyOperatorGeneralDefaults(normalizeGeneralSettings(raw), operatorDefaults);
|
|
}
|
|
|
|
function toExperimentalView(raw: unknown): InstanceExperimentalSettingsWithManaged {
|
|
const stored = raw && typeof raw === "object" && !Array.isArray(raw) ? raw : {};
|
|
const defaults = staging ? { ...managedConfig?.features, ...stored } : raw;
|
|
const { experimental, managedKeys } = applyManagedExperimentalOverlay(
|
|
applyCloudCatalogDefaults(normalizeExperimentalSettings(defaults), defaults, managedConfig),
|
|
enforcedManagedConfig,
|
|
);
|
|
// Self-hosted responses stay byte-identical: no managedKeys field at all.
|
|
return managedConfig ? { ...experimental, managedKeys } : experimental;
|
|
}
|
|
|
|
function toInstanceSettings(row: typeof instanceSettings.$inferSelect): InstanceSettings {
|
|
return {
|
|
id: row.id,
|
|
defaultEnvironmentId: row.defaultEnvironmentId ?? null,
|
|
general: toGeneralView(row.general),
|
|
experimental: toExperimentalView(row.experimental),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
} as InstanceSettings;
|
|
}
|
|
async function getOrCreateRow(runner: InstanceSettingsWriteDb = db) {
|
|
const existing = await runner
|
|
.select()
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (existing) return existing;
|
|
|
|
const now = new Date();
|
|
const [created] = await runner
|
|
.insert(instanceSettings)
|
|
.values({
|
|
singletonKey: DEFAULT_SINGLETON_KEY,
|
|
general: {},
|
|
experimental: {},
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoUpdate({
|
|
target: [instanceSettings.singletonKey],
|
|
set: {
|
|
updatedAt: now,
|
|
},
|
|
})
|
|
.returning();
|
|
|
|
if (created) return created;
|
|
|
|
const raced = await runner
|
|
.select()
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (raced) return raced;
|
|
|
|
throw new Error("Failed to initialize instance settings row");
|
|
}
|
|
|
|
return {
|
|
get: async (): Promise<InstanceSettings> => toInstanceSettings(await getOrCreateRow()),
|
|
|
|
update: async (
|
|
patch: PatchInstanceSettings,
|
|
writeOptions?: { db?: InstanceSettingsWriteDb },
|
|
): Promise<InstanceSettings> => {
|
|
// The write may run inside a caller-supplied transaction so it commits
|
|
// atomically with a sibling write (e.g. clearing the managed-default
|
|
// stamp on the environment row alongside a defaultEnvironmentId
|
|
// change). Reads use the same runner so the row is visible to the tx.
|
|
const runner = writeOptions?.db ?? db;
|
|
const current = await getOrCreateRow(runner);
|
|
const now = new Date();
|
|
const [updated] = await runner
|
|
.update(instanceSettings)
|
|
.set({
|
|
...(Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId")
|
|
? { defaultEnvironmentId: patch.defaultEnvironmentId ?? null }
|
|
: {}),
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
getGeneral: async (
|
|
readOptions?: { db?: InstanceSettingsWriteDb },
|
|
): Promise<InstanceGeneralSettings> => {
|
|
const row = await getOrCreateRow(readOptions?.db);
|
|
return toGeneralView(row.general);
|
|
},
|
|
|
|
getExperimental: async (): Promise<InstanceExperimentalSettingsWithManaged> => {
|
|
const row = await getOrCreateRow();
|
|
return toExperimentalView(row.experimental);
|
|
},
|
|
|
|
updateGeneral: async (patch: PatchInstanceGeneralSettings): Promise<InstanceSettings> => {
|
|
const current = await getOrCreateRow();
|
|
const storedGeneral = normalizeGeneralSettings(current.general);
|
|
// A full-GET echo carries the overlaid operator value for a field the
|
|
// user never chose; stripping it keeps the overlay strictly read-time,
|
|
// so changing or unsetting the variable later still takes effect.
|
|
const nextGeneral = stripOperatorGeneralEchoes(
|
|
storedGeneral,
|
|
normalizeGeneralSettings({ ...storedGeneral, ...patch }),
|
|
operatorDefaults,
|
|
);
|
|
const now = new Date();
|
|
const [updated] = await db
|
|
.update(instanceSettings)
|
|
.set({
|
|
general: { ...nextGeneral },
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
updateExperimental: async (patch: PatchInstanceExperimentalSettings): Promise<InstanceSettings> => {
|
|
for (const [key, enabled] of Object.entries(managedConfig?.features ?? {})) {
|
|
if (enabled && patch[key as ManagedExperimentalFeatureKey] === false) {
|
|
throw forbidden(`${key} is enabled by Paperclip Cloud and cannot be disabled on this instance`, {
|
|
code: "settings_operator_managed",
|
|
});
|
|
}
|
|
}
|
|
const current = await getOrCreateRow();
|
|
// Guarded Cloud flags stay absent from the row unless chosen, so the
|
|
// read-time catalog default keeps applying (see stripCloudCatalogDefaultEchoes).
|
|
const nextExperimental = staging
|
|
? applyExperimentalSettingsPatch(toExperimentalView(current.experimental), patch, options)
|
|
: stripCloudCatalogDefaultEchoes(
|
|
current.experimental,
|
|
patch,
|
|
applyExperimentalSettingsPatch(current.experimental, patch, options),
|
|
managedConfig,
|
|
);
|
|
if (staging) {
|
|
const stored = current.experimental as Record<string, unknown> | null;
|
|
for (const key of Object.keys(INSTANCE_FEATURE_CATALOG) as ManagedExperimentalFeatureKey[]) {
|
|
if (typeof stored?.[key] !== "boolean" && !Object.prototype.hasOwnProperty.call(patch, key)) {
|
|
delete nextExperimental[key];
|
|
}
|
|
}
|
|
}
|
|
const now = new Date();
|
|
const [updated] = await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*");
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*");
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch);
|
|
if (patch.enableWorktreeRunExecution !== undefined) await notifyScheduledRetryCancellation(tx);
|
|
if (patch.enableWorktreeRunExecution !== undefined) {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyExternalAgentSettlement(tx);
|
|
await notifyInstructionCommit(tx);
|
|
await notifyDirectoryCleanup(tx);
|
|
await notifySkillVersionRetentionReconstruction(tx);
|
|
await notifyInstructionCollection(tx);
|
|
await notifyCancelledWorkspacePreservation(tx);
|
|
await notifyExternalConversationSettlement(tx);
|
|
await notifyNativeRecoveryNotices(tx);
|
|
await notifyNativeRunProjection(tx);
|
|
await notifyNativeEnvironmentSettlement(tx);
|
|
await notifyNativeResultCommit(tx);
|
|
await notifyNativeWorkspaceFinalization(tx);
|
|
}
|
|
if (patch.enableChatConnectors !== undefined) await notifyVoiceChange(tx);
|
|
await notifyToolConnectionHealth(tx, "*");
|
|
return tx
|
|
.update(instanceSettings)
|
|
.set({
|
|
experimental: { ...nextExperimental },
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(instanceSettings.id, current.id))
|
|
.returning();
|
|
});
|
|
return toInstanceSettings(updated ?? current);
|
|
},
|
|
|
|
listCompanyIds: async (): Promise<string[]> =>
|
|
db
|
|
.select({ id: companies.id })
|
|
.from(companies)
|
|
.then((rows) => rows.map((row) => row.id)),
|
|
};
|
|
}
|