Files
PaperClipAI/server/src/services/instance-settings.ts
T
Devin Foley 916c13501f Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A company accumulates real state — issues, labels, blockers,
documents, work products, monitors, attachments, agents, routines — and
people need to move that state between instances: self-hosted to cloud,
cloud back to self-hosted, or plain backups
> - The experimental, flag-gated Cloud Sync transport (#6548) tried to
solve this host-to-host: the source pushed into a receiver over HTTPS
with a cross-instance consent/token handshake, which required the
destination to be publicly reachable and broke for common self-hosted
topologies (plain-HTTP LAN/VPN origins); the receiver half never landed
upstream at all
> - Meanwhile the portability bundle and the existing export/import
pages already move companies offline with none of those networking
constraints — but silently dropped labels, blockers, issue documents,
work products, monitors, and every attachment
> - This pull request removes the host-to-host transport and makes
Import/Export the single data-movement path: the pages become
first-class company-settings destinations, exports declare exactly what
they do not carry, and bundle schemaVersion 6 now carries all of the
above, with attachments as content-addressed sha256 blobs verified
before a single row is written
> - The benefit is a migration and backup flow that works between any
two instances with no reachability requirements, no cross-instance auth,
and no silent data loss

## Linked Issues or Issue Description

- Refs #6548 — the original Cloud Sync sender this PR supersedes and
removes.
- Related, not duplicates: #1697 (goals in the portability manifest —
orthogonal field addition), #954 (an earlier import/export +
skill-visibility proposal predating the current portability bundle).
- No open issue describes this directly, so in brief (feature-request
shape): **Problem** — moving a company between instances silently lost
labels (imports with label references actually hard-failed), blocker
relations, issue documents, work products, monitor state, and all
attachments, and the alternative Cloud Sync transport required the
destination to be publicly reachable over HTTPS plus a consent
handshake, which failed for typical self-hosted setups. **Desired
behavior** — one Import/Export flow in company settings that produces a
portable bundle carrying all of that data, tells the operator up front
what it cannot carry, imports with automations paused, and offers real
one-click activation afterwards.

## What Changed

- New export fidelity report (`GET
/api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel
on the Export page listing anything a bundle will not include (now only:
approvals, cost history, activity history)
- Imports accept `pauseAutomations`; imported agents and routines land
paused, the import result reports created routines, and the Import page
ends in an activation panel that actually resumes selected
agents/activates routines
- Export and Import pages promoted into the company-settings nav; the
Cloud Upstream wizard, ux-lab page, and API client removed; the old
settings route redirects to Export
- Host-to-host transport removed: upstream-sync/receiver-client routes
and services, CLI `cloud connect`/`cloud push` + keypair store, the
shared upstream transfer contract, and the `enableCloudSync` flag;
migration `0196` drops the two experimental `cloud_upstream_*` sender
tables
- Bundle schemaVersion 6: labels (definitions + per-task names, remapped
by name on import), blocker relations (`blockedBy` slugs,
cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`),
work products (system refs nulled), monitors (notes/scheduledBy
restored, imported un-armed)
- Attachments travel as content-addressed `blobs/<sha256>` entries
(deduped; comment-scoped attachments re-link via comment index); every
blob is hash-verified **before any write**, so a corrupted bundle cannot
leave a partially imported company; both zip codecs now round-trip
extensionless/binary entries byte-exactly; the Import page preflights
the inline body limit and offers continue-without-attachments
- v5 (and older) bundles still import, with an informational warning;
bundles newer than v6 are rejected cleanly
- Docs: board-operator import/export guide, CLI README, README/ROADMAP
updated

## Verification

- `pnpm -r` typechecks (shared, db incl. migration numbering/safety
checks, server, ui, cli) and `pnpm check:token-gates` — clean
- Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the
only 3 failures being pre-existing on `master` (2×
heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port;
reproduced identically with this change stashed); ui + cli suites green;
the embedded-Postgres export-fidelity suite applies the full migration
chain including the new `0196` against a fresh database
- Live end-to-end on a scratch instance: seeded a company with labels, a
blocker pair, an issue document, a work product, a monitor, an agent, a
routine, and two binary attachments (one comment-scoped) → export →
import into a fresh company → labels remapped to new ids, blocker edge
and document restored, monitor un-armed with notes intact, attachments
byte-identical (sha256-compared through the API), agents/routines paused
→ activation panel resumed them; a v5-shaped bundle imported with only
the info warning; flipping one byte in a blob made the import 422 with
**zero** rows created
- Reviewer repro: create a company with a labeled issue + attachment →
Settings → Export → download → Settings → Import on another
company/instance → watch the preview, apply with "start paused", then
activate

## Risks

- Migration `0196` drops
`cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables
behind a default-off flag; their connection/run history is intentionally
discarded
- Breaking removals are all of experimental, flag-gated surface:
`/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai
cloud connect|push`, and the `enableCloudSync` flag (stale keys in
stored instance settings parse harmlessly)
- Import remains non-atomic on mid-apply errors generally (pre-existing
behavior); the new blob verification specifically moved ahead of all
writes so tampered bundles cannot create partial state
- GitHub-sourced imports do not fetch `blobs/*` and skip attachments
with a warning

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with
extended thinking, tool use, and subagent orchestration; implementation
and review split across Fable 5 subagents, with live end-to-end
verification against a running instance

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-30 11:37:00 -07:00

437 lines
16 KiB
TypeScript

import type { Db } from "@paperclipai/db";
import { companies, instanceSettings } from "@paperclipai/db";
import {
DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
DEFAULT_BACKUP_RETENTION,
DEFAULT_ISSUE_GRAPH_LIVENESS_AUTO_RECOVERY_LOOKBACK_HOURS,
PAPERCLIP_CLOUD_MANAGED_BY,
instanceGeneralSettingsSchema,
type InstanceGeneralSettings,
instanceExperimentalSettingsSchema,
type InstanceExperimentalSettings,
type InstanceExperimentalSettingsWithManaged,
type ManagedExperimentalFeatureKey,
type ManagedSettingMetadata,
type PatchInstanceGeneralSettings,
type InstanceSettings,
type PatchInstanceSettings,
type PatchInstanceExperimentalSettings,
} from "@paperclipai/shared";
import { eq } from "drizzle-orm";
import { getManagedInstanceConfig, type ManagedInstanceConfig } from "./managed-config.js";
const DEFAULT_SINGLETON_KEY = "default";
const instanceGeneralSettingsStorageSchema = instanceGeneralSettingsSchema.strip();
const instanceExperimentalSettingsStorageSchema = instanceExperimentalSettingsSchema.strip();
const TRUTHY_RUNTIME_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
interface InstanceSettingsServiceOptions {
runtimeEnv?: Record<string, string | undefined>;
now?: () => Date;
}
type WorktreeRunExecutionSuppressedReason =
| "not_worktree_runtime"
| "flag_disabled"
| "missing_cutoff"
| "missing_instance_id"
| "instance_id_mismatch"
| "settings_read_error";
export type WorktreeRunExecutionActivationState =
| {
armed: true;
cutoff: string;
activationInstanceId: string;
reason: null;
}
| {
armed: false;
cutoff: null;
activationInstanceId: string | null;
reason: WorktreeRunExecutionSuppressedReason;
};
export function isTruthyRuntimeEnvValue(value: string | undefined) {
return typeof value === "string" && TRUTHY_RUNTIME_ENV_VALUES.has(value.trim().toLowerCase());
}
function getRuntimeInstanceId(env: Record<string, string | undefined>) {
const instanceId = env.PAPERCLIP_INSTANCE_ID?.trim();
return instanceId ? instanceId : null;
}
function stripServerManagedExperimentalPatchFields(
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
): PatchInstanceExperimentalSettings {
const {
worktreeRunExecutionActivatedAt: _ignoredActivatedAt,
worktreeRunExecutionActivationInstanceId: _ignoredActivationInstanceId,
...patchable
} = patch as Record<string, unknown>;
return patchable as PatchInstanceExperimentalSettings;
}
export function applyExperimentalSettingsPatch(
current: unknown,
patch: PatchInstanceExperimentalSettings | Record<string, unknown>,
options: InstanceSettingsServiceOptions = {},
): InstanceExperimentalSettings {
const previousExperimental = normalizeExperimentalSettings(current);
const patchable = stripServerManagedExperimentalPatchFields(patch);
const nextExperimental = normalizeExperimentalSettings({
...previousExperimental,
...patchable,
});
const hasWorktreeRunExecutionPatch = Object.prototype.hasOwnProperty.call(
patchable,
"enableWorktreeRunExecution",
);
if (!hasWorktreeRunExecutionPatch) {
return nextExperimental;
}
if (nextExperimental.enableWorktreeRunExecution !== true) {
return {
...nextExperimental,
worktreeRunExecutionActivatedAt: null,
worktreeRunExecutionActivationInstanceId: null,
};
}
if (previousExperimental.enableWorktreeRunExecution === true) {
return nextExperimental;
}
const runtimeEnv = options.runtimeEnv ?? process.env;
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
return nextExperimental;
}
return {
...nextExperimental,
worktreeRunExecutionActivatedAt: (options.now ?? (() => new Date()))().toISOString(),
worktreeRunExecutionActivationInstanceId: getRuntimeInstanceId(runtimeEnv),
};
}
function suppressWorktreeRunExecution(
reason: WorktreeRunExecutionSuppressedReason,
activationInstanceId: string | null = null,
): WorktreeRunExecutionActivationState {
return {
armed: false,
cutoff: null,
activationInstanceId,
reason,
};
}
export function resolveWorktreeRunExecutionActivation(
experimental: InstanceExperimentalSettings,
currentInstanceId: string | null | undefined,
): WorktreeRunExecutionActivationState {
if (experimental.enableWorktreeRunExecution !== true) {
return suppressWorktreeRunExecution(
"flag_disabled",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (!experimental.worktreeRunExecutionActivatedAt) {
return suppressWorktreeRunExecution(
"missing_cutoff",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (!currentInstanceId) {
return suppressWorktreeRunExecution(
"missing_instance_id",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
if (experimental.worktreeRunExecutionActivationInstanceId !== currentInstanceId) {
return suppressWorktreeRunExecution(
"instance_id_mismatch",
experimental.worktreeRunExecutionActivationInstanceId,
);
}
return {
armed: true,
cutoff: experimental.worktreeRunExecutionActivatedAt,
activationInstanceId: currentInstanceId,
reason: null,
};
}
export async function resolveWorktreeRunExecutionActivationState(options: {
getExperimental: () => Promise<InstanceExperimentalSettings>;
runtimeEnv?: Record<string, string | undefined>;
}): Promise<WorktreeRunExecutionActivationState> {
const runtimeEnv = options.runtimeEnv ?? process.env;
if (!isTruthyRuntimeEnvValue(runtimeEnv.PAPERCLIP_IN_WORKTREE)) {
return suppressWorktreeRunExecution("not_worktree_runtime");
}
try {
return resolveWorktreeRunExecutionActivation(
await options.getExperimental(),
getRuntimeInstanceId(runtimeEnv),
);
} catch {
return suppressWorktreeRunExecution("settings_read_error");
}
}
function normalizeGeneralSettings(raw: unknown): InstanceGeneralSettings {
const parsed = instanceGeneralSettingsStorageSchema.safeParse(raw ?? {});
if (parsed.success) {
return {
censorUsernameInLogs: parsed.data.censorUsernameInLogs ?? false,
keyboardShortcuts: parsed.data.keyboardShortcuts ?? false,
feedbackDataSharingPreference:
parsed.data.feedbackDataSharingPreference ?? DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
backupRetention: parsed.data.backupRetention ?? DEFAULT_BACKUP_RETENTION,
// Absent => unrestricted; only carry through an explicit policy.
...(parsed.data.executionMode ? { executionMode: parsed.data.executionMode } : {}),
};
}
return {
censorUsernameInLogs: false,
keyboardShortcuts: false,
feedbackDataSharingPreference: DEFAULT_FEEDBACK_DATA_SHARING_PREFERENCE,
backupRetention: DEFAULT_BACKUP_RETENTION,
};
}
export function normalizeExperimentalSettings(raw: unknown): InstanceExperimentalSettings {
const parsed = instanceExperimentalSettingsStorageSchema.safeParse(raw ?? {});
if (parsed.success) {
return {
enableEnvironments: parsed.data.enableEnvironments ?? false,
enableIsolatedWorkspaces: parsed.data.enableIsolatedWorkspaces ?? false,
enableStreamlinedLeftNavigation: parsed.data.enableStreamlinedLeftNavigation ?? true,
enableApps: parsed.data.enableApps ?? false,
enablePipelines: parsed.data.enablePipelines ?? false,
enableCases: parsed.data.enableCases ?? false,
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
enableTaskWatchdogs: parsed.data.enableTaskWatchdogs ?? false,
enableExternalObjects: parsed.data.enableExternalObjects ?? false,
enableSmokeLab: parsed.data.enableSmokeLab ?? false,
enableBuiltInAgents: parsed.data.enableBuiltInAgents ?? false,
enableBetaSkills: parsed.data.enableBetaSkills ?? false,
enableSummaries: parsed.data.enableSummaries ?? false,
enableStatusCards: parsed.data.enableStatusCards ?? false,
enableDecisions: parsed.data.enableDecisions ?? false,
enableGoalsSidebarLink: parsed.data.enableGoalsSidebarLink ?? false,
enableServerInfoDebugView: parsed.data.enableServerInfoDebugView ?? false,
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
enableIssueGraphLivenessAutoRecovery: parsed.data.enableIssueGraphLivenessAutoRecovery ?? false,
enableWorkspaceBranchReconcileForward: parsed.data.enableWorkspaceBranchReconcileForward ?? true,
enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true,
enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false,
enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false,
worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null,
worktreeRunExecutionActivationInstanceId:
parsed.data.worktreeRunExecutionActivationInstanceId ?? null,
issueGraphLivenessAutoRecoveryLookbackHours:
parsed.data.issueGraphLivenessAutoRecoveryLookbackHours ??
DEFAULT_ISSUE_GRAPH_LIVENESS_AUTO_RECOVERY_LOOKBACK_HOURS,
};
}
return {
enableEnvironments: false,
enableIsolatedWorkspaces: false,
enableStreamlinedLeftNavigation: true,
enableApps: false,
enablePipelines: false,
enableCases: false,
enableConferenceRoomChat: false,
enableTaskWatchdogs: false,
enableIssuePlanDecompositions: false,
enableExperimentalFileViewer: false,
enableExternalObjects: false,
enableSmokeLab: false,
enableBuiltInAgents: false,
enableBetaSkills: false,
enableSummaries: false,
enableStatusCards: false,
enableDecisions: false,
enableGoalsSidebarLink: false,
enableServerInfoDebugView: false,
autoRestartDevServerWhenIdle: false,
enableIssueGraphLivenessAutoRecovery: false,
enableWorkspaceBranchReconcileForward: true,
enableWorkspaceDirtyQuarantineRepair: true,
enableOwnerInstanceAdmin: false,
enableWorktreeRunExecution: false,
worktreeRunExecutionActivatedAt: null,
worktreeRunExecutionActivationInstanceId: null,
issueGraphLivenessAutoRecoveryLookbackHours:
DEFAULT_ISSUE_GRAPH_LIVENESS_AUTO_RECOVERY_LOOKBACK_HOURS,
};
}
export type ManagedExperimentalKeyMetadata = Partial<
Record<ManagedExperimentalFeatureKey, ManagedSettingMetadata>
>;
/**
* Overlay the cloud managed-config feature values over normalized settings.
*
* Read-time precedence: code floor (cloud) > managed overlay > tenant DB
* value > schema default. (No code floors are expressed as flags today —
* floors are enforced in code at the guarded routes, independent of any
* flag value.) The overlay is deliberately never persisted: it re-asserts on
* every read, so a DB restore or manual row edit cannot resurrect a
* capability the harness has disabled.
*/
export function applyManagedExperimentalOverlay(
experimental: InstanceExperimentalSettings,
managedConfig: ManagedInstanceConfig | null,
): { experimental: InstanceExperimentalSettings; managedKeys: ManagedExperimentalKeyMetadata } {
if (!managedConfig) return { experimental, managedKeys: {} };
const next: InstanceExperimentalSettings = { ...experimental };
const managedKeys: ManagedExperimentalKeyMetadata = {};
for (const [key, value] of Object.entries(managedConfig.features) as Array<
[ManagedExperimentalFeatureKey, boolean]
>) {
next[key] = value;
managedKeys[key] = { managed: true, managedBy: PAPERCLIP_CLOUD_MANAGED_BY };
}
return { experimental: next, managedKeys };
}
export function instanceSettingsService(db: Db, options: InstanceSettingsServiceOptions = {}) {
// Fail closed: a malformed PAPERCLIP_MANAGED_CONFIG throws here (and at
// boot in index.ts) rather than silently running without the overlay.
const managedConfig = getManagedInstanceConfig(options.runtimeEnv ?? process.env);
function toExperimentalView(raw: unknown): InstanceExperimentalSettingsWithManaged {
const { experimental, managedKeys } = applyManagedExperimentalOverlay(
normalizeExperimentalSettings(raw),
managedConfig,
);
// Self-hosted responses stay byte-identical: no managedKeys field at all.
return managedConfig ? { ...experimental, managedKeys } : experimental;
}
function toInstanceSettings(row: typeof instanceSettings.$inferSelect): InstanceSettings {
return {
id: row.id,
defaultEnvironmentId: row.defaultEnvironmentId ?? null,
general: normalizeGeneralSettings(row.general),
experimental: toExperimentalView(row.experimental),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
} as InstanceSettings;
}
async function getOrCreateRow() {
const existing = await db
.select()
.from(instanceSettings)
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
.then((rows) => rows[0] ?? null);
if (existing) return existing;
const now = new Date();
const [created] = await db
.insert(instanceSettings)
.values({
singletonKey: DEFAULT_SINGLETON_KEY,
general: {},
experimental: {},
createdAt: now,
updatedAt: now,
})
.onConflictDoUpdate({
target: [instanceSettings.singletonKey],
set: {
updatedAt: now,
},
})
.returning();
if (created) return created;
const raced = await db
.select()
.from(instanceSettings)
.where(eq(instanceSettings.singletonKey, DEFAULT_SINGLETON_KEY))
.then((rows) => rows[0] ?? null);
if (raced) return raced;
throw new Error("Failed to initialize instance settings row");
}
return {
get: async (): Promise<InstanceSettings> => toInstanceSettings(await getOrCreateRow()),
update: async (patch: PatchInstanceSettings): Promise<InstanceSettings> => {
const current = await getOrCreateRow();
const now = new Date();
const [updated] = await db
.update(instanceSettings)
.set({
...(Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId")
? { defaultEnvironmentId: patch.defaultEnvironmentId ?? null }
: {}),
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
return toInstanceSettings(updated ?? current);
},
getGeneral: async (): Promise<InstanceGeneralSettings> => {
const row = await getOrCreateRow();
return normalizeGeneralSettings(row.general);
},
getExperimental: async (): Promise<InstanceExperimentalSettingsWithManaged> => {
const row = await getOrCreateRow();
return toExperimentalView(row.experimental);
},
updateGeneral: async (patch: PatchInstanceGeneralSettings): Promise<InstanceSettings> => {
const current = await getOrCreateRow();
const nextGeneral = normalizeGeneralSettings({
...normalizeGeneralSettings(current.general),
...patch,
});
const now = new Date();
const [updated] = await db
.update(instanceSettings)
.set({
general: { ...nextGeneral },
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
return toInstanceSettings(updated ?? current);
},
updateExperimental: async (patch: PatchInstanceExperimentalSettings): Promise<InstanceSettings> => {
const current = await getOrCreateRow();
const nextExperimental = applyExperimentalSettingsPatch(current.experimental, patch, options);
const now = new Date();
const [updated] = await db
.update(instanceSettings)
.set({
experimental: { ...nextExperimental },
updatedAt: now,
})
.where(eq(instanceSettings.id, current.id))
.returning();
return toInstanceSettings(updated ?? current);
},
listCompanyIds: async (): Promise<string[]> =>
db
.select({ id: companies.id })
.from(companies)
.then((rows) => rows.map((row) => row.id)),
};
}