mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A company accumulates real state — issues, labels, blockers, documents, work products, monitors, attachments, agents, routines — and people need to move that state between instances: self-hosted to cloud, cloud back to self-hosted, or plain backups > - The experimental, flag-gated Cloud Sync transport (#6548) tried to solve this host-to-host: the source pushed into a receiver over HTTPS with a cross-instance consent/token handshake, which required the destination to be publicly reachable and broke for common self-hosted topologies (plain-HTTP LAN/VPN origins); the receiver half never landed upstream at all > - Meanwhile the portability bundle and the existing export/import pages already move companies offline with none of those networking constraints — but silently dropped labels, blockers, issue documents, work products, monitors, and every attachment > - This pull request removes the host-to-host transport and makes Import/Export the single data-movement path: the pages become first-class company-settings destinations, exports declare exactly what they do not carry, and bundle schemaVersion 6 now carries all of the above, with attachments as content-addressed sha256 blobs verified before a single row is written > - The benefit is a migration and backup flow that works between any two instances with no reachability requirements, no cross-instance auth, and no silent data loss ## Linked Issues or Issue Description - Refs #6548 — the original Cloud Sync sender this PR supersedes and removes. - Related, not duplicates: #1697 (goals in the portability manifest — orthogonal field addition), #954 (an earlier import/export + skill-visibility proposal predating the current portability bundle). - No open issue describes this directly, so in brief (feature-request shape): **Problem** — moving a company between instances silently lost labels (imports with label references actually hard-failed), blocker relations, issue documents, work products, monitor state, and all attachments, and the alternative Cloud Sync transport required the destination to be publicly reachable over HTTPS plus a consent handshake, which failed for typical self-hosted setups. **Desired behavior** — one Import/Export flow in company settings that produces a portable bundle carrying all of that data, tells the operator up front what it cannot carry, imports with automations paused, and offers real one-click activation afterwards. ## What Changed - New export fidelity report (`GET /api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel on the Export page listing anything a bundle will not include (now only: approvals, cost history, activity history) - Imports accept `pauseAutomations`; imported agents and routines land paused, the import result reports created routines, and the Import page ends in an activation panel that actually resumes selected agents/activates routines - Export and Import pages promoted into the company-settings nav; the Cloud Upstream wizard, ux-lab page, and API client removed; the old settings route redirects to Export - Host-to-host transport removed: upstream-sync/receiver-client routes and services, CLI `cloud connect`/`cloud push` + keypair store, the shared upstream transfer contract, and the `enableCloudSync` flag; migration `0196` drops the two experimental `cloud_upstream_*` sender tables - Bundle schemaVersion 6: labels (definitions + per-task names, remapped by name on import), blocker relations (`blockedBy` slugs, cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`), work products (system refs nulled), monitors (notes/scheduledBy restored, imported un-armed) - Attachments travel as content-addressed `blobs/<sha256>` entries (deduped; comment-scoped attachments re-link via comment index); every blob is hash-verified **before any write**, so a corrupted bundle cannot leave a partially imported company; both zip codecs now round-trip extensionless/binary entries byte-exactly; the Import page preflights the inline body limit and offers continue-without-attachments - v5 (and older) bundles still import, with an informational warning; bundles newer than v6 are rejected cleanly - Docs: board-operator import/export guide, CLI README, README/ROADMAP updated ## Verification - `pnpm -r` typechecks (shared, db incl. migration numbering/safety checks, server, ui, cli) and `pnpm check:token-gates` — clean - Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the only 3 failures being pre-existing on `master` (2× heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port; reproduced identically with this change stashed); ui + cli suites green; the embedded-Postgres export-fidelity suite applies the full migration chain including the new `0196` against a fresh database - Live end-to-end on a scratch instance: seeded a company with labels, a blocker pair, an issue document, a work product, a monitor, an agent, a routine, and two binary attachments (one comment-scoped) → export → import into a fresh company → labels remapped to new ids, blocker edge and document restored, monitor un-armed with notes intact, attachments byte-identical (sha256-compared through the API), agents/routines paused → activation panel resumed them; a v5-shaped bundle imported with only the info warning; flipping one byte in a blob made the import 422 with **zero** rows created - Reviewer repro: create a company with a labeled issue + attachment → Settings → Export → download → Settings → Import on another company/instance → watch the preview, apply with "start paused", then activate ## Risks - Migration `0196` drops `cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables behind a default-off flag; their connection/run history is intentionally discarded - Breaking removals are all of experimental, flag-gated surface: `/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai cloud connect|push`, and the `enableCloudSync` flag (stale keys in stored instance settings parse harmlessly) - Import remains non-atomic on mid-apply errors generally (pre-existing behavior); the new blob verification specifically moved ahead of all writes so tampered bundles cannot create partial state - GitHub-sourced imports do not fetch `blobs/*` and skip attachments with a warning ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with extended thinking, tool use, and subagent orchestration; implementation and review split across Fable 5 subagents, with live end-to-end verification against a running instance ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
273 lines
8.7 KiB
TypeScript
273 lines
8.7 KiB
TypeScript
import type {
|
|
CompanyPortabilityEmbeddedAssetManifestEntry,
|
|
CompanyPortabilityIssueManifestEntry,
|
|
} from "@paperclipai/shared";
|
|
|
|
/**
|
|
* Export selection is category-driven: instead of per-file checkboxes the
|
|
* export page exposes a small set of toggles, and the checked-file set is
|
|
* derived from them. Root-level files (README.md, COMPANY.md,
|
|
* .paperclip.yaml, images, company logo) always export and have no toggle.
|
|
*/
|
|
export type ExportCategoryKey =
|
|
| "agents"
|
|
| "projects"
|
|
| "skills"
|
|
| "routines"
|
|
| "tasks"
|
|
| "attachments";
|
|
|
|
export type ExportCategorySelection = Record<ExportCategoryKey, boolean>;
|
|
|
|
export type ExportSelectionIssue = Pick<
|
|
CompanyPortabilityIssueManifestEntry,
|
|
"path" | "recurring" | "attachments"
|
|
>;
|
|
|
|
export type ExportSelectionEmbeddedAsset = Pick<
|
|
CompanyPortabilityEmbeddedAssetManifestEntry,
|
|
"sha256" | "ownedBy"
|
|
>;
|
|
|
|
export const EXPORT_CATEGORY_ORDER: ExportCategoryKey[] = [
|
|
"agents",
|
|
"projects",
|
|
"skills",
|
|
"routines",
|
|
"tasks",
|
|
"attachments",
|
|
];
|
|
|
|
export const EXPORT_CATEGORY_LABELS: Record<ExportCategoryKey, string> = {
|
|
agents: "Agents",
|
|
projects: "Projects",
|
|
skills: "Skills",
|
|
routines: "Routines",
|
|
tasks: "Tasks",
|
|
attachments: "Attachments",
|
|
};
|
|
|
|
/** Everything exports by default. */
|
|
export function buildDefaultExportCategorySelection(): ExportCategorySelection {
|
|
return {
|
|
agents: true,
|
|
projects: true,
|
|
skills: true,
|
|
routines: true,
|
|
tasks: true,
|
|
attachments: true,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Attachment blobs only travel with the task (or routine) that references
|
|
* them — a blob without its owning task is an orphan the importer cannot
|
|
* attach. The Attachments toggle is therefore only meaningful while at
|
|
* least one of Tasks or Routines is still selected. Routines count because
|
|
* the bundle schema lets a recurring task manifest entry carry attachments
|
|
* (e.g. bundles that were imported and re-exported), even though natively
|
|
* exported routines emit only their TASK.md.
|
|
*/
|
|
export function isAttachmentsCategoryEnabled(
|
|
categories: Pick<ExportCategorySelection, "tasks" | "routines">,
|
|
): boolean {
|
|
return categories.tasks || categories.routines;
|
|
}
|
|
|
|
function buildRecurringTaskPrefixes(issues: ExportSelectionIssue[]): Set<string> {
|
|
const prefixes = new Set<string>();
|
|
|
|
for (const issue of issues) {
|
|
if (!issue.recurring) continue;
|
|
|
|
const filePath = issue.path.trim();
|
|
if (!filePath) continue;
|
|
|
|
prefixes.add(filePath);
|
|
|
|
const lastSlash = filePath.lastIndexOf("/");
|
|
if (lastSlash >= 0) {
|
|
prefixes.add(`${filePath.slice(0, lastSlash + 1)}`);
|
|
}
|
|
}
|
|
|
|
return prefixes;
|
|
}
|
|
|
|
function isRecurringTaskFile(filePath: string, recurringTaskPrefixes: Set<string>): boolean {
|
|
for (const prefix of recurringTaskPrefixes) {
|
|
if (filePath === prefix || filePath.startsWith(prefix)) return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* Map each attachment blob sha to the kind of task that references it, so
|
|
* blob inclusion can follow its owning task's toggle.
|
|
*/
|
|
function buildAttachmentShaOwnership(issues: ExportSelectionIssue[]): {
|
|
recurring: Set<string>;
|
|
oneOff: Set<string>;
|
|
} {
|
|
const recurring = new Set<string>();
|
|
const oneOff = new Set<string>();
|
|
|
|
for (const issue of issues) {
|
|
for (const attachment of issue.attachments ?? []) {
|
|
const sha = attachment.sha256.trim().toLowerCase();
|
|
if (!sha) continue;
|
|
(issue.recurring ? recurring : oneOff).add(sha);
|
|
}
|
|
}
|
|
|
|
return { recurring, oneOff };
|
|
}
|
|
|
|
const EMBEDDED_ASSET_OWNER_CATEGORY_KEYS = new Set<ExportCategoryKey>([
|
|
"agents",
|
|
"projects",
|
|
"skills",
|
|
"routines",
|
|
"tasks",
|
|
]);
|
|
|
|
/**
|
|
* Map each embedded-asset blob sha to the export categories whose files
|
|
* reference it (from the manifest's embeddedAssets ownedBy field). Multiple
|
|
* asset entries can share one content-addressed blob; ownership unions.
|
|
* Entries without ownership data count as always included.
|
|
*/
|
|
function buildEmbeddedAssetShaOwnership(
|
|
embeddedAssets: ExportSelectionEmbeddedAsset[],
|
|
): Map<string, Set<string>> {
|
|
const ownersBySha = new Map<string, Set<string>>();
|
|
for (const entry of embeddedAssets) {
|
|
const sha = entry.sha256.trim().toLowerCase();
|
|
if (!sha) continue;
|
|
const owners = ownersBySha.get(sha) ?? new Set<string>();
|
|
const ownedBy = entry.ownedBy && entry.ownedBy.length > 0 ? entry.ownedBy : ["always"];
|
|
for (const owner of ownedBy) owners.add(owner);
|
|
ownersBySha.set(sha, owners);
|
|
}
|
|
return ownersBySha;
|
|
}
|
|
|
|
/**
|
|
* An embedded-asset blob follows the toggles of the files that reference it,
|
|
* independent of the Attachments toggle: it stays while any owning category
|
|
* is still selected. "always" owners (root-file references) and owners this
|
|
* build does not recognize keep the bytes rather than silently dropping them.
|
|
*/
|
|
function isEmbeddedAssetBlobIncluded(
|
|
owners: Set<string>,
|
|
categories: ExportCategorySelection,
|
|
): boolean {
|
|
for (const owner of owners) {
|
|
if (!EMBEDDED_ASSET_OWNER_CATEGORY_KEYS.has(owner as ExportCategoryKey)) return true;
|
|
if (categories[owner as ExportCategoryKey]) return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* Classify a bundle file path into its export category, or null for
|
|
* root-level files that always export (README.md, COMPANY.md,
|
|
* .paperclip.yaml, images, company logo).
|
|
*/
|
|
export function categorizeExportFile(
|
|
filePath: string,
|
|
recurringTaskPrefixes: Set<string>,
|
|
): ExportCategoryKey | null {
|
|
if (filePath.startsWith("agents/")) return "agents";
|
|
if (filePath.startsWith("projects/")) return "projects";
|
|
if (filePath.startsWith("skills/")) return "skills";
|
|
if (filePath.startsWith("tasks/")) {
|
|
return isRecurringTaskFile(filePath, recurringTaskPrefixes) ? "routines" : "tasks";
|
|
}
|
|
if (filePath.startsWith("blobs/")) return "attachments";
|
|
return null;
|
|
}
|
|
|
|
/** Count bundle files per category (root always-exported files are not counted). */
|
|
export function countExportFilesByCategory(
|
|
filePaths: string[],
|
|
issues: ExportSelectionIssue[],
|
|
): Record<ExportCategoryKey, number> {
|
|
const recurringTaskPrefixes = buildRecurringTaskPrefixes(issues);
|
|
const counts: Record<ExportCategoryKey, number> = {
|
|
agents: 0,
|
|
projects: 0,
|
|
skills: 0,
|
|
routines: 0,
|
|
tasks: 0,
|
|
attachments: 0,
|
|
};
|
|
for (const filePath of filePaths) {
|
|
const category = categorizeExportFile(filePath, recurringTaskPrefixes);
|
|
if (category) counts[category] += 1;
|
|
}
|
|
return counts;
|
|
}
|
|
|
|
/**
|
|
* Compute the set of files the export will include for the given category
|
|
* selection. Pure: same inputs, same set. The Tasks/Routines → Attachments
|
|
* dependency is enforced here, so callers cannot produce orphan blobs even
|
|
* with an inconsistent selection.
|
|
*/
|
|
export function buildExportCheckedFiles(input: {
|
|
filePaths: string[];
|
|
issues: ExportSelectionIssue[];
|
|
categories: ExportCategorySelection;
|
|
embeddedAssets?: ExportSelectionEmbeddedAsset[];
|
|
}): Set<string> {
|
|
const { filePaths, issues, categories } = input;
|
|
const recurringTaskPrefixes = buildRecurringTaskPrefixes(issues);
|
|
const blobOwnership = buildAttachmentShaOwnership(issues);
|
|
const embeddedOwnership = buildEmbeddedAssetShaOwnership(input.embeddedAssets ?? []);
|
|
const attachmentsOn = categories.attachments && isAttachmentsCategoryEnabled(categories);
|
|
|
|
const checked = new Set<string>();
|
|
for (const filePath of filePaths) {
|
|
const category = categorizeExportFile(filePath, recurringTaskPrefixes);
|
|
|
|
if (category === null) {
|
|
// Root files always export.
|
|
checked.add(filePath);
|
|
continue;
|
|
}
|
|
|
|
if (category === "attachments") {
|
|
const sha = filePath.slice("blobs/".length).trim().toLowerCase();
|
|
// Embedded-asset blobs travel with the files that reference them,
|
|
// regardless of the Attachments toggle. A blob shared with an
|
|
// attachment can still qualify through the attachment path below.
|
|
const embeddedOwners = embeddedOwnership.get(sha);
|
|
if (embeddedOwners && isEmbeddedAssetBlobIncluded(embeddedOwners, categories)) {
|
|
checked.add(filePath);
|
|
continue;
|
|
}
|
|
if (!attachmentsOn) continue;
|
|
const ownedByRecurring = blobOwnership.recurring.has(sha);
|
|
const ownedByOneOff = blobOwnership.oneOff.has(sha);
|
|
if (!ownedByRecurring && !ownedByOneOff) {
|
|
// Unreferenced blob: no owning task or embedded reference to
|
|
// follow, gate on the attachments toggle alone (already dependent
|
|
// on tasks/routines). Embedded-owned blobs whose owners are all
|
|
// disabled stay dropped instead of falling back to this gate.
|
|
if (!embeddedOwners) checked.add(filePath);
|
|
} else if (
|
|
(ownedByOneOff && categories.tasks)
|
|
|| (ownedByRecurring && categories.routines)
|
|
) {
|
|
checked.add(filePath);
|
|
}
|
|
continue;
|
|
}
|
|
|
|
if (categories[category]) checked.add(filePath);
|
|
}
|
|
|
|
return checked;
|
|
}
|