mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 06:02:18 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A company accumulates real state — issues, labels, blockers, documents, work products, monitors, attachments, agents, routines — and people need to move that state between instances: self-hosted to cloud, cloud back to self-hosted, or plain backups > - The experimental, flag-gated Cloud Sync transport (#6548) tried to solve this host-to-host: the source pushed into a receiver over HTTPS with a cross-instance consent/token handshake, which required the destination to be publicly reachable and broke for common self-hosted topologies (plain-HTTP LAN/VPN origins); the receiver half never landed upstream at all > - Meanwhile the portability bundle and the existing export/import pages already move companies offline with none of those networking constraints — but silently dropped labels, blockers, issue documents, work products, monitors, and every attachment > - This pull request removes the host-to-host transport and makes Import/Export the single data-movement path: the pages become first-class company-settings destinations, exports declare exactly what they do not carry, and bundle schemaVersion 6 now carries all of the above, with attachments as content-addressed sha256 blobs verified before a single row is written > - The benefit is a migration and backup flow that works between any two instances with no reachability requirements, no cross-instance auth, and no silent data loss ## Linked Issues or Issue Description - Refs #6548 — the original Cloud Sync sender this PR supersedes and removes. - Related, not duplicates: #1697 (goals in the portability manifest — orthogonal field addition), #954 (an earlier import/export + skill-visibility proposal predating the current portability bundle). - No open issue describes this directly, so in brief (feature-request shape): **Problem** — moving a company between instances silently lost labels (imports with label references actually hard-failed), blocker relations, issue documents, work products, monitor state, and all attachments, and the alternative Cloud Sync transport required the destination to be publicly reachable over HTTPS plus a consent handshake, which failed for typical self-hosted setups. **Desired behavior** — one Import/Export flow in company settings that produces a portable bundle carrying all of that data, tells the operator up front what it cannot carry, imports with automations paused, and offers real one-click activation afterwards. ## What Changed - New export fidelity report (`GET /api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel on the Export page listing anything a bundle will not include (now only: approvals, cost history, activity history) - Imports accept `pauseAutomations`; imported agents and routines land paused, the import result reports created routines, and the Import page ends in an activation panel that actually resumes selected agents/activates routines - Export and Import pages promoted into the company-settings nav; the Cloud Upstream wizard, ux-lab page, and API client removed; the old settings route redirects to Export - Host-to-host transport removed: upstream-sync/receiver-client routes and services, CLI `cloud connect`/`cloud push` + keypair store, the shared upstream transfer contract, and the `enableCloudSync` flag; migration `0196` drops the two experimental `cloud_upstream_*` sender tables - Bundle schemaVersion 6: labels (definitions + per-task names, remapped by name on import), blocker relations (`blockedBy` slugs, cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`), work products (system refs nulled), monitors (notes/scheduledBy restored, imported un-armed) - Attachments travel as content-addressed `blobs/<sha256>` entries (deduped; comment-scoped attachments re-link via comment index); every blob is hash-verified **before any write**, so a corrupted bundle cannot leave a partially imported company; both zip codecs now round-trip extensionless/binary entries byte-exactly; the Import page preflights the inline body limit and offers continue-without-attachments - v5 (and older) bundles still import, with an informational warning; bundles newer than v6 are rejected cleanly - Docs: board-operator import/export guide, CLI README, README/ROADMAP updated ## Verification - `pnpm -r` typechecks (shared, db incl. migration numbering/safety checks, server, ui, cli) and `pnpm check:token-gates` — clean - Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the only 3 failures being pre-existing on `master` (2× heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port; reproduced identically with this change stashed); ui + cli suites green; the embedded-Postgres export-fidelity suite applies the full migration chain including the new `0196` against a fresh database - Live end-to-end on a scratch instance: seeded a company with labels, a blocker pair, an issue document, a work product, a monitor, an agent, a routine, and two binary attachments (one comment-scoped) → export → import into a fresh company → labels remapped to new ids, blocker edge and document restored, monitor un-armed with notes intact, attachments byte-identical (sha256-compared through the API), agents/routines paused → activation panel resumed them; a v5-shaped bundle imported with only the info warning; flipping one byte in a blob made the import 422 with **zero** rows created - Reviewer repro: create a company with a labeled issue + attachment → Settings → Export → download → Settings → Import on another company/instance → watch the preview, apply with "start paused", then activate ## Risks - Migration `0196` drops `cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables behind a default-off flag; their connection/run history is intentionally discarded - Breaking removals are all of experimental, flag-gated surface: `/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai cloud connect|push`, and the `enableCloudSync` flag (stale keys in stored instance settings parse harmlessly) - Import remains non-atomic on mid-apply errors generally (pre-existing behavior); the new blob verification specifically moved ahead of all writes so tampered bundles cannot create partial state - GitHub-sourced imports do not fetch `blobs/*` and skip attachments with a warning ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with extended thinking, tool use, and subagent orchestration; implementation and review split across Fable 5 subagents, with live end-to-end verification against a running instance ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
177 lines
7.4 KiB
TypeScript
177 lines
7.4 KiB
TypeScript
export type TableSizeBucket = "large" | "medium" | "small";
|
|
|
|
export type LocalTableRowCount = {
|
|
readonly table: string;
|
|
readonly localRows: number;
|
|
};
|
|
|
|
export type TableSizeEstimate = LocalTableRowCount & {
|
|
readonly estimateFactor: number;
|
|
readonly estimatedRows: number;
|
|
readonly bucket: TableSizeBucket;
|
|
};
|
|
|
|
export const TABLE_SIZE_ESTIMATE_SOURCE = {
|
|
collectedAt: "2026-07-06",
|
|
database: "default local dev embedded Postgres",
|
|
method: "SELECT count(*) in a read-only transaction",
|
|
} as const;
|
|
|
|
export const TABLE_SIZE_ESTIMATE_FACTOR = 250;
|
|
|
|
export const TABLE_SIZE_BUCKET_THRESHOLDS = {
|
|
largeRows: 1_000_000,
|
|
mediumRows: 100_000,
|
|
} as const;
|
|
|
|
export const LOCAL_TABLE_ROW_COUNTS = [
|
|
{ table: "agent_wakeup_requests", localRows: 52_791 },
|
|
{ table: "activity_log", localRows: 22_930 },
|
|
{ table: "issue_reference_mentions", localRows: 13_218 },
|
|
{ table: "heartbeat_run_events", localRows: 10_833 },
|
|
{ table: "issue_comments", localRows: 5_034 },
|
|
{ table: "document_revisions", localRows: 3_906 },
|
|
{ table: "heartbeat_runs", localRows: 3_620 },
|
|
{ table: "workspace_operations", localRows: 3_600 },
|
|
{ table: "environment_leases", localRows: 3_492 },
|
|
{ table: "cost_events", localRows: 3_330 },
|
|
{ table: "agent_task_sessions", localRows: 1_743 },
|
|
{ table: "documents", localRows: 1_692 },
|
|
{ table: "issue_documents", localRows: 1_678 },
|
|
{ table: "issues", localRows: 1_609 },
|
|
{ table: "secret_access_events", localRows: 933 },
|
|
{ table: "issue_read_states", localRows: 608 },
|
|
{ table: "issue_thread_interactions", localRows: 455 },
|
|
{ table: "issue_relations", localRows: 445 },
|
|
{ table: "execution_workspaces", localRows: 414 },
|
|
{ table: "agent_config_revisions", localRows: 145 },
|
|
{ table: "document_annotation_anchor_snapshots", localRows: 132 },
|
|
{ table: "document_annotation_comments", localRows: 122 },
|
|
{ table: "document_annotation_threads", localRows: 80 },
|
|
{ table: "routine_revisions", localRows: 64 },
|
|
{ table: "issue_tree_hold_members", localRows: 62 },
|
|
{ table: "routine_runs", localRows: 52 },
|
|
{ table: "issue_recovery_actions", localRows: 50 },
|
|
{ table: "assets", localRows: 46 },
|
|
{ table: "issue_attachments", localRows: 44 },
|
|
{ table: "principal_permission_grants", localRows: 25 },
|
|
{ table: "company_memberships", localRows: 19 },
|
|
{ table: "agent_runtime_state", localRows: 18 },
|
|
{ table: "agents", localRows: 18 },
|
|
{ table: "routine_triggers", localRows: 16 },
|
|
{ table: "company_skills", localRows: 15 },
|
|
{ table: "routine_documents", localRows: 14 },
|
|
{ table: "routines", localRows: 14 },
|
|
{ table: "issue_inbox_archives", localRows: 13 },
|
|
{ table: "issue_approvals", localRows: 12 },
|
|
{ table: "company_secret_bindings", localRows: 11 },
|
|
{ table: "issue_tree_holds", localRows: 10 },
|
|
{ table: "approvals", localRows: 9 },
|
|
{ table: "session", localRows: 9 },
|
|
{ table: "issue_work_products", localRows: 7 },
|
|
{ table: "company_skill_versions", localRows: 6 },
|
|
{ table: "feedback_exports", localRows: 6 },
|
|
{ table: "feedback_votes", localRows: 6 },
|
|
{ table: "issue_execution_decisions", localRows: 6 },
|
|
{ table: "company_secret_versions", localRows: 3 },
|
|
{ table: "company_secrets", localRows: 3 },
|
|
{ table: "projects", localRows: 3 },
|
|
{ table: "approval_comments", localRows: 2 },
|
|
{ table: "goals", localRows: 2 },
|
|
{ table: "project_goals", localRows: 2 },
|
|
{ table: "project_workspaces", localRows: 2 },
|
|
{ table: "account", localRows: 1 },
|
|
{ table: "companies", localRows: 1 },
|
|
{ table: "environments", localRows: 1 },
|
|
{ table: "instance_settings", localRows: 1 },
|
|
{ table: "instance_user_roles", localRows: 1 },
|
|
{ table: "user", localRows: 1 },
|
|
{ table: "agent_api_keys", localRows: 0 },
|
|
{ table: "agent_memberships", localRows: 0 },
|
|
{ table: "board_api_keys", localRows: 0 },
|
|
{ table: "budget_incidents", localRows: 0 },
|
|
{ table: "budget_policies", localRows: 0 },
|
|
{ table: "cli_auth_challenges", localRows: 0 },
|
|
{ table: "company_logos", localRows: 0 },
|
|
{ table: "company_secret_provider_configs", localRows: 0 },
|
|
{ table: "company_skill_comments", localRows: 0 },
|
|
{ table: "company_skill_stars", localRows: 0 },
|
|
{ table: "company_user_sidebar_preferences", localRows: 0 },
|
|
{ table: "environment_custom_image_setup_sessions", localRows: 0 },
|
|
{ table: "environment_custom_image_templates", localRows: 0 },
|
|
{ table: "external_object_mentions", localRows: 0 },
|
|
{ table: "external_objects", localRows: 0 },
|
|
{ table: "finance_events", localRows: 0 },
|
|
{ table: "heartbeat_run_watchdog_decisions", localRows: 0 },
|
|
{ table: "inbox_dismissals", localRows: 0 },
|
|
{ table: "invites", localRows: 0 },
|
|
{ table: "issue_labels", localRows: 0 },
|
|
{ table: "issue_plan_decompositions", localRows: 0 },
|
|
{ table: "issue_watchdogs", localRows: 0 },
|
|
{ table: "join_requests", localRows: 0 },
|
|
{ table: "labels", localRows: 0 },
|
|
{ table: "pipeline_automation_executions", localRows: 0 },
|
|
{ table: "pipeline_case_blockers", localRows: 0 },
|
|
{ table: "pipeline_case_documents", localRows: 0 },
|
|
{ table: "pipeline_case_events", localRows: 0 },
|
|
{ table: "pipeline_case_issue_links", localRows: 0 },
|
|
{ table: "pipeline_cases", localRows: 0 },
|
|
{ table: "pipeline_documents", localRows: 0 },
|
|
{ table: "pipeline_stages", localRows: 0 },
|
|
{ table: "pipeline_transitions", localRows: 0 },
|
|
{ table: "pipelines", localRows: 0 },
|
|
{ table: "plugin_company_settings", localRows: 0 },
|
|
{ table: "plugin_config", localRows: 0 },
|
|
{ table: "plugin_database_namespaces", localRows: 0 },
|
|
{ table: "plugin_entities", localRows: 0 },
|
|
{ table: "plugin_job_runs", localRows: 0 },
|
|
{ table: "plugin_jobs", localRows: 0 },
|
|
{ table: "plugin_logs", localRows: 0 },
|
|
{ table: "plugin_managed_resources", localRows: 0 },
|
|
{ table: "plugin_migrations", localRows: 0 },
|
|
{ table: "plugin_state", localRows: 0 },
|
|
{ table: "plugin_webhook_deliveries", localRows: 0 },
|
|
{ table: "plugins", localRows: 0 },
|
|
{ table: "project_memberships", localRows: 0 },
|
|
{ table: "user_secret_declarations", localRows: 0 },
|
|
{ table: "user_secret_definitions", localRows: 0 },
|
|
{ table: "user_sidebar_preferences", localRows: 0 },
|
|
{ table: "verification", localRows: 0 },
|
|
{ table: "workspace_runtime_services", localRows: 0 },
|
|
] as const satisfies readonly LocalTableRowCount[];
|
|
|
|
function bucketForEstimatedRows(estimatedRows: number): TableSizeBucket {
|
|
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.largeRows) return "large";
|
|
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.mediumRows) return "medium";
|
|
return "small";
|
|
}
|
|
|
|
export const TABLE_SIZE_ESTIMATES: readonly TableSizeEstimate[] = LOCAL_TABLE_ROW_COUNTS.map(
|
|
({ table, localRows }) => {
|
|
const estimatedRows = localRows * TABLE_SIZE_ESTIMATE_FACTOR;
|
|
return {
|
|
table,
|
|
localRows,
|
|
estimateFactor: TABLE_SIZE_ESTIMATE_FACTOR,
|
|
estimatedRows,
|
|
bucket: bucketForEstimatedRows(estimatedRows),
|
|
};
|
|
},
|
|
);
|
|
|
|
export const TABLE_SIZE_ESTIMATES_BY_TABLE: ReadonlyMap<string, TableSizeEstimate> = new Map(
|
|
TABLE_SIZE_ESTIMATES.map((estimate) => [estimate.table, estimate]),
|
|
);
|
|
|
|
export function getTableSizeEstimate(table: string): TableSizeEstimate | undefined {
|
|
return TABLE_SIZE_ESTIMATES_BY_TABLE.get(table);
|
|
}
|
|
|
|
export function getTableSizeBucket(table: string): TableSizeBucket {
|
|
return getTableSizeEstimate(table)?.bucket ?? "small";
|
|
}
|
|
|
|
export function isKnownLargeTable(table: string): boolean {
|
|
return getTableSizeBucket(table) === "large";
|
|
}
|