Files
PaperClipAI/packages/db/src/table-size-estimates.ts
T
Devin Foley 916c13501f Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A company accumulates real state — issues, labels, blockers,
documents, work products, monitors, attachments, agents, routines — and
people need to move that state between instances: self-hosted to cloud,
cloud back to self-hosted, or plain backups
> - The experimental, flag-gated Cloud Sync transport (#6548) tried to
solve this host-to-host: the source pushed into a receiver over HTTPS
with a cross-instance consent/token handshake, which required the
destination to be publicly reachable and broke for common self-hosted
topologies (plain-HTTP LAN/VPN origins); the receiver half never landed
upstream at all
> - Meanwhile the portability bundle and the existing export/import
pages already move companies offline with none of those networking
constraints — but silently dropped labels, blockers, issue documents,
work products, monitors, and every attachment
> - This pull request removes the host-to-host transport and makes
Import/Export the single data-movement path: the pages become
first-class company-settings destinations, exports declare exactly what
they do not carry, and bundle schemaVersion 6 now carries all of the
above, with attachments as content-addressed sha256 blobs verified
before a single row is written
> - The benefit is a migration and backup flow that works between any
two instances with no reachability requirements, no cross-instance auth,
and no silent data loss

## Linked Issues or Issue Description

- Refs #6548 — the original Cloud Sync sender this PR supersedes and
removes.
- Related, not duplicates: #1697 (goals in the portability manifest —
orthogonal field addition), #954 (an earlier import/export +
skill-visibility proposal predating the current portability bundle).
- No open issue describes this directly, so in brief (feature-request
shape): **Problem** — moving a company between instances silently lost
labels (imports with label references actually hard-failed), blocker
relations, issue documents, work products, monitor state, and all
attachments, and the alternative Cloud Sync transport required the
destination to be publicly reachable over HTTPS plus a consent
handshake, which failed for typical self-hosted setups. **Desired
behavior** — one Import/Export flow in company settings that produces a
portable bundle carrying all of that data, tells the operator up front
what it cannot carry, imports with automations paused, and offers real
one-click activation afterwards.

## What Changed

- New export fidelity report (`GET
/api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel
on the Export page listing anything a bundle will not include (now only:
approvals, cost history, activity history)
- Imports accept `pauseAutomations`; imported agents and routines land
paused, the import result reports created routines, and the Import page
ends in an activation panel that actually resumes selected
agents/activates routines
- Export and Import pages promoted into the company-settings nav; the
Cloud Upstream wizard, ux-lab page, and API client removed; the old
settings route redirects to Export
- Host-to-host transport removed: upstream-sync/receiver-client routes
and services, CLI `cloud connect`/`cloud push` + keypair store, the
shared upstream transfer contract, and the `enableCloudSync` flag;
migration `0196` drops the two experimental `cloud_upstream_*` sender
tables
- Bundle schemaVersion 6: labels (definitions + per-task names, remapped
by name on import), blocker relations (`blockedBy` slugs,
cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`),
work products (system refs nulled), monitors (notes/scheduledBy
restored, imported un-armed)
- Attachments travel as content-addressed `blobs/<sha256>` entries
(deduped; comment-scoped attachments re-link via comment index); every
blob is hash-verified **before any write**, so a corrupted bundle cannot
leave a partially imported company; both zip codecs now round-trip
extensionless/binary entries byte-exactly; the Import page preflights
the inline body limit and offers continue-without-attachments
- v5 (and older) bundles still import, with an informational warning;
bundles newer than v6 are rejected cleanly
- Docs: board-operator import/export guide, CLI README, README/ROADMAP
updated

## Verification

- `pnpm -r` typechecks (shared, db incl. migration numbering/safety
checks, server, ui, cli) and `pnpm check:token-gates` — clean
- Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the
only 3 failures being pre-existing on `master` (2×
heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port;
reproduced identically with this change stashed); ui + cli suites green;
the embedded-Postgres export-fidelity suite applies the full migration
chain including the new `0196` against a fresh database
- Live end-to-end on a scratch instance: seeded a company with labels, a
blocker pair, an issue document, a work product, a monitor, an agent, a
routine, and two binary attachments (one comment-scoped) → export →
import into a fresh company → labels remapped to new ids, blocker edge
and document restored, monitor un-armed with notes intact, attachments
byte-identical (sha256-compared through the API), agents/routines paused
→ activation panel resumed them; a v5-shaped bundle imported with only
the info warning; flipping one byte in a blob made the import 422 with
**zero** rows created
- Reviewer repro: create a company with a labeled issue + attachment →
Settings → Export → download → Settings → Import on another
company/instance → watch the preview, apply with "start paused", then
activate

## Risks

- Migration `0196` drops
`cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables
behind a default-off flag; their connection/run history is intentionally
discarded
- Breaking removals are all of experimental, flag-gated surface:
`/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai
cloud connect|push`, and the `enableCloudSync` flag (stale keys in
stored instance settings parse harmlessly)
- Import remains non-atomic on mid-apply errors generally (pre-existing
behavior); the new blob verification specifically moved ahead of all
writes so tampered bundles cannot create partial state
- GitHub-sourced imports do not fetch `blobs/*` and skip attachments
with a warning

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with
extended thinking, tool use, and subagent orchestration; implementation
and review split across Fable 5 subagents, with live end-to-end
verification against a running instance

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-30 11:37:00 -07:00

177 lines
7.4 KiB
TypeScript

export type TableSizeBucket = "large" | "medium" | "small";
export type LocalTableRowCount = {
readonly table: string;
readonly localRows: number;
};
export type TableSizeEstimate = LocalTableRowCount & {
readonly estimateFactor: number;
readonly estimatedRows: number;
readonly bucket: TableSizeBucket;
};
export const TABLE_SIZE_ESTIMATE_SOURCE = {
collectedAt: "2026-07-06",
database: "default local dev embedded Postgres",
method: "SELECT count(*) in a read-only transaction",
} as const;
export const TABLE_SIZE_ESTIMATE_FACTOR = 250;
export const TABLE_SIZE_BUCKET_THRESHOLDS = {
largeRows: 1_000_000,
mediumRows: 100_000,
} as const;
export const LOCAL_TABLE_ROW_COUNTS = [
{ table: "agent_wakeup_requests", localRows: 52_791 },
{ table: "activity_log", localRows: 22_930 },
{ table: "issue_reference_mentions", localRows: 13_218 },
{ table: "heartbeat_run_events", localRows: 10_833 },
{ table: "issue_comments", localRows: 5_034 },
{ table: "document_revisions", localRows: 3_906 },
{ table: "heartbeat_runs", localRows: 3_620 },
{ table: "workspace_operations", localRows: 3_600 },
{ table: "environment_leases", localRows: 3_492 },
{ table: "cost_events", localRows: 3_330 },
{ table: "agent_task_sessions", localRows: 1_743 },
{ table: "documents", localRows: 1_692 },
{ table: "issue_documents", localRows: 1_678 },
{ table: "issues", localRows: 1_609 },
{ table: "secret_access_events", localRows: 933 },
{ table: "issue_read_states", localRows: 608 },
{ table: "issue_thread_interactions", localRows: 455 },
{ table: "issue_relations", localRows: 445 },
{ table: "execution_workspaces", localRows: 414 },
{ table: "agent_config_revisions", localRows: 145 },
{ table: "document_annotation_anchor_snapshots", localRows: 132 },
{ table: "document_annotation_comments", localRows: 122 },
{ table: "document_annotation_threads", localRows: 80 },
{ table: "routine_revisions", localRows: 64 },
{ table: "issue_tree_hold_members", localRows: 62 },
{ table: "routine_runs", localRows: 52 },
{ table: "issue_recovery_actions", localRows: 50 },
{ table: "assets", localRows: 46 },
{ table: "issue_attachments", localRows: 44 },
{ table: "principal_permission_grants", localRows: 25 },
{ table: "company_memberships", localRows: 19 },
{ table: "agent_runtime_state", localRows: 18 },
{ table: "agents", localRows: 18 },
{ table: "routine_triggers", localRows: 16 },
{ table: "company_skills", localRows: 15 },
{ table: "routine_documents", localRows: 14 },
{ table: "routines", localRows: 14 },
{ table: "issue_inbox_archives", localRows: 13 },
{ table: "issue_approvals", localRows: 12 },
{ table: "company_secret_bindings", localRows: 11 },
{ table: "issue_tree_holds", localRows: 10 },
{ table: "approvals", localRows: 9 },
{ table: "session", localRows: 9 },
{ table: "issue_work_products", localRows: 7 },
{ table: "company_skill_versions", localRows: 6 },
{ table: "feedback_exports", localRows: 6 },
{ table: "feedback_votes", localRows: 6 },
{ table: "issue_execution_decisions", localRows: 6 },
{ table: "company_secret_versions", localRows: 3 },
{ table: "company_secrets", localRows: 3 },
{ table: "projects", localRows: 3 },
{ table: "approval_comments", localRows: 2 },
{ table: "goals", localRows: 2 },
{ table: "project_goals", localRows: 2 },
{ table: "project_workspaces", localRows: 2 },
{ table: "account", localRows: 1 },
{ table: "companies", localRows: 1 },
{ table: "environments", localRows: 1 },
{ table: "instance_settings", localRows: 1 },
{ table: "instance_user_roles", localRows: 1 },
{ table: "user", localRows: 1 },
{ table: "agent_api_keys", localRows: 0 },
{ table: "agent_memberships", localRows: 0 },
{ table: "board_api_keys", localRows: 0 },
{ table: "budget_incidents", localRows: 0 },
{ table: "budget_policies", localRows: 0 },
{ table: "cli_auth_challenges", localRows: 0 },
{ table: "company_logos", localRows: 0 },
{ table: "company_secret_provider_configs", localRows: 0 },
{ table: "company_skill_comments", localRows: 0 },
{ table: "company_skill_stars", localRows: 0 },
{ table: "company_user_sidebar_preferences", localRows: 0 },
{ table: "environment_custom_image_setup_sessions", localRows: 0 },
{ table: "environment_custom_image_templates", localRows: 0 },
{ table: "external_object_mentions", localRows: 0 },
{ table: "external_objects", localRows: 0 },
{ table: "finance_events", localRows: 0 },
{ table: "heartbeat_run_watchdog_decisions", localRows: 0 },
{ table: "inbox_dismissals", localRows: 0 },
{ table: "invites", localRows: 0 },
{ table: "issue_labels", localRows: 0 },
{ table: "issue_plan_decompositions", localRows: 0 },
{ table: "issue_watchdogs", localRows: 0 },
{ table: "join_requests", localRows: 0 },
{ table: "labels", localRows: 0 },
{ table: "pipeline_automation_executions", localRows: 0 },
{ table: "pipeline_case_blockers", localRows: 0 },
{ table: "pipeline_case_documents", localRows: 0 },
{ table: "pipeline_case_events", localRows: 0 },
{ table: "pipeline_case_issue_links", localRows: 0 },
{ table: "pipeline_cases", localRows: 0 },
{ table: "pipeline_documents", localRows: 0 },
{ table: "pipeline_stages", localRows: 0 },
{ table: "pipeline_transitions", localRows: 0 },
{ table: "pipelines", localRows: 0 },
{ table: "plugin_company_settings", localRows: 0 },
{ table: "plugin_config", localRows: 0 },
{ table: "plugin_database_namespaces", localRows: 0 },
{ table: "plugin_entities", localRows: 0 },
{ table: "plugin_job_runs", localRows: 0 },
{ table: "plugin_jobs", localRows: 0 },
{ table: "plugin_logs", localRows: 0 },
{ table: "plugin_managed_resources", localRows: 0 },
{ table: "plugin_migrations", localRows: 0 },
{ table: "plugin_state", localRows: 0 },
{ table: "plugin_webhook_deliveries", localRows: 0 },
{ table: "plugins", localRows: 0 },
{ table: "project_memberships", localRows: 0 },
{ table: "user_secret_declarations", localRows: 0 },
{ table: "user_secret_definitions", localRows: 0 },
{ table: "user_sidebar_preferences", localRows: 0 },
{ table: "verification", localRows: 0 },
{ table: "workspace_runtime_services", localRows: 0 },
] as const satisfies readonly LocalTableRowCount[];
function bucketForEstimatedRows(estimatedRows: number): TableSizeBucket {
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.largeRows) return "large";
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.mediumRows) return "medium";
return "small";
}
export const TABLE_SIZE_ESTIMATES: readonly TableSizeEstimate[] = LOCAL_TABLE_ROW_COUNTS.map(
({ table, localRows }) => {
const estimatedRows = localRows * TABLE_SIZE_ESTIMATE_FACTOR;
return {
table,
localRows,
estimateFactor: TABLE_SIZE_ESTIMATE_FACTOR,
estimatedRows,
bucket: bucketForEstimatedRows(estimatedRows),
};
},
);
export const TABLE_SIZE_ESTIMATES_BY_TABLE: ReadonlyMap<string, TableSizeEstimate> = new Map(
TABLE_SIZE_ESTIMATES.map((estimate) => [estimate.table, estimate]),
);
export function getTableSizeEstimate(table: string): TableSizeEstimate | undefined {
return TABLE_SIZE_ESTIMATES_BY_TABLE.get(table);
}
export function getTableSizeBucket(table: string): TableSizeBucket {
return getTableSizeEstimate(table)?.bucket ?? "small";
}
export function isKnownLargeTable(table: string): boolean {
return getTableSizeBucket(table) === "large";
}