mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A company accumulates real state — issues, labels, blockers, documents, work products, monitors, attachments, agents, routines — and people need to move that state between instances: self-hosted to cloud, cloud back to self-hosted, or plain backups > - The experimental, flag-gated Cloud Sync transport (#6548) tried to solve this host-to-host: the source pushed into a receiver over HTTPS with a cross-instance consent/token handshake, which required the destination to be publicly reachable and broke for common self-hosted topologies (plain-HTTP LAN/VPN origins); the receiver half never landed upstream at all > - Meanwhile the portability bundle and the existing export/import pages already move companies offline with none of those networking constraints — but silently dropped labels, blockers, issue documents, work products, monitors, and every attachment > - This pull request removes the host-to-host transport and makes Import/Export the single data-movement path: the pages become first-class company-settings destinations, exports declare exactly what they do not carry, and bundle schemaVersion 6 now carries all of the above, with attachments as content-addressed sha256 blobs verified before a single row is written > - The benefit is a migration and backup flow that works between any two instances with no reachability requirements, no cross-instance auth, and no silent data loss ## Linked Issues or Issue Description - Refs #6548 — the original Cloud Sync sender this PR supersedes and removes. - Related, not duplicates: #1697 (goals in the portability manifest — orthogonal field addition), #954 (an earlier import/export + skill-visibility proposal predating the current portability bundle). - No open issue describes this directly, so in brief (feature-request shape): **Problem** — moving a company between instances silently lost labels (imports with label references actually hard-failed), blocker relations, issue documents, work products, monitor state, and all attachments, and the alternative Cloud Sync transport required the destination to be publicly reachable over HTTPS plus a consent handshake, which failed for typical self-hosted setups. **Desired behavior** — one Import/Export flow in company settings that produces a portable bundle carrying all of that data, tells the operator up front what it cannot carry, imports with automations paused, and offers real one-click activation afterwards. ## What Changed - New export fidelity report (`GET /api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel on the Export page listing anything a bundle will not include (now only: approvals, cost history, activity history) - Imports accept `pauseAutomations`; imported agents and routines land paused, the import result reports created routines, and the Import page ends in an activation panel that actually resumes selected agents/activates routines - Export and Import pages promoted into the company-settings nav; the Cloud Upstream wizard, ux-lab page, and API client removed; the old settings route redirects to Export - Host-to-host transport removed: upstream-sync/receiver-client routes and services, CLI `cloud connect`/`cloud push` + keypair store, the shared upstream transfer contract, and the `enableCloudSync` flag; migration `0196` drops the two experimental `cloud_upstream_*` sender tables - Bundle schemaVersion 6: labels (definitions + per-task names, remapped by name on import), blocker relations (`blockedBy` slugs, cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`), work products (system refs nulled), monitors (notes/scheduledBy restored, imported un-armed) - Attachments travel as content-addressed `blobs/<sha256>` entries (deduped; comment-scoped attachments re-link via comment index); every blob is hash-verified **before any write**, so a corrupted bundle cannot leave a partially imported company; both zip codecs now round-trip extensionless/binary entries byte-exactly; the Import page preflights the inline body limit and offers continue-without-attachments - v5 (and older) bundles still import, with an informational warning; bundles newer than v6 are rejected cleanly - Docs: board-operator import/export guide, CLI README, README/ROADMAP updated ## Verification - `pnpm -r` typechecks (shared, db incl. migration numbering/safety checks, server, ui, cli) and `pnpm check:token-gates` — clean - Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the only 3 failures being pre-existing on `master` (2× heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port; reproduced identically with this change stashed); ui + cli suites green; the embedded-Postgres export-fidelity suite applies the full migration chain including the new `0196` against a fresh database - Live end-to-end on a scratch instance: seeded a company with labels, a blocker pair, an issue document, a work product, a monitor, an agent, a routine, and two binary attachments (one comment-scoped) → export → import into a fresh company → labels remapped to new ids, blocker edge and document restored, monitor un-armed with notes intact, attachments byte-identical (sha256-compared through the API), agents/routines paused → activation panel resumed them; a v5-shaped bundle imported with only the info warning; flipping one byte in a blob made the import 422 with **zero** rows created - Reviewer repro: create a company with a labeled issue + attachment → Settings → Export → download → Settings → Import on another company/instance → watch the preview, apply with "start paused", then activate ## Risks - Migration `0196` drops `cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables behind a default-off flag; their connection/run history is intentionally discarded - Breaking removals are all of experimental, flag-gated surface: `/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai cloud connect|push`, and the `enableCloudSync` flag (stale keys in stored instance settings parse harmlessly) - Import remains non-atomic on mid-apply errors generally (pre-existing behavior); the new blob verification specifically moved ahead of all writes so tampered bundles cannot create partial state - GitHub-sourced imports do not fetch `blobs/*` and skip attachments with a warning ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with extended thinking, tool use, and subagent orchestration; implementation and review split across Fable 5 subagents, with live end-to-end verification against a running instance ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
309 lines
10 KiB
TypeScript
309 lines
10 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import {
|
|
type ExportCategorySelection,
|
|
type ExportSelectionEmbeddedAsset,
|
|
type ExportSelectionIssue,
|
|
buildDefaultExportCategorySelection,
|
|
buildExportCheckedFiles,
|
|
countExportFilesByCategory,
|
|
isAttachmentsCategoryEnabled,
|
|
} from "./company-export-selection";
|
|
|
|
function attachment(sha256: string) {
|
|
return {
|
|
sha256,
|
|
contentType: "application/octet-stream",
|
|
originalFilename: null,
|
|
byteSize: 1,
|
|
commentIndex: null,
|
|
};
|
|
}
|
|
|
|
const ROOT_FILES = [
|
|
"README.md",
|
|
"COMPANY.md",
|
|
".paperclip.yaml",
|
|
"images/org-chart.png",
|
|
"images/logo.png",
|
|
];
|
|
|
|
const FILE_PATHS = [
|
|
...ROOT_FILES,
|
|
"agents/ceo/AGENT.md",
|
|
"agents/ceo/instructions.md",
|
|
"projects/growth/PROJECT.md",
|
|
"skills/research/SKILL.md",
|
|
"tasks/one-off/TASK.md",
|
|
"tasks/one-off/documents/spec.md",
|
|
"tasks/weekly-report/TASK.md",
|
|
"blobs/aaa111",
|
|
"blobs/bbb222",
|
|
];
|
|
|
|
const ISSUES: ExportSelectionIssue[] = [
|
|
{
|
|
path: "tasks/one-off/TASK.md",
|
|
recurring: false,
|
|
attachments: [attachment("aaa111")],
|
|
},
|
|
{
|
|
path: "tasks/weekly-report/TASK.md",
|
|
recurring: true,
|
|
attachments: [attachment("bbb222")],
|
|
},
|
|
];
|
|
|
|
function selection(overrides: Partial<ExportCategorySelection> = {}): ExportCategorySelection {
|
|
return { ...buildDefaultExportCategorySelection(), ...overrides };
|
|
}
|
|
|
|
function checkedWith(overrides: Partial<ExportCategorySelection> = {}): string[] {
|
|
return Array.from(
|
|
buildExportCheckedFiles({
|
|
filePaths: FILE_PATHS,
|
|
issues: ISSUES,
|
|
categories: selection(overrides),
|
|
}),
|
|
).sort();
|
|
}
|
|
|
|
describe("buildExportCheckedFiles", () => {
|
|
it("checks every file by default", () => {
|
|
expect(checkedWith()).toEqual([...FILE_PATHS].sort());
|
|
});
|
|
|
|
it("excludes agent files when agents is off", () => {
|
|
const checked = checkedWith({ agents: false });
|
|
expect(checked).not.toContain("agents/ceo/AGENT.md");
|
|
expect(checked).not.toContain("agents/ceo/instructions.md");
|
|
expect(checked).toEqual(
|
|
[...FILE_PATHS].filter((p) => !p.startsWith("agents/")).sort(),
|
|
);
|
|
});
|
|
|
|
it("excludes project files when projects is off", () => {
|
|
expect(checkedWith({ projects: false })).toEqual(
|
|
[...FILE_PATHS].filter((p) => !p.startsWith("projects/")).sort(),
|
|
);
|
|
});
|
|
|
|
it("excludes skill files when skills is off", () => {
|
|
expect(checkedWith({ skills: false })).toEqual(
|
|
[...FILE_PATHS].filter((p) => !p.startsWith("skills/")).sort(),
|
|
);
|
|
});
|
|
|
|
it("splits recurring task files from one-off task files", () => {
|
|
const withoutTasks = checkedWith({ tasks: false });
|
|
expect(withoutTasks).not.toContain("tasks/one-off/TASK.md");
|
|
expect(withoutTasks).not.toContain("tasks/one-off/documents/spec.md");
|
|
expect(withoutTasks).toContain("tasks/weekly-report/TASK.md");
|
|
|
|
const withoutRoutines = checkedWith({ routines: false });
|
|
expect(withoutRoutines).toContain("tasks/one-off/TASK.md");
|
|
expect(withoutRoutines).toContain("tasks/one-off/documents/spec.md");
|
|
expect(withoutRoutines).not.toContain("tasks/weekly-report/TASK.md");
|
|
});
|
|
|
|
it("drops a blob when its owning one-off task is excluded", () => {
|
|
const checked = checkedWith({ tasks: false });
|
|
expect(checked).not.toContain("blobs/aaa111");
|
|
// Routine-owned blob still travels with its routine.
|
|
expect(checked).toContain("blobs/bbb222");
|
|
});
|
|
|
|
it("drops a blob when its owning routine is excluded", () => {
|
|
const checked = checkedWith({ routines: false });
|
|
expect(checked).toContain("blobs/aaa111");
|
|
expect(checked).not.toContain("blobs/bbb222");
|
|
});
|
|
|
|
it("excludes all blobs when attachments is off, keeping task files", () => {
|
|
const checked = checkedWith({ attachments: false });
|
|
expect(checked).not.toContain("blobs/aaa111");
|
|
expect(checked).not.toContain("blobs/bbb222");
|
|
expect(checked).toContain("tasks/one-off/TASK.md");
|
|
expect(checked).toContain("tasks/weekly-report/TASK.md");
|
|
});
|
|
|
|
it("forces attachments off when both tasks and routines are off", () => {
|
|
const checked = checkedWith({ tasks: false, routines: false, attachments: true });
|
|
expect(checked.some((p) => p.startsWith("blobs/"))).toBe(false);
|
|
expect(checked.some((p) => p.startsWith("tasks/"))).toBe(false);
|
|
});
|
|
|
|
it("always includes root files, even with every category off", () => {
|
|
const checked = checkedWith({
|
|
agents: false,
|
|
projects: false,
|
|
skills: false,
|
|
routines: false,
|
|
tasks: false,
|
|
attachments: false,
|
|
});
|
|
expect(checked).toEqual([...ROOT_FILES].sort());
|
|
});
|
|
|
|
it("keeps a blob referenced by both a one-off task and a routine while either is on", () => {
|
|
const issues: ExportSelectionIssue[] = [
|
|
{ path: "tasks/one-off/TASK.md", recurring: false, attachments: [attachment("shared")] },
|
|
{ path: "tasks/weekly-report/TASK.md", recurring: true, attachments: [attachment("shared")] },
|
|
];
|
|
const filePaths = ["tasks/one-off/TASK.md", "tasks/weekly-report/TASK.md", "blobs/shared"];
|
|
|
|
for (const overrides of [{ tasks: false }, { routines: false }] as const) {
|
|
const checked = buildExportCheckedFiles({
|
|
filePaths,
|
|
issues,
|
|
categories: selection(overrides),
|
|
});
|
|
expect(checked.has("blobs/shared")).toBe(true);
|
|
}
|
|
|
|
const bothOff = buildExportCheckedFiles({
|
|
filePaths,
|
|
issues,
|
|
categories: selection({ tasks: false, routines: false }),
|
|
});
|
|
expect(bothOff.has("blobs/shared")).toBe(false);
|
|
});
|
|
|
|
it("gates unreferenced blobs on the attachments toggle alone", () => {
|
|
const filePaths = ["tasks/one-off/TASK.md", "blobs/orphan"];
|
|
const issues: ExportSelectionIssue[] = [
|
|
{ path: "tasks/one-off/TASK.md", recurring: false, attachments: [] },
|
|
];
|
|
|
|
expect(
|
|
buildExportCheckedFiles({ filePaths, issues, categories: selection() }).has("blobs/orphan"),
|
|
).toBe(true);
|
|
expect(
|
|
buildExportCheckedFiles({ filePaths, issues, categories: selection({ attachments: false }) }).has("blobs/orphan"),
|
|
).toBe(false);
|
|
expect(
|
|
buildExportCheckedFiles({
|
|
filePaths,
|
|
issues,
|
|
categories: selection({ tasks: false, routines: false }),
|
|
}).has("blobs/orphan"),
|
|
).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("embedded asset blob ownership", () => {
|
|
const embeddedFilePaths = [
|
|
"COMPANY.md",
|
|
"agents/ceo/AGENT.md",
|
|
"tasks/one-off/TASK.md",
|
|
"blobs/embed-task",
|
|
"blobs/embed-agent",
|
|
"blobs/embed-root",
|
|
];
|
|
const embeddedIssues: ExportSelectionIssue[] = [
|
|
{ path: "tasks/one-off/TASK.md", recurring: false, attachments: [] },
|
|
];
|
|
const embeddedAssets: ExportSelectionEmbeddedAsset[] = [
|
|
{ sha256: "embed-task", ownedBy: ["tasks"] },
|
|
{ sha256: "embed-agent", ownedBy: ["agents"] },
|
|
{ sha256: "embed-root", ownedBy: ["always"] },
|
|
];
|
|
|
|
function checkedEmbedded(overrides: Partial<ExportCategorySelection> = {}): Set<string> {
|
|
return buildExportCheckedFiles({
|
|
filePaths: embeddedFilePaths,
|
|
issues: embeddedIssues,
|
|
categories: selection(overrides),
|
|
embeddedAssets,
|
|
});
|
|
}
|
|
|
|
it("keeps embedded blobs while their owning category is selected", () => {
|
|
const checked = checkedEmbedded();
|
|
expect(checked.has("blobs/embed-task")).toBe(true);
|
|
expect(checked.has("blobs/embed-agent")).toBe(true);
|
|
expect(checked.has("blobs/embed-root")).toBe(true);
|
|
});
|
|
|
|
it("drops an embedded blob when all its owners are off, even with attachments on", () => {
|
|
const checked = checkedEmbedded({ tasks: false, attachments: true });
|
|
expect(checked.has("blobs/embed-task")).toBe(false);
|
|
expect(checked.has("blobs/embed-agent")).toBe(true);
|
|
|
|
const agentsOff = checkedEmbedded({ agents: false });
|
|
expect(agentsOff.has("blobs/embed-agent")).toBe(false);
|
|
expect(agentsOff.has("blobs/embed-task")).toBe(true);
|
|
});
|
|
|
|
it("keeps embedded blobs independently of the attachments toggle", () => {
|
|
const checked = checkedEmbedded({ attachments: false });
|
|
expect(checked.has("blobs/embed-task")).toBe(true);
|
|
expect(checked.has("blobs/embed-agent")).toBe(true);
|
|
expect(checked.has("blobs/embed-root")).toBe(true);
|
|
});
|
|
|
|
it("always keeps root-owned and ownerless embedded blobs", () => {
|
|
const checked = buildExportCheckedFiles({
|
|
filePaths: ["blobs/embed-root", "blobs/no-owner"],
|
|
issues: [],
|
|
categories: selection({
|
|
agents: false,
|
|
projects: false,
|
|
skills: false,
|
|
routines: false,
|
|
tasks: false,
|
|
attachments: false,
|
|
}),
|
|
embeddedAssets: [
|
|
{ sha256: "embed-root", ownedBy: ["always"] },
|
|
{ sha256: "no-owner" },
|
|
],
|
|
});
|
|
expect(checked.has("blobs/embed-root")).toBe(true);
|
|
expect(checked.has("blobs/no-owner")).toBe(true);
|
|
});
|
|
|
|
it("lets a shared blob qualify through its attachment owner when its embedded owners are off", () => {
|
|
const issues: ExportSelectionIssue[] = [
|
|
{ path: "tasks/one-off/TASK.md", recurring: false, attachments: [] },
|
|
{ path: "tasks/weekly-report/TASK.md", recurring: true, attachments: [attachment("shared-sha")] },
|
|
];
|
|
const checked = buildExportCheckedFiles({
|
|
filePaths: ["tasks/weekly-report/TASK.md", "blobs/shared-sha"],
|
|
issues,
|
|
categories: selection({ tasks: false }),
|
|
embeddedAssets: [{ sha256: "shared-sha", ownedBy: ["tasks"] }],
|
|
});
|
|
expect(checked.has("blobs/shared-sha")).toBe(true);
|
|
|
|
const bothOff = buildExportCheckedFiles({
|
|
filePaths: ["tasks/weekly-report/TASK.md", "blobs/shared-sha"],
|
|
issues,
|
|
categories: selection({ tasks: false, routines: false }),
|
|
embeddedAssets: [{ sha256: "shared-sha", ownedBy: ["tasks"] }],
|
|
});
|
|
expect(bothOff.has("blobs/shared-sha")).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("isAttachmentsCategoryEnabled", () => {
|
|
it("is enabled while tasks or routines remain selected", () => {
|
|
expect(isAttachmentsCategoryEnabled(selection())).toBe(true);
|
|
expect(isAttachmentsCategoryEnabled(selection({ tasks: false }))).toBe(true);
|
|
expect(isAttachmentsCategoryEnabled(selection({ routines: false }))).toBe(true);
|
|
expect(isAttachmentsCategoryEnabled(selection({ tasks: false, routines: false }))).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("countExportFilesByCategory", () => {
|
|
it("counts files per category, leaving root files uncounted", () => {
|
|
expect(countExportFilesByCategory(FILE_PATHS, ISSUES)).toEqual({
|
|
agents: 2,
|
|
projects: 1,
|
|
skills: 1,
|
|
routines: 1,
|
|
tasks: 2,
|
|
attachments: 2,
|
|
});
|
|
});
|
|
});
|