mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 16:35:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company Import/Export (#10507) moves whole companies between instances as portability bundles > - Real-world use on a large company (1,418 issues, ~10.6k comments) surfaced a cluster of related failures: the import took hours and the browser connection died while the server kept running, a retry silently produced a second partial import, the progress/error UI gave no durable signal, and a cloud-tenant user couldn't even open the companies afterward > - Root cause of the slowness: importBundle inserted every issue, comment, and document as a separate round-trip to a network Postgres — an N+1-over-network pattern > - This pull request hardens the whole import path: durable progress/error UI, an async server-side job so imports survive dropped connections (with a duplicate-submit guard), a fail-closed guard against incomplete payloads, and batched inserts that cut a large import from hours to minutes > - The benefit is that migrating a real, large company actually completes, is legible while it runs, and can't half-import twice ## Linked Issues or Issue Description - Refs #10507 (the Import/Export feature this hardens). Supersedes #10513 (the progress/error-UI piece, folded in here). No open issue; problem described above (large-company import: slow, connection-fragile, silently duplicable, opaque UI). ## What Changed - **Batched inserts (perf):** importBundle pre-generates entity ids in JS and inserts in chunked multi-row statements, so children no longer wait on parents' generated ids. A 1,418-issue import drops from ~15,600 insert statements to **82** (190×); benchmark below. Import semantics — collision handling, pause-on-import, label/blocker/monitor/attachment/embedded-asset handling, blob sha verification — are unchanged (full portability suite green). - **Async import jobs for board sessions:** the existing cloud-tenant async job path opens to board sessions with per-actor job keys; the import page submits, polls, and resumes watching after a reload or dropped connection instead of holding one fragile request. A non-terminal job blocks a duplicate submit (409 returns the running job), preventing the double-import. - **Fail-closed completeness guard:** an optional `expectedFileCount` on inline imports; the server rejects (422 `import_payload_incomplete`) a body carrying fewer files than declared, so a re-framed/short payload fails loudly instead of half-importing. - **Durable progress/error UI (was #10513):** persistent progress panels with size-aware copy, persistent error panels with retry guidance, and inline explanation when the preview button is disabled; request-lifecycle guards so stale previews/imports can't publish or detach. ## Verification - `pnpm -r` typechecks (shared, server, ui) clean. - `company-portability.test.ts` (76) + `company-portability-routes.test.ts` (30) green — the import correctness net — plus new `CompanyImport.test.tsx` async/resume/409 coverage and a new batching regression test (a 50-issue import issues <50 issue-insert statements; rows land unchanged). - **Batching benchmark (embedded Postgres):** at 1,418 issues × 7 comments × 1 doc — 82 insert statements vs ~15,598 one-per-row (190×), ~1s wall-clock; a row-verifying run at that scale imports all 1,418 issues / 9,926 comments / 1,418 documents with unique identifiers and no warnings (no rows dropped by chunking). Over a network DB the round-trip reduction is the hours→minutes lever. - What is NOT directly measured here: wall-clock against a real network Postgres (that happens on a staging deploy); the local timing is network-free. ## Risks - Batching is the load-bearing change: it rewrites the import write path. Mitigated by the unchanged 106-test correctness suite, a new scale/row-integrity test, and per-writer transactions (a failure rolls back its table group; not a single outer transaction across writers — noted, correctness preserved). - Async jobs are in-memory (lost on server restart → pollers 404 and can resubmit); matches the pre-existing cloud-tenant job semantics. - `expectedFileCount` is optional (older callers unaffected); over-count is allowed, only under-count fails closed. ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), Claude Code CLI, extended thinking + tool use; implementation across Fable 5 subagents with live diagnosis against a running instance. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
99 lines
3.2 KiB
TypeScript
99 lines
3.2 KiB
TypeScript
/**
|
|
* Client-side bookkeeping for async company import jobs.
|
|
*
|
|
* The import page submits imports as server-side jobs and polls their status,
|
|
* so a dropped connection or a page reload must not lose track of a running
|
|
* import. The job id is mirrored into `sessionStorage` (keyed per company and
|
|
* package) when a job is accepted and removed once it reaches a terminal
|
|
* state, letting the page resume watching after a reload. Jobs live in server
|
|
* memory only: a stored id the server no longer knows (restart, retention
|
|
* expiry) surfaces as a 404 and the stored entry is discarded.
|
|
*/
|
|
|
|
export const IMPORT_JOB_POLL_INTERVAL_MS = 3000;
|
|
|
|
const STORAGE_PREFIX = "paperclip:company-import-job";
|
|
|
|
export interface StoredImportJob {
|
|
jobId: string;
|
|
pauseAutomations: boolean;
|
|
}
|
|
|
|
export function importJobStorageKey(companyId: string, packageName: string): string {
|
|
return `${STORAGE_PREFIX}:${companyId}:${packageName}`;
|
|
}
|
|
|
|
export function writeStoredImportJob(storageKey: string, value: StoredImportJob): void {
|
|
try {
|
|
sessionStorage.setItem(storageKey, JSON.stringify(value));
|
|
} catch {
|
|
// Storage may be unavailable (quota, privacy mode); polling still works,
|
|
// only reload-resume is lost.
|
|
}
|
|
}
|
|
|
|
export function clearStoredImportJob(storageKey: string): void {
|
|
try {
|
|
sessionStorage.removeItem(storageKey);
|
|
} catch {
|
|
// Ignore storage failures; a stale entry is discarded on the next read.
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Find a stored job for this company. Keys are scoped per package name, so
|
|
* scan the company's prefix; the server allows one live job per user, so at
|
|
* most one non-terminal entry is expected.
|
|
*/
|
|
export function readStoredImportJob(
|
|
companyId: string,
|
|
): (StoredImportJob & { storageKey: string }) | null {
|
|
try {
|
|
const prefix = `${STORAGE_PREFIX}:${companyId}:`;
|
|
for (let index = 0; index < sessionStorage.length; index += 1) {
|
|
const key = sessionStorage.key(index);
|
|
if (!key || !key.startsWith(prefix)) continue;
|
|
const raw = sessionStorage.getItem(key);
|
|
if (raw) {
|
|
try {
|
|
const parsed: unknown = JSON.parse(raw);
|
|
if (
|
|
typeof parsed === "object" && parsed !== null &&
|
|
typeof (parsed as { jobId?: unknown }).jobId === "string"
|
|
) {
|
|
return {
|
|
jobId: (parsed as { jobId: string }).jobId,
|
|
pauseAutomations: (parsed as { pauseAutomations?: unknown }).pauseAutomations === true,
|
|
storageKey: key,
|
|
};
|
|
}
|
|
} catch {
|
|
// Fall through to discard the malformed entry.
|
|
}
|
|
}
|
|
sessionStorage.removeItem(key);
|
|
index -= 1;
|
|
}
|
|
} catch {
|
|
// Storage unavailable — nothing to resume.
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function sleep(ms: number): Promise<void> {
|
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
}
|
|
|
|
/**
|
|
* Wait one poll interval before the next status request. While the tab is
|
|
* hidden, keep waiting instead of polling: the job runs server-side, so a
|
|
* backgrounded page skips status requests until it is visible again.
|
|
*/
|
|
export async function waitForNextImportJobPoll(
|
|
intervalMs: number = IMPORT_JOB_POLL_INTERVAL_MS,
|
|
): Promise<void> {
|
|
do {
|
|
await sleep(intervalMs);
|
|
} while (typeof document !== "undefined" && document.hidden);
|
|
}
|