Files
PaperClipAI/ui/src/lib/import-job-watch.ts
T
Devin Foley 276ae3a75d Harden company import: durable UI, async jobs, integrity guard, batched inserts (#10523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Company Import/Export (#10507) moves whole companies between
instances as portability bundles
> - Real-world use on a large company (1,418 issues, ~10.6k comments)
surfaced a cluster of related failures: the import took hours and the
browser connection died while the server kept running, a retry silently
produced a second partial import, the progress/error UI gave no durable
signal, and a cloud-tenant user couldn't even open the companies
afterward
> - Root cause of the slowness: importBundle inserted every issue,
comment, and document as a separate round-trip to a network Postgres —
an N+1-over-network pattern
> - This pull request hardens the whole import path: durable
progress/error UI, an async server-side job so imports survive dropped
connections (with a duplicate-submit guard), a fail-closed guard against
incomplete payloads, and batched inserts that cut a large import from
hours to minutes
> - The benefit is that migrating a real, large company actually
completes, is legible while it runs, and can't half-import twice

## Linked Issues or Issue Description

- Refs #10507 (the Import/Export feature this hardens). Supersedes
#10513 (the progress/error-UI piece, folded in here). No open issue;
problem described above (large-company import: slow, connection-fragile,
silently duplicable, opaque UI).

## What Changed

- **Batched inserts (perf):** importBundle pre-generates entity ids in
JS and inserts in chunked multi-row statements, so children no longer
wait on parents' generated ids. A 1,418-issue import drops from ~15,600
insert statements to **82** (190×); benchmark below. Import semantics —
collision handling, pause-on-import,
label/blocker/monitor/attachment/embedded-asset handling, blob sha
verification — are unchanged (full portability suite green).
- **Async import jobs for board sessions:** the existing cloud-tenant
async job path opens to board sessions with per-actor job keys; the
import page submits, polls, and resumes watching after a reload or
dropped connection instead of holding one fragile request. A
non-terminal job blocks a duplicate submit (409 returns the running
job), preventing the double-import.
- **Fail-closed completeness guard:** an optional `expectedFileCount` on
inline imports; the server rejects (422 `import_payload_incomplete`) a
body carrying fewer files than declared, so a re-framed/short payload
fails loudly instead of half-importing.
- **Durable progress/error UI (was #10513):** persistent progress panels
with size-aware copy, persistent error panels with retry guidance, and
inline explanation when the preview button is disabled;
request-lifecycle guards so stale previews/imports can't publish or
detach.

## Verification

- `pnpm -r` typechecks (shared, server, ui) clean.
- `company-portability.test.ts` (76) +
`company-portability-routes.test.ts` (30) green — the import correctness
net — plus new `CompanyImport.test.tsx` async/resume/409 coverage and a
new batching regression test (a 50-issue import issues <50 issue-insert
statements; rows land unchanged).
- **Batching benchmark (embedded Postgres):** at 1,418 issues × 7
comments × 1 doc — 82 insert statements vs ~15,598 one-per-row (190×),
~1s wall-clock; a row-verifying run at that scale imports all 1,418
issues / 9,926 comments / 1,418 documents with unique identifiers and no
warnings (no rows dropped by chunking). Over a network DB the round-trip
reduction is the hours→minutes lever.
- What is NOT directly measured here: wall-clock against a real network
Postgres (that happens on a staging deploy); the local timing is
network-free.

## Risks

- Batching is the load-bearing change: it rewrites the import write
path. Mitigated by the unchanged 106-test correctness suite, a new
scale/row-integrity test, and per-writer transactions (a failure rolls
back its table group; not a single outer transaction across writers —
noted, correctness preserved).
- Async jobs are in-memory (lost on server restart → pollers 404 and can
resubmit); matches the pre-existing cloud-tenant job semantics.
- `expectedFileCount` is optional (older callers unaffected); over-count
is allowed, only under-count fails closed.

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), Claude Code CLI,
extended thinking + tool use; implementation across Fable 5 subagents
with live diagnosis against a running instance.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-30 16:52:58 -07:00

99 lines
3.2 KiB
TypeScript

/**
* Client-side bookkeeping for async company import jobs.
*
* The import page submits imports as server-side jobs and polls their status,
* so a dropped connection or a page reload must not lose track of a running
* import. The job id is mirrored into `sessionStorage` (keyed per company and
* package) when a job is accepted and removed once it reaches a terminal
* state, letting the page resume watching after a reload. Jobs live in server
* memory only: a stored id the server no longer knows (restart, retention
* expiry) surfaces as a 404 and the stored entry is discarded.
*/
export const IMPORT_JOB_POLL_INTERVAL_MS = 3000;
const STORAGE_PREFIX = "paperclip:company-import-job";
export interface StoredImportJob {
jobId: string;
pauseAutomations: boolean;
}
export function importJobStorageKey(companyId: string, packageName: string): string {
return `${STORAGE_PREFIX}:${companyId}:${packageName}`;
}
export function writeStoredImportJob(storageKey: string, value: StoredImportJob): void {
try {
sessionStorage.setItem(storageKey, JSON.stringify(value));
} catch {
// Storage may be unavailable (quota, privacy mode); polling still works,
// only reload-resume is lost.
}
}
export function clearStoredImportJob(storageKey: string): void {
try {
sessionStorage.removeItem(storageKey);
} catch {
// Ignore storage failures; a stale entry is discarded on the next read.
}
}
/**
* Find a stored job for this company. Keys are scoped per package name, so
* scan the company's prefix; the server allows one live job per user, so at
* most one non-terminal entry is expected.
*/
export function readStoredImportJob(
companyId: string,
): (StoredImportJob & { storageKey: string }) | null {
try {
const prefix = `${STORAGE_PREFIX}:${companyId}:`;
for (let index = 0; index < sessionStorage.length; index += 1) {
const key = sessionStorage.key(index);
if (!key || !key.startsWith(prefix)) continue;
const raw = sessionStorage.getItem(key);
if (raw) {
try {
const parsed: unknown = JSON.parse(raw);
if (
typeof parsed === "object" && parsed !== null &&
typeof (parsed as { jobId?: unknown }).jobId === "string"
) {
return {
jobId: (parsed as { jobId: string }).jobId,
pauseAutomations: (parsed as { pauseAutomations?: unknown }).pauseAutomations === true,
storageKey: key,
};
}
} catch {
// Fall through to discard the malformed entry.
}
}
sessionStorage.removeItem(key);
index -= 1;
}
} catch {
// Storage unavailable — nothing to resume.
}
return null;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/**
* Wait one poll interval before the next status request. While the tab is
* hidden, keep waiting instead of polling: the job runs server-side, so a
* backgrounded page skips status requests until it is visible again.
*/
export async function waitForNextImportJobPoll(
intervalMs: number = IMPORT_JOB_POLL_INTERVAL_MS,
): Promise<void> {
do {
await sleep(intervalMs);
} while (typeof document !== "undefined" && document.hidden);
}