Files
PaperClipAI/ui/src/api/client.ts
T
Devin Foley 5ec7ce76e5 Upload company import packages as compressed zip uploads (fix large-company imports through Cloud) (#10531)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Company Import (#10507, hardened in #10523) lets a user upload a
company package on the Import page
> - The page expanded the user's `.zip` into a files map and POSTed it
as ONE inline JSON body — ~40MB for a real company because attachment
blobs get base64-inflated
> - On Paperclip Cloud that body travels browser → harness proxy →
tenant, where it truncated in transit → body-parser 400 → the browser
saw "Failed to fetch", and nothing imported
> - Two compounding causes: the giant inline body itself, and the board
async opt-in riding an `x-paperclip-cloud-*` header that the Cloud
harness strips as anti-spoofing (so async never engaged and the import
held one fragile synchronous connection)
> - This pull request uploads the raw compressed `.zip` as a multipart
request (about a third the size, already compressed) parsed server-side
into the same bundle the importer consumes, and moves the async opt-in
to a proxy-safe `?async=1`
> - The benefit is that a large-company import actually completes
through Cloud: a small compressed upload, a real async job that survives
dropped connections

## Linked Issues or Issue Description

- Refs #10507 / #10523 (Import/Export and its hardening). No open issue;
problem described above (large-company browser import through a proxy:
inline JSON body truncates → 400 → "Failed to fetch"; async opt-in
header stripped by the front door → async never engages).

## What Changed

- **Multipart zip transport.** The Import page uploads the raw `File` as
`multipart/form-data` (field `package`, import options in a JSON `meta`
field); the server unzips it into `{ rootPath, files }` and runs the
exact existing preview/import logic. The `application/json` inline path
is byte-identical for CLI/programmatic callers. Bare `application/zip`
(meta via `?meta=`) is also accepted for programmatic use.
- **Shared node zip reader.** `packages/shared/src/portability-zip.ts`
(node-only subpath, not re-exported to the browser bundle — same pattern
as `portability-hash.ts`); the CLI's `zip.ts` becomes a thin re-export.
Identical codec (STORE + DEFLATE via `inflateRawSync`, rejects data
descriptors/zip64).
- **Proxy-safe async signal.** `wantsAsyncImport` = `?async=1` (board
browsers, survives the harness) OR the existing
`x-paperclip-cloud-async-import` header (cloud tenants, set
server-side). The UI async client now uses `?async=1`. Backward
compatible.
- **Size + preflight.** New `PORTABLE_ZIP_UPLOAD_LIMIT_BYTES = 128MB`;
the inline 56MB preflight no longer gates the zip path (it shows the
compressed size instead). Async submit/poll/resume, the duplicate-guard
fingerprint (now over the resolved bundle), pause-on-import,
progress/error panels, and activation all apply to the multipart path.
- OpenAPI documents json + multipart + zip bodies and the `async` query
param.

## Verification

- Full typecheck chain (shared, server, ui, cli) clean.
- 152 tests across 8 files: new `portability-zip.test.ts`
(STORE/DEFLATE/base64-blob byte-exact round-trip, truncation throws,
data-descriptor rejection); `company-portability-routes.test.ts` +7
(multipart import+preview equals the inline bundle; async multipart
202→poll→success; board async via `?async=1` with no cloud header;
cloud-tenant async via header; sync fallback with neither; truncated-zip
400, nothing imported); `CompanyImport.test.tsx` asserts the local zip
sends the raw File and the inline preflight no longer blocks;
`openapi-routes.test.ts` green.
- NOT yet measured: the end-to-end browser upload through the live Cloud
harness — verified on staging after deploy before closing out.

## Risks

- Import semantics unchanged — only transport changed; the JSON inline
path is byte-identical, the cloud-tenant header async path untouched.
Multipart parsing is server-side (memory-bound: a ~13MB zip → ~30MB
files map, fine on the server).
- The bare `application/zip` path is programmatic-only and covered by
content-type dispatch but not a dedicated route test (the multipart path
is).

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), Claude Code CLI,
extended thinking + tool use; root-caused against live logs/DB and the
harness proxy source.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-30 18:48:22 -07:00

178 lines
6.1 KiB
TypeScript

import { getPageVisibility, getVisibilityHeaderValue } from "@/lib/page-visibility";
const BASE = "/api";
export class ApiError extends Error {
status: number;
body: unknown;
constructor(message: string, status: number, body: unknown) {
super(message);
this.name = "ApiError";
this.status = status;
this.body = body;
}
}
export interface RequestOptions {
/** Abort signal wired through to `fetch` and coalescing (per-caller). */
signal?: AbortSignal;
/** Extra request headers (e.g. the async-import opt-in). Mutations only. */
headers?: Record<string, string>;
}
function abortError(): DOMException {
return new DOMException("The operation was aborted.", "AbortError");
}
/**
* Non-authoritative observability hints (PAP-12556 / Phase 1). The server treats
* these as scheduling/telemetry only and never as security signals.
*/
function applyObservabilityHeaders(headers: Headers) {
if (headers.has("X-Paperclip-Tab-Visible")) return; // caller override wins
const visibility = getPageVisibility();
headers.set("X-Paperclip-Tab-Visible", getVisibilityHeaderValue(visibility));
if (typeof window !== "undefined" && window.location) {
headers.set("X-Paperclip-Route", window.location.pathname);
}
}
async function request<T>(path: string, init?: RequestInit): Promise<T> {
const headers = new Headers(init?.headers ?? undefined);
const body = init?.body;
if (!(body instanceof FormData) && !headers.has("Content-Type")) {
headers.set("Content-Type", "application/json");
}
applyObservabilityHeaders(headers);
const res = await fetch(`${BASE}${path}`, {
headers,
credentials: "include",
...init,
});
if (!res.ok) {
const errorBody = await res.json().catch(() => null);
throw new ApiError(
(errorBody as { error?: string } | null)?.error ?? `Request failed: ${res.status}`,
res.status,
errorBody,
);
}
if (res.status === 204) return undefined as T;
return res.json();
}
// --- In-tab request coalescing for identical safe GETs -----------------------
//
// Multiple callers issuing the same GET while one is in flight share a single
// underlying fetch. Each caller keeps its own abort semantics: aborting one
// caller only cancels the shared fetch when *every* caller has aborted.
// Mutations are never coalesced.
interface InflightGet {
promise: Promise<unknown>;
controller: AbortController;
refs: Set<symbol>;
}
const inflightGets = new Map<string, InflightGet>();
function coalescedGet<T>(path: string, options?: RequestOptions): Promise<T> {
const signal = options?.signal;
if (signal?.aborted) return Promise.reject(abortError());
let entry = inflightGets.get(path);
if (!entry) {
const controller = new AbortController();
const promise = request<T>(path, { method: "GET", signal: controller.signal });
const created: InflightGet = { promise, controller, refs: new Set() };
// Clear the shared entry once settled so later calls issue a fresh request.
promise.then(
() => {
if (inflightGets.get(path) === created) inflightGets.delete(path);
},
() => {
if (inflightGets.get(path) === created) inflightGets.delete(path);
},
);
inflightGets.set(path, created);
entry = created;
}
const activeEntry = entry;
const ref = Symbol("caller");
activeEntry.refs.add(ref);
const releaseRef = () => {
if (!activeEntry.refs.delete(ref)) return;
// Last caller gone before the fetch settled → abort the shared request.
if (activeEntry.refs.size === 0 && inflightGets.get(path) === activeEntry) {
inflightGets.delete(path);
activeEntry.controller.abort();
}
};
return new Promise<T>((resolve, reject) => {
const onAbort = () => {
signal?.removeEventListener("abort", onAbort);
releaseRef();
reject(abortError());
};
if (signal) signal.addEventListener("abort", onAbort);
activeEntry.promise.then(
(value) => {
signal?.removeEventListener("abort", onAbort);
activeEntry.refs.delete(ref);
resolve(value as T);
},
(err) => {
signal?.removeEventListener("abort", onAbort);
activeEntry.refs.delete(ref);
reject(err);
},
);
});
}
/** Test-only: number of in-flight coalesced GET keys. */
export function __inflightGetCount(): number {
return inflightGets.size;
}
function isRequestOptions(value: unknown): value is RequestOptions {
return typeof value === "object" && value !== null && "signal" in value;
}
export const api = {
get: <T>(path: string, options?: RequestOptions) => coalescedGet<T>(path, options),
post: <T>(path: string, body: unknown, options?: RequestOptions) =>
request<T>(path, {
method: "POST",
body: JSON.stringify(body),
signal: options?.signal,
...(options?.headers ? { headers: options.headers } : {}),
}),
postForm: <T>(path: string, body: FormData, options?: RequestOptions) =>
request<T>(path, {
method: "POST",
body,
signal: options?.signal,
// Never set Content-Type here — the browser sets multipart/form-data with
// the boundary. Extra headers (e.g. an async opt-in) may still ride along.
...(options?.headers ? { headers: options.headers } : {}),
}),
put: <T>(path: string, body: unknown, options?: RequestOptions) =>
request<T>(path, { method: "PUT", body: JSON.stringify(body), signal: options?.signal }),
patch: <T>(path: string, body: unknown, options?: RequestOptions) =>
request<T>(path, { method: "PATCH", body: JSON.stringify(body), signal: options?.signal }),
delete: <T>(path: string, bodyOrOptions?: unknown, options?: RequestOptions) => {
const requestOptions = isRequestOptions(bodyOrOptions) ? bodyOrOptions : options;
const body = bodyOrOptions === undefined || isRequestOptions(bodyOrOptions) ? undefined : JSON.stringify(bodyOrOptions);
return request<T>(path, { method: "DELETE", ...(body === undefined ? {} : { body }), signal: requestOptions?.signal });
},
deleteWithBody: <T>(path: string, body: unknown, options?: RequestOptions) =>
request<T>(path, { method: "DELETE", body: JSON.stringify(body), signal: options?.signal }),
};