mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 05:31:46 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company Import (#10507, hardened in #10523) lets a user upload a company package on the Import page > - The page expanded the user's `.zip` into a files map and POSTed it as ONE inline JSON body — ~40MB for a real company because attachment blobs get base64-inflated > - On Paperclip Cloud that body travels browser → harness proxy → tenant, where it truncated in transit → body-parser 400 → the browser saw "Failed to fetch", and nothing imported > - Two compounding causes: the giant inline body itself, and the board async opt-in riding an `x-paperclip-cloud-*` header that the Cloud harness strips as anti-spoofing (so async never engaged and the import held one fragile synchronous connection) > - This pull request uploads the raw compressed `.zip` as a multipart request (about a third the size, already compressed) parsed server-side into the same bundle the importer consumes, and moves the async opt-in to a proxy-safe `?async=1` > - The benefit is that a large-company import actually completes through Cloud: a small compressed upload, a real async job that survives dropped connections ## Linked Issues or Issue Description - Refs #10507 / #10523 (Import/Export and its hardening). No open issue; problem described above (large-company browser import through a proxy: inline JSON body truncates → 400 → "Failed to fetch"; async opt-in header stripped by the front door → async never engages). ## What Changed - **Multipart zip transport.** The Import page uploads the raw `File` as `multipart/form-data` (field `package`, import options in a JSON `meta` field); the server unzips it into `{ rootPath, files }` and runs the exact existing preview/import logic. The `application/json` inline path is byte-identical for CLI/programmatic callers. Bare `application/zip` (meta via `?meta=`) is also accepted for programmatic use. - **Shared node zip reader.** `packages/shared/src/portability-zip.ts` (node-only subpath, not re-exported to the browser bundle — same pattern as `portability-hash.ts`); the CLI's `zip.ts` becomes a thin re-export. Identical codec (STORE + DEFLATE via `inflateRawSync`, rejects data descriptors/zip64). - **Proxy-safe async signal.** `wantsAsyncImport` = `?async=1` (board browsers, survives the harness) OR the existing `x-paperclip-cloud-async-import` header (cloud tenants, set server-side). The UI async client now uses `?async=1`. Backward compatible. - **Size + preflight.** New `PORTABLE_ZIP_UPLOAD_LIMIT_BYTES = 128MB`; the inline 56MB preflight no longer gates the zip path (it shows the compressed size instead). Async submit/poll/resume, the duplicate-guard fingerprint (now over the resolved bundle), pause-on-import, progress/error panels, and activation all apply to the multipart path. - OpenAPI documents json + multipart + zip bodies and the `async` query param. ## Verification - Full typecheck chain (shared, server, ui, cli) clean. - 152 tests across 8 files: new `portability-zip.test.ts` (STORE/DEFLATE/base64-blob byte-exact round-trip, truncation throws, data-descriptor rejection); `company-portability-routes.test.ts` +7 (multipart import+preview equals the inline bundle; async multipart 202→poll→success; board async via `?async=1` with no cloud header; cloud-tenant async via header; sync fallback with neither; truncated-zip 400, nothing imported); `CompanyImport.test.tsx` asserts the local zip sends the raw File and the inline preflight no longer blocks; `openapi-routes.test.ts` green. - NOT yet measured: the end-to-end browser upload through the live Cloud harness — verified on staging after deploy before closing out. ## Risks - Import semantics unchanged — only transport changed; the JSON inline path is byte-identical, the cloud-tenant header async path untouched. Multipart parsing is server-side (memory-bound: a ~13MB zip → ~30MB files map, fine on the server). - The bare `application/zip` path is programmatic-only and covered by content-type dispatch but not a dedicated route test (the multipart path is). ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), Claude Code CLI, extended thinking + tool use; root-caused against live logs/DB and the harness proxy source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
178 lines
6.1 KiB
TypeScript
178 lines
6.1 KiB
TypeScript
import { getPageVisibility, getVisibilityHeaderValue } from "@/lib/page-visibility";
|
|
|
|
const BASE = "/api";
|
|
|
|
export class ApiError extends Error {
|
|
status: number;
|
|
body: unknown;
|
|
|
|
constructor(message: string, status: number, body: unknown) {
|
|
super(message);
|
|
this.name = "ApiError";
|
|
this.status = status;
|
|
this.body = body;
|
|
}
|
|
}
|
|
|
|
export interface RequestOptions {
|
|
/** Abort signal wired through to `fetch` and coalescing (per-caller). */
|
|
signal?: AbortSignal;
|
|
/** Extra request headers (e.g. the async-import opt-in). Mutations only. */
|
|
headers?: Record<string, string>;
|
|
}
|
|
|
|
function abortError(): DOMException {
|
|
return new DOMException("The operation was aborted.", "AbortError");
|
|
}
|
|
|
|
/**
|
|
* Non-authoritative observability hints (PAP-12556 / Phase 1). The server treats
|
|
* these as scheduling/telemetry only and never as security signals.
|
|
*/
|
|
function applyObservabilityHeaders(headers: Headers) {
|
|
if (headers.has("X-Paperclip-Tab-Visible")) return; // caller override wins
|
|
const visibility = getPageVisibility();
|
|
headers.set("X-Paperclip-Tab-Visible", getVisibilityHeaderValue(visibility));
|
|
if (typeof window !== "undefined" && window.location) {
|
|
headers.set("X-Paperclip-Route", window.location.pathname);
|
|
}
|
|
}
|
|
|
|
async function request<T>(path: string, init?: RequestInit): Promise<T> {
|
|
const headers = new Headers(init?.headers ?? undefined);
|
|
const body = init?.body;
|
|
if (!(body instanceof FormData) && !headers.has("Content-Type")) {
|
|
headers.set("Content-Type", "application/json");
|
|
}
|
|
applyObservabilityHeaders(headers);
|
|
|
|
const res = await fetch(`${BASE}${path}`, {
|
|
headers,
|
|
credentials: "include",
|
|
...init,
|
|
});
|
|
if (!res.ok) {
|
|
const errorBody = await res.json().catch(() => null);
|
|
throw new ApiError(
|
|
(errorBody as { error?: string } | null)?.error ?? `Request failed: ${res.status}`,
|
|
res.status,
|
|
errorBody,
|
|
);
|
|
}
|
|
if (res.status === 204) return undefined as T;
|
|
return res.json();
|
|
}
|
|
|
|
// --- In-tab request coalescing for identical safe GETs -----------------------
|
|
//
|
|
// Multiple callers issuing the same GET while one is in flight share a single
|
|
// underlying fetch. Each caller keeps its own abort semantics: aborting one
|
|
// caller only cancels the shared fetch when *every* caller has aborted.
|
|
// Mutations are never coalesced.
|
|
|
|
interface InflightGet {
|
|
promise: Promise<unknown>;
|
|
controller: AbortController;
|
|
refs: Set<symbol>;
|
|
}
|
|
|
|
const inflightGets = new Map<string, InflightGet>();
|
|
|
|
function coalescedGet<T>(path: string, options?: RequestOptions): Promise<T> {
|
|
const signal = options?.signal;
|
|
if (signal?.aborted) return Promise.reject(abortError());
|
|
|
|
let entry = inflightGets.get(path);
|
|
if (!entry) {
|
|
const controller = new AbortController();
|
|
const promise = request<T>(path, { method: "GET", signal: controller.signal });
|
|
const created: InflightGet = { promise, controller, refs: new Set() };
|
|
// Clear the shared entry once settled so later calls issue a fresh request.
|
|
promise.then(
|
|
() => {
|
|
if (inflightGets.get(path) === created) inflightGets.delete(path);
|
|
},
|
|
() => {
|
|
if (inflightGets.get(path) === created) inflightGets.delete(path);
|
|
},
|
|
);
|
|
inflightGets.set(path, created);
|
|
entry = created;
|
|
}
|
|
|
|
const activeEntry = entry;
|
|
const ref = Symbol("caller");
|
|
activeEntry.refs.add(ref);
|
|
|
|
const releaseRef = () => {
|
|
if (!activeEntry.refs.delete(ref)) return;
|
|
// Last caller gone before the fetch settled → abort the shared request.
|
|
if (activeEntry.refs.size === 0 && inflightGets.get(path) === activeEntry) {
|
|
inflightGets.delete(path);
|
|
activeEntry.controller.abort();
|
|
}
|
|
};
|
|
|
|
return new Promise<T>((resolve, reject) => {
|
|
const onAbort = () => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
releaseRef();
|
|
reject(abortError());
|
|
};
|
|
if (signal) signal.addEventListener("abort", onAbort);
|
|
|
|
activeEntry.promise.then(
|
|
(value) => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
activeEntry.refs.delete(ref);
|
|
resolve(value as T);
|
|
},
|
|
(err) => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
activeEntry.refs.delete(ref);
|
|
reject(err);
|
|
},
|
|
);
|
|
});
|
|
}
|
|
|
|
/** Test-only: number of in-flight coalesced GET keys. */
|
|
export function __inflightGetCount(): number {
|
|
return inflightGets.size;
|
|
}
|
|
|
|
function isRequestOptions(value: unknown): value is RequestOptions {
|
|
return typeof value === "object" && value !== null && "signal" in value;
|
|
}
|
|
|
|
export const api = {
|
|
get: <T>(path: string, options?: RequestOptions) => coalescedGet<T>(path, options),
|
|
post: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, {
|
|
method: "POST",
|
|
body: JSON.stringify(body),
|
|
signal: options?.signal,
|
|
...(options?.headers ? { headers: options.headers } : {}),
|
|
}),
|
|
postForm: <T>(path: string, body: FormData, options?: RequestOptions) =>
|
|
request<T>(path, {
|
|
method: "POST",
|
|
body,
|
|
signal: options?.signal,
|
|
// Never set Content-Type here — the browser sets multipart/form-data with
|
|
// the boundary. Extra headers (e.g. an async opt-in) may still ride along.
|
|
...(options?.headers ? { headers: options.headers } : {}),
|
|
}),
|
|
put: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "PUT", body: JSON.stringify(body), signal: options?.signal }),
|
|
patch: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "PATCH", body: JSON.stringify(body), signal: options?.signal }),
|
|
delete: <T>(path: string, bodyOrOptions?: unknown, options?: RequestOptions) => {
|
|
const requestOptions = isRequestOptions(bodyOrOptions) ? bodyOrOptions : options;
|
|
const body = bodyOrOptions === undefined || isRequestOptions(bodyOrOptions) ? undefined : JSON.stringify(bodyOrOptions);
|
|
return request<T>(path, { method: "DELETE", ...(body === undefined ? {} : { body }), signal: requestOptions?.signal });
|
|
},
|
|
deleteWithBody: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "DELETE", body: JSON.stringify(body), signal: options?.signal }),
|
|
};
|