mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 14:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, reliable CI feedback. > - Diff-aware CI selects test files and partitions them into runner jobs. > - Stale cost estimates put several slow suites on the same runner. Broad imports also couple small tests to large modules. > - This PR refreshes test costs, runs independent graph discovery in parallel, and narrows three dependency boundaries. > - Selection safeguards and CODEOWNER-only AWS routing stay in force. ## Linked Issues or Issue Description Refs #18398. This follows the audit of [PR #18427's verification run](https://github.com/paperclipai/paperclip/actions/runs/41169978784). The audit found 403 selected general-server files without timing weights. Three suites measured at 232s, 122s and 116s shared one shard because each received a roughly 2s fallback weight. Local graph discovery across four projects took 43.7s serially and 21.7s with two workers. A reverse-order repeat took 37.7s serially and 22.6s with two workers. Both comparisons produced identical selection results. This is a local measurement, not a claim about production CI speed. ## What Changed - Refresh general-server costs for 1,285 of 1,309 current files and serialized costs for all 163 files from two successful protected-master Cloud readiness runs. Retain prior costs for unmeasured files, including dedicated chat and native-runner suites. - Record passing server file costs, including collection, setup and environment preparation. A weekly hosted workflow validates trusted source runs and opens a timing-only PR. It tests shard coverage, pauses while a refresh PR is open, and never merges automatically. - Discover at most two project graphs concurrently in isolated, heap-limited Node processes. Cache repeated inventory reads within one selection. Worker errors and timeouts retain the full task plan. - Fetch only the merge commit and parents for the selector. A real shallow-clone regression proves the complete diff still includes more than 300 files, renames and deletions. - Share the identical log viewer and run details between both AgentDetail pages. The viewer test imports the shared implementation and runs each of its 13 distinct cases once. - Extract the pure API error class while preserving the API client's public exports. Import delivery work signals through their existing narrow DB export. ## Verification - PASS: `pnpm -r typecheck` and `pnpm build`. - PASS: 284 CI selection, timing, shallow-diff, sharding and routing tests. - PASS: 74 further cache, routing and graph-proof tests. - PASS: 94 targeted UI assertions across the viewer, run invocation, API errors/client and progress helpers. All 76 assertions in the 12 server suites made independently selectable also pass. - PASS: `pnpm check:token-gates`, module-boundary checks, YAML parsing and `git diff --check`. - PASS: [Full PR CI and required `ci / verify` aggregate](https://github.com/paperclipai/paperclip/actions/runs/41206683345), including all 12 general-server shards, all nine serialized shards, workspace groups, chat suites, Runner lanes and build checks. - PASS: [GitHub hosted real-workspace selector and cache/routing qualification](https://github.com/paperclipai/paperclip/actions/runs/41206683205). - PASS: Both Greptile and Paperclip reviews are 5/5 with no actionable findings or inline comments on `7dc65c51b8`. The PR has no merge conflicts. - PASS: The production refresh parser accepted all 107 structured file-cost records from a real successful server CI job, including setup/loading cost. - Local `pnpm test:run` completed all six chat shards successfully. The duplicate monolithic local run was stopped after the complete remote suite passed; it was not completed locally. The focused log-viewer command improved from 18.84s (26 duplicate variant cases) to 8.49s (all 13 distinct behavior cases). The dependency audit confirms 12 server files and the API-error test no longer need conservative always-run retention. Repartitioning the audited PR with the new costs changes the slowest shard from 810s to 591s of summed historical test execution (1,124 of 1,142 files measured). This is a workload simulation, not measured new CI wall time. ## Risks - Two graph processes use more peak memory. Each has a 2 GiB heap limit and a three-minute timeout. Any failure preserves the original task plan. Real workspace qualification runs in GitHub CI. - Historical logs contain test/hook costs only. New reporter records add loading/setup costs after merge. The manifests identify measurement coverage and source runs. These weights change placement, never permission to skip tests. - Runner load adds timing noise. The refresh uses medians and requires at least 90% measurement coverage in each lane. - GitHub may require approval to run checks on a PR created with `GITHUB_TOKEN`. Refreshes remain reviewable and do not merge themselves. - The shared viewer is a mechanical extraction. Both pages retain their existing exports, and the behavior assertions remain covered. AWS runner authorization is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, reasoning, code execution and GitHub tools. The exact deployment ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and six chat shards; complete suite passed in CI as detailed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
287 lines
11 KiB
TypeScript
287 lines
11 KiB
TypeScript
import { getPageVisibility, getVisibilityHeaderValue } from "@/lib/page-visibility";
|
|
import { tenantSessionRecovery } from "@/lib/tenant-session-recovery";
|
|
import { ApiError, apiErrorFromResponse } from "./api-error";
|
|
export { ApiError, apiErrorFromResponse, type ApiErrorOptions } from "./api-error";
|
|
import { readApiJson } from "./response";
|
|
|
|
const BASE = "/api";
|
|
|
|
// --- Dev/QA outage simulator ----------------------------------------------------
|
|
//
|
|
// Reproduces a deploy blip without a deploy. Dev and QA builds only:
|
|
//
|
|
// localStorage.paperclipSimulateOutage = "503:tenant_app_unavailable" // until removed
|
|
// localStorage.paperclipSimulateOutage = "503:tenant_app_unavailable@20" // for 20 seconds
|
|
// localStorage.paperclipSimulateOutage = "502" // non-JSON proxy page
|
|
// localStorage.paperclipSimulateOutage = "network" // fetch rejects (connection drop)
|
|
//
|
|
// Every same-origin API request made through `apiFetch` (the client, health,
|
|
// session) fails with the chosen shape. Remove the key to end the outage.
|
|
|
|
export const OUTAGE_SIMULATOR_STORAGE_KEY = "paperclipSimulateOutage";
|
|
|
|
export type SimulatedOutage =
|
|
| { kind: "network"; durationMs: number | null }
|
|
| { kind: "status"; status: number; code: string | null; durationMs: number | null };
|
|
|
|
export function parseSimulatedOutage(raw: string | null | undefined): SimulatedOutage | null {
|
|
const match = /^\s*(network|(\d{3})(?::([A-Za-z0-9_.-]+))?)(?:@(\d+(?:\.\d+)?))?\s*$/.exec(raw ?? "");
|
|
if (!match) return null;
|
|
const durationMs = match[4] ? Math.round(Number(match[4]) * 1000) : null;
|
|
if (match[1] === "network") return { kind: "network", durationMs };
|
|
const status = Number(match[2]);
|
|
if (status < 400 || status > 599) return null;
|
|
return { kind: "status", status, code: match[3] ?? null, durationMs };
|
|
}
|
|
|
|
function outageSimulatorEnabled(): boolean {
|
|
return import.meta.env.DEV || import.meta.env.MODE === "qa";
|
|
}
|
|
|
|
let simulatedOutageWindow: { raw: string; endsAt: number | null } | null = null;
|
|
|
|
/** The outage to simulate for the next request, or null. Never throws. */
|
|
export function activeSimulatedOutage(now = Date.now()): SimulatedOutage | null {
|
|
if (!outageSimulatorEnabled() || typeof window === "undefined") return null;
|
|
let raw: string | null = null;
|
|
try {
|
|
raw = window.localStorage?.getItem(OUTAGE_SIMULATOR_STORAGE_KEY) ?? null;
|
|
} catch {
|
|
return null;
|
|
}
|
|
const outage = parseSimulatedOutage(raw);
|
|
if (!raw || !outage) {
|
|
simulatedOutageWindow = null;
|
|
return null;
|
|
}
|
|
if (simulatedOutageWindow?.raw !== raw) {
|
|
simulatedOutageWindow = { raw, endsAt: outage.durationMs === null ? null : now + outage.durationMs };
|
|
}
|
|
if (simulatedOutageWindow.endsAt !== null && now >= simulatedOutageWindow.endsAt) {
|
|
simulatedOutageWindow = null;
|
|
try {
|
|
window.localStorage.removeItem(OUTAGE_SIMULATOR_STORAGE_KEY);
|
|
} catch {
|
|
// Storage can be unavailable; the outage still ends for this tab.
|
|
}
|
|
return null;
|
|
}
|
|
return outage;
|
|
}
|
|
|
|
function simulatedOutageResponse(outage: Extract<SimulatedOutage, { kind: "status" }>): Response {
|
|
if (!outage.code) {
|
|
// What a proxy serves while the app is down: an HTML page, not JSON.
|
|
return new Response("<!doctype html><title>Unavailable</title>", {
|
|
status: outage.status,
|
|
headers: { "Content-Type": "text/html" },
|
|
});
|
|
}
|
|
return new Response(JSON.stringify({ error: outage.code }), {
|
|
status: outage.status,
|
|
headers: { "Content-Type": "application/json" },
|
|
});
|
|
}
|
|
|
|
/**
|
|
* `fetch` for same-origin API calls. Identical to `fetch` except that the
|
|
* dev/QA outage simulator can intercept it.
|
|
*/
|
|
export async function apiFetch(input: string, init?: RequestInit): Promise<Response> {
|
|
const outage = activeSimulatedOutage();
|
|
if (outage) {
|
|
if (init?.signal?.aborted) throw abortError();
|
|
if (outage.kind === "network") throw new TypeError("Failed to fetch");
|
|
return simulatedOutageResponse(outage);
|
|
}
|
|
return fetch(input, init);
|
|
}
|
|
|
|
export interface RequestOptions {
|
|
/** Abort signal wired through to `fetch` and coalescing (per-caller). */
|
|
signal?: AbortSignal;
|
|
/** Extra request headers (e.g. the async-import opt-in). Mutations only. */
|
|
headers?: Record<string, string>;
|
|
/** The `fetch` cache mode. Use `"no-store"` for a response that must never
|
|
* come from the browser's HTTP cache. */
|
|
cache?: RequestCache;
|
|
}
|
|
|
|
function abortError(): DOMException {
|
|
return new DOMException("The operation was aborted.", "AbortError");
|
|
}
|
|
|
|
/**
|
|
* Non-authoritative observability hints (PAP-12556 / Phase 1). The server treats
|
|
* these as scheduling/telemetry only and never as security signals.
|
|
*/
|
|
function applyObservabilityHeaders(headers: Headers) {
|
|
if (headers.has("X-Paperclip-Tab-Visible")) return; // caller override wins
|
|
const visibility = getPageVisibility();
|
|
headers.set("X-Paperclip-Tab-Visible", getVisibilityHeaderValue(visibility));
|
|
if (typeof window !== "undefined" && window.location) {
|
|
headers.set("X-Paperclip-Route", window.location.pathname);
|
|
}
|
|
}
|
|
|
|
export async function requestResponse(path: string, init?: RequestInit): Promise<Response> {
|
|
const headers = new Headers(init?.headers ?? undefined);
|
|
const body = init?.body;
|
|
if (!(body instanceof FormData) && !headers.has("Content-Type")) {
|
|
headers.set("Content-Type", "application/json");
|
|
}
|
|
applyObservabilityHeaders(headers);
|
|
|
|
const res = await apiFetch(`${BASE}${path}`, {
|
|
credentials: "include",
|
|
...init,
|
|
headers,
|
|
});
|
|
if (!res.ok) {
|
|
const errorBody = await readApiJson(res);
|
|
const recovery = tenantSessionRecovery.recoverIfNeeded(res.status, errorBody);
|
|
if (recovery) return recovery;
|
|
throw apiErrorFromResponse(res, errorBody);
|
|
}
|
|
return res;
|
|
}
|
|
|
|
async function request<T>(path: string, init?: RequestInit): Promise<T> {
|
|
const res = await requestResponse(path, init);
|
|
if (res.status === 204) return undefined as T;
|
|
return readApiJson<T>(res);
|
|
}
|
|
|
|
// --- In-tab request coalescing for identical safe GETs -----------------------
|
|
//
|
|
// Multiple callers issuing the same GET while one is in flight share a single
|
|
// underlying fetch. Each caller keeps its own abort semantics: aborting one
|
|
// caller only cancels the shared fetch when *every* caller has aborted.
|
|
// Mutations are never coalesced.
|
|
|
|
interface InflightGet {
|
|
promise: Promise<unknown>;
|
|
controller: AbortController;
|
|
refs: Set<symbol>;
|
|
}
|
|
|
|
const inflightGets = new Map<string, InflightGet>();
|
|
|
|
function coalescedGet<T>(path: string, options?: RequestOptions): Promise<T> {
|
|
const signal = options?.signal;
|
|
if (signal?.aborted) return Promise.reject(abortError());
|
|
|
|
let entry = inflightGets.get(path);
|
|
if (!entry) {
|
|
const controller = new AbortController();
|
|
const promise = request<T>(path, {
|
|
method: "GET",
|
|
signal: controller.signal,
|
|
...(options?.cache ? { cache: options.cache } : {}),
|
|
});
|
|
const created: InflightGet = { promise, controller, refs: new Set() };
|
|
// Clear the shared entry once settled so later calls issue a fresh request.
|
|
promise.then(
|
|
() => {
|
|
if (inflightGets.get(path) === created) inflightGets.delete(path);
|
|
},
|
|
() => {
|
|
if (inflightGets.get(path) === created) inflightGets.delete(path);
|
|
},
|
|
);
|
|
inflightGets.set(path, created);
|
|
entry = created;
|
|
}
|
|
|
|
const activeEntry = entry;
|
|
const ref = Symbol("caller");
|
|
activeEntry.refs.add(ref);
|
|
|
|
const releaseRef = () => {
|
|
if (!activeEntry.refs.delete(ref)) return;
|
|
// Last caller gone before the fetch settled → abort the shared request.
|
|
if (activeEntry.refs.size === 0 && inflightGets.get(path) === activeEntry) {
|
|
inflightGets.delete(path);
|
|
activeEntry.controller.abort();
|
|
}
|
|
};
|
|
|
|
return new Promise<T>((resolve, reject) => {
|
|
const onAbort = () => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
releaseRef();
|
|
reject(abortError());
|
|
};
|
|
if (signal) signal.addEventListener("abort", onAbort);
|
|
|
|
activeEntry.promise.then(
|
|
(value) => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
activeEntry.refs.delete(ref);
|
|
resolve(value as T);
|
|
},
|
|
(err) => {
|
|
signal?.removeEventListener("abort", onAbort);
|
|
activeEntry.refs.delete(ref);
|
|
reject(err);
|
|
},
|
|
);
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Stop later callers from joining the in-flight GET for `path`.
|
|
*
|
|
* Coalescing keys on the path alone, so a GET issued under one account's session
|
|
* can be joined by a caller that runs after the account changed — and handed the
|
|
* previous account's response. Detaching leaves that request to settle for the
|
|
* callers that asked for it, and makes the next call issue a fresh one. It does
|
|
* not abort, because those callers still want what they asked for.
|
|
*/
|
|
export function detachInflightGet(path: string): void {
|
|
inflightGets.delete(path);
|
|
}
|
|
|
|
function isRequestOptions(value: unknown): value is RequestOptions {
|
|
return typeof value === "object" && value !== null && "signal" in value;
|
|
}
|
|
|
|
export const api = {
|
|
get: <T>(path: string, options?: RequestOptions) => coalescedGet<T>(path, options),
|
|
post: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, {
|
|
method: "POST",
|
|
body: JSON.stringify(body),
|
|
signal: options?.signal,
|
|
...(options?.headers ? { headers: options.headers } : {}),
|
|
}),
|
|
postForm: <T>(path: string, body: FormData, options?: RequestOptions) =>
|
|
request<T>(path, {
|
|
method: "POST",
|
|
body,
|
|
signal: options?.signal,
|
|
// Never set Content-Type here — the browser sets multipart/form-data with
|
|
// the boundary. Extra headers (e.g. an async opt-in) may still ride along.
|
|
...(options?.headers ? { headers: options.headers } : {}),
|
|
}),
|
|
put: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "PUT", body: JSON.stringify(body), signal: options?.signal }),
|
|
/** Raw binary upload (e.g. one chunked import-transfer part); the body travels as-is. */
|
|
putRaw: <T>(path: string, body: Blob, options?: RequestOptions) =>
|
|
request<T>(path, {
|
|
method: "PUT",
|
|
body,
|
|
signal: options?.signal,
|
|
headers: { "Content-Type": "application/octet-stream", ...(options?.headers ?? {}) },
|
|
}),
|
|
patch: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "PATCH", body: JSON.stringify(body), signal: options?.signal }),
|
|
delete: <T>(path: string, bodyOrOptions?: unknown, options?: RequestOptions) => {
|
|
const requestOptions = isRequestOptions(bodyOrOptions) ? bodyOrOptions : options;
|
|
const body = bodyOrOptions === undefined || isRequestOptions(bodyOrOptions) ? undefined : JSON.stringify(bodyOrOptions);
|
|
return request<T>(path, { method: "DELETE", ...(body === undefined ? {} : { body }), signal: requestOptions?.signal });
|
|
},
|
|
deleteWithBody: <T>(path: string, body: unknown, options?: RequestOptions) =>
|
|
request<T>(path, { method: "DELETE", body: JSON.stringify(body), signal: options?.signal }),
|
|
};
|