Files
PaperClipAI/server/src/__tests__/cloud-routes.test.ts
Devin FoleyandPaperclip 90182b4f8b Report bounded Cloud portfolio failure diagnostics (#15340)
## Thinking Path

> - Paperclip manages work for AI agents and their companies.
> - Cloud instances proxy a trusted user's portfolio through the control
plane.
> - A failed request returns a generic error to the client.
> - That replacement error loses the failure phase and network code in
Sentry.
> - Operators need bounded evidence without upstream messages or
credentials.
> - This PR adds safe diagnostics while preserving the existing request
behavior.

## Linked Issues or Issue Description

Refs #10850, which added the portfolio proxy. No open PR for this
diagnostic gap was found.

**What happened?**

A rejected portfolio fetch becomes a generic 502 in Sentry. The event
cannot distinguish a connection reset, deadline, HTTP response failure,
or body failure. The route's previous warning also included the original
error and a stack identifier.

**Steps to reproduce**

Make the portfolio proxy's fetch reject with a TypeError whose cause has
`code: ECONNRESET`. The client correctly receives the generic 502, but
the captured replacement error loses that code.

**Expected behavior**

Keep the existing client response. Attach only bounded server-side
diagnostic fields to the failure event. Do not retry the request or
expose the original error.

## What Changed

- Add a typed portfolio error with a private frozen diagnostic record:
phase, upstream HTTP status, elapsed milliseconds, and an allowlisted
network code.
- Read at most four error/cause objects through own data properties.
Unknown codes, messages, getters, and out-of-range values do not enter
the record.
- Replace the route's raw-error warnings with safe fields. Send a plain
error plus event-local context through the existing optional Sentry
gate. Keep the route callsite and default fingerprint policy.
- Preserve authentication, trusted headers, cookies, exact HTTP error
bodies, cache behavior, the ten-second deadline, and one fetch per
request. Public responses receive no diagnostic fields.
- Document the fields and test HTTP behavior, privacy, and event
isolation with the real Sentry SDK.

## Verification

- `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run
server/src/__tests__/cloud-portfolio-error.test.ts
server/src/__tests__/cloud-routes.test.ts
server/src/__tests__/sentry.test.ts
server/src/__tests__/run-failure-sentry-real-sdk.test.ts`: 65 passed,
with the audited optional SDK installed.
- Full local `pnpm -r typecheck` and `pnpm build` passed on Node 24.21.0
and pnpm 9.15.4.
- Independent review found no blockers and independently passed all 65
tests, including the real SDK checks, on this exact commit.
- Full Linux CI passed on this exact commit and provides aggregate suite
coverage (54 successful checks, 2 intentional skips). A duplicate full
local aggregate was not run.
- The first SDK contract job failed before tests when npm could not
resolve an OpenTelemetry transitive package. A subsequent empty-cache
install first encountered a missing tarball, then succeeded after the
registry artifact became available. All 6 real-SDK tests passed against
that fresh install; the single unchanged-head CI retry passed. The SDK
pin, workflow, and dependency files are unchanged.
- The first browser shard 8 run timed out waiting for the inbox retry
reply after 45 seconds. The unchanged isolated case passed (1/1), and
the test, UI handler, fixture, and recovery files match the base commit.
The failed log contains no wakeup POST before the test's immediate
navigation; a navigation/request timing race is suspected but unproven
without a trace. The single unchanged-head shard retry passed (20
passed, 1 skipped); no timeout or source change was made.
- Greptile reviewed this exact commit at 5/5 with no unresolved review
threads.
- No live portfolio request was replayed. Route tests use controlled
local upstream responses.
- The added diff passed the secret and PII scan and `git diff --check`.

## Risks

This is a diagnostic change. It does not identify or repair the origin
of a connection reset. Unknown transport failures remain `unknown`.
Elapsed values outside 0–60,000 ms become null. Default Sentry
fingerprinting remains enabled; exact historical group membership is not
guaranteed. No retry, migration, deployment, or configuration change is
included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, code
execution, and independent agent review. The exact deployment model ID
and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 06:58:36 -07:00

247 lines
12 KiB
TypeScript

import express from "express";
import request from "supertest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { errorHandler } from "../middleware/index.js";
import { cloudRoutes } from "../routes/cloud.js";
import { CloudPortfolioError } from "../services/cloud-portfolio-error.js";
import * as sentry from "../sentry.js";
import { logger } from "../middleware/logger.js";
afterEach(() => vi.restoreAllMocks());
const cloudEnv = {
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "tenant-secret",
PAPERCLIP_CLOUD_STACK_ID: "stack-current",
PAPERCLIP_CLOUD_API_ORIGIN: "https://cloud.example.test/control-plane",
};
function cloudActor(userId: string) {
return {
type: "board" as const,
source: "cloud_tenant" as const,
userId,
companyIds: ["company-1"],
};
}
function createApp(options: {
actor?: ReturnType<typeof cloudActor> | {
type: "board";
source: "session";
userId: string;
};
runtimeEnv?: Record<string, string | undefined>;
fetchImpl: typeof fetch;
now?: () => number;
}) {
const app = express();
app.use((req, _res, next) => {
(req as any).actor = options.actor ?? cloudActor("actor-user");
next();
});
app.use("/api/cloud", cloudRoutes({
runtimeEnv: options.runtimeEnv ?? cloudEnv,
fetchImpl: options.fetchImpl,
now: options.now,
}));
app.use(errorHandler);
return app;
}
function jsonResponse(payload: unknown, status = 200) {
return new Response(JSON.stringify(payload), {
status,
headers: { "content-type": "application/json" },
});
}
describe("GET /api/cloud/stacks", () => {
it.each([401, 403, 429, 503])("preserves the response contract for upstream HTTP %s without reading the body", async (status) => {
const upstream = jsonResponse({ secret: "private upstream payload" }, status);
const json = vi.spyOn(upstream, "json");
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks").set("Cookie", "client-session=private-cookie");
expect(response.status).toBe(502);
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
expect(response.headers["set-cookie"]).toBeUndefined();
expect(fetchImpl).toHaveBeenCalledTimes(1);
expect(new Headers(fetchImpl.mock.calls[0]![1]!.headers).has("cookie")).toBe(false);
expect(json).not.toHaveBeenCalled();
expect(capture).toHaveBeenCalledWith(expect.any(CloudPortfolioError));
const error = capture.mock.calls[0]![0] as CloudPortfolioError;
expect(error.diagnostics).toMatchObject({ phase: "http_response", upstreamStatus: status, networkCode: "unknown", elapsedMs: expect.any(Number) });
expect(error.stack).toContain("routes/cloud.ts:");
expect(error.stack).not.toContain("at portfolioError");
expect(warn).toHaveBeenCalledWith({ cloudPortfolio: error.diagnostics }, "Paperclip Cloud portfolio request failed");
expect(JSON.stringify(warn.mock.calls)).not.toContain("private");
});
it("classifies a fetch reset without retrying or caching the failure", async () => {
const fetchImpl = vi.fn<typeof fetch>()
.mockRejectedValueOnce(new TypeError("private fetch message", { cause: Object.assign(new Error("private cause"), { code: "ECONNRESET" }) }))
.mockResolvedValueOnce(jsonResponse({ stacks: [] }));
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
const app = createApp({ fetchImpl });
const failure = await request(app).get("/api/cloud/stacks");
expect(failure.status).toBe(502);
expect(failure.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
expect(fetchImpl).toHaveBeenCalledTimes(1);
const error = capture.mock.calls[0]![0] as CloudPortfolioError;
expect(error.diagnostics).toMatchObject({ phase: "fetch", upstreamStatus: null, networkCode: "ECONNRESET" });
expect(JSON.stringify(warn.mock.calls)).not.toContain("private");
expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] });
expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] });
expect(fetchImpl).toHaveBeenCalledTimes(2);
});
it.each(["invalid_json", "body_reset", "deadline"] as const)("keeps body-phase %s failures separate from fetch failures", async (kind) => {
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
const controller = new AbortController();
const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(controller.signal);
const upstream = jsonResponse({});
vi.spyOn(upstream, "json").mockImplementation(async () => {
if (kind === "deadline") controller.abort(new DOMException("private timeout", "TimeoutError"));
throw kind === "body_reset" ? Object.assign(new Error("private body"), { code: "ECONNRESET" }) : new SyntaxError("private body");
});
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
const app = createApp({ fetchImpl });
for (let count = 1; count <= 2; count++) {
const response = await request(app).get("/api/cloud/stacks");
expect(response.status).toBe(502);
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio returned invalid JSON", code: "cloud_portfolio_invalid_response", details: { code: "cloud_portfolio_invalid_response" } });
expect(fetchImpl).toHaveBeenCalledTimes(count);
}
expect(timeout).toHaveBeenCalledWith(10_000);
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({
phase: "response_body", upstreamStatus: 200,
networkCode: kind === "deadline" ? "DEADLINE_EXCEEDED" : kind === "body_reset" ? "ECONNRESET" : "unknown",
});
});
it("records the owned fetch deadline without changing the ten-second signal", async () => {
const signal = AbortSignal.abort(new DOMException("private timeout", "TimeoutError"));
const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(signal);
const fetchImpl = vi.fn<typeof fetch>().mockRejectedValue(signal.reason);
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks");
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
expect(fetchImpl).toHaveBeenCalledTimes(1);
expect(fetchImpl.mock.calls[0]![1]?.signal).toBe(signal);
expect(timeout).toHaveBeenCalledWith(10_000);
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "fetch", networkCode: "DEADLINE_EXCEEDED" });
});
it("does not mislabel a response serialization failure as an upstream fetch error", async () => {
const payload: { cycle?: unknown } = {};
payload.cycle = payload;
const upstream = jsonResponse({});
vi.spyOn(upstream, "json").mockResolvedValue(payload);
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks");
expect(response.status).toBe(502);
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
expect(fetchImpl).toHaveBeenCalledTimes(1);
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "response_write", upstreamStatus: 200, networkCode: "unknown" });
});
it("returns the actor's portfolio without forwarding client-supplied identity", async () => {
const portfolio = { stacks: [{ slug: "current", displayName: "Current" }] };
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(jsonResponse(portfolio));
const app = createApp({ fetchImpl });
const res = await request(app)
.get("/api/cloud/stacks?userId=client-supplied-user")
.set("x-paperclip-cloud-user-id", "spoofed-header-user")
.set("authorization", "Bearer client-token");
expect(res.status).toBe(200);
expect(res.body).toEqual(portfolio);
expect(res.headers["cache-control"]).toBe("no-store");
expect(fetchImpl).toHaveBeenCalledTimes(1);
const [url, init] = fetchImpl.mock.calls[0]!;
expect(url.toString()).toBe("https://cloud.example.test/v1/tenant/portfolio");
expect(init).toMatchObject({ method: "GET" });
expect(init?.headers).toEqual({
accept: "application/json",
authorization: "Bearer tenant-secret",
"x-paperclip-cloud-user-id": "actor-user",
"x-paperclip-cloud-stack-id": "stack-current",
});
expect(JSON.stringify(init)).not.toContain("client-supplied-user");
expect(JSON.stringify(init)).not.toContain("spoofed-header-user");
expect(JSON.stringify(init)).not.toContain("client-token");
});
it("caches successful portfolios per actor for 30 seconds", async () => {
let currentTime = 1_000;
const fetchImpl = vi.fn<typeof fetch>()
.mockResolvedValueOnce(jsonResponse({ generation: 1 }))
.mockResolvedValueOnce(jsonResponse({ generation: 2 }));
const app = createApp({ fetchImpl, now: () => currentTime });
const first = await request(app).get("/api/cloud/stacks");
currentTime += 29_999;
const cached = await request(app).get("/api/cloud/stacks");
currentTime += 1;
const refreshed = await request(app).get("/api/cloud/stacks");
expect(first.body).toEqual({ generation: 1 });
expect(cached.body).toEqual({ generation: 1 });
expect(refreshed.body).toEqual({ generation: 2 });
expect(fetchImpl).toHaveBeenCalledTimes(2);
});
it("keeps cache entries isolated by the server-derived actor user id", async () => {
const fetchImpl = vi.fn<typeof fetch>().mockImplementation(async (_url, init) => {
const headers = new Headers(init?.headers);
return jsonResponse({ userId: headers.get("x-paperclip-cloud-user-id") });
});
const app = express();
app.use((req, _res, next) => {
const userId = req.header("x-test-actor-user") ?? "user-a";
(req as any).actor = cloudActor(userId);
next();
});
app.use("/api/cloud", cloudRoutes({ runtimeEnv: cloudEnv, fetchImpl }));
app.use(errorHandler);
const first = await request(app).get("/api/cloud/stacks").set("x-test-actor-user", "user-a");
const second = await request(app).get("/api/cloud/stacks").set("x-test-actor-user", "user-b");
const firstAgain = await request(app).get("/api/cloud/stacks").set("x-test-actor-user", "user-a");
expect(first.body).toEqual({ userId: "user-a" });
expect(second.body).toEqual({ userId: "user-b" });
expect(firstAgain.body).toEqual({ userId: "user-a" });
expect(fetchImpl).toHaveBeenCalledTimes(2);
});
it("returns 404 on self-hosted instances without calling upstream", async () => {
const fetchImpl = vi.fn<typeof fetch>();
const app = createApp({ runtimeEnv: {}, fetchImpl });
const res = await request(app).get("/api/cloud/stacks");
expect(res.status).toBe(404);
expect(fetchImpl).not.toHaveBeenCalled();
});
it("rejects non-tenant actors on managed instances without calling upstream", async () => {
const fetchImpl = vi.fn<typeof fetch>();
const app = createApp({
actor: { type: "board", source: "session", userId: "session-user" },
fetchImpl,
});
const res = await request(app).get("/api/cloud/stacks");
expect(res.status).toBe(403);
expect(res.body.code).toBe("cloud_tenant_required");
expect(fetchImpl).not.toHaveBeenCalled();
});
});