From 90182b4f8b40d6ee217937ba61199b4abc31dee7 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 6 Oct 2026 06:58:36 -0700 Subject: [PATCH] Report bounded Cloud portfolio failure diagnostics (#15340) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages work for AI agents and their companies. > - Cloud instances proxy a trusted user's portfolio through the control plane. > - A failed request returns a generic error to the client. > - That replacement error loses the failure phase and network code in Sentry. > - Operators need bounded evidence without upstream messages or credentials. > - This PR adds safe diagnostics while preserving the existing request behavior. ## Linked Issues or Issue Description Refs #10850, which added the portfolio proxy. No open PR for this diagnostic gap was found. **What happened?** A rejected portfolio fetch becomes a generic 502 in Sentry. The event cannot distinguish a connection reset, deadline, HTTP response failure, or body failure. The route's previous warning also included the original error and a stack identifier. **Steps to reproduce** Make the portfolio proxy's fetch reject with a TypeError whose cause has `code: ECONNRESET`. The client correctly receives the generic 502, but the captured replacement error loses that code. **Expected behavior** Keep the existing client response. Attach only bounded server-side diagnostic fields to the failure event. Do not retry the request or expose the original error. ## What Changed - Add a typed portfolio error with a private frozen diagnostic record: phase, upstream HTTP status, elapsed milliseconds, and an allowlisted network code. - Read at most four error/cause objects through own data properties. Unknown codes, messages, getters, and out-of-range values do not enter the record. - Replace the route's raw-error warnings with safe fields. Send a plain error plus event-local context through the existing optional Sentry gate. Keep the route callsite and default fingerprint policy. - Preserve authentication, trusted headers, cookies, exact HTTP error bodies, cache behavior, the ten-second deadline, and one fetch per request. Public responses receive no diagnostic fields. - Document the fields and test HTTP behavior, privacy, and event isolation with the real Sentry SDK. ## Verification - `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run server/src/__tests__/cloud-portfolio-error.test.ts server/src/__tests__/cloud-routes.test.ts server/src/__tests__/sentry.test.ts server/src/__tests__/run-failure-sentry-real-sdk.test.ts`: 65 passed, with the audited optional SDK installed. - Full local `pnpm -r typecheck` and `pnpm build` passed on Node 24.21.0 and pnpm 9.15.4. - Independent review found no blockers and independently passed all 65 tests, including the real SDK checks, on this exact commit. - Full Linux CI passed on this exact commit and provides aggregate suite coverage (54 successful checks, 2 intentional skips). A duplicate full local aggregate was not run. - The first SDK contract job failed before tests when npm could not resolve an OpenTelemetry transitive package. A subsequent empty-cache install first encountered a missing tarball, then succeeded after the registry artifact became available. All 6 real-SDK tests passed against that fresh install; the single unchanged-head CI retry passed. The SDK pin, workflow, and dependency files are unchanged. - The first browser shard 8 run timed out waiting for the inbox retry reply after 45 seconds. The unchanged isolated case passed (1/1), and the test, UI handler, fixture, and recovery files match the base commit. The failed log contains no wakeup POST before the test's immediate navigation; a navigation/request timing race is suspected but unproven without a trace. The single unchanged-head shard retry passed (20 passed, 1 skipped); no timeout or source change was made. - Greptile reviewed this exact commit at 5/5 with no unresolved review threads. - No live portfolio request was replayed. Route tests use controlled local upstream responses. - The added diff passed the secret and PII scan and `git diff --check`. ## Risks This is a diagnostic change. It does not identify or repair the origin of a connection reset. Unknown transport failures remain `unknown`. Elapsed values outside 0–60,000 ms become null. Default Sentry fingerprinting remains enabled; exact historical group membership is not guaranteed. No retry, migration, deployment, or configuration change is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, code execution, and independent agent review. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/observability.md | 13 +++ .../__tests__/cloud-portfolio-error.test.ts | 45 +++++++++ server/src/__tests__/cloud-routes.test.ts | 99 ++++++++++++++++++- .../run-failure-sentry-real-sdk.test.ts | 42 ++++++++ server/src/__tests__/sentry.test.ts | 23 +++++ server/src/routes/cloud.ts | 44 ++++++--- server/src/sentry.ts | 9 ++ server/src/services/cloud-portfolio-error.ts | 70 +++++++++++++ 8 files changed, 331 insertions(+), 14 deletions(-) create mode 100644 server/src/__tests__/cloud-portfolio-error.test.ts create mode 100644 server/src/services/cloud-portfolio-error.ts diff --git a/doc/observability.md b/doc/observability.md index e877610b3d0..5c1e57c1784 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -467,6 +467,19 @@ native failure only while its saved review is still current and pending; caller payloads cannot select a different review. Dispatch checks that assignment again. Superseded or completed reviews retain their authority. +Cloud portfolio proxy failures have an event-local `cloud_portfolio` context. +It contains only the phase (`fetch`, `http_response`, `response_body`, or +`response_write`), upstream HTTP status when available, elapsed milliseconds, +and an allowlisted network error code. Unknown values become `unknown` or +`null`; elapsed time outside 0–60,000 ms becomes `null`. Codes come from at +most four error/cause objects' own data properties, never error messages. +`DEADLINE_EXCEEDED` means the request's own ten-second signal was aborted. +The route's warning uses the same safe fields. Neither record includes URLs, +headers, cookies, bodies, user/stack IDs, or the original exception. Sentry +receives a plain error with the existing generic message and default grouping. +The context does not change the HTTP response, authentication, cookies, cache, +deadline, or single-fetch behavior, and does not carry into unrelated events. + An unconfirmed adapter Stop timeout has an event-local `adapter_stop` context: the run UUID, built-in adapter type, native/legacy runtime mode, configured wait duration, and whether abort was requested. Invalid identities become diff --git a/server/src/__tests__/cloud-portfolio-error.test.ts b/server/src/__tests__/cloud-portfolio-error.test.ts new file mode 100644 index 00000000000..32e35eff953 --- /dev/null +++ b/server/src/__tests__/cloud-portfolio-error.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from "vitest"; +import { CloudPortfolioError } from "../services/cloud-portfolio-error.js"; + +const input = { phase: "fetch" as const, elapsedMs: 270.2, upstreamStatus: null }; + +describe("CloudPortfolioError", () => { + it("retains only an allowlisted nested network code, not the original exception", () => { + const cause = Object.assign(new Error("private upstream payload"), { code: "ECONNRESET", url: "https://private.example.test" }); + const error = new CloudPortfolioError("upstream", input, new TypeError("fetch failed", { cause })); + expect(error.diagnostics).toEqual({ phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" }); + expect(error).not.toHaveProperty("cause"); + expect(JSON.stringify(error)).not.toContain("private"); + expect(Object.isFrozen(error.diagnostics)).toBe(true); + }); + + it.each([ + new Error("ECONNRESET in a private message is not evidence"), + { code: "private-provider-code" }, + { get code() { throw new Error("private getter"); }, get cause() { throw new Error("private getter"); } }, + new Proxy({}, { getOwnPropertyDescriptor() { throw new Error("private proxy"); } }), + ])("fails closed for unknown errors without invoking getters", (cause) => { + expect(new CloudPortfolioError("upstream", input, cause).diagnostics.networkCode).toBe("unknown"); + }); + + it("bounds cause traversal and handles cycles", () => { + const cycle: { cause?: unknown } = {}; + cycle.cause = cycle; + expect(new CloudPortfolioError("upstream", input, cycle).diagnostics.networkCode).toBe("unknown"); + const deep = { cause: { cause: { cause: { cause: { code: "ECONNRESET" } } } } }; + expect(new CloudPortfolioError("upstream", input, deep).diagnostics.networkCode).toBe("unknown"); + }); + + it("normalizes unsafe diagnostics and keeps non-network phases distinct", () => { + const error = new CloudPortfolioError("upstream", { + phase: "private-phase" as never, elapsedMs: Infinity, upstreamStatus: 900, + }, { code: "ECONNRESET" }); + expect(error.diagnostics).toEqual({ phase: "unknown", elapsedMs: null, upstreamStatus: null, networkCode: "unknown" }); + for (const phase of ["http_response", "response_write"] as const) { + expect(new CloudPortfolioError("upstream", { ...input, phase, upstreamStatus: 503, deadlineExceeded: true }, { code: "ECONNRESET" }).diagnostics) + .toMatchObject({ phase, upstreamStatus: 503, networkCode: "unknown" }); + } + expect(new CloudPortfolioError("upstream", { ...input, elapsedMs: 60_001 }).diagnostics.elapsedMs).toBeNull(); + expect(new CloudPortfolioError("upstream", { ...input, elapsedMs: -1 }).diagnostics.elapsedMs).toBeNull(); + }); +}); diff --git a/server/src/__tests__/cloud-routes.test.ts b/server/src/__tests__/cloud-routes.test.ts index a6453e05a25..a85c4c4087b 100644 --- a/server/src/__tests__/cloud-routes.test.ts +++ b/server/src/__tests__/cloud-routes.test.ts @@ -1,8 +1,13 @@ import express from "express"; import request from "supertest"; -import { describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { errorHandler } from "../middleware/index.js"; import { cloudRoutes } from "../routes/cloud.js"; +import { CloudPortfolioError } from "../services/cloud-portfolio-error.js"; +import * as sentry from "../sentry.js"; +import { logger } from "../middleware/logger.js"; + +afterEach(() => vi.restoreAllMocks()); const cloudEnv = { PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "tenant-secret", @@ -51,6 +56,98 @@ function jsonResponse(payload: unknown, status = 200) { } describe("GET /api/cloud/stacks", () => { + it.each([401, 403, 429, 503])("preserves the response contract for upstream HTTP %s without reading the body", async (status) => { + const upstream = jsonResponse({ secret: "private upstream payload" }, status); + const json = vi.spyOn(upstream, "json"); + const fetchImpl = vi.fn().mockResolvedValue(upstream); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const warn = vi.spyOn(logger, "warn").mockImplementation(() => {}); + const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks").set("Cookie", "client-session=private-cookie"); + expect(response.status).toBe(502); + expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } }); + expect(response.headers["set-cookie"]).toBeUndefined(); + expect(fetchImpl).toHaveBeenCalledTimes(1); + expect(new Headers(fetchImpl.mock.calls[0]![1]!.headers).has("cookie")).toBe(false); + expect(json).not.toHaveBeenCalled(); + expect(capture).toHaveBeenCalledWith(expect.any(CloudPortfolioError)); + const error = capture.mock.calls[0]![0] as CloudPortfolioError; + expect(error.diagnostics).toMatchObject({ phase: "http_response", upstreamStatus: status, networkCode: "unknown", elapsedMs: expect.any(Number) }); + expect(error.stack).toContain("routes/cloud.ts:"); + expect(error.stack).not.toContain("at portfolioError"); + expect(warn).toHaveBeenCalledWith({ cloudPortfolio: error.diagnostics }, "Paperclip Cloud portfolio request failed"); + expect(JSON.stringify(warn.mock.calls)).not.toContain("private"); + }); + + it("classifies a fetch reset without retrying or caching the failure", async () => { + const fetchImpl = vi.fn() + .mockRejectedValueOnce(new TypeError("private fetch message", { cause: Object.assign(new Error("private cause"), { code: "ECONNRESET" }) })) + .mockResolvedValueOnce(jsonResponse({ stacks: [] })); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const warn = vi.spyOn(logger, "warn").mockImplementation(() => {}); + const app = createApp({ fetchImpl }); + const failure = await request(app).get("/api/cloud/stacks"); + expect(failure.status).toBe(502); + expect(failure.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } }); + expect(fetchImpl).toHaveBeenCalledTimes(1); + const error = capture.mock.calls[0]![0] as CloudPortfolioError; + expect(error.diagnostics).toMatchObject({ phase: "fetch", upstreamStatus: null, networkCode: "ECONNRESET" }); + expect(JSON.stringify(warn.mock.calls)).not.toContain("private"); + expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] }); + expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] }); + expect(fetchImpl).toHaveBeenCalledTimes(2); + }); + + it.each(["invalid_json", "body_reset", "deadline"] as const)("keeps body-phase %s failures separate from fetch failures", async (kind) => { + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const controller = new AbortController(); + const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(controller.signal); + const upstream = jsonResponse({}); + vi.spyOn(upstream, "json").mockImplementation(async () => { + if (kind === "deadline") controller.abort(new DOMException("private timeout", "TimeoutError")); + throw kind === "body_reset" ? Object.assign(new Error("private body"), { code: "ECONNRESET" }) : new SyntaxError("private body"); + }); + const fetchImpl = vi.fn().mockResolvedValue(upstream); + const app = createApp({ fetchImpl }); + for (let count = 1; count <= 2; count++) { + const response = await request(app).get("/api/cloud/stacks"); + expect(response.status).toBe(502); + expect(response.body).toEqual({ error: "Paperclip Cloud portfolio returned invalid JSON", code: "cloud_portfolio_invalid_response", details: { code: "cloud_portfolio_invalid_response" } }); + expect(fetchImpl).toHaveBeenCalledTimes(count); + } + expect(timeout).toHaveBeenCalledWith(10_000); + expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ + phase: "response_body", upstreamStatus: 200, + networkCode: kind === "deadline" ? "DEADLINE_EXCEEDED" : kind === "body_reset" ? "ECONNRESET" : "unknown", + }); + }); + + it("records the owned fetch deadline without changing the ten-second signal", async () => { + const signal = AbortSignal.abort(new DOMException("private timeout", "TimeoutError")); + const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(signal); + const fetchImpl = vi.fn().mockRejectedValue(signal.reason); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks"); + expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } }); + expect(fetchImpl).toHaveBeenCalledTimes(1); + expect(fetchImpl.mock.calls[0]![1]?.signal).toBe(signal); + expect(timeout).toHaveBeenCalledWith(10_000); + expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "fetch", networkCode: "DEADLINE_EXCEEDED" }); + }); + + it("does not mislabel a response serialization failure as an upstream fetch error", async () => { + const payload: { cycle?: unknown } = {}; + payload.cycle = payload; + const upstream = jsonResponse({}); + vi.spyOn(upstream, "json").mockResolvedValue(payload); + const fetchImpl = vi.fn().mockResolvedValue(upstream); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks"); + expect(response.status).toBe(502); + expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } }); + expect(fetchImpl).toHaveBeenCalledTimes(1); + expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "response_write", upstreamStatus: 200, networkCode: "unknown" }); + }); + it("returns the actor's portfolio without forwarding client-supplied identity", async () => { const portfolio = { stacks: [{ slug: "current", displayName: "Current" }] }; const fetchImpl = vi.fn().mockResolvedValue(jsonResponse(portfolio)); diff --git a/server/src/__tests__/run-failure-sentry-real-sdk.test.ts b/server/src/__tests__/run-failure-sentry-real-sdk.test.ts index fd518304020..5cac0081e31 100644 --- a/server/src/__tests__/run-failure-sentry-real-sdk.test.ts +++ b/server/src/__tests__/run-failure-sentry-real-sdk.test.ts @@ -42,6 +42,48 @@ afterEach(async () => { }); describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK", () => { + it("keeps portfolio diagnostics bounded and isolated from unrelated captures", async () => { + const Sentry = sentryPackage!; + const events: Array> = []; + vi.stubEnv("SENTRY_DSN_BACKEND", "https://public@example.invalid/1"); + vi.doMock("../peer-version-check.js", () => ({ checkExactPeerVersions: () => ({ ok: true }) })); + vi.doMock("@sentry/node", () => ({ + ...Sentry, + init: (options: Record) => Sentry.init({ + ...options, + transport: () => ({ send: async () => ({}), flush: async () => true }), + beforeSend: (event: Record) => { events.push(event); return event; }, + }), + })); + vi.resetModules(); + const { sentryReady, captureException } = await import("../sentry.js"); + const { CloudPortfolioError } = await import("../services/cloud-portfolio-error.js"); + await sentryReady; + const failure = new CloudPortfolioError("upstream", { + phase: "fetch", elapsedMs: 270, upstreamStatus: null, + }, new TypeError("private-portfolio-message", { cause: { code: "ECONNRESET", token: "private-portfolio-token" } })); + Object.assign(failure, { + cause: new Error("private-portfolio-cause"), + request: { url: "https://private-portfolio.invalid", headers: { cookie: "private-portfolio-cookie" } }, + }); + Object.defineProperty(failure, "diagnostics", { value: { extra: "private-portfolio-replacement" } }); + captureException(failure); + captureException(new Error("unrelated portfolio fixture")); + await Sentry.flush(2000); + expect(events).toHaveLength(2); + const captured = events.find((event) => (event.exception as { values: Array<{ value: string }> }).values.some((entry) => entry.value === failure.message)); + expect(captured).toMatchObject({ + tags: { error_code: "cloud_portfolio_failure" }, fingerprint: ["{{ default }}"], + contexts: { cloud_portfolio: { phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" } }, + }); + expect((captured!.exception as { values: unknown[] }).values).toHaveLength(1); + expect(JSON.stringify(events)).not.toContain("private-portfolio-"); + const unrelated = events.find((event) => event !== captured)!; + expect(unrelated).not.toHaveProperty("contexts.cloud_portfolio"); + expect(unrelated).not.toHaveProperty("tags.error_code"); + expect(unrelated).not.toHaveProperty("fingerprint"); + }); + it.each([ { phase: "workspace_restore", phaseElapsedMs: 60_123, expectedPhase: "workspace_restore", expectedElapsedMs: 60_123 }, { phase: "private-provider-phase", phaseElapsedMs: 100, expectedPhase: "unknown", expectedElapsedMs: null }, diff --git a/server/src/__tests__/sentry.test.ts b/server/src/__tests__/sentry.test.ts index ba044e8b1b2..865ae21fc4c 100644 --- a/server/src/__tests__/sentry.test.ts +++ b/server/src/__tests__/sentry.test.ts @@ -127,6 +127,29 @@ describe("sentryReady", () => { }); describe("captureException", () => { + it("sends only private normalized portfolio diagnostics on the matching event", async () => { + process.env[BACKEND_DSN_ENV] = "https://fixture@example.com/1"; + const sdk = mockSentryPackage(); + const { captureException, sentryReady } = await importFreshSentry(); + const { CloudPortfolioError } = await import("../services/cloud-portfolio-error.js"); + await sentryReady; + const error = new CloudPortfolioError("upstream", { phase: "fetch", elapsedMs: 270, upstreamStatus: null }, { code: "ECONNRESET" }); + Object.assign(error, { cause: new Error("private cause"), headers: { authorization: "private token" } }); + Object.defineProperty(error, "diagnostics", { value: { token: "private replacement" } }); + captureException(error); + const unrelated = new Error("unrelated"); + captureException(unrelated); + expect(sdk.captureException.mock.calls[0]).toEqual([ + expect.objectContaining({ message: error.message, stack: error.stack }), + { tags: { error_code: "cloud_portfolio_failure" }, fingerprint: ["{{ default }}"], contexts: { + cloud_portfolio: { phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" }, + } }, + ]); + expect(JSON.stringify(sdk.captureException.mock.calls[0])).not.toContain("private"); + expect(sdk.captureException.mock.calls[0]![0]).not.toHaveProperty("cause"); + expect(sdk.captureException.mock.calls[1]).toEqual([unrelated]); + }); + it("adds bounded Stop timeout context only to that event", async () => { process.env[BACKEND_DSN_ENV] = "https://fixture@example.com/1"; const sdk = mockSentryPackage(); diff --git a/server/src/routes/cloud.ts b/server/src/routes/cloud.ts index 0b4aefc8fc6..e8456d7e40f 100644 --- a/server/src/routes/cloud.ts +++ b/server/src/routes/cloud.ts @@ -1,6 +1,7 @@ import { Router } from "express"; import { forbidden, HttpError, notFound } from "../errors.js"; import { logger } from "../middleware/logger.js"; +import { CloudPortfolioError, type CloudPortfolioPhase } from "../services/cloud-portfolio-error.js"; import { getCloudStackContext, type CloudInstanceEnv, @@ -67,7 +68,24 @@ export function cloudRoutes(opts: { }); } + const startedAt = performance.now(); + let phase: CloudPortfolioPhase = "fetch"; + let upstreamStatus: number | null = null; + let upstreamSignal: AbortSignal | undefined; + const portfolioError = (kind: "upstream" | "invalid_response", error?: unknown) => { + const reportableError = new CloudPortfolioError(kind, { + phase, + elapsedMs: performance.now() - startedAt, + upstreamStatus, + deadlineExceeded: upstreamSignal?.aborted === true, + }, error); + // Preserve the route callsite instead of grouping by this helper frame. + Error.captureStackTrace?.(reportableError, portfolioError); + return reportableError; + }; + try { + upstreamSignal = AbortSignal.timeout(10_000); const upstream = await fetchImpl(portfolioUrl, { method: "GET", headers: { @@ -76,26 +94,27 @@ export function cloudRoutes(opts: { "x-paperclip-cloud-user-id": actorUserId, "x-paperclip-cloud-stack-id": context.stackId, }, - signal: AbortSignal.timeout(10_000), + signal: upstreamSignal, }); + phase = "http_response"; + upstreamStatus = upstream.status; if (!upstream.ok) { + const error = portfolioError("upstream"); logger.warn( - { status: upstream.status, stackId: context.stackId }, + { cloudPortfolio: error.diagnostics }, "Paperclip Cloud portfolio request failed", ); - throw new HttpError(502, "Paperclip Cloud portfolio request failed", { - code: "cloud_portfolio_upstream_error", - }); + throw error; } + phase = "response_body"; let payload: unknown; try { payload = await upstream.json(); - } catch { - throw new HttpError(502, "Paperclip Cloud portfolio returned invalid JSON", { - code: "cloud_portfolio_invalid_response", - }); + } catch (error) { + throw portfolioError("invalid_response", error); } + phase = "response_write"; portfolioCache.set(cacheKey, { expiresAt: currentTime + cacheTtlMs, payload, @@ -104,13 +123,12 @@ export function cloudRoutes(opts: { res.json(payload); } catch (error) { if (error instanceof HttpError) throw error; + const reportableError = portfolioError("upstream", error); logger.warn( - { err: error, stackId: context.stackId }, + { cloudPortfolio: reportableError.diagnostics }, "Paperclip Cloud portfolio request failed", ); - throw new HttpError(502, "Paperclip Cloud portfolio request failed", { - code: "cloud_portfolio_upstream_error", - }); + throw reportableError; } }); diff --git a/server/src/sentry.ts b/server/src/sentry.ts index 665cab811f5..1ecd5c3935e 100644 --- a/server/src/sentry.ts +++ b/server/src/sentry.ts @@ -57,6 +57,7 @@ import os from "node:os"; import { AdapterStopTimeoutError } from "./services/adapter-stop-timeout.js"; +import { CloudPortfolioError } from "./services/cloud-portfolio-error.js"; import type { RunFailureDiagnostics } from "./services/run-failure-diagnostics.js"; import { readBuildCommit } from "./build-commit.js"; import { checkExactPeerVersions } from "./peer-version-check.js"; @@ -116,6 +117,14 @@ export function captureException(error: unknown): void { contexts: { adapter_stop: { ...error.diagnostics } }, fingerprint: ["{{ default }}"], }); + } else if (error instanceof CloudPortfolioError) { + const exception = new Error(error.message); + exception.stack = error.stack; + sentryHandle.captureException(exception, { + tags: { error_code: "cloud_portfolio_failure" }, + contexts: { cloud_portfolio: { ...CloudPortfolioError.diagnosticsFor(error) } }, + fingerprint: ["{{ default }}"], + }); } else { sentryHandle.captureException(error); } diff --git a/server/src/services/cloud-portfolio-error.ts b/server/src/services/cloud-portfolio-error.ts new file mode 100644 index 00000000000..6868830eb28 --- /dev/null +++ b/server/src/services/cloud-portfolio-error.ts @@ -0,0 +1,70 @@ +import { HttpError } from "../errors.js"; + +const phases = ["fetch", "http_response", "response_body", "response_write"] as const; +export type CloudPortfolioPhase = typeof phases[number]; +const networkCodes = new Set([ + "ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EAI_AGAIN", + "ENETUNREACH", "EHOSTUNREACH", "UND_ERR_CONNECT_TIMEOUT", + "UND_ERR_HEADERS_TIMEOUT", "UND_ERR_BODY_TIMEOUT", "UND_ERR_SOCKET", + "ERR_TLS_CERT_ALTNAME_INVALID", "CERT_HAS_EXPIRED", + "DEPTH_ZERO_SELF_SIGNED_CERT", "UNABLE_TO_VERIFY_LEAF_SIGNATURE", "ABORT_ERR", +]); + +// Inspect data properties only. A provider error can contain getters, cycles, +// arbitrary messages or nested request objects; none are diagnostic payloads. +function ownValue(value: unknown, key: string): unknown { + if (!value || typeof value !== "object") return undefined; + try { + const descriptor = Object.getOwnPropertyDescriptor(value, key); + return descriptor && "value" in descriptor ? descriptor.value : undefined; + } catch { + return undefined; + } +} + +function networkCode(error: unknown): string { + let current = error; + for (let depth = 0; depth < 4; depth++) { + const code = ownValue(current, "code"); + if (typeof code === "string" && networkCodes.has(code)) return code; + current = ownValue(current, "cause"); + } + return "unknown"; +} + +/** Server-only diagnostics. The existing HTTP response details stay unchanged. */ +export class CloudPortfolioError extends HttpError { + readonly #diagnostics: Readonly>; + + constructor( + kind: "upstream" | "invalid_response", + input: { phase: CloudPortfolioPhase; elapsedMs: number; upstreamStatus: number | null; deadlineExceeded?: boolean }, + error?: unknown, + ) { + super(502, kind === "invalid_response" + ? "Paperclip Cloud portfolio returned invalid JSON" + : "Paperclip Cloud portfolio request failed", { + code: kind === "invalid_response" ? "cloud_portfolio_invalid_response" : "cloud_portfolio_upstream_error", + }); + Error.captureStackTrace?.(this, CloudPortfolioError); + const phase = phases.includes(input.phase) ? input.phase : "unknown"; + this.#diagnostics = Object.freeze({ + phase, + upstreamStatus: Number.isInteger(input.upstreamStatus) && input.upstreamStatus! >= 100 && input.upstreamStatus! <= 599 + ? input.upstreamStatus : null, + elapsedMs: Number.isFinite(input.elapsedMs) && input.elapsedMs >= 0 && input.elapsedMs <= 60_000 + ? Math.round(input.elapsedMs) : null, + networkCode: phase === "fetch" || phase === "response_body" + ? input.deadlineExceeded === true ? "DEADLINE_EXCEEDED" : networkCode(error) + : "unknown", + }); + } + + get diagnostics() { + return this.#diagnostics; + } + + static diagnosticsFor(error: CloudPortfolioError) { + return error.#diagnostics; + } +}