mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 10:13:11 +02:00
Report bounded Cloud portfolio failure diagnostics (#15340)
## Thinking Path > - Paperclip manages work for AI agents and their companies. > - Cloud instances proxy a trusted user's portfolio through the control plane. > - A failed request returns a generic error to the client. > - That replacement error loses the failure phase and network code in Sentry. > - Operators need bounded evidence without upstream messages or credentials. > - This PR adds safe diagnostics while preserving the existing request behavior. ## Linked Issues or Issue Description Refs #10850, which added the portfolio proxy. No open PR for this diagnostic gap was found. **What happened?** A rejected portfolio fetch becomes a generic 502 in Sentry. The event cannot distinguish a connection reset, deadline, HTTP response failure, or body failure. The route's previous warning also included the original error and a stack identifier. **Steps to reproduce** Make the portfolio proxy's fetch reject with a TypeError whose cause has `code: ECONNRESET`. The client correctly receives the generic 502, but the captured replacement error loses that code. **Expected behavior** Keep the existing client response. Attach only bounded server-side diagnostic fields to the failure event. Do not retry the request or expose the original error. ## What Changed - Add a typed portfolio error with a private frozen diagnostic record: phase, upstream HTTP status, elapsed milliseconds, and an allowlisted network code. - Read at most four error/cause objects through own data properties. Unknown codes, messages, getters, and out-of-range values do not enter the record. - Replace the route's raw-error warnings with safe fields. Send a plain error plus event-local context through the existing optional Sentry gate. Keep the route callsite and default fingerprint policy. - Preserve authentication, trusted headers, cookies, exact HTTP error bodies, cache behavior, the ten-second deadline, and one fetch per request. Public responses receive no diagnostic fields. - Document the fields and test HTTP behavior, privacy, and event isolation with the real Sentry SDK. ## Verification - `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run server/src/__tests__/cloud-portfolio-error.test.ts server/src/__tests__/cloud-routes.test.ts server/src/__tests__/sentry.test.ts server/src/__tests__/run-failure-sentry-real-sdk.test.ts`: 65 passed, with the audited optional SDK installed. - Full local `pnpm -r typecheck` and `pnpm build` passed on Node 24.21.0 and pnpm 9.15.4. - Independent review found no blockers and independently passed all 65 tests, including the real SDK checks, on this exact commit. - Full Linux CI passed on this exact commit and provides aggregate suite coverage (54 successful checks, 2 intentional skips). A duplicate full local aggregate was not run. - The first SDK contract job failed before tests when npm could not resolve an OpenTelemetry transitive package. A subsequent empty-cache install first encountered a missing tarball, then succeeded after the registry artifact became available. All 6 real-SDK tests passed against that fresh install; the single unchanged-head CI retry passed. The SDK pin, workflow, and dependency files are unchanged. - The first browser shard 8 run timed out waiting for the inbox retry reply after 45 seconds. The unchanged isolated case passed (1/1), and the test, UI handler, fixture, and recovery files match the base commit. The failed log contains no wakeup POST before the test's immediate navigation; a navigation/request timing race is suspected but unproven without a trace. The single unchanged-head shard retry passed (20 passed, 1 skipped); no timeout or source change was made. - Greptile reviewed this exact commit at 5/5 with no unresolved review threads. - No live portfolio request was replayed. Route tests use controlled local upstream responses. - The added diff passed the secret and PII scan and `git diff --check`. ## Risks This is a diagnostic change. It does not identify or repair the origin of a connection reset. Unknown transport failures remain `unknown`. Elapsed values outside 0–60,000 ms become null. Default Sentry fingerprinting remains enabled; exact historical group membership is not guaranteed. No retry, migration, deployment, or configuration change is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, code execution, and independent agent review. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
3290d97417
commit
90182b4f8b
8 files changed
+331
-14
No files matched your search
@@ -467,6 +467,19 @@ native failure only while its saved review is still current and pending; caller
|
||||
payloads cannot select a different review. Dispatch checks that assignment again.
|
||||
Superseded or completed reviews retain their authority.
|
||||
|
||||
Cloud portfolio proxy failures have an event-local `cloud_portfolio` context.
|
||||
It contains only the phase (`fetch`, `http_response`, `response_body`, or
|
||||
`response_write`), upstream HTTP status when available, elapsed milliseconds,
|
||||
and an allowlisted network error code. Unknown values become `unknown` or
|
||||
`null`; elapsed time outside 0–60,000 ms becomes `null`. Codes come from at
|
||||
most four error/cause objects' own data properties, never error messages.
|
||||
`DEADLINE_EXCEEDED` means the request's own ten-second signal was aborted.
|
||||
The route's warning uses the same safe fields. Neither record includes URLs,
|
||||
headers, cookies, bodies, user/stack IDs, or the original exception. Sentry
|
||||
receives a plain error with the existing generic message and default grouping.
|
||||
The context does not change the HTTP response, authentication, cookies, cache,
|
||||
deadline, or single-fetch behavior, and does not carry into unrelated events.
|
||||
|
||||
An unconfirmed adapter Stop timeout has an event-local `adapter_stop` context:
|
||||
the run UUID, built-in adapter type, native/legacy runtime mode, configured
|
||||
wait duration, and whether abort was requested. Invalid identities become
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { CloudPortfolioError } from "../services/cloud-portfolio-error.js";
|
||||
|
||||
const input = { phase: "fetch" as const, elapsedMs: 270.2, upstreamStatus: null };
|
||||
|
||||
describe("CloudPortfolioError", () => {
|
||||
it("retains only an allowlisted nested network code, not the original exception", () => {
|
||||
const cause = Object.assign(new Error("private upstream payload"), { code: "ECONNRESET", url: "https://private.example.test" });
|
||||
const error = new CloudPortfolioError("upstream", input, new TypeError("fetch failed", { cause }));
|
||||
expect(error.diagnostics).toEqual({ phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" });
|
||||
expect(error).not.toHaveProperty("cause");
|
||||
expect(JSON.stringify(error)).not.toContain("private");
|
||||
expect(Object.isFrozen(error.diagnostics)).toBe(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
new Error("ECONNRESET in a private message is not evidence"),
|
||||
{ code: "private-provider-code" },
|
||||
{ get code() { throw new Error("private getter"); }, get cause() { throw new Error("private getter"); } },
|
||||
new Proxy({}, { getOwnPropertyDescriptor() { throw new Error("private proxy"); } }),
|
||||
])("fails closed for unknown errors without invoking getters", (cause) => {
|
||||
expect(new CloudPortfolioError("upstream", input, cause).diagnostics.networkCode).toBe("unknown");
|
||||
});
|
||||
|
||||
it("bounds cause traversal and handles cycles", () => {
|
||||
const cycle: { cause?: unknown } = {};
|
||||
cycle.cause = cycle;
|
||||
expect(new CloudPortfolioError("upstream", input, cycle).diagnostics.networkCode).toBe("unknown");
|
||||
const deep = { cause: { cause: { cause: { cause: { code: "ECONNRESET" } } } } };
|
||||
expect(new CloudPortfolioError("upstream", input, deep).diagnostics.networkCode).toBe("unknown");
|
||||
});
|
||||
|
||||
it("normalizes unsafe diagnostics and keeps non-network phases distinct", () => {
|
||||
const error = new CloudPortfolioError("upstream", {
|
||||
phase: "private-phase" as never, elapsedMs: Infinity, upstreamStatus: 900,
|
||||
}, { code: "ECONNRESET" });
|
||||
expect(error.diagnostics).toEqual({ phase: "unknown", elapsedMs: null, upstreamStatus: null, networkCode: "unknown" });
|
||||
for (const phase of ["http_response", "response_write"] as const) {
|
||||
expect(new CloudPortfolioError("upstream", { ...input, phase, upstreamStatus: 503, deadlineExceeded: true }, { code: "ECONNRESET" }).diagnostics)
|
||||
.toMatchObject({ phase, upstreamStatus: 503, networkCode: "unknown" });
|
||||
}
|
||||
expect(new CloudPortfolioError("upstream", { ...input, elapsedMs: 60_001 }).diagnostics.elapsedMs).toBeNull();
|
||||
expect(new CloudPortfolioError("upstream", { ...input, elapsedMs: -1 }).diagnostics.elapsedMs).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,8 +1,13 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { cloudRoutes } from "../routes/cloud.js";
|
||||
import { CloudPortfolioError } from "../services/cloud-portfolio-error.js";
|
||||
import * as sentry from "../sentry.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
const cloudEnv = {
|
||||
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "tenant-secret",
|
||||
@@ -51,6 +56,98 @@ function jsonResponse(payload: unknown, status = 200) {
|
||||
}
|
||||
|
||||
describe("GET /api/cloud/stacks", () => {
|
||||
it.each([401, 403, 429, 503])("preserves the response contract for upstream HTTP %s without reading the body", async (status) => {
|
||||
const upstream = jsonResponse({ secret: "private upstream payload" }, status);
|
||||
const json = vi.spyOn(upstream, "json");
|
||||
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
|
||||
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
|
||||
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
|
||||
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks").set("Cookie", "client-session=private-cookie");
|
||||
expect(response.status).toBe(502);
|
||||
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
|
||||
expect(response.headers["set-cookie"]).toBeUndefined();
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(1);
|
||||
expect(new Headers(fetchImpl.mock.calls[0]![1]!.headers).has("cookie")).toBe(false);
|
||||
expect(json).not.toHaveBeenCalled();
|
||||
expect(capture).toHaveBeenCalledWith(expect.any(CloudPortfolioError));
|
||||
const error = capture.mock.calls[0]![0] as CloudPortfolioError;
|
||||
expect(error.diagnostics).toMatchObject({ phase: "http_response", upstreamStatus: status, networkCode: "unknown", elapsedMs: expect.any(Number) });
|
||||
expect(error.stack).toContain("routes/cloud.ts:");
|
||||
expect(error.stack).not.toContain("at portfolioError");
|
||||
expect(warn).toHaveBeenCalledWith({ cloudPortfolio: error.diagnostics }, "Paperclip Cloud portfolio request failed");
|
||||
expect(JSON.stringify(warn.mock.calls)).not.toContain("private");
|
||||
});
|
||||
|
||||
it("classifies a fetch reset without retrying or caching the failure", async () => {
|
||||
const fetchImpl = vi.fn<typeof fetch>()
|
||||
.mockRejectedValueOnce(new TypeError("private fetch message", { cause: Object.assign(new Error("private cause"), { code: "ECONNRESET" }) }))
|
||||
.mockResolvedValueOnce(jsonResponse({ stacks: [] }));
|
||||
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
|
||||
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
|
||||
const app = createApp({ fetchImpl });
|
||||
const failure = await request(app).get("/api/cloud/stacks");
|
||||
expect(failure.status).toBe(502);
|
||||
expect(failure.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(1);
|
||||
const error = capture.mock.calls[0]![0] as CloudPortfolioError;
|
||||
expect(error.diagnostics).toMatchObject({ phase: "fetch", upstreamStatus: null, networkCode: "ECONNRESET" });
|
||||
expect(JSON.stringify(warn.mock.calls)).not.toContain("private");
|
||||
expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] });
|
||||
expect((await request(app).get("/api/cloud/stacks")).body).toEqual({ stacks: [] });
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it.each(["invalid_json", "body_reset", "deadline"] as const)("keeps body-phase %s failures separate from fetch failures", async (kind) => {
|
||||
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
|
||||
const controller = new AbortController();
|
||||
const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(controller.signal);
|
||||
const upstream = jsonResponse({});
|
||||
vi.spyOn(upstream, "json").mockImplementation(async () => {
|
||||
if (kind === "deadline") controller.abort(new DOMException("private timeout", "TimeoutError"));
|
||||
throw kind === "body_reset" ? Object.assign(new Error("private body"), { code: "ECONNRESET" }) : new SyntaxError("private body");
|
||||
});
|
||||
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
|
||||
const app = createApp({ fetchImpl });
|
||||
for (let count = 1; count <= 2; count++) {
|
||||
const response = await request(app).get("/api/cloud/stacks");
|
||||
expect(response.status).toBe(502);
|
||||
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio returned invalid JSON", code: "cloud_portfolio_invalid_response", details: { code: "cloud_portfolio_invalid_response" } });
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(count);
|
||||
}
|
||||
expect(timeout).toHaveBeenCalledWith(10_000);
|
||||
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({
|
||||
phase: "response_body", upstreamStatus: 200,
|
||||
networkCode: kind === "deadline" ? "DEADLINE_EXCEEDED" : kind === "body_reset" ? "ECONNRESET" : "unknown",
|
||||
});
|
||||
});
|
||||
|
||||
it("records the owned fetch deadline without changing the ten-second signal", async () => {
|
||||
const signal = AbortSignal.abort(new DOMException("private timeout", "TimeoutError"));
|
||||
const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(signal);
|
||||
const fetchImpl = vi.fn<typeof fetch>().mockRejectedValue(signal.reason);
|
||||
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
|
||||
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks");
|
||||
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(1);
|
||||
expect(fetchImpl.mock.calls[0]![1]?.signal).toBe(signal);
|
||||
expect(timeout).toHaveBeenCalledWith(10_000);
|
||||
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "fetch", networkCode: "DEADLINE_EXCEEDED" });
|
||||
});
|
||||
|
||||
it("does not mislabel a response serialization failure as an upstream fetch error", async () => {
|
||||
const payload: { cycle?: unknown } = {};
|
||||
payload.cycle = payload;
|
||||
const upstream = jsonResponse({});
|
||||
vi.spyOn(upstream, "json").mockResolvedValue(payload);
|
||||
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(upstream);
|
||||
const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {});
|
||||
const response = await request(createApp({ fetchImpl })).get("/api/cloud/stacks");
|
||||
expect(response.status).toBe(502);
|
||||
expect(response.body).toEqual({ error: "Paperclip Cloud portfolio request failed", code: "cloud_portfolio_upstream_error", details: { code: "cloud_portfolio_upstream_error" } });
|
||||
expect(fetchImpl).toHaveBeenCalledTimes(1);
|
||||
expect((capture.mock.calls[0]![0] as CloudPortfolioError).diagnostics).toMatchObject({ phase: "response_write", upstreamStatus: 200, networkCode: "unknown" });
|
||||
});
|
||||
|
||||
it("returns the actor's portfolio without forwarding client-supplied identity", async () => {
|
||||
const portfolio = { stacks: [{ slug: "current", displayName: "Current" }] };
|
||||
const fetchImpl = vi.fn<typeof fetch>().mockResolvedValue(jsonResponse(portfolio));
|
||||
|
||||
@@ -42,6 +42,48 @@ afterEach(async () => {
|
||||
});
|
||||
|
||||
describe.skipIf(!sentryPackage)("run failure context with the real Sentry SDK", () => {
|
||||
it("keeps portfolio diagnostics bounded and isolated from unrelated captures", async () => {
|
||||
const Sentry = sentryPackage!;
|
||||
const events: Array<Record<string, unknown>> = [];
|
||||
vi.stubEnv("SENTRY_DSN_BACKEND", "https://public@example.invalid/1");
|
||||
vi.doMock("../peer-version-check.js", () => ({ checkExactPeerVersions: () => ({ ok: true }) }));
|
||||
vi.doMock("@sentry/node", () => ({
|
||||
...Sentry,
|
||||
init: (options: Record<string, unknown>) => Sentry.init({
|
||||
...options,
|
||||
transport: () => ({ send: async () => ({}), flush: async () => true }),
|
||||
beforeSend: (event: Record<string, unknown>) => { events.push(event); return event; },
|
||||
}),
|
||||
}));
|
||||
vi.resetModules();
|
||||
const { sentryReady, captureException } = await import("../sentry.js");
|
||||
const { CloudPortfolioError } = await import("../services/cloud-portfolio-error.js");
|
||||
await sentryReady;
|
||||
const failure = new CloudPortfolioError("upstream", {
|
||||
phase: "fetch", elapsedMs: 270, upstreamStatus: null,
|
||||
}, new TypeError("private-portfolio-message", { cause: { code: "ECONNRESET", token: "private-portfolio-token" } }));
|
||||
Object.assign(failure, {
|
||||
cause: new Error("private-portfolio-cause"),
|
||||
request: { url: "https://private-portfolio.invalid", headers: { cookie: "private-portfolio-cookie" } },
|
||||
});
|
||||
Object.defineProperty(failure, "diagnostics", { value: { extra: "private-portfolio-replacement" } });
|
||||
captureException(failure);
|
||||
captureException(new Error("unrelated portfolio fixture"));
|
||||
await Sentry.flush(2000);
|
||||
expect(events).toHaveLength(2);
|
||||
const captured = events.find((event) => (event.exception as { values: Array<{ value: string }> }).values.some((entry) => entry.value === failure.message));
|
||||
expect(captured).toMatchObject({
|
||||
tags: { error_code: "cloud_portfolio_failure" }, fingerprint: ["{{ default }}"],
|
||||
contexts: { cloud_portfolio: { phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" } },
|
||||
});
|
||||
expect((captured!.exception as { values: unknown[] }).values).toHaveLength(1);
|
||||
expect(JSON.stringify(events)).not.toContain("private-portfolio-");
|
||||
const unrelated = events.find((event) => event !== captured)!;
|
||||
expect(unrelated).not.toHaveProperty("contexts.cloud_portfolio");
|
||||
expect(unrelated).not.toHaveProperty("tags.error_code");
|
||||
expect(unrelated).not.toHaveProperty("fingerprint");
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ phase: "workspace_restore", phaseElapsedMs: 60_123, expectedPhase: "workspace_restore", expectedElapsedMs: 60_123 },
|
||||
{ phase: "private-provider-phase", phaseElapsedMs: 100, expectedPhase: "unknown", expectedElapsedMs: null },
|
||||
|
||||
@@ -127,6 +127,29 @@ describe("sentryReady", () => {
|
||||
});
|
||||
|
||||
describe("captureException", () => {
|
||||
it("sends only private normalized portfolio diagnostics on the matching event", async () => {
|
||||
process.env[BACKEND_DSN_ENV] = "https://fixture@example.com/1";
|
||||
const sdk = mockSentryPackage();
|
||||
const { captureException, sentryReady } = await importFreshSentry();
|
||||
const { CloudPortfolioError } = await import("../services/cloud-portfolio-error.js");
|
||||
await sentryReady;
|
||||
const error = new CloudPortfolioError("upstream", { phase: "fetch", elapsedMs: 270, upstreamStatus: null }, { code: "ECONNRESET" });
|
||||
Object.assign(error, { cause: new Error("private cause"), headers: { authorization: "private token" } });
|
||||
Object.defineProperty(error, "diagnostics", { value: { token: "private replacement" } });
|
||||
captureException(error);
|
||||
const unrelated = new Error("unrelated");
|
||||
captureException(unrelated);
|
||||
expect(sdk.captureException.mock.calls[0]).toEqual([
|
||||
expect.objectContaining({ message: error.message, stack: error.stack }),
|
||||
{ tags: { error_code: "cloud_portfolio_failure" }, fingerprint: ["{{ default }}"], contexts: {
|
||||
cloud_portfolio: { phase: "fetch", elapsedMs: 270, upstreamStatus: null, networkCode: "ECONNRESET" },
|
||||
} },
|
||||
]);
|
||||
expect(JSON.stringify(sdk.captureException.mock.calls[0])).not.toContain("private");
|
||||
expect(sdk.captureException.mock.calls[0]![0]).not.toHaveProperty("cause");
|
||||
expect(sdk.captureException.mock.calls[1]).toEqual([unrelated]);
|
||||
});
|
||||
|
||||
it("adds bounded Stop timeout context only to that event", async () => {
|
||||
process.env[BACKEND_DSN_ENV] = "https://fixture@example.com/1";
|
||||
const sdk = mockSentryPackage();
|
||||
|
||||
+31
-13
@@ -1,6 +1,7 @@
|
||||
import { Router } from "express";
|
||||
import { forbidden, HttpError, notFound } from "../errors.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { CloudPortfolioError, type CloudPortfolioPhase } from "../services/cloud-portfolio-error.js";
|
||||
import {
|
||||
getCloudStackContext,
|
||||
type CloudInstanceEnv,
|
||||
@@ -67,7 +68,24 @@ export function cloudRoutes(opts: {
|
||||
});
|
||||
}
|
||||
|
||||
const startedAt = performance.now();
|
||||
let phase: CloudPortfolioPhase = "fetch";
|
||||
let upstreamStatus: number | null = null;
|
||||
let upstreamSignal: AbortSignal | undefined;
|
||||
const portfolioError = (kind: "upstream" | "invalid_response", error?: unknown) => {
|
||||
const reportableError = new CloudPortfolioError(kind, {
|
||||
phase,
|
||||
elapsedMs: performance.now() - startedAt,
|
||||
upstreamStatus,
|
||||
deadlineExceeded: upstreamSignal?.aborted === true,
|
||||
}, error);
|
||||
// Preserve the route callsite instead of grouping by this helper frame.
|
||||
Error.captureStackTrace?.(reportableError, portfolioError);
|
||||
return reportableError;
|
||||
};
|
||||
|
||||
try {
|
||||
upstreamSignal = AbortSignal.timeout(10_000);
|
||||
const upstream = await fetchImpl(portfolioUrl, {
|
||||
method: "GET",
|
||||
headers: {
|
||||
@@ -76,26 +94,27 @@ export function cloudRoutes(opts: {
|
||||
"x-paperclip-cloud-user-id": actorUserId,
|
||||
"x-paperclip-cloud-stack-id": context.stackId,
|
||||
},
|
||||
signal: AbortSignal.timeout(10_000),
|
||||
signal: upstreamSignal,
|
||||
});
|
||||
phase = "http_response";
|
||||
upstreamStatus = upstream.status;
|
||||
if (!upstream.ok) {
|
||||
const error = portfolioError("upstream");
|
||||
logger.warn(
|
||||
{ status: upstream.status, stackId: context.stackId },
|
||||
{ cloudPortfolio: error.diagnostics },
|
||||
"Paperclip Cloud portfolio request failed",
|
||||
);
|
||||
throw new HttpError(502, "Paperclip Cloud portfolio request failed", {
|
||||
code: "cloud_portfolio_upstream_error",
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
|
||||
phase = "response_body";
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = await upstream.json();
|
||||
} catch {
|
||||
throw new HttpError(502, "Paperclip Cloud portfolio returned invalid JSON", {
|
||||
code: "cloud_portfolio_invalid_response",
|
||||
});
|
||||
} catch (error) {
|
||||
throw portfolioError("invalid_response", error);
|
||||
}
|
||||
phase = "response_write";
|
||||
portfolioCache.set(cacheKey, {
|
||||
expiresAt: currentTime + cacheTtlMs,
|
||||
payload,
|
||||
@@ -104,13 +123,12 @@ export function cloudRoutes(opts: {
|
||||
res.json(payload);
|
||||
} catch (error) {
|
||||
if (error instanceof HttpError) throw error;
|
||||
const reportableError = portfolioError("upstream", error);
|
||||
logger.warn(
|
||||
{ err: error, stackId: context.stackId },
|
||||
{ cloudPortfolio: reportableError.diagnostics },
|
||||
"Paperclip Cloud portfolio request failed",
|
||||
);
|
||||
throw new HttpError(502, "Paperclip Cloud portfolio request failed", {
|
||||
code: "cloud_portfolio_upstream_error",
|
||||
});
|
||||
throw reportableError;
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@
|
||||
|
||||
import os from "node:os";
|
||||
import { AdapterStopTimeoutError } from "./services/adapter-stop-timeout.js";
|
||||
import { CloudPortfolioError } from "./services/cloud-portfolio-error.js";
|
||||
import type { RunFailureDiagnostics } from "./services/run-failure-diagnostics.js";
|
||||
import { readBuildCommit } from "./build-commit.js";
|
||||
import { checkExactPeerVersions } from "./peer-version-check.js";
|
||||
@@ -116,6 +117,14 @@ export function captureException(error: unknown): void {
|
||||
contexts: { adapter_stop: { ...error.diagnostics } },
|
||||
fingerprint: ["{{ default }}"],
|
||||
});
|
||||
} else if (error instanceof CloudPortfolioError) {
|
||||
const exception = new Error(error.message);
|
||||
exception.stack = error.stack;
|
||||
sentryHandle.captureException(exception, {
|
||||
tags: { error_code: "cloud_portfolio_failure" },
|
||||
contexts: { cloud_portfolio: { ...CloudPortfolioError.diagnosticsFor(error) } },
|
||||
fingerprint: ["{{ default }}"],
|
||||
});
|
||||
} else {
|
||||
sentryHandle.captureException(error);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
import { HttpError } from "../errors.js";
|
||||
|
||||
const phases = ["fetch", "http_response", "response_body", "response_write"] as const;
|
||||
export type CloudPortfolioPhase = typeof phases[number];
|
||||
const networkCodes = new Set([
|
||||
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EAI_AGAIN",
|
||||
"ENETUNREACH", "EHOSTUNREACH", "UND_ERR_CONNECT_TIMEOUT",
|
||||
"UND_ERR_HEADERS_TIMEOUT", "UND_ERR_BODY_TIMEOUT", "UND_ERR_SOCKET",
|
||||
"ERR_TLS_CERT_ALTNAME_INVALID", "CERT_HAS_EXPIRED",
|
||||
"DEPTH_ZERO_SELF_SIGNED_CERT", "UNABLE_TO_VERIFY_LEAF_SIGNATURE", "ABORT_ERR",
|
||||
]);
|
||||
|
||||
// Inspect data properties only. A provider error can contain getters, cycles,
|
||||
// arbitrary messages or nested request objects; none are diagnostic payloads.
|
||||
function ownValue(value: unknown, key: string): unknown {
|
||||
if (!value || typeof value !== "object") return undefined;
|
||||
try {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(value, key);
|
||||
return descriptor && "value" in descriptor ? descriptor.value : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function networkCode(error: unknown): string {
|
||||
let current = error;
|
||||
for (let depth = 0; depth < 4; depth++) {
|
||||
const code = ownValue(current, "code");
|
||||
if (typeof code === "string" && networkCodes.has(code)) return code;
|
||||
current = ownValue(current, "cause");
|
||||
}
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
/** Server-only diagnostics. The existing HTTP response details stay unchanged. */
|
||||
export class CloudPortfolioError extends HttpError {
|
||||
readonly #diagnostics: Readonly<Record<string, string | number | null>>;
|
||||
|
||||
constructor(
|
||||
kind: "upstream" | "invalid_response",
|
||||
input: { phase: CloudPortfolioPhase; elapsedMs: number; upstreamStatus: number | null; deadlineExceeded?: boolean },
|
||||
error?: unknown,
|
||||
) {
|
||||
super(502, kind === "invalid_response"
|
||||
? "Paperclip Cloud portfolio returned invalid JSON"
|
||||
: "Paperclip Cloud portfolio request failed", {
|
||||
code: kind === "invalid_response" ? "cloud_portfolio_invalid_response" : "cloud_portfolio_upstream_error",
|
||||
});
|
||||
Error.captureStackTrace?.(this, CloudPortfolioError);
|
||||
const phase = phases.includes(input.phase) ? input.phase : "unknown";
|
||||
this.#diagnostics = Object.freeze({
|
||||
phase,
|
||||
upstreamStatus: Number.isInteger(input.upstreamStatus) && input.upstreamStatus! >= 100 && input.upstreamStatus! <= 599
|
||||
? input.upstreamStatus : null,
|
||||
elapsedMs: Number.isFinite(input.elapsedMs) && input.elapsedMs >= 0 && input.elapsedMs <= 60_000
|
||||
? Math.round(input.elapsedMs) : null,
|
||||
networkCode: phase === "fetch" || phase === "response_body"
|
||||
? input.deadlineExceeded === true ? "DEADLINE_EXCEEDED" : networkCode(error)
|
||||
: "unknown",
|
||||
});
|
||||
}
|
||||
|
||||
get diagnostics() {
|
||||
return this.#diagnostics;
|
||||
}
|
||||
|
||||
static diagnosticsFor(error: CloudPortfolioError) {
|
||||
return error.#diagnostics;
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user