fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent setup lists the models that an OpenRouter key can select. The
server gets this list from the public OpenRouter catalog.
> - The catalog request has its own 10 second deadline. When the
deadline expired, fetch rejected with a raw `TimeoutError` (a
`DOMException`).
> - The error handler answered 500 "Internal server error" and sent a
crash report with no first-party stack frames. The user did not see the
existing retry guidance.
> - A slow or unavailable third-party catalog is not a fault in the
Paperclip server. It must not create a crash report.
> - This pull request maps catalog outages to a 503 response with the
existing retry message and a stable code. Only those errors skip crash
reporting.
> - The benefit: users see "Retry or enter a model ID manually", and
Sentry does not get noise from a provider outage.

## Linked Issues or Issue Description

**What happened?**

A staging server answered `GET
/api/companies/:companyId/adapters/opencode_local/models` with `500 —
The operation was aborted due to timeout`. At the same time, other
outbound requests from the host were slow. Sentry recorded
`TimeoutError: The operation was aborted due to timeout`. The event had
only Node internal frames (`AbortSignal.timeout`).

**Expected behavior**

The route returns a clear "catalog unavailable" response. The UI shows
the retry guidance. A third-party outage does not create a crash report.

**Steps to reproduce**

1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more
than 10 seconds), or make it unreachable.
2. Request the OpenCode model list with `provider=openrouter`.
3. See a 500 response and a Sentry `TimeoutError` event.

**Paperclip version or commit**

`e23549cd9c631e4595a3213e03d0c960360452c6`

I searched for duplicate PRs. #17083, #17961 and #10290 change the
OpenRouter catalog contents or model validation. They do not change how
the server handles a catalog timeout.

## What Changed

- `server/src/services/openrouter-models.ts`: The catalog now maps three
failure types to `HttpError(503)` with code
`openrouter_catalog_unavailable` and the existing retry message: its own
deadline, a classified transport failure (uses
`classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A
module-local `WeakSet` records only the errors that it creates for these
outages. `isOpenRouterCatalogUnavailable()` reads that record.
- Other non-OK statuses become a 502 with code
`openrouter_catalog_error`. These are still reported. Malformed JSON and
an invalid catalog shape are also still reported.
- `server/src/middleware/error-handler.ts`: The handler does not send a
crash report for an error from `isOpenRouterCatalogUnavailable()`. It
still answers 503 with the message and code. The new check is next to
the existing MCP connection-failure exclusion.
- Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for
the failures that stay reportable. A route-level test sends a stalled
catalog request through the real error handler. It checks for a 503
response and no crash report. It also checks that a different 503 is
still reported.

## Verification

- `vitest run server/src/services/openrouter-models.test.ts`: 7 tests
pass. I ran this file alone because the full workspace install did not
fit in the agent sandbox disk.
- Red check: with the original `openrouter-models.ts`, 5 of the 7 tests
fail. The deadline test receives `DOMException TimeoutError`, which is
the same error as the production event.
- `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs
the full workspace dependencies. CI runs it.

## Risks

- Low risk. Successful responses do not change. The cache and in-flight
deduplication do not change.
- The failure status changes from 500 to 503 (or 502) for the three
routes that read this catalog: adapter models, fast-response models and
prompt-injection-prevention models. The UI already shows
`error.message`, so users now see the retry guidance in place of
"Internal server error".
- The crash-report exclusion is narrow. It applies only to error objects
that this module created after an actual outage. A copied message or
code does not match.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry, Railway and
GitHub connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleyandClaude Opus 5.5 authored and GitHub committed 2026-11-13 13:00:24 -08:00
1 parent 0ebe6d846d
commit 2756be2f32
4 files changed
+148 -4

No files matched your search

@@ -0,0 +1,51 @@
import express from "express";
import request from "supertest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { HttpError } from "../errors.js";
import { errorHandler } from "../middleware/error-handler.js";
import { listOpenRouterModels } from "../services/openrouter-models.js";
const captureExceptionMock = vi.hoisted(() => vi.fn());
vi.mock("../sentry.js", () => ({ captureException: captureExceptionMock }));
vi.mock("../telemetry.js", () => ({ getTelemetryClient: () => null }));
function createApp() {
const app = express();
app.get("/models", async (_req, res) => { res.json(await listOpenRouterModels(true)); });
app.get("/other-outage", async () => { throw new HttpError(503, "Durable agent files require shared object storage"); });
app.use(errorHandler);
return app;
}
describe("OpenRouter catalog outages", () => {
beforeEach(() => captureExceptionMock.mockReset());
afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); });
it("answers a stalled public catalog with retry guidance and no crash report", async () => {
// PAPERCLIP-BACKEND-8D: the catalog request's own 10s deadline used to
// reach this handler as a raw TimeoutError, answering 500 and paging Sentry.
const timeout = AbortSignal.timeout.bind(AbortSignal);
vi.spyOn(AbortSignal, "timeout").mockImplementation(() => timeout(5));
vi.stubGlobal("fetch", vi.fn((_url: string, init: { signal: AbortSignal }) => new Promise((_resolve, reject) => {
init.signal.addEventListener("abort", () => reject(init.signal.reason));
})));
const res = await request(createApp()).get("/models");
expect(res.status).toBe(503);
expect(res.body).toEqual({
error: "Could not load OpenRouter models. Retry or enter a model ID manually.",
code: "openrouter_catalog_unavailable",
details: { code: "openrouter_catalog_unavailable" },
});
expect(captureExceptionMock).not.toHaveBeenCalled();
});
it("still reports other server-side 503 failures", async () => {
const res = await request(createApp()).get("/other-outage");
expect(res.status).toBe(503);
expect(captureExceptionMock).toHaveBeenCalledTimes(1);
});
});
+4 -1
View File
@@ -20,6 +20,7 @@ import {
import { recordResponsibleUserDenialOnActiveRun } from "../services/responsible-user-denial-run-outcomes.js";
import { isObjectJournalBusyError } from "../services/object-byte-journal.js";
import { isExpectedMcpConnectionFailure } from "../services/mcp-connection-failure.js";
import { isOpenRouterCatalogUnavailable } from "../services/openrouter-models.js";
import { isEndedDbPoolError } from "../database-retry.js";
import { ToolGatewayHttpError } from "../services/tool-gateway.js";
@@ -214,7 +215,9 @@ export function errorHandler(
},
reportableError,
);
reportCrash(reportableError, req, !isExpectedMcpConnectionFailure(err));
// An unreachable public provider catalog still answers 503 with its
// retry guidance, but it is the provider's outage, not a server fault.
reportCrash(reportableError, req, !isExpectedMcpConnectionFailure(err) && !isOpenRouterCatalogUnavailable(err));
}
const secretSensitiveServerError =
err.status >= 500 &&
@@ -27,3 +27,55 @@ it("allows retry after a public catalog failure", async () => {
await expect(listOpenRouterModels()).rejects.toThrow("Retry or enter a model ID manually");
await expect(listOpenRouterModels()).resolves.toEqual([]);
});
/** A fetch that only settles when its signal fires, as a stalled upstream does. */
function stalledFetch() {
return vi.fn((_url: string, init: { signal: AbortSignal }) => new Promise((_resolve, reject) => {
init.signal.addEventListener("abort", () => reject(init.signal.reason));
}));
}
function shortCatalogDeadline() {
const timeout = AbortSignal.timeout.bind(AbortSignal);
return vi.spyOn(AbortSignal, "timeout").mockImplementation(() => timeout(5));
}
it("answers its own catalog deadline as an unavailable provider, not a raw TimeoutError", async () => {
const deadline = shortCatalogDeadline();
vi.stubGlobal("fetch", stalledFetch());
const { listOpenRouterModels, isOpenRouterCatalogUnavailable } = await import("./openrouter-models.js");
const error = await listOpenRouterModels().catch((err: unknown) => err);
deadline.mockRestore();
expect(error).toMatchObject({ status: 503, message: "Could not load OpenRouter models. Retry or enter a model ID manually.",
details: { code: "openrouter_catalog_unavailable" } });
expect(isOpenRouterCatalogUnavailable(error)).toBe(true);
});
it.each([
["a DNS failure", () => Promise.reject(Object.assign(new TypeError("fetch failed"), { cause: Object.assign(new Error("getaddrinfo ENOTFOUND openrouter.ai"), { code: "ENOTFOUND" }) }))],
["a rate limit", () => Promise.resolve({ ok: false, status: 429 })],
["a provider server error", () => Promise.resolve({ ok: false, status: 503 })],
])("marks %s as an unavailable catalog", async (_label, response) => {
vi.stubGlobal("fetch", vi.fn(response));
const { listOpenRouterModels, isOpenRouterCatalogUnavailable } = await import("./openrouter-models.js");
const error = await listOpenRouterModels().catch((err: unknown) => err);
expect(error).toMatchObject({ status: 503, details: { code: "openrouter_catalog_unavailable" } });
expect(isOpenRouterCatalogUnavailable(error)).toBe(true);
});
it("keeps contract failures and unrelated errors reportable", async () => {
const { HttpError } = await import("../errors.js");
vi.stubGlobal("fetch", vi.fn()
.mockResolvedValueOnce({ ok: false, status: 404 })
.mockResolvedValueOnce({ ok: true, json: async () => { throw new SyntaxError("Unexpected token <"); } })
.mockResolvedValueOnce({ ok: true, json: async () => ({ data: null }) }));
const { listOpenRouterModels, isOpenRouterCatalogUnavailable } = await import("./openrouter-models.js");
const notFound = await listOpenRouterModels().catch((err: unknown) => err);
expect(notFound).toMatchObject({ status: 502, details: { code: "openrouter_catalog_error", status: 404 } });
expect(isOpenRouterCatalogUnavailable(notFound)).toBe(false);
const malformed = await listOpenRouterModels().catch((err: unknown) => err);
expect(malformed).toBeInstanceOf(SyntaxError);
expect(isOpenRouterCatalogUnavailable(malformed)).toBe(false);
await expect(listOpenRouterModels()).rejects.toThrow("OpenRouter returned an invalid model catalog.");
expect(isOpenRouterCatalogUnavailable(new HttpError(503, "Could not load OpenRouter models. Retry or enter a model ID manually.", { code: "openrouter_catalog_unavailable" }))).toBe(false);
});
+41 -3
View File
@@ -1,16 +1,54 @@
import type { AdapterModel } from "@paperclipai/adapter-utils";
import { HttpError } from "../errors.js";
import { classifyRemoteConnectionError } from "./remote-connection-failure.js";
const CATALOG_URL = "https://openrouter.ai/api/v1/models?sort=most-popular";
const CATALOG_TIMEOUT_MS = 10_000;
const CATALOG_UNAVAILABLE = "Could not load OpenRouter models. Retry or enter a model ID manually.";
let cached: { until: number; models: AdapterModel[] } | undefined;
let pending: Promise<AdapterModel[]> | undefined;
// Local provenance: only errors this module created for an actual outage of
// the public catalog (its own deadline, a classified transport failure, or a
// rate-limit/server status). They affect crash reporting only.
const unavailable = new WeakSet<Error>();
function catalogUnavailable(): HttpError {
const error = new HttpError(503, CATALOG_UNAVAILABLE, { code: "openrouter_catalog_unavailable" });
unavailable.add(error);
return error;
}
/** The public OpenRouter catalog was unreachable; this is not a Paperclip fault. */
export function isOpenRouterCatalogUnavailable(error: unknown): boolean {
return error instanceof HttpError && unavailable.has(error);
}
/** Map a failure of the outbound catalog request; rethrow anything else unchanged. */
function requestFailure(error: unknown, signal: AbortSignal): unknown {
return signal.aborted || classifyRemoteConnectionError(error) ? catalogUnavailable() : error;
}
/** OpenRouter's public catalog does not require access to anyone's credentials. */
export async function listOpenRouterModels(refresh = false): Promise<AdapterModel[]> {
if (!refresh && cached && cached.until > Date.now()) return cached.models;
if (pending) return pending;
pending = (async () => {
const response = await fetch("https://openrouter.ai/api/v1/models?sort=most-popular", { signal: AbortSignal.timeout(10_000) });
if (!response.ok) throw new Error("Could not load OpenRouter models. Retry or enter a model ID manually.");
const body = await response.json() as { data?: Array<{ id?: unknown; name?: unknown }> };
const signal = AbortSignal.timeout(CATALOG_TIMEOUT_MS);
let body: { data?: Array<{ id?: unknown; name?: unknown }> };
try {
const response = await fetch(CATALOG_URL, { signal });
if (!response.ok) {
await response.body?.cancel().catch(() => {});
if (response.status === 408 || response.status === 429 || response.status >= 500) throw catalogUnavailable();
throw new HttpError(502, CATALOG_UNAVAILABLE, { code: "openrouter_catalog_error", status: response.status });
}
body = await response.json() as typeof body;
} catch (error) {
if (error instanceof HttpError) throw error;
throw requestFailure(error, signal);
}
if (!Array.isArray(body.data)) throw new Error("OpenRouter returned an invalid model catalog.");
const models = body.data.flatMap(model => typeof model.id === "string" && model.id.includes("/")
? [{ id: `openrouter/${model.id}`, label: typeof model.name === "string" ? model.name : model.id }]