mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent setup lists the models that an OpenRouter key can select. The server gets this list from the public OpenRouter catalog. > - The catalog request has its own 10 second deadline. When the deadline expired, fetch rejected with a raw `TimeoutError` (a `DOMException`). > - The error handler answered 500 "Internal server error" and sent a crash report with no first-party stack frames. The user did not see the existing retry guidance. > - A slow or unavailable third-party catalog is not a fault in the Paperclip server. It must not create a crash report. > - This pull request maps catalog outages to a 503 response with the existing retry message and a stable code. Only those errors skip crash reporting. > - The benefit: users see "Retry or enter a model ID manually", and Sentry does not get noise from a provider outage. ## Linked Issues or Issue Description **What happened?** A staging server answered `GET /api/companies/:companyId/adapters/opencode_local/models` with `500 — The operation was aborted due to timeout`. At the same time, other outbound requests from the host were slow. Sentry recorded `TimeoutError: The operation was aborted due to timeout`. The event had only Node internal frames (`AbortSignal.timeout`). **Expected behavior** The route returns a clear "catalog unavailable" response. The UI shows the retry guidance. A third-party outage does not create a crash report. **Steps to reproduce** 1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more than 10 seconds), or make it unreachable. 2. Request the OpenCode model list with `provider=openrouter`. 3. See a 500 response and a Sentry `TimeoutError` event. **Paperclip version or commit** `e23549cd9c631e4595a3213e03d0c960360452c6` I searched for duplicate PRs. #17083, #17961 and #10290 change the OpenRouter catalog contents or model validation. They do not change how the server handles a catalog timeout. ## What Changed - `server/src/services/openrouter-models.ts`: The catalog now maps three failure types to `HttpError(503)` with code `openrouter_catalog_unavailable` and the existing retry message: its own deadline, a classified transport failure (uses `classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A module-local `WeakSet` records only the errors that it creates for these outages. `isOpenRouterCatalogUnavailable()` reads that record. - Other non-OK statuses become a 502 with code `openrouter_catalog_error`. These are still reported. Malformed JSON and an invalid catalog shape are also still reported. - `server/src/middleware/error-handler.ts`: The handler does not send a crash report for an error from `isOpenRouterCatalogUnavailable()`. It still answers 503 with the message and code. The new check is next to the existing MCP connection-failure exclusion. - Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for the failures that stay reportable. A route-level test sends a stalled catalog request through the real error handler. It checks for a 503 response and no crash report. It also checks that a different 503 is still reported. ## Verification - `vitest run server/src/services/openrouter-models.test.ts`: 7 tests pass. I ran this file alone because the full workspace install did not fit in the agent sandbox disk. - Red check: with the original `openrouter-models.ts`, 5 of the 7 tests fail. The deadline test receives `DOMException TimeoutError`, which is the same error as the production event. - `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs the full workspace dependencies. CI runs it. ## Risks - Low risk. Successful responses do not change. The cache and in-flight deduplication do not change. - The failure status changes from 500 to 503 (or 502) for the three routes that read this catalog: adapter models, fast-response models and prompt-injection-prevention models. The UI already shows `error.message`, so users now see the retry guidance in place of "Internal server error". - The crash-report exclusion is narrow. It applies only to error objects that this module created after an actual outage. A copied message or code does not match. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness with tool use (shell, file edits, Sentry, Railway and GitHub connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
377 lines
13 KiB
TypeScript
377 lines
13 KiB
TypeScript
import { AgentLifecycleConflict } from "../modules/agent-lifecycle/index.js";
|
|
import type { Request, Response, NextFunction } from "express";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { ZodError } from "zod";
|
|
import { HttpError } from "../errors.js";
|
|
import { trackErrorHandlerCrash } from "@paperclipai/shared/telemetry";
|
|
import { getTelemetryClient } from "../telemetry.js";
|
|
import { captureException } from "../sentry.js";
|
|
import { COMPANY_IMPORT_API_PATH } from "../routes/company-import-paths.js";
|
|
import { logger } from "./logger.js";
|
|
import {
|
|
isSecretSensitiveHttpRequest,
|
|
secretSensitiveRouteFamily,
|
|
} from "./http-log-policy.js";
|
|
import { SecretSensitiveRequestError } from "./secret-sensitive-request-error.js";
|
|
import {
|
|
collectSensitiveStringValues,
|
|
redactSensitiveValueOccurrences,
|
|
} from "./redact-sensitive.js";
|
|
import { recordResponsibleUserDenialOnActiveRun } from "../services/responsible-user-denial-run-outcomes.js";
|
|
import { isObjectJournalBusyError } from "../services/object-byte-journal.js";
|
|
import { isExpectedMcpConnectionFailure } from "../services/mcp-connection-failure.js";
|
|
import { isOpenRouterCatalogUnavailable } from "../services/openrouter-models.js";
|
|
import { isEndedDbPoolError } from "../database-retry.js";
|
|
import { ToolGatewayHttpError } from "../services/tool-gateway.js";
|
|
|
|
export interface ErrorContext {
|
|
error: {
|
|
message: string;
|
|
stack?: string;
|
|
name?: string;
|
|
details?: unknown;
|
|
raw?: unknown;
|
|
};
|
|
method: string;
|
|
url: string;
|
|
reqBody?: unknown;
|
|
reqParams?: unknown;
|
|
reqQuery?: unknown;
|
|
}
|
|
|
|
function isRedactedSkillPolicyDenial(details: Record<string, unknown> | null) {
|
|
return details?.code === "skill_policy_denied";
|
|
}
|
|
|
|
function readZodIssues(err: unknown): unknown[] | null {
|
|
if (err instanceof ZodError) return err.issues;
|
|
if (
|
|
!err ||
|
|
typeof err !== "object" ||
|
|
(err as { name?: unknown }).name !== "ZodError"
|
|
)
|
|
return null;
|
|
const issues = (err as { issues?: unknown }).issues;
|
|
return Array.isArray(issues) ? issues : null;
|
|
}
|
|
|
|
function attachErrorContext(
|
|
req: Request,
|
|
res: Response,
|
|
payload: ErrorContext["error"],
|
|
rawError?: Error,
|
|
) {
|
|
(res as any).__errorContext = {
|
|
error: payload,
|
|
method: req.method,
|
|
url: req.originalUrl,
|
|
reqBody: req.body,
|
|
reqParams: req.params,
|
|
reqQuery: req.query,
|
|
} satisfies ErrorContext;
|
|
if (rawError) {
|
|
(res as any).err = rawError;
|
|
}
|
|
}
|
|
|
|
function sanitizeSecretSensitiveError(req: Request, error: Error): Error {
|
|
const routeFamily = secretSensitiveRouteFamily(req.method, req.originalUrl);
|
|
if (!routeFamily) return error;
|
|
return new SecretSensitiveRequestError(error, {
|
|
routeFamily,
|
|
method: req.method,
|
|
routePath: (req.route as { path?: unknown } | undefined)?.path,
|
|
});
|
|
}
|
|
|
|
function sanitizeSecretSensitiveResponse(
|
|
req: Request,
|
|
value: unknown,
|
|
): unknown {
|
|
if (!isSecretSensitiveHttpRequest(req.method, req.originalUrl)) return value;
|
|
return redactSensitiveValueOccurrences(
|
|
value,
|
|
collectSensitiveStringValues(req.body),
|
|
);
|
|
}
|
|
|
|
/** Report a server-side crash to every error sink. */
|
|
function reportCrash(error: Error, req: Request, reportToSentry = true): void {
|
|
const tc = getTelemetryClient();
|
|
if (tc) trackErrorHandlerCrash(tc, { errorCode: error.name });
|
|
if (reportToSentry) {
|
|
if (req.actor?.type === "agent" && req.actor.runId && req.actor.companyId && req.actor.agentId) {
|
|
captureException(error, { runId: req.actor.runId, companyId: req.actor.companyId, agentId: req.actor.agentId });
|
|
} else captureException(error);
|
|
}
|
|
}
|
|
|
|
/** Set by the orderly shutdown just before it ends the database pools. */
|
|
function isDatabaseClosing(req: Request): boolean {
|
|
const locals = req.app?.locals as { paperclipDatabaseClosing?: boolean } | undefined;
|
|
return locals?.paperclipDatabaseClosing === true;
|
|
}
|
|
|
|
function getPaperclipDb(req: Request): Db | null {
|
|
const locals = req.app?.locals as { paperclipDb?: Db; db?: Db } | undefined;
|
|
return locals?.paperclipDb ?? locals?.db ?? null;
|
|
}
|
|
|
|
function recordResponsibleUserDenialFromHttpError(
|
|
req: Request,
|
|
details: Record<string, unknown> | null,
|
|
) {
|
|
if (req.actor?.type !== "agent") return;
|
|
const db = getPaperclipDb(req);
|
|
if (!db) return;
|
|
|
|
void recordResponsibleUserDenialOnActiveRun(db, {
|
|
runId: req.actor.runId ?? null,
|
|
agentId: req.actor.agentId ?? null,
|
|
companyId: req.actor.companyId ?? null,
|
|
code: details?.code,
|
|
}).catch((recordErr) => {
|
|
logger.warn(
|
|
{
|
|
err: recordErr,
|
|
runId: req.actor?.runId ?? null,
|
|
agentId:
|
|
req.actor?.type === "agent" ? (req.actor.agentId ?? null) : null,
|
|
},
|
|
"failed to record responsible-user denial on heartbeat run",
|
|
);
|
|
});
|
|
}
|
|
|
|
/**
|
|
* The tool gateway rejects requests with its own error type. A route that lets
|
|
* one reach this handler (such as approving a tool review from a task card)
|
|
* would otherwise turn an expected rejection, for example a 409 asking for a
|
|
* new review, into a 500 crash report. Gateway server failures stay on the
|
|
* crash path so their provider text is not exposed.
|
|
*/
|
|
function clientToolGatewayErrorAsHttpError(err: unknown): unknown {
|
|
if (!(err instanceof ToolGatewayHttpError) || err.status < 400 || err.status >= 500) return err;
|
|
const mapped = new HttpError(err.status, err.message, { ...err.details, code: err.reasonCode });
|
|
mapped.stack = err.stack;
|
|
return mapped;
|
|
}
|
|
|
|
export function errorHandler(
|
|
thrown: unknown,
|
|
req: Request,
|
|
res: Response,
|
|
_next: NextFunction,
|
|
) {
|
|
const err = clientToolGatewayErrorAsHttpError(thrown);
|
|
if (err instanceof AgentLifecycleConflict) { res.status(409).json({ error: err.message }); return; }
|
|
if (err instanceof HttpError) {
|
|
const details =
|
|
err.details &&
|
|
typeof err.details === "object" &&
|
|
!Array.isArray(err.details)
|
|
? (err.details as Record<string, unknown>)
|
|
: null;
|
|
const redactedSkillPolicyDenial = isRedactedSkillPolicyDenial(details);
|
|
const workspaceRepairPreconditionFailure =
|
|
details?.code === "workspace_repair_precondition_failed";
|
|
// Its code and remediation are already top-level; repeating them only lengthens agent context.
|
|
const boardAccessDenial = details?.code === "board_access_required";
|
|
const structuredConnectionError = new Set([
|
|
"user_authorization_required",
|
|
"organization_authorization_required",
|
|
"grant_audience_denied",
|
|
"grant_revoked",
|
|
"needs_reauthorization",
|
|
"installation_required",
|
|
"connection_not_installed",
|
|
"subject_not_permitted",
|
|
"standing_delegation_required",
|
|
"grant_owner_membership_inactive",
|
|
]).has(typeof details?.code === "string" ? details.code : "");
|
|
const responseDetailsValue = sanitizeSecretSensitiveResponse(
|
|
req,
|
|
err.details,
|
|
);
|
|
const responseDetails =
|
|
responseDetailsValue &&
|
|
typeof responseDetailsValue === "object" &&
|
|
!Array.isArray(responseDetailsValue)
|
|
? (responseDetailsValue as Record<string, unknown>)
|
|
: null;
|
|
recordResponsibleUserDenialFromHttpError(req, details);
|
|
if (err.status >= 500) {
|
|
const reportableError = sanitizeSecretSensitiveError(req, err);
|
|
attachErrorContext(
|
|
req,
|
|
res,
|
|
isSecretSensitiveHttpRequest(req.method, req.originalUrl)
|
|
? { message: reportableError.message, name: reportableError.name }
|
|
: {
|
|
message: err.message,
|
|
stack: err.stack,
|
|
name: err.name,
|
|
details: err.details,
|
|
},
|
|
reportableError,
|
|
);
|
|
// An unreachable public provider catalog still answers 503 with its
|
|
// retry guidance, but it is the provider's outage, not a server fault.
|
|
reportCrash(reportableError, req, !isExpectedMcpConnectionFailure(err) && !isOpenRouterCatalogUnavailable(err));
|
|
}
|
|
const secretSensitiveServerError =
|
|
err.status >= 500 &&
|
|
isSecretSensitiveHttpRequest(req.method, req.originalUrl);
|
|
res.status(err.status).json(
|
|
secretSensitiveServerError
|
|
? { error: "Internal server error" }
|
|
: {
|
|
error: sanitizeSecretSensitiveResponse(req, err.message),
|
|
...(typeof responseDetails?.code === "string"
|
|
? { code: responseDetails.code }
|
|
: {}),
|
|
...(redactedSkillPolicyDenial &&
|
|
typeof responseDetails?.reason === "string"
|
|
? { reason: responseDetails.reason }
|
|
: {}),
|
|
...(workspaceRepairPreconditionFailure &&
|
|
typeof responseDetails?.reason === "string"
|
|
? { reason: responseDetails.reason }
|
|
: {}),
|
|
...(workspaceRepairPreconditionFailure &&
|
|
typeof responseDetails?.repairPhase === "string"
|
|
? { repairPhase: responseDetails.repairPhase }
|
|
: {}),
|
|
...(typeof responseDetails?.remediation === "string" ||
|
|
(structuredConnectionError &&
|
|
responseDetails?.remediation &&
|
|
typeof responseDetails.remediation === "object")
|
|
? { remediation: responseDetails.remediation }
|
|
: {}),
|
|
...(structuredConnectionError && responseDetails?.connection
|
|
? { connection: responseDetails.connection }
|
|
: {}),
|
|
...(structuredConnectionError && responseDetails?.subject
|
|
? { subject: responseDetails.subject }
|
|
: {}),
|
|
...(structuredConnectionError &&
|
|
typeof responseDetails?.grantId === "string"
|
|
? { grantId: responseDetails.grantId }
|
|
: {}),
|
|
...(!redactedSkillPolicyDenial &&
|
|
!workspaceRepairPreconditionFailure &&
|
|
!boardAccessDenial &&
|
|
responseDetailsValue
|
|
? { details: responseDetailsValue }
|
|
: {}),
|
|
},
|
|
);
|
|
return;
|
|
}
|
|
|
|
const zodIssues = readZodIssues(err);
|
|
if (zodIssues) {
|
|
res.status(400).json({
|
|
error: "Validation error",
|
|
details: sanitizeSecretSensitiveResponse(req, zodIssues),
|
|
});
|
|
return;
|
|
}
|
|
|
|
// Only body-parser's malformed-JSON errors are client input failures.
|
|
// Parser messages can quote request bytes; return a constant response and
|
|
// keep the raw error out of crash reporting and HTTP error context.
|
|
if (
|
|
err instanceof SyntaxError &&
|
|
"status" in err && err.status === 400 &&
|
|
"type" in err && err.type === "entity.parse.failed"
|
|
) {
|
|
res.status(400).json({ error: "Invalid JSON body" });
|
|
return;
|
|
}
|
|
|
|
const rootError = err instanceof Error ? err : new Error(String(err));
|
|
|
|
// The client tore down the connection mid-request (closed tab, dropped
|
|
// mobile network, cancelled upload): Node surfaces it as `Error: aborted`
|
|
// with ECONNRESET. There is no server fault to report and nobody left to
|
|
// answer, so skip the error sinks and just close out the response.
|
|
if (
|
|
rootError.message === "aborted" &&
|
|
(rootError as NodeJS.ErrnoException).code === "ECONNRESET"
|
|
) {
|
|
if (!res.headersSent) res.status(499);
|
|
res.end();
|
|
return;
|
|
}
|
|
|
|
// A route that cancels its work when the response closes (for example a
|
|
// storage read) rejects with AbortError after the client has gone. That
|
|
// rejection is the cancellation, not a server fault. An AbortError while
|
|
// the client is still connected is still reported.
|
|
if (rootError.name === "AbortError" && res.destroyed && !res.writableFinished) {
|
|
if (!res.headersSent) res.status(499);
|
|
res.end();
|
|
return;
|
|
}
|
|
|
|
// Shutdown drains the HTTP listener before it ends the pool, but a handler
|
|
// whose client already disconnected (or whose socket the drain timeout
|
|
// closed) keeps running. Its next query then fails with CONNECTION_ENDED
|
|
// against the pool this process deliberately ended. That is shutdown, not a
|
|
// server fault, so skip the error sinks. Any other error during shutdown is
|
|
// still reported.
|
|
if (isDatabaseClosing(req) && isEndedDbPoolError(rootError)) {
|
|
if (!res.headersSent) {
|
|
res.status(503).json({ error: "Server is shutting down" });
|
|
} else {
|
|
res.end();
|
|
}
|
|
return;
|
|
}
|
|
|
|
if (isObjectJournalBusyError(rootError)) {
|
|
if (!res.headersSent) {
|
|
res.set("Retry-After", "2");
|
|
res.status(503).json({ error: rootError.message, code: rootError.code });
|
|
} else {
|
|
res.end();
|
|
}
|
|
return;
|
|
}
|
|
|
|
const reportableError = sanitizeSecretSensitiveError(req, rootError);
|
|
attachErrorContext(
|
|
req,
|
|
res,
|
|
isSecretSensitiveHttpRequest(req.method, req.originalUrl)
|
|
? { message: reportableError.message, name: reportableError.name }
|
|
: err instanceof Error
|
|
? { message: err.message, stack: err.stack, name: err.name }
|
|
: {
|
|
message: String(err),
|
|
raw: err,
|
|
stack: rootError.stack,
|
|
name: rootError.name,
|
|
},
|
|
reportableError,
|
|
);
|
|
|
|
reportCrash(reportableError, req);
|
|
|
|
res.status(500).json({
|
|
error: "Internal server error",
|
|
...(shouldExposeTrustedCloudTenantImportError(req)
|
|
? { message: rootError.message }
|
|
: {}),
|
|
});
|
|
}
|
|
|
|
function shouldExposeTrustedCloudTenantImportError(req: Request) {
|
|
return (
|
|
req.actor?.source === "cloud_tenant" &&
|
|
req.method === "POST" &&
|
|
req.originalUrl.split("?")[0] === COMPANY_IMPORT_API_PATH
|
|
);
|
|
}
|