From 9cc979116e4d4f49d3a2504b32d14189fad41728 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 13 Oct 2026 18:26:03 -0700 Subject: [PATCH] fix(server): stop reporting queries that outlive the database shutdown (#16243) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server reports unexpected route errors to Sentry through the shared error handler. > - An orderly shutdown drains the HTTP listener, and then it ends the database pool. > - `server.close()` completes when the sockets close. A route handler whose client disconnected, or whose socket the drain timeout closed, can still run. > - The next query of that handler fails with postgres.js `CONNECTION_ENDED`, because this process ended the pool. The error handler reports this as a 500 crash. > - This pull request marks the pool as closing just before it ends. The error handler then skips the error sinks for that one error code during shutdown. > - Public MCP calls run in their own internal Express app, so the flag is also shared with that app. > - The benefit is that a normal restart does not create false Sentry crash reports. Real errors are still reported. ## Linked Issues or Issue Description No public GitHub issue exists. This is the bug description. **What happened?** A staging server restarted. During the restart, `GET /api/companies/:companyId/heartbeat-runs` failed in the per-row run privacy check (`canActorReadHeartbeatRun`). The error was `Failed query: select "id" from "issues" ...`. Its cause was `write CONNECTION_ENDED :5432`. postgres.js raises this code only for a query that starts after the process called `sql.end()`. The error handler sent the error to Sentry as a crash. **Expected behavior** A query that runs after the shutdown ends the pool is an expected part of the shutdown. The server must not report it as a crash. If the response is still open, the server answers `503`. The server must continue to report all other errors. **Steps to reproduce** 1. Start the server, and send a slow request (for example, a large heartbeat run list). 2. Close the client connection, or let the 5 second drain timeout close it, while the handler still runs. 3. Send SIGTERM. The shutdown drains HTTP and ends the pool. 4. The handler's next query fails with `CONNECTION_ENDED`. Before this change, the error goes to Sentry. ## What Changed - `server/src/database-retry.ts`: Add `isEndedDbPoolError`. It finds the postgres.js `CONNECTION_ENDED` code anywhere in the error `cause` chain. `isTransientDbConnectionError` now uses the same chain walk. Its behavior does not change. - `server/src/index.ts`: The orderly shutdown sets `app.locals.paperclipDatabaseClosing = true` immediately before it ends the database pools. - `server/src/middleware/error-handler.ts`: When the flag is set and the error is `CONNECTION_ENDED`, the handler answers `503 { error: "Server is shutting down" }`, or it ends the response if the headers are already sent. It does not call Sentry or crash telemetry. This follows the existing client-abort (`ECONNRESET`) case. - `server/src/services/public-mcp/capabilities.ts` and `server/src/app.ts`: `createMcpApiDispatch` takes the main app's locals. Each per-call dispatch app reads `paperclipDatabaseClosing` through a live getter, so a public MCP call that outlives the pool is handled the same way (Greptile review finding). - `server/src/__tests__/error-handler.test.ts`: Add tests for the 503 path, the headers-sent path, `CONNECTION_ENDED` outside shutdown (still reported), and `CONNECTION_CLOSED` during shutdown (still reported). - `server/src/__tests__/public-mcp-dispatch-shutdown.test.ts`: Send an ended-pool error through the real MCP dispatch, both during shutdown (503, not reported) and outside shutdown (reported). ## Verification - `cd server && vitest run src/__tests__/public-mcp-dispatch-shutdown.test.ts src/__tests__/error-handler.test.ts src/__tests__/cloud-tenant-transient-db-retry.test.ts`: 25 tests pass. - With only the error-handler change reverted, the two new error-handler shutdown tests fail. With only the dispatch change reverted, the MCP shutdown test fails. The negative-control tests pass in both cases. - I could not run the full `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` locally, or `public-mcp.test.ts`, which needs built workspace packages. The sandbox has a 4 GB memory limit. CI runs the full gates. ## Risks - Low risk. The skip applies only to `CONNECTION_ENDED`, and only after the shutdown starts to end the pools. postgres.js raises that code only for a pool that this process ended. Network and database failures (`CONNECTION_CLOSED`, `CONNECTION_DESTROYED`, and others) are still reported. - `CONNECTION_ENDED` outside shutdown (for example, a dedicated client used after it ended) is still reported as a crash. - The MCP dispatch app gets only the shutdown flag from the main app's locals. It does not get any other locals. - No schema, API contract, or UI change. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (repository inspection, shell, test execution) and the Sentry and GitHub connectors. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- server/src/__tests__/error-handler.test.ts | 61 +++++++++++++++++++ .../public-mcp-dispatch-shutdown.test.ts | 51 ++++++++++++++++ server/src/app.ts | 2 +- server/src/database-retry.ts | 17 +++++- server/src/index.ts | 7 ++- server/src/middleware/error-handler.ts | 22 +++++++ .../src/services/public-mcp/capabilities.ts | 9 ++- 7 files changed, 164 insertions(+), 5 deletions(-) create mode 100644 server/src/__tests__/public-mcp-dispatch-shutdown.test.ts diff --git a/server/src/__tests__/error-handler.test.ts b/server/src/__tests__/error-handler.test.ts index 934cf7aa7c0..8ef380af6f9 100644 --- a/server/src/__tests__/error-handler.test.ts +++ b/server/src/__tests__/error-handler.test.ts @@ -83,6 +83,67 @@ describe("errorHandler", () => { expect(telemetryMocks.trackErrorHandlerCrash).not.toHaveBeenCalled(); }); + function endedPoolQueryError() { + // Drizzle wraps the driver error; postgres.js uses CONNECTION_ENDED for a + // query issued after this process called `sql.end()`. + const driverError = Object.assign(new Error("write CONNECTION_ENDED db.example:5432"), { + code: "CONNECTION_ENDED", + }); + return new Error('Failed query: select "id" from "issues"', { cause: driverError }); + } + + it("answers 503 without reporting a query that outlived the database shutdown", () => { + const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request; + const res = makeRes() as any; + const next = vi.fn() as unknown as NextFunction; + + errorHandler(endedPoolQueryError(), req, res, next); + + expect(res.status).toHaveBeenCalledWith(503); + expect(res.json).toHaveBeenCalledWith({ error: "Server is shutting down" }); + expect(res.err).toBeUndefined(); + expect(captureExceptionMock).not.toHaveBeenCalled(); + expect(telemetryMocks.trackErrorHandlerCrash).not.toHaveBeenCalled(); + }); + + it("only ends the response when headers were already sent during database shutdown", () => { + const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request; + const res = { ...makeRes(), end: vi.fn(), headersSent: true } as any; + const next = vi.fn() as unknown as NextFunction; + + errorHandler(endedPoolQueryError(), req, res, next); + + expect(res.status).not.toHaveBeenCalled(); + expect(res.end).toHaveBeenCalled(); + expect(captureExceptionMock).not.toHaveBeenCalled(); + }); + + it("still reports an ended-pool query error outside database shutdown", () => { + const req = { ...makeReq(), app: { locals: {} } } as unknown as Request; + const res = makeRes() as any; + const next = vi.fn() as unknown as NextFunction; + const err = endedPoolQueryError(); + + errorHandler(err, req, res, next); + + expect(res.status).toHaveBeenCalledWith(500); + expect(captureExceptionMock).toHaveBeenCalledWith(err); + }); + + it("still reports other connection failures during database shutdown", () => { + const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request; + const res = makeRes() as any; + const next = vi.fn() as unknown as NextFunction; + const err = new Error("Failed query", { + cause: Object.assign(new Error("write CONNECTION_CLOSED"), { code: "CONNECTION_CLOSED" }), + }); + + errorHandler(err, req, res, next); + + expect(res.status).toHaveBeenCalledWith(500); + expect(captureExceptionMock).toHaveBeenCalledWith(err); + }); + it("exposes raw 500 messages for trusted Cloud tenant imports", () => { const req = { ...makeReq(), diff --git a/server/src/__tests__/public-mcp-dispatch-shutdown.test.ts b/server/src/__tests__/public-mcp-dispatch-shutdown.test.ts new file mode 100644 index 00000000000..7540922f3e4 --- /dev/null +++ b/server/src/__tests__/public-mcp-dispatch-shutdown.test.ts @@ -0,0 +1,51 @@ +import express from "express"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { createMcpApiDispatch } from "../services/public-mcp/capabilities.js"; +import type { McpPrincipal } from "../services/public-mcp/oauth.js"; + +const captureExceptionMock = vi.hoisted(() => vi.fn()); +const trackErrorHandlerCrashMock = vi.hoisted(() => vi.fn()); + +vi.mock("../sentry.js", () => ({ captureException: captureExceptionMock })); +vi.mock("../telemetry.js", () => ({ getTelemetryClient: () => ({}) })); +vi.mock("@paperclipai/shared/telemetry", () => ({ + trackErrorHandlerCrash: trackErrorHandlerCrashMock, +})); + +const principal = { actor: { type: "board", userId: "user-1" } } as unknown as McpPrincipal; + +// The internal dispatch runs each public MCP call in its own Express app, so +// the shutdown flag set on the main app must reach that app's error handler. +function endedPoolApi() { + const api = express.Router(); + api.get("/tasks", () => { + throw new Error('Failed query: select "id" from "issues"', { + cause: Object.assign(new Error("write CONNECTION_ENDED db.example:5432"), { code: "CONNECTION_ENDED" }), + }); + }); + return api; +} + +describe("public MCP dispatch during database shutdown", () => { + beforeEach(() => { + captureExceptionMock.mockReset(); + trackErrorHandlerCrashMock.mockReset(); + }); + + it("does not report a call that outlived the database pool", async () => { + const parentLocals: Record = {}; + const dispatch = createMcpApiDispatch(endedPoolApi(), parentLocals); + parentLocals.paperclipDatabaseClosing = true; + + await expect(dispatch(principal, "GET", "/tasks", undefined)).rejects.toMatchObject({ status: 503 }); + expect(captureExceptionMock).not.toHaveBeenCalled(); + expect(trackErrorHandlerCrashMock).not.toHaveBeenCalled(); + }); + + it("still reports an ended-pool error outside database shutdown", async () => { + const dispatch = createMcpApiDispatch(endedPoolApi(), {}); + + await expect(dispatch(principal, "GET", "/tasks", undefined)).rejects.toMatchObject({ status: 500 }); + expect(captureExceptionMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/server/src/app.ts b/server/src/app.ts index bd8685fe49d..b320dea6913 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -1061,7 +1061,7 @@ export async function createApp( let publicMcpEvents: PublicMcpEvents | null = null; let dotMcpEvents: PublicMcpEvents | null = null; if (publicMcpOAuth) { - const dispatch = createMcpApiDispatch(api); + const dispatch = createMcpApiDispatch(api, app.locals); publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, { isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(), }); diff --git a/server/src/database-retry.ts b/server/src/database-retry.ts index d56429f7140..17cc9448510 100644 --- a/server/src/database-retry.ts +++ b/server/src/database-retry.ts @@ -6,14 +6,27 @@ const transientDbConnectionCodes = new Set([ "CONNECTION_DESTROYED", ]); -export function isTransientDbConnectionError(error: unknown): boolean { +function hasDbConnectionErrorCode(error: unknown, matches: (code: string) => boolean): boolean { for (let current: unknown = error; current instanceof Error; current = current.cause) { const code = (current as { code?: unknown }).code; - if (typeof code === "string" && transientDbConnectionCodes.has(code)) return true; + if (typeof code === "string" && matches(code)) return true; } return false; } +export function isTransientDbConnectionError(error: unknown): boolean { + return hasDbConnectionErrorCode(error, (code) => transientDbConnectionCodes.has(code)); +} + +/** + * postgres.js rejects every query issued after this process called + * `sql.end()` with `CONNECTION_ENDED`. Unlike a dropped socket, it never comes + * from the database or the network: the pool was ended locally. + */ +export function isEndedDbPoolError(error: unknown): boolean { + return hasDbConnectionErrorCode(error, (code) => code === "CONNECTION_ENDED"); +} + /** * Retry only an explicitly idempotent operation, such as a known read-only * lookup or receipt-protected synchronization. Neither a SQL prefix nor a diff --git a/server/src/index.ts b/server/src/index.ts index ab0d225ad95..be6b6bd701f 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -2062,7 +2062,12 @@ async function startServerWithDatabaseTeardown( closeHttpListener: () => closeHttpListenerForShutdown({ server, signal, log: logger }), drainPendingRunFailureReports: waitForPendingRunFailureReports, - closeDatabase: closeDatabaseClients, + // Mark the pool as ending before `sql.end()` rejects the first query, + // so the error handler recognizes a request that outlived the drain. + closeDatabase: () => { + (app.locals as { paperclipDatabaseClosing?: boolean }).paperclipDatabaseClosing = true; + return closeDatabaseClients(); + }, stopEmbeddedPostgres, shutdownInstrumentation, shutdownSentry, diff --git a/server/src/middleware/error-handler.ts b/server/src/middleware/error-handler.ts index f138c06e4d6..4300267c884 100644 --- a/server/src/middleware/error-handler.ts +++ b/server/src/middleware/error-handler.ts @@ -15,6 +15,7 @@ import { } from "./redact-sensitive.js"; import { recordResponsibleUserDenialOnActiveRun } from "../services/responsible-user-denial-run-outcomes.js"; import { isExpectedMcpConnectionFailure } from "../services/mcp-connection-failure.js"; +import { isEndedDbPoolError } from "../database-retry.js"; export interface ErrorContext { error: { @@ -93,6 +94,12 @@ function reportCrash(error: Error, reportToSentry = true): void { if (reportToSentry) captureException(error); } +/** Set by the orderly shutdown just before it ends the database pools. */ +function isDatabaseClosing(req: Request): boolean { + const locals = req.app?.locals as { paperclipDatabaseClosing?: boolean } | undefined; + return locals?.paperclipDatabaseClosing === true; +} + function getPaperclipDb(req: Request): Db | null { const locals = req.app?.locals as { paperclipDb?: Db; db?: Db } | undefined; return locals?.paperclipDb ?? locals?.db ?? null; @@ -266,6 +273,21 @@ export function errorHandler( return; } + // Shutdown drains the HTTP listener before it ends the pool, but a handler + // whose client already disconnected (or whose socket the drain timeout + // closed) keeps running. Its next query then fails with CONNECTION_ENDED + // against the pool this process deliberately ended. That is shutdown, not a + // server fault, so skip the error sinks. Any other error during shutdown is + // still reported. + if (isDatabaseClosing(req) && isEndedDbPoolError(rootError)) { + if (!res.headersSent) { + res.status(503).json({ error: "Server is shutting down" }); + } else { + res.end(); + } + return; + } + const reportableError = sanitizeSecretSensitiveError(req, rootError); attachErrorContext( req, diff --git a/server/src/services/public-mcp/capabilities.ts b/server/src/services/public-mcp/capabilities.ts index ca0eaf27cc7..a56683cff4e 100644 --- a/server/src/services/public-mcp/capabilities.ts +++ b/server/src/services/public-mcp/capabilities.ts @@ -19,10 +19,17 @@ import { expandedMcpCapabilities } from "./expanded-capabilities.js"; /** Internal HTTP dispatch reuses domain authorization, validation, audit and scheduling. * Neither an arbitrary path nor an actor is accepted from the MCP client. * Each invocation has its own Express request and verified actor. + * `parentLocals` is the main app's locals; the orderly-shutdown flag is read + * through it so a call that outlives the database pool is recognized. */ -export function createMcpApiDispatch(api: Router): ApiDispatch { +export function createMcpApiDispatch(api: Router, parentLocals?: Record): ApiDispatch { return async (principal, method, path, body) => { const app = express(); + if (parentLocals) { + Object.defineProperty(app.locals, "paperclipDatabaseClosing", { + get: () => parentLocals.paperclipDatabaseClosing === true, + }); + } // light-my-request adapts the parent prototypes of app.request/response. // Clone Express's shared parents so injection cannot alter real HTTP traffic. for (const target of [app.request, app.response]) {