fix(server): stop reporting queries that outlive the database shutdown (#16243)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server reports unexpected route errors to Sentry through the
shared error handler.
> - An orderly shutdown drains the HTTP listener, and then it ends the
database pool.
> - `server.close()` completes when the sockets close. A route handler
whose client disconnected, or whose socket the drain timeout closed, can
still run.
> - The next query of that handler fails with postgres.js
`CONNECTION_ENDED`, because this process ended the pool. The error
handler reports this as a 500 crash.
> - This pull request marks the pool as closing just before it ends. The
error handler then skips the error sinks for that one error code during
shutdown.
> - Public MCP calls run in their own internal Express app, so the flag
is also shared with that app.
> - The benefit is that a normal restart does not create false Sentry
crash reports. Real errors are still reported.

## Linked Issues or Issue Description

No public GitHub issue exists. This is the bug description.

**What happened?**

A staging server restarted. During the restart, `GET
/api/companies/:companyId/heartbeat-runs` failed in the per-row run
privacy check (`canActorReadHeartbeatRun`). The error was `Failed query:
select "id" from "issues" ...`. Its cause was `write CONNECTION_ENDED
<host>:5432`. postgres.js raises this code only for a query that starts
after the process called `sql.end()`. The error handler sent the error
to Sentry as a crash.

**Expected behavior**

A query that runs after the shutdown ends the pool is an expected part
of the shutdown. The server must not report it as a crash. If the
response is still open, the server answers `503`. The server must
continue to report all other errors.

**Steps to reproduce**

1. Start the server, and send a slow request (for example, a large
heartbeat run list).
2. Close the client connection, or let the 5 second drain timeout close
it, while the handler still runs.
3. Send SIGTERM. The shutdown drains HTTP and ends the pool.
4. The handler's next query fails with `CONNECTION_ENDED`. Before this
change, the error goes to Sentry.

## What Changed

- `server/src/database-retry.ts`: Add `isEndedDbPoolError`. It finds the
postgres.js `CONNECTION_ENDED` code anywhere in the error `cause` chain.
`isTransientDbConnectionError` now uses the same chain walk. Its
behavior does not change.
- `server/src/index.ts`: The orderly shutdown sets
`app.locals.paperclipDatabaseClosing = true` immediately before it ends
the database pools.
- `server/src/middleware/error-handler.ts`: When the flag is set and the
error is `CONNECTION_ENDED`, the handler answers `503 { error: "Server
is shutting down" }`, or it ends the response if the headers are already
sent. It does not call Sentry or crash telemetry. This follows the
existing client-abort (`ECONNRESET`) case.
- `server/src/services/public-mcp/capabilities.ts` and
`server/src/app.ts`: `createMcpApiDispatch` takes the main app's locals.
Each per-call dispatch app reads `paperclipDatabaseClosing` through a
live getter, so a public MCP call that outlives the pool is handled the
same way (Greptile review finding).
- `server/src/__tests__/error-handler.test.ts`: Add tests for the 503
path, the headers-sent path, `CONNECTION_ENDED` outside shutdown (still
reported), and `CONNECTION_CLOSED` during shutdown (still reported).
- `server/src/__tests__/public-mcp-dispatch-shutdown.test.ts`: Send an
ended-pool error through the real MCP dispatch, both during shutdown
(503, not reported) and outside shutdown (reported).

## Verification

- `cd server && vitest run
src/__tests__/public-mcp-dispatch-shutdown.test.ts
src/__tests__/error-handler.test.ts
src/__tests__/cloud-tenant-transient-db-retry.test.ts`: 25 tests pass.
- With only the error-handler change reverted, the two new error-handler
shutdown tests fail. With only the dispatch change reverted, the MCP
shutdown test fails. The negative-control tests pass in both cases.
- I could not run the full `pnpm -r typecheck`, `pnpm test:run`, and
`pnpm build` locally, or `public-mcp.test.ts`, which needs built
workspace packages. The sandbox has a 4 GB memory limit. CI runs the
full gates.

## Risks

- Low risk. The skip applies only to `CONNECTION_ENDED`, and only after
the shutdown starts to end the pools. postgres.js raises that code only
for a pool that this process ended. Network and database failures
(`CONNECTION_CLOSED`, `CONNECTION_DESTROYED`, and others) are still
reported.
- `CONNECTION_ENDED` outside shutdown (for example, a dedicated client
used after it ended) is still reported as a crash.
- The MCP dispatch app gets only the shutdown flag from the main app's
locals. It does not get any other locals.
- No schema, API contract, or UI change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool
use (repository inspection, shell, test execution) and the Sentry and
GitHub connectors.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-10-13 18:26:03 -07:00
1 parent f987f95d6a
commit 9cc979116e
7 files changed
+164 -5

No files matched your search

@@ -83,6 +83,67 @@ describe("errorHandler", () => {
expect(telemetryMocks.trackErrorHandlerCrash).not.toHaveBeenCalled();
});
function endedPoolQueryError() {
// Drizzle wraps the driver error; postgres.js uses CONNECTION_ENDED for a
// query issued after this process called `sql.end()`.
const driverError = Object.assign(new Error("write CONNECTION_ENDED db.example:5432"), {
code: "CONNECTION_ENDED",
});
return new Error('Failed query: select "id" from "issues"', { cause: driverError });
}
it("answers 503 without reporting a query that outlived the database shutdown", () => {
const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request;
const res = makeRes() as any;
const next = vi.fn() as unknown as NextFunction;
errorHandler(endedPoolQueryError(), req, res, next);
expect(res.status).toHaveBeenCalledWith(503);
expect(res.json).toHaveBeenCalledWith({ error: "Server is shutting down" });
expect(res.err).toBeUndefined();
expect(captureExceptionMock).not.toHaveBeenCalled();
expect(telemetryMocks.trackErrorHandlerCrash).not.toHaveBeenCalled();
});
it("only ends the response when headers were already sent during database shutdown", () => {
const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request;
const res = { ...makeRes(), end: vi.fn(), headersSent: true } as any;
const next = vi.fn() as unknown as NextFunction;
errorHandler(endedPoolQueryError(), req, res, next);
expect(res.status).not.toHaveBeenCalled();
expect(res.end).toHaveBeenCalled();
expect(captureExceptionMock).not.toHaveBeenCalled();
});
it("still reports an ended-pool query error outside database shutdown", () => {
const req = { ...makeReq(), app: { locals: {} } } as unknown as Request;
const res = makeRes() as any;
const next = vi.fn() as unknown as NextFunction;
const err = endedPoolQueryError();
errorHandler(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(500);
expect(captureExceptionMock).toHaveBeenCalledWith(err);
});
it("still reports other connection failures during database shutdown", () => {
const req = { ...makeReq(), app: { locals: { paperclipDatabaseClosing: true } } } as unknown as Request;
const res = makeRes() as any;
const next = vi.fn() as unknown as NextFunction;
const err = new Error("Failed query", {
cause: Object.assign(new Error("write CONNECTION_CLOSED"), { code: "CONNECTION_CLOSED" }),
});
errorHandler(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(500);
expect(captureExceptionMock).toHaveBeenCalledWith(err);
});
it("exposes raw 500 messages for trusted Cloud tenant imports", () => {
const req = {
...makeReq(),
@@ -0,0 +1,51 @@
import express from "express";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { createMcpApiDispatch } from "../services/public-mcp/capabilities.js";
import type { McpPrincipal } from "../services/public-mcp/oauth.js";
const captureExceptionMock = vi.hoisted(() => vi.fn());
const trackErrorHandlerCrashMock = vi.hoisted(() => vi.fn());
vi.mock("../sentry.js", () => ({ captureException: captureExceptionMock }));
vi.mock("../telemetry.js", () => ({ getTelemetryClient: () => ({}) }));
vi.mock("@paperclipai/shared/telemetry", () => ({
trackErrorHandlerCrash: trackErrorHandlerCrashMock,
}));
const principal = { actor: { type: "board", userId: "user-1" } } as unknown as McpPrincipal;
// The internal dispatch runs each public MCP call in its own Express app, so
// the shutdown flag set on the main app must reach that app's error handler.
function endedPoolApi() {
const api = express.Router();
api.get("/tasks", () => {
throw new Error('Failed query: select "id" from "issues"', {
cause: Object.assign(new Error("write CONNECTION_ENDED db.example:5432"), { code: "CONNECTION_ENDED" }),
});
});
return api;
}
describe("public MCP dispatch during database shutdown", () => {
beforeEach(() => {
captureExceptionMock.mockReset();
trackErrorHandlerCrashMock.mockReset();
});
it("does not report a call that outlived the database pool", async () => {
const parentLocals: Record<string, unknown> = {};
const dispatch = createMcpApiDispatch(endedPoolApi(), parentLocals);
parentLocals.paperclipDatabaseClosing = true;
await expect(dispatch(principal, "GET", "/tasks", undefined)).rejects.toMatchObject({ status: 503 });
expect(captureExceptionMock).not.toHaveBeenCalled();
expect(trackErrorHandlerCrashMock).not.toHaveBeenCalled();
});
it("still reports an ended-pool error outside database shutdown", async () => {
const dispatch = createMcpApiDispatch(endedPoolApi(), {});
await expect(dispatch(principal, "GET", "/tasks", undefined)).rejects.toMatchObject({ status: 500 });
expect(captureExceptionMock).toHaveBeenCalledTimes(1);
});
});
+1 -1
View File
@@ -1061,7 +1061,7 @@ export async function createApp(
let publicMcpEvents: PublicMcpEvents | null = null;
let dotMcpEvents: PublicMcpEvents | null = null;
if (publicMcpOAuth) {
const dispatch = createMcpApiDispatch(api);
const dispatch = createMcpApiDispatch(api, app.locals);
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
});
+15 -2
View File
@@ -6,14 +6,27 @@ const transientDbConnectionCodes = new Set([
"CONNECTION_DESTROYED",
]);
export function isTransientDbConnectionError(error: unknown): boolean {
function hasDbConnectionErrorCode(error: unknown, matches: (code: string) => boolean): boolean {
for (let current: unknown = error; current instanceof Error; current = current.cause) {
const code = (current as { code?: unknown }).code;
if (typeof code === "string" && transientDbConnectionCodes.has(code)) return true;
if (typeof code === "string" && matches(code)) return true;
}
return false;
}
export function isTransientDbConnectionError(error: unknown): boolean {
return hasDbConnectionErrorCode(error, (code) => transientDbConnectionCodes.has(code));
}
/**
* postgres.js rejects every query issued after this process called
* `sql.end()` with `CONNECTION_ENDED`. Unlike a dropped socket, it never comes
* from the database or the network: the pool was ended locally.
*/
export function isEndedDbPoolError(error: unknown): boolean {
return hasDbConnectionErrorCode(error, (code) => code === "CONNECTION_ENDED");
}
/**
* Retry only an explicitly idempotent operation, such as a known read-only
* lookup or receipt-protected synchronization. Neither a SQL prefix nor a
+6 -1
View File
@@ -2062,7 +2062,12 @@ async function startServerWithDatabaseTeardown(
closeHttpListener: () =>
closeHttpListenerForShutdown({ server, signal, log: logger }),
drainPendingRunFailureReports: waitForPendingRunFailureReports,
closeDatabase: closeDatabaseClients,
// Mark the pool as ending before `sql.end()` rejects the first query,
// so the error handler recognizes a request that outlived the drain.
closeDatabase: () => {
(app.locals as { paperclipDatabaseClosing?: boolean }).paperclipDatabaseClosing = true;
return closeDatabaseClients();
},
stopEmbeddedPostgres,
shutdownInstrumentation,
shutdownSentry,
+22
View File
@@ -15,6 +15,7 @@ import {
} from "./redact-sensitive.js";
import { recordResponsibleUserDenialOnActiveRun } from "../services/responsible-user-denial-run-outcomes.js";
import { isExpectedMcpConnectionFailure } from "../services/mcp-connection-failure.js";
import { isEndedDbPoolError } from "../database-retry.js";
export interface ErrorContext {
error: {
@@ -93,6 +94,12 @@ function reportCrash(error: Error, reportToSentry = true): void {
if (reportToSentry) captureException(error);
}
/** Set by the orderly shutdown just before it ends the database pools. */
function isDatabaseClosing(req: Request): boolean {
const locals = req.app?.locals as { paperclipDatabaseClosing?: boolean } | undefined;
return locals?.paperclipDatabaseClosing === true;
}
function getPaperclipDb(req: Request): Db | null {
const locals = req.app?.locals as { paperclipDb?: Db; db?: Db } | undefined;
return locals?.paperclipDb ?? locals?.db ?? null;
@@ -266,6 +273,21 @@ export function errorHandler(
return;
}
// Shutdown drains the HTTP listener before it ends the pool, but a handler
// whose client already disconnected (or whose socket the drain timeout
// closed) keeps running. Its next query then fails with CONNECTION_ENDED
// against the pool this process deliberately ended. That is shutdown, not a
// server fault, so skip the error sinks. Any other error during shutdown is
// still reported.
if (isDatabaseClosing(req) && isEndedDbPoolError(rootError)) {
if (!res.headersSent) {
res.status(503).json({ error: "Server is shutting down" });
} else {
res.end();
}
return;
}
const reportableError = sanitizeSecretSensitiveError(req, rootError);
attachErrorContext(
req,
@@ -19,10 +19,17 @@ import { expandedMcpCapabilities } from "./expanded-capabilities.js";
/** Internal HTTP dispatch reuses domain authorization, validation, audit and scheduling.
* Neither an arbitrary path nor an actor is accepted from the MCP client.
* Each invocation has its own Express request and verified actor.
* `parentLocals` is the main app's locals; the orderly-shutdown flag is read
* through it so a call that outlives the database pool is recognized.
*/
export function createMcpApiDispatch(api: Router): ApiDispatch {
export function createMcpApiDispatch(api: Router, parentLocals?: Record<string, unknown>): ApiDispatch {
return async (principal, method, path, body) => {
const app = express();
if (parentLocals) {
Object.defineProperty(app.locals, "paperclipDatabaseClosing", {
get: () => parentLocals.paperclipDatabaseClosing === true,
});
}
// light-my-request adapts the parent prototypes of app.request/response.
// Clone Express's shared parents so injection cannot alter real HTTP traffic.
for (const target of [app.request, app.response]) {