Commit Graph
1 Commits
Author SHA1 Message Date
Devin Foley 6c7c0fd1f2 fix(db): stop the postgres driver from crashing the process on a write/close race (#12227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server and its test suites talk to PostgreSQL through the
`postgres` (postgres.js) driver, and tests routinely tear their
databases down while connections still carry traffic
> - The driver flushes small buffered frames from a `setImmediate`, and
that deferred flush calls `socket.write()` without checking that the
socket still exists; a reserved connection whose backend died keeps
accepting queries, so the flush can fire with a null socket
> - The resulting `TypeError` escapes from a timer callback with no
try/catch above it, crashing the process — in CI this fails suites whose
tests all passed ("Vitest caught 1 unhandled error"), and the same crash
is reported against the driver in the wild after ECONNRESET
> - The latest driver release (3.4.9) still has the bug, so this pull
request adds a pnpm patch guarding the flush and normalizing timer state
on close, plus a deterministic regression test
> - The benefit is CI that no longer fails randomly on a teardown race,
and production processes that survive a database connection dying at the
wrong moment

## Linked Issues or Issue Description

No public issue exists; the underlying problem follows the bug-report
template.

**What happened?**

CI jobs fail with all tests passing: vitest reports `Vitest caught 1
unhandled error during the test run` with `TypeError: Cannot read
properties of null (reading 'write')` at `postgres/src/connection.js`
`Immediate.nextWrite`. The attribution points at whichever test file
happened to be running (e.g. `native-codex-runner.integration.test.ts`),
because the throw comes from a process-level timer callback, not from a
test. The identical crash is reported against the upstream driver by
other projects after `ECONNRESET` (e.g. immich-app/immich#25098).

**Expected behavior**

A connection dying between a write being scheduled and its deferred
flush must settle the affected queries through the driver's normal
connection-error path, never throw from a bare timer callback.

**Steps to reproduce**

Run the new `packages/db/src/postgres-driver-teardown.test.ts` with the
patch removed: reserve a connection (`sql.reserve()` — the same surface
`sql.begin()` uses), destroy the backend socket, wait for the client to
process the close, then issue one query on the reserved connection. The
deferred flush fires one tick later with `socket === null` and crashes
the process with exactly the CI signature.

**Paperclip version or commit**

master `198fc8b28`, `postgres@3.4.9` (latest release; bug still present
on the driver's master branch).

## What Changed

- `patches/postgres@3.4.9.patch` (new, wired via
`pnpm.patchedDependencies`): `nextWrite` returns without writing when
`socket === null`, dropping the buffered bytes — the close path has
already settled every in-flight query, so those bytes have nowhere to
go. The `closed()` and `terminate()` handlers additionally reset
`nextWriteTimer`/`chunk` after `clearImmediate`, so a stale cleared
handle cannot silently block a future reconnect's first flush. All three
shipped builds (`src`, `cjs`, `cf`) get the identical change.
- `packages/db/src/postgres-driver-teardown.test.ts` (new):
deterministic reproduction against a minimal in-process fake wire server
(startup auth + an empty result for the `fetch_types` bootstrap).
Asserts the late query settles with `CONNECTION_DESTROYED` through
`sql.end()` instead of crashing the process.

## Verification

- The regression test fails against unpatched `postgres@3.4.9` with the
exact CI signature (verified by running the same scenario against an
unpatched checkout) and passes with the patch.
- Full `packages/db` suite: 27 files / 101 tests pass.
- Spot-checked server suites that exercise the database through the
patched driver.

## Risks

- Low. The behavioral change activates only in a state that previously
crashed the process (write flush with no socket). Dropping the buffered
bytes matches what the connection's close path already promised callers:
every in-flight query has been settled with a connection error.
- The timer/chunk reset in `closed()`/`terminate()` prevents a
theoretical stale-handle hang after reconnect; on the normal path both
were already reset by `nextWrite`.
- The patch pins to `postgres@3.4.9`; a future driver upgrade will
surface the patch for re-evaluation (pnpm fails loudly on version
mismatch), and the guard can be dropped if the fix lands upstream.

## Model Used

Claude Fable 5 (`claude-fable-5`, Anthropic) via Claude Code — agentic
coding session with tool use (driver source analysis, wire-protocol fake
server, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-26 10:19:19 -07:00