fix(db): release a postgres.js connection's receive buffer once it is fully read (#21432)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - A hosted server keeps about 0.5 GB of `external` memory (Buffers)
for its whole life, with 340 open database connections
> - postgres.js keeps a zero-length view of the buffer of each
connection's last read. The view keeps the whole buffer alive until that
connection reads again
> - An idle connection therefore pins the socket chunk or the whole
buffer of its largest recent message, which can be one MB or more
> - This pull request replaces a fully read buffer with an empty one in
the driver patch
> - The benefit is that idle connections hold no receive buffers. At 340
connections that each read a 1 MB row, external memory drops from about
320 MB to 0

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server holds 455 to 520 MB of `external` memory for
its whole life. It has about 340 open database connections. Each idle
postgres.js connection keeps the buffer of its last read.

**Steps to reproduce**
1. Open 80 or 340 pool connections.
2. Read a 1 MB row on each connection.
3. Drop the results, let the connections go idle, and force garbage
collection.
4. Measure `process.memoryUsage().external`. Each idle connection still
holds about 0.95 MB.

**Expected behavior**
An idle connection holds no buffer from a result that it has completely
read.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18.

## What Changed

- `patches/postgres@3.4.9.patch`: in `data()` of `src/connection.js` and
`cjs/src/connection.js`, the driver now replaces the receive buffer with
an empty buffer when it has handled every message in it. A partly read
buffer is not changed. The other hunks of the patch are unchanged. I
regenerated the patch with `pnpm patch postgres@3.4.9`, edited the two
files, and ran `pnpm patch-commit`. I did not commit the lockfile change
from that command.
- `packages/db/src/postgres-receive-buffer.test.ts` and
`packages/db/src/__fixtures__/postgres-receive-buffer.mjs`: a regression
test for ESM and CommonJS on real PostgreSQL.
- It checks that results are reassembled correctly for a 3 MB row, for
161 row sizes around one 64 KB socket chunk (to land message boundaries
at the end of a chunk), and for 20,000 small rows that arrive many to a
chunk.
- It records a weak reference to each large receive buffer and checks
that none of them stays alive after the connection becomes idle.
- `packages/db/scripts/benchmark-postgres-receive-buffer.ts`: one
command that reports heap, external and rss per connection, before and
after large rows.
- `doc/DATABASE.md`: documents the new patch behavior, the test and the
benchmark.

## Verification

- Red/green: with the new line removed from the installed driver, the
fixture fails for ESM and CommonJS ("an idle connection must not keep
the receive buffer of its last result"). With the line in place it
passes, repeated twice for each format.
- `pnpm --filter @paperclipai/db exec vitest run
src/postgres-receive-buffer.test.ts
src/postgres-connection-recovery.test.ts`: 20 passed.
- `pnpm --filter @paperclipai/db exec vitest run src/backup-lib.test.ts
src/backup-lib-binary.test.ts src/client-disconnect-replay.test.ts
src/client-teardown-registry.test.ts src/work-signals.test.ts`: 36
passed. In 1 of 4 runs, the JavaScript backup test (COPY and cursor
reads) timed out on a machine with load average 20 to 34. It took 39 s
and 25 s with the change and 21 s without it in passing runs. Read
throughput is the same with and without the change: a 200,000-row select
plus a COPY TO STDOUT has a median of 319 and 348 ms with it, and 329 ms
without it.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/prepared-statements-driver.test.ts
src/__tests__/prepared-statements.test.ts`: 5 passed.
- `pnpm --filter @paperclipai/db typecheck`: passed.
- Benchmark: `node --expose-gc --import tsx
packages/db/scripts/benchmark-postgres-receive-buffer.ts` with a 1 MB
row on each connection, numbers after garbage collection:

| Connections | External per idle connection, before | After | External
held in total, before | After |
| --- | --- | --- | --- | --- |
| 80 | 917 KB | 1 KB | 72 MB | 0 MB |
| 340 | 960 KB | 0 KB | 319 MB | 0 MB |

The rss per connection also went down (80 connections: 1,555 to 1,375
KB; 340 connections: 1,308 to 714 KB). The allocator keeps some freed
memory, so rss changes less than external memory.

## Risks

- Low risk. The change runs only when the buffer has no unread bytes. A
message split across chunks keeps the same reassembly path. The test
covers split messages, exact chunk boundaries and many messages in one
chunk.
- This saves external memory (Buffers outside the JavaScript heap). It
does not reduce `heapUsed`. It does not fix the out-of-memory crashes of
the hosted server by itself.
- The patch must be removed or ported when Paperclip upgrades
postgres.js. The test fails if a driver version keeps the buffer again.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), used in
Claude Code with tool use (shell, file editing, local PostgreSQL), heap
snapshots and allocation measurement.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 11:38:32 -06:00
1 parent 426e8a35dc
commit 2de4787f41
5 files changed
+217 -6

No files matched your search

+12
View File
@@ -1095,6 +1095,18 @@ with slow background work. It does not make the SQL itself faster. The default
pipeline limit remains unchanged. Test transaction behavior before using this
setting with a driver build that does not include the patch.
The patch also releases a connection's receive buffer when the connection has
read every message in it. Without this fix, an idle connection keeps the buffer
of its last read until its next read: the socket chunk, or the whole buffer of
its largest recent message. At 340 open connections that each read a 1 MB row,
this held about 320 MB of `external` memory. The memory is outside the
JavaScript heap. `packages/db/src/postgres-receive-buffer.test.ts` checks that
the buffer is released. It also checks that results are reassembled correctly
when socket chunks end exactly at message boundaries, when they split messages
in the middle, and when the operating system chooses the chunks. Measure it with
`pnpm --filter @paperclipai/db exec node --expose-gc --import tsx scripts/benchmark-postgres-receive-buffer.ts`
(`CONNECTIONS` and `ROW_BYTES` are optional).
Trusted-header actor synchronization retries transient connection failures,
including `CONNECT_TIMEOUT`, at most twice. This retry applies only to the
idempotent actor synchronization operations, not arbitrary transactions. A
@@ -0,0 +1,38 @@
import postgres from "postgres";
import { startEmbeddedPostgresTestDatabase } from "../src/test-embedded-postgres.js";
const gc = (globalThis as { gc?: () => void }).gc;
if (!gc) throw new Error("Run with node --expose-gc");
const connections = Number(process.env.CONNECTIONS ?? 80);
const rowBytes = Number(process.env.ROW_BYTES ?? 1_000_000);
const temporary = process.env.DATABASE_URL ? null : await startEmbeddedPostgresTestDatabase("postgres-receive-buffer-");
const sql = postgres(process.env.DATABASE_URL ?? temporary!.connectionString, {
max: connections, idle_timeout: 60, prepare: false, onnotice: () => {},
connection: { application_name: "receive-buffer-benchmark" },
});
function memory() {
gc!(); gc!();
const { heapUsed, external, rss } = process.memoryUsage();
return { heapUsed, external, rss };
}
try {
await sql.unsafe("select 1");
const before = memory();
await Promise.all(Array.from({ length: connections }, () => sql.unsafe("select pg_sleep(0.5)")));
const open = memory();
await Promise.all(Array.from({ length: connections }, () => sql.unsafe("select repeat('x', $1::int) as value", [rowBytes]).then(() => undefined)));
const idleAfterLargeRow = memory();
const perConnectionKB = (from: ReturnType<typeof memory>, to: ReturnType<typeof memory>, key: keyof ReturnType<typeof memory>) =>
Math.round((to[key] - from[key]) / connections / 1024);
console.log(JSON.stringify({
connections, rowBytes,
openIdle: { heapKB: perConnectionKB(before, open, "heapUsed"), externalKB: perConnectionKB(before, open, "external"), rssKB: perConnectionKB(before, open, "rss") },
idleAfterLargeRow: { heapKB: perConnectionKB(before, idleAfterLargeRow, "heapUsed"), externalKB: perConnectionKB(before, idleAfterLargeRow, "external"), rssKB: perConnectionKB(before, idleAfterLargeRow, "rss") },
pinnedExternalMB: Math.round((idleAfterLargeRow.external - open.external) / 1048576),
}));
} finally {
await sql.end({ timeout: 1 });
await temporary?.cleanup();
}
@@ -0,0 +1,108 @@
import assert from "node:assert/strict";
import { createRequire } from "node:module";
import net from "node:net";
import { setImmediate } from "node:timers/promises";
const mode = process.env.CHUNKING ?? "socket";
const databasePort = Number(new URL(process.env.TEST_DATABASE_URL).port || 5432);
const chunking = { exactEnds: 0, splitInside: 0 };
const parsers = new WeakMap();
const emit = net.Socket.prototype.emit;
net.Socket.prototype.emit = function (event, ...args) {
if (event !== "data" || mode === "socket" || this.remotePort !== databasePort) return emit.call(this, event, ...args);
const chunk = args[0];
const state = parsers.get(this) ?? { header: Buffer.alloc(0), remaining: 0 };
parsers.set(this, state);
const cuts = [];
let offset = 0;
while (offset < chunk.length) {
if (state.remaining === 0) {
const take = Math.min(5 - state.header.length, chunk.length - offset);
state.header = Buffer.concat([state.header, chunk.subarray(offset, offset + take)]);
offset += take;
if (state.header.length < 5) break;
state.remaining = state.header.readUInt32BE(1) - 4;
state.header = Buffer.alloc(0);
if (mode === "straddle" && state.remaining > 1) {
const middle = offset + Math.min(Math.floor(state.remaining / 2), chunk.length - offset);
if (middle > offset && middle < chunk.length) { cuts.push(middle); chunking.splitInside += 1; }
}
}
const take = Math.min(state.remaining, chunk.length - offset);
offset += take;
state.remaining -= take;
if (state.remaining === 0 && state.header.length === 0) {
cuts.push(offset);
chunking.exactEnds += 1;
}
}
let start = 0;
for (const cut of [...new Set(cuts)].sort((a, b) => a - b).concat(chunk.length)) {
if (cut > start) emit.call(this, "data", chunk.subarray(start, cut));
start = Math.max(start, cut);
}
return true;
};
const concat = Buffer.concat;
const received = [];
Buffer.concat = function (...args) {
const result = concat.apply(this, args);
if (result.length >= 512 * 1024) received.push(new WeakRef(result.buffer));
return result;
};
const postgres = process.env.DRIVER_FORMAT === "cjs"
? createRequire(import.meta.url)("postgres")
: (await import("postgres")).default;
const sql = postgres(process.env.TEST_DATABASE_URL, {
max: 1,
connection: { application_name: "driver-receive-buffer" },
onnotice() {},
});
async function collected() {
for (let attempt = 0; attempt < 5; attempt++) {
await setImmediate();
globalThis.gc();
}
return received.filter(ref => ref.deref() !== undefined).length;
}
try {
const [large] = await sql.unsafe("select repeat('x', 3000000) as value");
assert.equal(large.value.length, 3_000_000);
assert.equal(large.value, "x".repeat(3_000_000));
for (let size = 65_400; size <= 65_560; size++) {
const [row] = await sql.unsafe("select repeat('z', $1::int) as value", [size]);
assert.equal(row.value.length, size);
}
const rows = await sql.unsafe("select g, repeat('y', 50) as value from generate_series(1, 20000) g");
assert.equal(rows.length, 20_000);
assert.equal(rows[19_999].g, 20_000);
assert.ok(rows.every((row, index) => row.g === index + 1 && row.value === "y".repeat(50)));
const [pair] = await sql.unsafe("select repeat('a', 1500000) as a, repeat('b', 1500000) as b");
assert.equal(pair.a, "a".repeat(1_500_000));
assert.equal(pair.b, "b".repeat(1_500_000));
assert.ok(received.length > 0, "a large row must be reassembled from several socket chunks");
assert.equal(await collected(), 0, "an idle connection must not keep the receive buffer of its last result");
for (let round = 0; round < 20; round++) {
const size = 1_000_000 + round * 100_003;
const [row] = await sql.unsafe("select repeat('r', $1::int) as value", [size]);
assert.equal(row.value.length, size);
assert.equal(await collected(), 0, `an idle connection kept the receive buffer of a ${size} byte row`);
}
const [after] = await sql.unsafe("select 1 as value");
assert.equal(after.value, 1);
if (mode === "message") assert.ok(chunking.exactEnds > 20_000, "chunks must end exactly at message boundaries");
if (mode === "straddle") assert.ok(chunking.splitInside > 20_000, "chunks must split messages in the middle");
console.log("released");
} finally {
await sql.end({ timeout: 1 });
}
@@ -0,0 +1,35 @@
import { execFile } from "node:child_process";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
type EmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const run = promisify(execFile);
const fixture = fileURLToPath(new URL("./__fixtures__/postgres-receive-buffer.mjs", import.meta.url));
describe.skipIf(!support.supported)("postgres receive buffer", () => {
let database: EmbeddedPostgresTestDatabase;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-driver-receive-buffer-");
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => { await database?.cleanup(); });
for (const format of ["esm", "cjs"]) {
for (const chunking of ["socket", "message", "straddle"]) {
it(`${format}: reassembles ${chunking}-chunked results and releases the buffer once the connection is idle`, async () => {
const { stdout, stderr } = await run(process.execPath, ["--expose-gc", fixture], {
env: { ...process.env, TEST_DATABASE_URL: database.connectionString, DRIVER_FORMAT: format, CHUNKING: chunking },
timeout: 60_000,
});
expect(stdout.trim()).toBe("released");
expect(stderr).toBe("");
}, 70_000);
}
}
});
+24 -6
View File
@@ -1,5 +1,5 @@
diff --git a/cjs/src/connection.js b/cjs/src/connection.js
index 07f6716702ac888215c86ad6e0071aaf55ae519c..85700681ccc7bf98f1fe987978db47e0097227f3 100644
index 07f6716702ac888215c86ad6e0071aaf55ae519c..4ee75faa8ae47555576e3b4e0ec6b7c4b140ec2f 100644
--- a/cjs/src/connection.js
+++ b/cjs/src/connection.js
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -12,7 +12,16 @@ index 07f6716702ac888215c86ad6e0071aaf55ae519c..85700681ccc7bf98f1fe987978db47e0
} catch (error) {
sent.length === 0 && write(Sync)
errored(error)
@@ -438,6 +438,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
}
+
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
}
async function connect() {
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
clearImmediate(nextWriteTimer)
@@ -20,7 +29,7 @@ index 07f6716702ac888215c86ad6e0071aaf55ae519c..85700681ccc7bf98f1fe987978db47e0
socket.removeListener('data', data)
socket.removeListener('connect', connected)
idleTimer.cancel()
@@ -451,6 +452,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return reconnect()
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))
@@ -94,7 +103,7 @@ index f09c61c7e998f411560b95c9a8e1ed6a97972c11..04dded5c7e2cdc34ac01b29a87cc9324
? queries.push(q)
: c.execute(q) || move(c, full)
diff --git a/src/connection.js b/src/connection.js
index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..81541f9cea75925256d7b02b2079c60802b2f92f 100644
index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..0a4c4cc185f14afa30c7c6a294bafe689e685683 100644
--- a/src/connection.js
+++ b/src/connection.js
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -107,7 +116,16 @@ index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..81541f9cea75925256d7b02b2079c608
} catch (error) {
sent.length === 0 && write(Sync)
errored(error)
@@ -438,6 +438,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
}
+
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
}
async function connect() {
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
clearImmediate(nextWriteTimer)
@@ -115,7 +133,7 @@ index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..81541f9cea75925256d7b02b2079c608
socket.removeListener('data', data)
socket.removeListener('connect', connected)
idleTimer.cancel()
@@ -451,6 +452,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return reconnect()
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))