Files
PaperClipAI/patches/postgres@3.4.9.patch
T
DottaandClaude Opus 5.5 d25667f370 perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.

The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.

**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.

**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.

## What Changed

- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:

  ```diff
       remove: (x) => {
  -      const index = xs.indexOf(x)
  -      return index === -1
  +      const at = xs.indexOf(x, index)
  +      return at === -1
           ? null
  -        : (xs.splice(index, 1), x)
  +        : (xs.splice(at, 1), x)
       },
  ...
         if (index === xs.length) {
           index = 0
           xs = []
  +      } else if (index > 1024 && index * 2 > xs.length) {
  +        xs = xs.slice(index)
  +        index = 0
         } else {
           xs[index - 1] = undefined
         }
  ```

The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:

  | Measurement | Before | After |
  | --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |

## Risks

- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:46:41 -06:00

265 lines
8.7 KiB
Diff

diff --git a/cjs/src/connection.js b/cjs/src/connection.js
index 07f6716702ac888215c86ad6e0071aaf55ae519c..4ee75faa8ae47555576e3b4e0ec6b7c4b140ec2f 100644
--- a/cjs/src/connection.js
+++ b/cjs/src/connection.js
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return write(toBuffer(q))
&& !q.describeFirst
&& !q.cursorFn
- && sent.length < max_pipeline
&& (!q.options.onexecute || q.options.onexecute(connection))
+ && sent.length < max_pipeline
} catch (error) {
sent.length === 0 && write(Sync)
errored(error)
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
}
+
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
}
async function connect() {
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
clearImmediate(nextWriteTimer)
+ chunk = nextWriteTimer = null
socket.removeListener('data', data)
socket.removeListener('connect', connected)
idleTimer.cancel()
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return reconnect()
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))
+ query = results = errorResponse = null
+ result = new Result()
+ rows = 0
closedTime = performance.now()
hadError && options.shared.retries++
delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
diff --git a/cjs/src/index.js b/cjs/src/index.js
index f09c61c7e998f411560b95c9a8e1ed6a97972c11..04dded5c7e2cdc34ac01b29a87cc93248ec8c9c6 100644
--- a/cjs/src/index.js
+++ b/cjs/src/index.js
@@ -216,8 +216,19 @@ function Postgres(a, b) {
: move(c, reserved)
c.reserved.release = true
+ let closedError
+ c.onclose = error => {
+ closedError = error
+ while (queue.length)
+ queue.shift().reject(error)
+ }
+
const sql = Sql(handler)
sql.release = () => {
+ if (closedError)
+ return
+ closedError = Errors.connection('CONNECTION_RELEASED', options)
+ c.onclose = null
c.reserved = null
onopen(c)
}
@@ -225,6 +236,8 @@ function Postgres(a, b) {
return sql
function handler(q) {
+ if (closedError)
+ return q.reject(closedError)
c.queue === full
? queue.push(q)
: c.execute(q) || move(c, full)
@@ -236,13 +249,19 @@ function Postgres(a, b) {
const queries = Queue()
let savepoints = 0
, connection
+ , closedError
, prepare = null
try {
await sql.unsafe('begin ' + options.replace(/[^a-z ]/ig, ''), [], { onexecute }).execute()
return await Promise.race([
scope(connection, fn),
- new Promise((_, reject) => connection.onclose = reject)
+ new Promise((_, reject) => connection.onclose = error => {
+ closedError = error
+ while (queries.length)
+ queries.shift().reject(error)
+ reject(error)
+ })
])
} catch (error) {
throw error
@@ -290,6 +309,8 @@ function Postgres(a, b) {
function handler(q) {
q.catch(e => uncaughtError || (uncaughtError = e))
+ if (closedError)
+ return q.reject(closedError)
c.queue === full
? queries.push(q)
: c.execute(q) || move(c, full)
diff --git a/cjs/src/queue.js b/cjs/src/queue.js
index 8438f5da92cf7637d98a33491d75c75b1c817c25..1c9a330fa173ecae84219055368cf630feadbda2 100644
--- a/cjs/src/queue.js
+++ b/cjs/src/queue.js
@@ -9,10 +9,10 @@ function Queue(initial = []) {
return xs.length - index
},
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
push: (x) => (xs.push(x), x),
shift: () => {
@@ -21,6 +21,9 @@ function Queue(initial = []) {
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
diff --git a/src/connection.js b/src/connection.js
index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..0a4c4cc185f14afa30c7c6a294bafe689e685683 100644
--- a/src/connection.js
+++ b/src/connection.js
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return write(toBuffer(q))
&& !q.describeFirst
&& !q.cursorFn
- && sent.length < max_pipeline
&& (!q.options.onexecute || q.options.onexecute(connection))
+ && sent.length < max_pipeline
} catch (error) {
sent.length === 0 && write(Sync)
errored(error)
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
}
+
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
}
async function connect() {
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
remaining = 0
incomings = null
clearImmediate(nextWriteTimer)
+ chunk = nextWriteTimer = null
socket.removeListener('data', data)
socket.removeListener('connect', connected)
idleTimer.cancel()
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
return reconnect()
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))
+ query = results = errorResponse = null
+ result = new Result()
+ rows = 0
closedTime = performance.now()
hadError && options.shared.retries++
delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
diff --git a/src/index.js b/src/index.js
index c7fba3dace20173f4cb13f3f138bb6d213b43683..f0888986add8ba02c8853f792ae42dd1cec88c82 100644
--- a/src/index.js
+++ b/src/index.js
@@ -216,8 +216,19 @@ function Postgres(a, b) {
: move(c, reserved)
c.reserved.release = true
+ let closedError
+ c.onclose = error => {
+ closedError = error
+ while (queue.length)
+ queue.shift().reject(error)
+ }
+
const sql = Sql(handler)
sql.release = () => {
+ if (closedError)
+ return
+ closedError = Errors.connection('CONNECTION_RELEASED', options)
+ c.onclose = null
c.reserved = null
onopen(c)
}
@@ -225,6 +236,8 @@ function Postgres(a, b) {
return sql
function handler(q) {
+ if (closedError)
+ return q.reject(closedError)
c.queue === full
? queue.push(q)
: c.execute(q) || move(c, full)
@@ -236,13 +249,19 @@ function Postgres(a, b) {
const queries = Queue()
let savepoints = 0
, connection
+ , closedError
, prepare = null
try {
await sql.unsafe('begin ' + options.replace(/[^a-z ]/ig, ''), [], { onexecute }).execute()
return await Promise.race([
scope(connection, fn),
- new Promise((_, reject) => connection.onclose = reject)
+ new Promise((_, reject) => connection.onclose = error => {
+ closedError = error
+ while (queries.length)
+ queries.shift().reject(error)
+ reject(error)
+ })
])
} catch (error) {
throw error
@@ -290,6 +309,8 @@ function Postgres(a, b) {
function handler(q) {
q.catch(e => uncaughtError || (uncaughtError = e))
+ if (closedError)
+ return q.reject(closedError)
c.queue === full
? queries.push(q)
: c.execute(q) || move(c, full)
diff --git a/src/queue.js b/src/queue.js
index c4ef97164f057db99dce1b9af3401dbb1af37e31..46e7ade55c0a3250137a21117c03ce9166b40bc2 100644
--- a/src/queue.js
+++ b/src/queue.js
@@ -9,10 +9,10 @@ function Queue(initial = []) {
return xs.length - index
},
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
push: (x) => (xs.push(x), x),
shift: () => {
@@ -21,6 +21,9 @@ function Queue(initial = []) {
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}