mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.
The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.
**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.
**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.
## What Changed
- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:
```diff
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
...
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
```
The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:
| Measurement | Before | After |
| --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |
## Risks
- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
265 lines
8.7 KiB
Diff
265 lines
8.7 KiB
Diff
diff --git a/cjs/src/connection.js b/cjs/src/connection.js
|
|
index 07f6716702ac888215c86ad6e0071aaf55ae519c..4ee75faa8ae47555576e3b4e0ec6b7c4b140ec2f 100644
|
|
--- a/cjs/src/connection.js
|
|
+++ b/cjs/src/connection.js
|
|
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
return write(toBuffer(q))
|
|
&& !q.describeFirst
|
|
&& !q.cursorFn
|
|
- && sent.length < max_pipeline
|
|
&& (!q.options.onexecute || q.options.onexecute(connection))
|
|
+ && sent.length < max_pipeline
|
|
} catch (error) {
|
|
sent.length === 0 && write(Sync)
|
|
errored(error)
|
|
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
remaining = 0
|
|
incomings = null
|
|
}
|
|
+
|
|
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
|
|
}
|
|
|
|
async function connect() {
|
|
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
remaining = 0
|
|
incomings = null
|
|
clearImmediate(nextWriteTimer)
|
|
+ chunk = nextWriteTimer = null
|
|
socket.removeListener('data', data)
|
|
socket.removeListener('connect', connected)
|
|
idleTimer.cancel()
|
|
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
return reconnect()
|
|
|
|
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))
|
|
+ query = results = errorResponse = null
|
|
+ result = new Result()
|
|
+ rows = 0
|
|
closedTime = performance.now()
|
|
hadError && options.shared.retries++
|
|
delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
|
|
diff --git a/cjs/src/index.js b/cjs/src/index.js
|
|
index f09c61c7e998f411560b95c9a8e1ed6a97972c11..04dded5c7e2cdc34ac01b29a87cc93248ec8c9c6 100644
|
|
--- a/cjs/src/index.js
|
|
+++ b/cjs/src/index.js
|
|
@@ -216,8 +216,19 @@ function Postgres(a, b) {
|
|
: move(c, reserved)
|
|
c.reserved.release = true
|
|
|
|
+ let closedError
|
|
+ c.onclose = error => {
|
|
+ closedError = error
|
|
+ while (queue.length)
|
|
+ queue.shift().reject(error)
|
|
+ }
|
|
+
|
|
const sql = Sql(handler)
|
|
sql.release = () => {
|
|
+ if (closedError)
|
|
+ return
|
|
+ closedError = Errors.connection('CONNECTION_RELEASED', options)
|
|
+ c.onclose = null
|
|
c.reserved = null
|
|
onopen(c)
|
|
}
|
|
@@ -225,6 +236,8 @@ function Postgres(a, b) {
|
|
return sql
|
|
|
|
function handler(q) {
|
|
+ if (closedError)
|
|
+ return q.reject(closedError)
|
|
c.queue === full
|
|
? queue.push(q)
|
|
: c.execute(q) || move(c, full)
|
|
@@ -236,13 +249,19 @@ function Postgres(a, b) {
|
|
const queries = Queue()
|
|
let savepoints = 0
|
|
, connection
|
|
+ , closedError
|
|
, prepare = null
|
|
|
|
try {
|
|
await sql.unsafe('begin ' + options.replace(/[^a-z ]/ig, ''), [], { onexecute }).execute()
|
|
return await Promise.race([
|
|
scope(connection, fn),
|
|
- new Promise((_, reject) => connection.onclose = reject)
|
|
+ new Promise((_, reject) => connection.onclose = error => {
|
|
+ closedError = error
|
|
+ while (queries.length)
|
|
+ queries.shift().reject(error)
|
|
+ reject(error)
|
|
+ })
|
|
])
|
|
} catch (error) {
|
|
throw error
|
|
@@ -290,6 +309,8 @@ function Postgres(a, b) {
|
|
|
|
function handler(q) {
|
|
q.catch(e => uncaughtError || (uncaughtError = e))
|
|
+ if (closedError)
|
|
+ return q.reject(closedError)
|
|
c.queue === full
|
|
? queries.push(q)
|
|
: c.execute(q) || move(c, full)
|
|
diff --git a/cjs/src/queue.js b/cjs/src/queue.js
|
|
index 8438f5da92cf7637d98a33491d75c75b1c817c25..1c9a330fa173ecae84219055368cf630feadbda2 100644
|
|
--- a/cjs/src/queue.js
|
|
+++ b/cjs/src/queue.js
|
|
@@ -9,10 +9,10 @@ function Queue(initial = []) {
|
|
return xs.length - index
|
|
},
|
|
remove: (x) => {
|
|
- const index = xs.indexOf(x)
|
|
- return index === -1
|
|
+ const at = xs.indexOf(x, index)
|
|
+ return at === -1
|
|
? null
|
|
- : (xs.splice(index, 1), x)
|
|
+ : (xs.splice(at, 1), x)
|
|
},
|
|
push: (x) => (xs.push(x), x),
|
|
shift: () => {
|
|
@@ -21,6 +21,9 @@ function Queue(initial = []) {
|
|
if (index === xs.length) {
|
|
index = 0
|
|
xs = []
|
|
+ } else if (index > 1024 && index * 2 > xs.length) {
|
|
+ xs = xs.slice(index)
|
|
+ index = 0
|
|
} else {
|
|
xs[index - 1] = undefined
|
|
}
|
|
diff --git a/src/connection.js b/src/connection.js
|
|
index 1b1cccde43b4570d5d071d6ffaab2669b3c2065a..0a4c4cc185f14afa30c7c6a294bafe689e685683 100644
|
|
--- a/src/connection.js
|
|
+++ b/src/connection.js
|
|
@@ -173,8 +173,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
return write(toBuffer(q))
|
|
&& !q.describeFirst
|
|
&& !q.cursorFn
|
|
- && sent.length < max_pipeline
|
|
&& (!q.options.onexecute || q.options.onexecute(connection))
|
|
+ && sent.length < max_pipeline
|
|
} catch (error) {
|
|
sent.length === 0 && write(Sync)
|
|
errored(error)
|
|
@@ -330,6 +330,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
remaining = 0
|
|
incomings = null
|
|
}
|
|
+
|
|
+ incoming.length === 0 && (incoming = Buffer.alloc(0))
|
|
}
|
|
|
|
async function connect() {
|
|
@@ -438,6 +440,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
remaining = 0
|
|
incomings = null
|
|
clearImmediate(nextWriteTimer)
|
|
+ chunk = nextWriteTimer = null
|
|
socket.removeListener('data', data)
|
|
socket.removeListener('connect', connected)
|
|
idleTimer.cancel()
|
|
@@ -451,6 +454,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
|
|
return reconnect()
|
|
|
|
!hadError && (query || sent.length) && error(Errors.connection('CONNECTION_CLOSED', options, socket))
|
|
+ query = results = errorResponse = null
|
|
+ result = new Result()
|
|
+ rows = 0
|
|
closedTime = performance.now()
|
|
hadError && options.shared.retries++
|
|
delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
|
|
diff --git a/src/index.js b/src/index.js
|
|
index c7fba3dace20173f4cb13f3f138bb6d213b43683..f0888986add8ba02c8853f792ae42dd1cec88c82 100644
|
|
--- a/src/index.js
|
|
+++ b/src/index.js
|
|
@@ -216,8 +216,19 @@ function Postgres(a, b) {
|
|
: move(c, reserved)
|
|
c.reserved.release = true
|
|
|
|
+ let closedError
|
|
+ c.onclose = error => {
|
|
+ closedError = error
|
|
+ while (queue.length)
|
|
+ queue.shift().reject(error)
|
|
+ }
|
|
+
|
|
const sql = Sql(handler)
|
|
sql.release = () => {
|
|
+ if (closedError)
|
|
+ return
|
|
+ closedError = Errors.connection('CONNECTION_RELEASED', options)
|
|
+ c.onclose = null
|
|
c.reserved = null
|
|
onopen(c)
|
|
}
|
|
@@ -225,6 +236,8 @@ function Postgres(a, b) {
|
|
return sql
|
|
|
|
function handler(q) {
|
|
+ if (closedError)
|
|
+ return q.reject(closedError)
|
|
c.queue === full
|
|
? queue.push(q)
|
|
: c.execute(q) || move(c, full)
|
|
@@ -236,13 +249,19 @@ function Postgres(a, b) {
|
|
const queries = Queue()
|
|
let savepoints = 0
|
|
, connection
|
|
+ , closedError
|
|
, prepare = null
|
|
|
|
try {
|
|
await sql.unsafe('begin ' + options.replace(/[^a-z ]/ig, ''), [], { onexecute }).execute()
|
|
return await Promise.race([
|
|
scope(connection, fn),
|
|
- new Promise((_, reject) => connection.onclose = reject)
|
|
+ new Promise((_, reject) => connection.onclose = error => {
|
|
+ closedError = error
|
|
+ while (queries.length)
|
|
+ queries.shift().reject(error)
|
|
+ reject(error)
|
|
+ })
|
|
])
|
|
} catch (error) {
|
|
throw error
|
|
@@ -290,6 +309,8 @@ function Postgres(a, b) {
|
|
|
|
function handler(q) {
|
|
q.catch(e => uncaughtError || (uncaughtError = e))
|
|
+ if (closedError)
|
|
+ return q.reject(closedError)
|
|
c.queue === full
|
|
? queries.push(q)
|
|
: c.execute(q) || move(c, full)
|
|
diff --git a/src/queue.js b/src/queue.js
|
|
index c4ef97164f057db99dce1b9af3401dbb1af37e31..46e7ade55c0a3250137a21117c03ce9166b40bc2 100644
|
|
--- a/src/queue.js
|
|
+++ b/src/queue.js
|
|
@@ -9,10 +9,10 @@ function Queue(initial = []) {
|
|
return xs.length - index
|
|
},
|
|
remove: (x) => {
|
|
- const index = xs.indexOf(x)
|
|
- return index === -1
|
|
+ const at = xs.indexOf(x, index)
|
|
+ return at === -1
|
|
? null
|
|
- : (xs.splice(index, 1), x)
|
|
+ : (xs.splice(at, 1), x)
|
|
},
|
|
push: (x) => (xs.push(x), x),
|
|
shift: () => {
|
|
@@ -21,6 +21,9 @@ function Queue(initial = []) {
|
|
if (index === xs.length) {
|
|
index = 0
|
|
xs = []
|
|
+ } else if (index > 1024 && index * 2 > xs.length) {
|
|
+ xs = xs.slice(index)
|
|
+ index = 0
|
|
} else {
|
|
xs[index - 1] = undefined
|
|
}
|