mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:48:15 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.
The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.
**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.
**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.
## What Changed
- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:
```diff
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
...
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
```
The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:
| Measurement | Before | After |
| --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |
## Risks
- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
70 lines
2.7 KiB
TypeScript
70 lines
2.7 KiB
TypeScript
import { execFile } from "node:child_process";
|
|
import { createRequire } from "node:module";
|
|
import path from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import { promisify } from "node:util";
|
|
import { startEmbeddedPostgresTestDatabase } from "../src/test-embedded-postgres.js";
|
|
|
|
type Item = { queue: Queue | null };
|
|
type Queue = { readonly length: number; push(item: Item): Item; shift(): Item | undefined; remove(item: Item): Item | null };
|
|
|
|
const simulatedQueries = Number(process.env.SIMULATED_QUERIES ?? 1_000_000);
|
|
const poolQueries = Number(process.env.QUERIES ?? 200_000);
|
|
const driverRoot = path.resolve(path.dirname(createRequire(import.meta.url).resolve("postgres")), "../..");
|
|
const Queue: () => Queue = (await import(pathToFileURL(path.join(driverRoot, "src/queue.js")).href)).default;
|
|
|
|
function simulate() {
|
|
const open = Queue();
|
|
const busy = Queue();
|
|
const move = (c: Item, queue: Queue) => {
|
|
c.queue?.remove(c);
|
|
queue.push(c);
|
|
c.queue = queue;
|
|
};
|
|
for (let id = 0; id < 117; id++) move({ queue: null }, open);
|
|
const running: Item[] = new Array(8);
|
|
const windows: number[] = [];
|
|
let started = performance.now();
|
|
for (let query = 1; query <= simulatedQueries; query++) {
|
|
const c = open.shift()!;
|
|
move(c, busy);
|
|
const slot = query % running.length;
|
|
if (running[slot]) move(running[slot], open);
|
|
running[slot] = c;
|
|
if (query % (simulatedQueries / 5) === 0) {
|
|
windows.push(Number((((performance.now() - started) * 1000) / (simulatedQueries / 5)).toFixed(3)));
|
|
started = performance.now();
|
|
}
|
|
}
|
|
return { queries: simulatedQueries, connections: 117, inFlight: 8, microsPerQueryByWindow: windows };
|
|
}
|
|
|
|
const temporary = process.env.DATABASE_URL ? null : await startEmbeddedPostgresTestDatabase("postgres-queue-benchmark-");
|
|
try {
|
|
const simulation = simulate();
|
|
const { stdout } = await promisify(execFile)(process.execPath, [
|
|
fileURLToPath(new URL("../src/__fixtures__/postgres-queue-pool.mjs", import.meta.url)),
|
|
], {
|
|
env: {
|
|
...process.env,
|
|
TEST_DATABASE_URL: process.env.DATABASE_URL ?? temporary!.connectionString,
|
|
QUERIES: String(poolQueries),
|
|
WINDOWS: "10",
|
|
MEASURE_SCANS: "0",
|
|
},
|
|
timeout: 600_000,
|
|
});
|
|
const pool = JSON.parse(stdout) as { windows: Array<{ cpuMicrosPerQuery: number; wallMicrosPerQuery: number }> };
|
|
console.log(JSON.stringify({
|
|
simulation,
|
|
pool: {
|
|
...pool,
|
|
windows: undefined,
|
|
cpuMicrosPerQueryByWindow: pool.windows.map((window) => window.cpuMicrosPerQuery),
|
|
wallMicrosPerQueryByWindow: pool.windows.map((window) => window.wallMicrosPerQuery),
|
|
},
|
|
}));
|
|
} finally {
|
|
await temporary?.cleanup();
|
|
}
|