mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a task stalls on a recovery action that needs the board, recovery wakes the agents of the tasks that wait on it: its parent and the tasks it blocks (`wakeWaitersOfStalledIssues` in `server/src/services/recovery/blocker-stall-wakeups.ts`). > - Each pass reads the waiters of every stalled task with one statement for each stall. The statement has 18 parameters and two correlated subqueries. Its SQL text is 0.8 KB, and it is the same for every stall of the same shape. > - The server wrote that text with the query builder for every stall on every pass. On a hosted instance the server main thread was near one full core and garbage collection used a quarter of it. A statistics sample showed this statement at about 7.8 executions per second. > - I measured it on a local PostgreSQL through the real client. The query builder allocates 103 KB to write and run this statement. The same statement from a prepared query allocates 15 KB. > - This pull request writes the SQL text of the waiter read one time for each client and shape. Each execution then only binds values. > - The benefit is 76% less allocation and 5 times fewer garbage collections for a pass over 20 stalled tasks. The statement, its parameters and its rows do not change. ## Linked Issues or Issue Description Related: #21195 (the same change for the native chat progress page read; it added the `preparedStatements` helper that this pull request uses), #21131 and #18965. No issue exists for this read. **What existing behavior does this improve?** The cost, in the server process, of the waiter read that the blocker-stall wake pass runs for each stalled task. **Subsystem affected** Recovery: blocker-stall wakes (`server/src/services/recovery/blocker-stall-wakeups.ts`). **Current behavior** `wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle query builder inside its loop, and Drizzle then writes the SQL text from it, one time for each stall and page. Measured on the merge base, Node 26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture has 20 stalled tasks, each with a parent and one dependent task that are already woken, so a pass writes nothing: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100 (163.7 ms) | | Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms) | At the sampled rate of 7.8 executions per second the read allocates about 0.8 MB per second. This rate is one sample of one instance, taken soon after a long outage, when recovery had a backlog. It is not a steady-state figure. **Proposed behavior** The query moves into one builder function, `stallWaiterPageQuery`. The module prepares it one time for each client and shape, with placeholders for the values. The shapes are: the stalled task has a parent or not; all waiters or one waiter; first page or later page. The loop executes the prepared statement. **Reason and benefit** Same benchmark, this branch: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8 ms) | | Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) | One waiter read costs about 88 KB less. At the sampled rate this removes about 0.7 MB per second of allocation, which is about 0.9% of the 77.7 MB per second that the instance allocated. It is a small part of the total. The CPU numbers come from a machine with other load and are less stable than the allocation numbers. **Breaking changes** None. ## What Changed - `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter query moves out of the loop into `stallWaiterPageQuery(db, binding)`. `stallWaiterPageStatement` returns the prepared statement for the shape and the values to bind. The loop executes it. The lines inside the two SQL fragments keep their indentation, so the SQL text is byte-identical. The pull request adds no comment. The one existing comment of this read stays at the call site, where the code still makes that choice. - `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts` (new): statement equivalence, row equivalence on PostgreSQL, and the red/green mechanism test. - `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the fixture that the test and the benchmark share. - `server/src/__tests__/prepared-statements-driver.test.ts` (new): proves at the driver that a prepared read and the query builder ask postgres.js for the same thing, with the driver's prepared-statement setting unset, false and true. It covers this read and the native chat progress page read of #21195. - `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no comments): the benchmark above. It loads the module at `--baseline-ref` next to the current module in one process. `sweep` is one company pass over `STALLS` stalled tasks whose waiters are already woken: one stall read, then one waiter read for each stall. `task` is the pass for one stalled task. Before it measures, the script also runs a company with a paused agent and 101 waiters on a stall with a parent and on a stall without one, and one pass for a single waiter of each. This covers six shapes: the first page, the later page and one waiter, each for a stall with a parent and for a stall without one. The script fails unless both versions send the same statements, byte for byte, with the same parameters and return the same result. Passes alternate between the versions with a full collection before each pass. CPU is the whole process. Allocated bytes are V8's `total_allocated_bytes`. Optional environment variables: `STALLS` (default 20), `ITERATIONS` (default 300 calls for each pass), `PASSES` (default 5 for each version). Not changed: which statements run, their order, the other reads of the pass (they are small and vary by waiter), the wake itself, the driver's prepared-statement setting, and the schema. The read takes no lock and runs on the root client outside a transaction. Its two lists (waiter statuses and wake statuses) are module constants, so the statement has no variable-length list parameter. ## Verification - Statement equivalence (`sends the SQL text and the parameters that the query builder writes, for every shape of the read`): for all eight shapes the prepared SQL text equals the text that the builder writes with the values, and the bound parameters are equal. The test also proves that a stall without a parent binds `null` for the parent. - Against the merge base at the driver: the benchmark compares every statement that the merge base module and this branch send for six of the eight waiter-read shapes. The texts are byte-identical and the parameters are equal. The two shapes that it does not reach (one waiter on a later page) cannot occur: a one-waiter read returns at most one row, so it has no later page. - Row equivalence on PostgreSQL (`returns the rows of the query builder: same values, types and order, and no other company's rows`): two companies. The fixture has a stall with a parent and six dependents, and a stall without a parent. One dependent is already woken, one is done, one is hidden and one has no assignee. One has a failed wake only, and one has a wake row with the same key in the other company; both stay waiters. The prepared statement and the builder return strictly equal rows for all waiters, one waiter, a later page, and a stall read under the other company's id (no rows). Null columns stay null. - Mechanism, red then green (`writes the SQL text of the waiter read one time and then only binds values`): a pass over three stalls writes no waiter SQL text and sends one identical text three times. On the merge base module the same test fails: the pass writes three texts. - Driver behavior (`asks the driver for the same thing as the query builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's `.prepare(name)` on postgres.js does not send the name to the database. `PostgresJsSession.prepareQuery` drops it (`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and `PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)` (lines 33 and 43). An ordinary query reaches the same call through `_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to 799). The test runs both reads through the builder and through the prepared statement on one connection for each setting. The arguments and the options of the driver call are strictly equal. With the setting unset or false, the server has no named statement after either path, so a transaction-mode pooler sees no change. With the setting true, the server has one named statement. Its name is the one that postgres.js generates for the connection, and both paths use that same statement. - Existing tests: `issue-blocker-stall-wakeups.test.ts` and `legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3 tests). `prepared-statements-driver.test.ts` passed (3 tests). - `pnpm --filter @paperclipai/server typecheck` passed. `pnpm check:module-boundaries` passed. - `pnpm verify` returns exit 2 (broader verification required, because the change adds a file under `server/scripts`). I did not run the full local suite. CI supplies that coverage. Reproduce: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \ src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \ src/__tests__/prepared-statements-driver.test.ts TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \ --baseline-ref=549816e6a0 ``` Not verified: the effect on a hosted instance. The rate above is one sample. The local numbers are from Node 26; the hosted instance runs Node 24. ## Risks - Low. The statement text and parameters are equal by test, the read takes no lock, and it runs on the root client outside a transaction. - A prepared statement keeps the text of the first build. The lists inside the statement are module constants, so the text cannot go stale in a running process. - A test that mocks `db.select` to observe this read no longer sees it. I found none for this read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude Code with tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>