Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 549816e6a0 perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run that came from a chat reports safe progress to that
chat. Each progress event wakes a queue, and each wake reads one page of
candidates (`nativeProgressPages` in
`server/src/services/chat-native-progress.ts`).
> - The page read joins six tables and has 44 parameters. Its SQL text
is 3.6 KB. The server wrote that text with the query builder on every
wake, and the text is the same for every wake of the same filter.
> - On a hosted instance the server main thread was near one full core
and garbage collection used a quarter of it. An allocation sample put
46% of all allocation under the query builder's SQL writing. The sample
does not name statements.
> - I measured single statements on a local PostgreSQL through the real
client. The query builder allocates about 20 KB for the smallest
statement, plus about 2 KB for each selected column and about 3 KB for
each predicate. This page read costs 296 KB and 0.34 ms of CPU to write.
The same statement from a prepared query costs 21 KB.
> - This pull request writes the SQL text of the page read one time for
each client and filter shape. Each execution then only binds values.
> - The benefit is 91% less allocation and 14 times fewer garbage
collections for one page read. The statement, its parameters and its
rows do not change.

## Linked Issues or Issue Description

Related: #21131 (fixed SQL text for the native run event append) and
#18965 (an earlier prepared read, in the GitHub review wait). No issue
exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the candidate page read that every
native chat progress wake runs.

**Subsystem affected**

Chat channels: native run progress publication
(`server/src/services/chat-native-progress.ts`).

**Current behavior**

`nativeProgressPages` builds the complete query with the Drizzle query
builder for each page, and Drizzle then writes the SQL text from it.
Measured on the merge base, Node 26, local PostgreSQL 18, median of 5
passes of 2,000 calls:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) |
| One wake for a task (page read + one transaction) | 7 | 681.3 KB |
2,238 µs | 34 (76.6 ms) |

A statistics sample of one hosted instance on one morning showed this
statement at about 3.9 executions per second. At that rate the read
allocates about 1.1 MB per second for SQL text. This rate is one sample,
taken soon after a long outage. It is not a steady-state figure.

**Proposed behavior**

The query stays in one builder function, `nativeProgressPageQuery`. The
module prepares it one time for each client and each of the six filter
shapes (all tasks, one company, one task; first page or later page),
with placeholders for the values. Each page read executes the prepared
statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) |
| One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) |

At the sampled rate this removes about 1.05 MB per second of allocation,
which is about 1.4% of the 77.7 MB per second that the instance
allocated. It is a small part of the total. The remaining six statements
of a wake run inside a transaction, where a prepared query gains
nothing, and this pull request does not change them.

**Breaking changes**

None.

## What Changed

- `server/src/services/chat-native-progress.ts`: the page query moves
into `nativeProgressPageQuery(db, binding)` without a change to its
text. `nativeProgressPageStatement` returns the prepared statement for
the filter shape and the values to bind. `nativeProgressPages` executes
it.
- `server/src/services/prepared-statements.ts` (new, 17 lines, no
comments):
- `preparedStatements(prepare)` keeps one prepared statement for each
client and shape. A prepared statement runs on the client that prepared
it, so the cache key is the client object. A transaction client gets its
own statement and gains nothing.
- `columnPlaceholder(name, column)` makes a placeholder whose value the
column encodes. A bare `sql.placeholder` skips the column's encoding, so
a `Date` would reach the driver unencoded. With this helper it reaches
the driver as the same ISO string as a value that the query builder
binds.
- `server/src/__tests__/chat-native-progress-statement.test.ts` (new):
statement equivalence, row equivalence on PostgreSQL, and the red/green
mechanism test.
- `server/src/__tests__/prepared-statements.test.ts` (new): the helper.
- `server/src/__tests__/chat-native-progress-work.test.ts` and
`chat-channels.integration.test.ts`: three tests watched or failed the
page read through `db.select`. A prepared statement does not call
`db.select`, so these tests now watch the driver (`db.$client.unsafe`).
The idle assertions are stricter: they now fail on any statement.
- `server/scripts/benchmark-native-progress-page.ts` (new, no comments):
the benchmark above. It loads the module at `--baseline-ref` next to the
current module in one process. `page` is `loadNativeChatProgressScopes`
for one task, which sends the page read and nothing else. `wake` is
`reconcileNativeChatProgress` for one task whose progress is already
published, so it writes nothing and every pass does the same work. The
script fails unless both versions send the same statements with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process (`process.cpuUsage`). Allocated bytes are V8's
`total_allocated_bytes`. A collection counts when it starts inside a
pass. Optional environment variables: `ITERATIONS` (default 2000 calls
for each pass) and `PASSES` (default 5 for each version).
`TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes.

Not changed: which statements run, their order, the transaction of a
wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's
prepared-statement setting, and the schema. The read has no
variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all six shapes,
and for an empty task id, the prepared SQL text equals the text the
builder writes with the values, and the bound parameters are equal. The
test also proves that the time filter is bound as the ISO string.
- One-time check against the merge base: I derived the builder of the
merge base from its source and compared 24 inputs (4 scopes, with and
without a cursor, 3 page sizes). The prepared SQL text is
byte-identical, and the parameters are strictly equal.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies, four candidate runs, and three runs that are not candidates.
The fixture has a null payload, a nested JSON payload, a run bound
through its task column only, and microsecond timestamps. The prepared
statement and the builder return strictly equal rows for every filter,
for pages of one row walked with the cursor, and for a task of another
company (no rows).
- Mechanism, red then green (`writes the SQL text of a read one time and
then only binds values`): three reads for different tasks write no SQL
text and send one identical text three times. On the merge base module
the same test fails with "expected sqlToQuery to not be called at all,
but actually been called 3 times".
- The benchmark asserts that both versions send the same statements with
the same parameters and return the same result, for the page read and
for a full wake.
- Existing tests: `chat-native-progress-work.test.ts` (19 passed),
`chat-channels.integration.test.ts` (1,954 of 1,963 passed in the
complete local run, including the 49 tests that match `native
progress|saved progress|safe native`. Two tests fail on this machine
because the worktree is inside another checkout: the bundled Slack skill
resolves to that checkout's skills directory, which has no
`slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests
failed in the same run from the state that those two left. The seven
pass when I run them by name. CI is the evidence for these nine),
`app-private-hostname-gate.test.ts` (passed).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returned exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \
  src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \
  --baseline-ref=267983356c
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, and the read
runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement (safe event types, conversation states, publication
states) are module constants, so the text cannot go stale in a running
process.
- The helper keys statements by client object. A caller that passes a
transaction client gets a statement for that transaction only. This is
correct, and it costs the same as before.
- Tests that mock `db.select` to observe this read no longer see it. I
found three and changed them to watch the driver.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 15:56:28 -06:00
DottaandPaperclip 1ed0af4186 fix(server): wake native chat progress from committed changes (#17639)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat users receive coarse progress updates while native runs
execute.
> - A recurring chat pass scans saved run events even when no progress
changes.
> - This traffic prevents an idle database from staying idle.
> - Events, lineage changes, and publication state already identify when
projection can change.
> - This PR gives native progress its own commit-driven worker and
schedules only unfinished work.
> - Idle progress stops scanning while restart and recovery preserve
delivery.

## Linked Issues or Issue Description

Refs #17606. This is slice 29 of the recurring-server-work stack, based
on `codex/sleep-28-chat-run-milestones`. It owns native chat progress
only. Related historical optimization: #16845. The parent and this PR
must land in stack order.

**What happened?**

Native chat progress is still called by the one-second chat
reconciliation loop. It repeatedly selects run events with no new work.
Its cursor also truncates PostgreSQL timestamp precision, which can
repeat a rejected full page forever.

**Expected behavior**

Committed events and eligibility changes should wake the projection
owner. Existing rows should reconstruct work after restart. A settled
worker should have no recurring database activity. Full batches, lock
contention, and errors must retain recovery.

**Steps to reproduce**

Start an otherwise idle instance with chat history. Observe the native
progress lane in the recurring chat coordinator. For the cursor defect,
create a full page of suppressed candidates sharing a microsecond
timestamp and one eligible later candidate. The old cursor fails to
advance.

## What Changed

- Extract native progress from lifecycle milestones into its owning
service. Keep the public combined API and budget compatible.
- Register a scoped worker with the existing delivery coordinator and
scheduler. Remove native progress from the recurring chat coordinator.
- Notify before allowlisted event writes in both persistence owners.
Preserve outer commit, rollback, duplicate detection, and the ordinary
non-progress log fast path.
- Extend the existing lineage/authority notification boundary once.
Failed-publication replay and operator retry carry their issue scope.
First working publications signal progress so commit ordering cannot
strand an already-inspected event.
- Limit notification payloads to company/task IDs so wider caller inputs
never serialize database handles or private fields. Load task-binding
fields only for progress event kinds; other native/recovery events
retain their narrow lock query.
- Preserve current safe phrases, cadence, phase deduplication, provider
policy, authorization, and final-output precedence.
- Retry concrete issue/run lock contention. Disarm deleted or terminal
candidates. Preserve scoped dirty work across batches, failures, drain
and shutdown.
- Preserve PostgreSQL timestamp precision in pagination. Add a bounded
regression for the former infinite rejected-page scan.
- Document the producer audit, lifecycle contract, failures, and
remaining limits in
`doc/plans/2026-10-25-native-chat-progress-sleep-audit.md`.

## Verification

Published head: `b7ee820e49fd06c780b36b79d9b1c0883557fb73`, targeting
master after #17606 merged at
`5f0da562dbfdf3229f3cd4bb3dc742ab541986b4`. A normal ancestry merge
preserves the complete 18-path native-progress patch byte-for-byte
relative to its verified parent, with all six unrelated current master
paths intact. No new production change was introduced during final
reconciliation.

- **163 distinct selected cases pass on the verified unchanged patch:**
137 across eight complete suites (native progress, atomic event append,
all five milestone worker suites, and app admission wiring), plus 26
real chat integration cases selected by `native progress`. The other
1,595 chat integration cases were skipped, not passed.
- The full worker suites cover restart, committed events, trace
reprojection, duplicate/no-progress events, first-working publication
ordering, concrete lock contention, issue scope, backlog, drain/resume,
failures and shutdown. Settled history leaves no recurring SQL. The real
chat cases verify operator retry and replay, microsecond pagination,
authority, cadence, phase deduplication, final/question precedence and
redaction.
- The sole earlier parent conflict preserves both this slice's
issue-scoped retry wake and the exact voice conversation scope. The
current parent's milestone producers, onboarding completion guard,
identity query/read-scope coverage, migration 0397 and voice
cleanup/retention repairs remain present.
- Server production typechecking, build and module-boundary checks pass.
The selected server tests live under the excluded `src/__tests__` path
and were executed separately. Each database owner uses a new isolated
app-migrated PostgreSQL fixture.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/b7ee820e49fd06c780b36b79d9b1c0883557fb73/doc/plans/2026-10-25-native-chat-progress-sleep-audit.md#current-milestone-parent-acceptance-october-28)
records current counts and scope. Older combined checks and failures
retain their original revision scope and are not added to 163. No new
table, dependency, workflow, migration, or scheduler contract is
introduced.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing verdict is waived. Older head/parent reviews are
not current passes. Actual failing Ponytail findings must be fixed.
Missing/stopped/paused turns without a verdict are authorized
nonblockers; active unfinished reviews retain the original per-head
timer with a 25-minute monitor cutoff respecting the 30-minute maximum.
- No complete local repository matrix, live provider tenant, external
wake, physical host/database suspension or savings claim is made.
Following stack duties remain separate.

## Risks

- Signals are process-local under the existing single active writer
contract. Independent SQL writers or another active process need an
explicit wake integration.
- Allowlisted direct log writes now use a transaction. Native bound runs
register transaction intent; unrelated direct event types retain their
atomic-statement path.
- The shared eligibility boundary wakes both lifecycle and native
projection. It does not decide provider policy or copy event payloads.
- The existing 24-hour selection window, event-time cadence and
authority checks remain. A suppressed event is not retried just because
wall-clock time passes.
- This establishes worker quiet, not whole-instance suspension. Cloud
alarm coverage and conservative sleep gates do not change. Other chat
maintenance remains on its existing cadence.
- This actual-master reconciliation still requires fresh exact-head
review and CI before protected merge. Subsequent slices retain their own
gates.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code execution
and browser-test tools. The exact serving variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:22:29 -05:00