PaperClipAI/doc/plans/2026-10-27-inbound-queue-query-proof.md

Inbound queue query proof

Contract and cause

Paperclip's inbound queue discovers its next deadline, selects each conversation head, and selects ordinary ready deliveries. A retained processed delivery can still own the head while an issued or processing inbound_wakeup action remains. This work keeps that obligation, company and endpoint scope, lease expiry, retry timestamps, ordering keys, lifecycle dependencies, and provider exclusions. It adds no cache, migration, producer, worker, or timer.

The two open-work selectors used a correlated EXISTS inside an OR. PostgreSQL 17.11 already used a hashed subplan in the measured fixture. The original hypothesis of repeated physical action scans was wrong. The estimate still included a large correlated alternative. It can enable JIT at the normal 100000 cost threshold. The deadline query also evaluated JSON conditions on retained processed history before rejecting rows with no pending action.

The treatment uses a set of (company_id, endpoint_id, delivery_id) for pending ownership. Null delivery IDs cannot match and are excluded, so the predicate still returns a boolean. Duplicate actions do not multiply delivery rows. Ready-only probes retain their original scoped EXISTS. This keeps the readiness and retry parsing boundary outside this optimization.

Reproduction

Run from the source checkout with its dependencies built:

pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e --require-work-budget

The harness starts disposable PostgreSQL through the repository test helper. It does not read a live database URL. It loads the pinned pre-fix queue, entry-point, and ledger sources. Other runtime helpers and dependencies must match. It captures the complete SQL and parameters through the real service, including projections, ordering, limits, and the head probe's lease path. It stops before provider effects. Every selected column is compared as a positional row on one reserved SQL connection. Fixture deadlines are fixed. The small control has 20 completed deliveries and actions. The retained-history case has 1500 of each, two open deliveries, and 32 KiB of synthetic incompressible message text per historical delivery.

Two warmups precede five alternating samples per arm. Plans use EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) outside the timed samples. The explicit fixture work budget rejects JIT for the two changed selectors and more than 1000 shared buffer accesses (hits plus reads) for the deadline selection. It has no CI timing threshold. Use --jit-off for a diagnostic control.

Local evidence uses Node 26.4.0 and Linux ARM64 PostgreSQL 17.11 in a disposable Docker container, matching the hosted major and minor version. The normal settings are shared_buffers 128 MiB, work_mem 4 MiB, JIT on, jit_above_cost 100000, and random_page_cost 4. Native macOS PostgreSQL could not start because the host had exhausted its SysV semaphore allocation. The scratch provider replaces only the test PostgreSQL constructor. Schema migrations, clients and query execution remain real. No persistent database is used.

Results

Matched pre-fix baseline: 09ad9adce2cf4742944353a3707a2f908e9bb35e. An earlier baseline at 96727b3 had identical ledger and conversation-queue sources. Master changed an unrelated health notification import in the service entry-point, so the final comparison was repeated from the newer pre-fix base. The ordinary-ready selector is an unchanged control. The head query still reads its existing JSON ordering/filter fields; do not claim it reduces physical history scans. Local SQL timings do not prove a hosted request speedup.

Final quiet PostgreSQL 17.11 comparison, in milliseconds. Each arm has two warmups and five alternating samples. Parentheses show minimum–maximum.

Actual selector	Baseline median (range)	Treatment median (range)	Ratio	Equality
deadline	18.78 (18.43–20.41)	1.58 (1.14–1.75)	11.90x	2 complete rows equal
head	5.71 (5.38–6.40)	6.18 (5.56–6.72)	0.92x	1 complete rows equal
ready	9.23 (7.97–9.41)	8.33 (7.72–9.21)	1.11x	1 complete rows equal

• deadline: shared buffer hits 10662 → 455; JIT present → absent.
• head: shared buffer hits 10600 → 10600; JIT absent → absent.
• ready: shared buffer hits 66 → 66; JIT present → present.

The matched small control is neutral (overlapping sample ranges). The JIT-off control remains diagnostic under other local build load: deadline 9.72 → 2.17 ms; head 9.32 → 9.08 ms; ordinary ready 1.52 → 1.56 ms. The last two ranges overlap. It separates deadline physical work from an earlier JIT-only head improvement. The earlier small-then-large fixture put the head estimate at 100369.79 and enabled JIT. The final pristine retained-only fixture put it at 98866.79, below the threshold. Its head timing is neutral. Plan-cost threshold crossings are data-sensitive; do not claim a head speedup for the final fixture.

Feedback loop

Question	Loop	Measured elapsed	Decision
Can the real source run locally?	Cached offline install, isolated build	27.7 s install; 141 s build log span	Reuse the built checkout for each narrow attempt
Does the initial scan hypothesis hold?	Captured queries and real PG plans	4.0 s inside the instrumented baseline harness, plus module loading	Reject repeated physical scans as the cause
Does the set improve the actual selectors?	Red work budget, alternating pair, JIT-off control	28.36 s for all three processes, including module and DB setup	Exploratory variant identified JIT cost; final scope keeps ready-only unchanged
Does a selective ready probe remain scoped?	Original and treatment SQL with unrelated malformed retry data	12 s focused attempt; 28 s after replay/module rebuild	Direct head probes return empty; actual OR selectors both return error 22007; keep readiness unchanged

Coordination waits for a quiet benchmark interval are separate from these command measurements. They are not a reason to run noisy parallel timing samples. The small real-database regression loop takes about 11 seconds including transforms and startup. Full repository checks and exact-head CI remain required.

Scope of benefit

The discovery sample attributed 4.731 summed SQL seconds to 90 deadline selections and 4.681 seconds to 97 head selections over about five minutes. The final pristine fixture proves a deadline improvement; the head is neutral at that data point. The ordinary-ready control accounted for a further 1.389 seconds over 90 calls. These are background selectors, not a correlated request critical path. The maximum improvement to a single affected selection cannot exceed its observed SQL duration (about 49–53 ms on average). No end-to-end latency or rollout claim follows from adding those overlapping aggregate times. The change is small and avoids a recurring cost under the hosted planner defaults, so it is worth shipping with the other proven fixes.

Correctness and failure accounting

Five real-PostgreSQL tests cover tuple scope, duplicate/null actions, exact boolean results, full payloads, future head ownership, independent reactions, paused/attention/agentmail exclusions, retry/processing deadlines, and provider sequence ordering. Six existing assembled chat tests also pass. They cover lease renewal failure, reverse-arrival Slack and Telegram events, lifecycle retry, edit/admission races, and deadline restoration.

An exploratory malformed-retry test compared an actual OR selector against an AND-only oracle. That comparison was inconclusive. It must not be presented as proof of an introduced defect. The complete actual old and new OR selectors both return PostgreSQL error 22007 for unrelated malformed pending actions in the measured fixture, including other-company and null-delivery controls. The selective direct head probe returns empty on both versions. The final fix leaves ready-only evaluation unchanged.

The initial repeated-action-scan hypothesis was also rejected. Both arms scan action history once in the measured plans. The benefits come from query costing and filter order, rather than fewer physical action scans.

Final verification

• pnpm -r typecheck: pass.
• pnpm build: pass.
• New and selected existing assembled PostgreSQL 17 suites: 11 passed.
• Complete local suite: not run because the native host PostgreSQL startup blocker is already established; complete final-head CI remains the gate.
• Final retained-history benchmark work budget: pass, complete positional equality for all three selectors.
• The final benchmark process took 34.19 seconds from its instrumented setup through proof. This includes the coordination wait after the fixture and queries were ready. The actual timing phase was brief.

The PR contains the source, fixture and harness. Raw local JSON includes every sample, query, parameter and plan. No hosted rollout was performed.

Additional failures retained

Greptile's first review found two proof gaps: the budget counted only cache hits, and the payload test read the row again instead of checking the captured selection. Both are fixed. The budget includes hits plus reads. The test compares the captured row's complete normalized payload with its seeded object.

A prepared timing attempt lost its idle pooled connection while waiting. Its now()-derived fixture value differed between queries, so equality stopped the run before samples. The harness now reserves its timing connection and uses an explicit follower deadline. The final quiet comparison passes.

The first CI head failed e2e shard 3 in agent-chat-projects cleanup: cancelling a heartbeat run timed out on a row lock and returned HTTP 500. This is outside the changed queue selector. The final head must still pass that check. The original log remains at https://github.com/paperclipai/paperclip/actions/runs/40624302918/job/122605509182.