mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.
**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.
**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.
**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.
**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.
**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).
## What Changed
- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.
## Verification
- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
- overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
- the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.
## Risks
- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>