From 689d62661d6d1fbf6ae6a4e20e5124888fa12dc0 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 06:02:54 -0600 Subject: [PATCH] fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for ` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- packages/db/src/work-signals.test.ts | 16 +++++ .../delivery-work-coordinator.test.ts | 66 ++++++++++++++++++- .../src/services/delivery-work-coordinator.ts | 8 ++- 3 files changed, 88 insertions(+), 2 deletions(-) diff --git a/packages/db/src/work-signals.test.ts b/packages/db/src/work-signals.test.ts index aabf0529911..b46e2c74b08 100644 --- a/packages/db/src/work-signals.test.ts +++ b/packages/db/src/work-signals.test.ts @@ -472,6 +472,22 @@ describe("database work scope identity", () => { } finally { await other.close(); } }); + it("scopes an unhinted signal alongside a hinted one to the hinted set, locally and in another process", async () => { + const other = receiving(["alongside"]); + const local: Array = []; + const stop = subscribeDatabaseWork(db, "alongside", (event, hints) => { if (event === "settled") local.push(hints ? [...hints] : null); }); + try { + const result = await run(root => root.transaction(async tx => { + await signalDatabaseWork(tx, "alongside"); + await signalDatabaseWork(tx, "alongside", "company-1"); + })); + expect(local).toEqual([["company-1"]]); + expect(result.payloads.map(payload => JSON.parse(payload).hints)).toEqual([["company-1"]]); + other.deliver(result.payloads); + expect(other.wakes).toEqual([["alongside", ["company-1"]]]); + } finally { stop(); await other.close(); } + }); + it("splits many hints into payloads an older receiver still reads, within the per-topic byte cap", async () => { const hints = Array.from({ length: 80 }, (_, index) => JSON.stringify({ companyId: `company-${String(index).padStart(4, "0")}` })); const other = receiving(["many"]); diff --git a/server/src/__tests__/delivery-work-coordinator.test.ts b/server/src/__tests__/delivery-work-coordinator.test.ts index 6fb1ca3471f..7cacd3f681f 100644 --- a/server/src/__tests__/delivery-work-coordinator.test.ts +++ b/server/src/__tests__/delivery-work-coordinator.test.ts @@ -1,5 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; +import { installDatabaseWorkSignals, publishDatabaseWorkSettled } from "../../../packages/db/src/work-signals.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { notifyDeliveryWork, DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { beginIdleTrackedWork, idleWorkSnapshot, isIdleTaskDrainActive, readTaskDrain, startTaskDrain, stopTaskDrain } from "../services/task-admission.js"; @@ -399,3 +399,67 @@ it("invalidates owner reconciliation synchronously before awaiting an active sto finish(); await worker.ready; await stopping; expect(stopped).toBe(true); }); + +describe("settled hints a task cannot apply", () => { + function settledFixture(onSettled: (hints?: ReadonlySet) => void) { + const s = setup(); + const t = { ...task(), onSettled: vi.fn(onSettled) }; + const worker = s.coordinator.register(DELIVERY_QUEUES.feedback, t); + return { s, t, worker }; + } + const received = (mock: ReturnType) => mock.mock.calls.map(([hints]) => hints === undefined ? null : [...hints as ReadonlySet]); + + it("still wakes the task, unscoped, and logs a warning when it rejects a hint", async () => { + const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {}); + try { + const { s, t, worker } = settledFixture(hints => { + if (hints && [...hints].some(hint => !hint.startsWith("company:"))) throw new Error("unknown hint shape"); + }); + await worker.ready; + t.run.mockClear(); + publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "{\"newShape\":1}"); + await vi.advanceTimersByTimeAsync(1); + expect(received(t.onSettled)).toEqual([["{\"newShape\":1}"], null]); + expect(t.run).toHaveBeenCalledOnce(); + expect(warn).toHaveBeenCalledOnce(); + expect(String(warn.mock.calls[0]![0])).toContain(`Delivery settlement could not be applied for ${DELIVERY_QUEUES.feedback}`); + expect(s.onError).not.toHaveBeenCalled(); + } finally { warn.mockRestore(); } + }); + + it("wakes the task once without retrying when even the unscoped settlement throws", async () => { + const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {}); + try { + const { s, t, worker } = settledFixture(() => { throw new Error("broken"); }); + await worker.ready; + t.run.mockClear(); + publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1"); + await vi.advanceTimersByTimeAsync(1); + expect(received(t.onSettled)).toEqual([["company:1"], null]); + expect(t.run).toHaveBeenCalledOnce(); + expect(warn).toHaveBeenCalledTimes(2); + t.onSettled.mockClear(); warn.mockClear(); + publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback); + await vi.advanceTimersByTimeAsync(1); + expect(received(t.onSettled)).toEqual([null]); + expect(t.run).toHaveBeenCalledTimes(2); + expect(warn).toHaveBeenCalledOnce(); + await vi.advanceTimersByTimeAsync(60_000); + expect(t.run).toHaveBeenCalledTimes(2); + } finally { warn.mockRestore(); } + }); + + it("passes recognised hints through once, without a warning", async () => { + const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {}); + try { + const { s, t, worker } = settledFixture(() => {}); + await worker.ready; + t.run.mockClear(); + publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1"); + await vi.advanceTimersByTimeAsync(1); + expect(received(t.onSettled)).toEqual([["company:1"]]); + expect(t.run).toHaveBeenCalledOnce(); + expect(warn).not.toHaveBeenCalled(); + } finally { warn.mockRestore(); } + }); +}); diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 7512f3768fd..a9bd83e5cd0 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -110,6 +110,12 @@ export function createDeliveryWorkCoordinator(input: { dirty = true; if (!running) schedule(task.coalesceMs ?? 0); } + function applySettled(hints?: ReadonlySet) { + for (const attempt of hints === undefined ? [undefined] : [hints, undefined]) { + try { task.onSettled?.(attempt); return; } + catch (error) { process.emitWarning(`Delivery settlement could not be applied for ${queue}: ${String(error)}`); } + } + } const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => { if (workerStopped) return; if (databaseWorkPending(input.owner, queue)) { @@ -118,7 +124,7 @@ export function createDeliveryWorkCoordinator(input: { // replaces this deadline with an immediate wake on the normal path. if (!running) schedule(task.retryMs); } - if (event === "settled") { task.onSettled?.(hints); wake(); } + if (event === "settled") { applySettled(hints); wake(); } }); const worker = { wake, async stop() { workerStopped = true;