fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The delivery work coordinator wakes background tasks when a database
work topic settles, and passes the settled scope hints to the task's
`onSettled`
> - The coordinator calls `onSettled(hints)` before `wake()`
> - If `onSettled` throws on a hint that it does not recognise, the task
is not woken. Only a generic listener warning is logged
> - Hints now cross server processes (#21681), so a hint shape from
another server version can arrive during a deploy overlap
> - This pull request makes the coordinator log a warning, apply the
settlement unscoped once, and always wake the task
> - The benefit is that a hint that a task cannot read never makes the
task miss committed work

## Linked Issues or Issue Description

Follow-up to #21681 (merged). This branch is based on master.

**What existing behavior does this improve?**
How the delivery work coordinator handles an `onSettled` callback that
throws.

**Subsystem affected**
`server/src/services/delivery-work-coordinator.ts`.

**Current behavior**
`subscribeDatabaseWork` calls the coordinator listener, which calls
`task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws,
`wake()` does not run. The work-signal publisher catches the error and
logs "Database work listener failed". The task stays asleep until a
different event or deadline wakes it.

**Proposed behavior**
If `onSettled(hints)` throws, the coordinator logs a warning and calls
`onSettled(undefined)` once, which is the unscoped settlement. If that
also throws, it logs a second warning. In all cases it then wakes the
task.

**Reason and benefit**
A settled event means committed work can be ready. A task must never
miss it because a hint could not be applied.

**Breaking changes**
None.

## What Changed

- `delivery-work-coordinator.ts`: a settled event goes through
`applySettled(hints)`. It tries `onSettled(hints)`. If that throws and
hints were given, it tries `onSettled(undefined)` one time. Each failure
logs `Delivery settlement could not be applied for <queue>` with
`process.emitWarning`. Then `wake()` runs.
- Unscoped means `onSettled(undefined)`. Each registered task already
defines what that means. For example, scoped deadline queues, dispatch
work, muse maintenance and the revocation index mark `*` dirty.
Counter-style tasks set their request flag. `chat-action-work` does
nothing. The fallback is bounded: it calls `onSettled` at most twice per
event, it does not recurse, and it does not schedule another settlement.
So it cannot loop.
- `work-signals.test.ts`: a new test pins one rule of #21681 for the
local path and the remote path. An unhinted `signalDatabaseWork` in the
same transaction as a hinted one gives a settlement that is scoped to
the hinted set. The owner of `work-signals.ts` confirmed that this is
deliberate. The repository does not allow code comments in pull
requests, so this sentence and the test document it.

## Verification

- `cd server && npx vitest run
src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests:
- An `onSettled` that throws on an unknown hint shape: the task gets
`[hint]` and then the unscoped settlement, it is woken once, and one
warning is logged. `onError` is not called.
- An `onSettled` that always throws: the task gets 2 calls and is woken
once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1
warning. No more runs happen after 60 seconds, so there is no loop.
- A normal `onSettled`: it gets the hints once, the task is woken once,
and no warning is logged.
- Red/green: with the coordinator from master, the 2 throwing tests
fail. `onSettled` is called only with the hints, and there is no
unscoped settlement and no wake. With this branch, all tests pass.
- `cd packages/db && npx vitest run src/work-signals.test.ts -t
alongside`: passes on real PostgreSQL. The local listener and the
receiving process both get `["company-1"]`.

## Risks

- Low risk. The change applies only when `onSettled` throws. Before,
that path skipped the wake. Now it wakes with an unscoped settlement.
For scoped deadline queues an unscoped settlement is a `*` reload. That
is the existing safe fallback, and the reconstruction interval of the
queue still throttles it.
- A task whose `onSettled` throws on every event now runs on every
settled event and logs a warning each time. Before, it did not run. The
warning makes such a defect visible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-15 06:02:54 -06:00
1 parent fb3b09aaef
commit 689d62661d
3 files changed
+88 -2

No files matched your search

+16
View File
@@ -472,6 +472,22 @@ describe("database work scope identity", () => {
} finally { await other.close(); }
});
it("scopes an unhinted signal alongside a hinted one to the hinted set, locally and in another process", async () => {
const other = receiving(["alongside"]);
const local: Array<string[] | null> = [];
const stop = subscribeDatabaseWork(db, "alongside", (event, hints) => { if (event === "settled") local.push(hints ? [...hints] : null); });
try {
const result = await run(root => root.transaction(async tx => {
await signalDatabaseWork(tx, "alongside");
await signalDatabaseWork(tx, "alongside", "company-1");
}));
expect(local).toEqual([["company-1"]]);
expect(result.payloads.map(payload => JSON.parse(payload).hints)).toEqual([["company-1"]]);
other.deliver(result.payloads);
expect(other.wakes).toEqual([["alongside", ["company-1"]]]);
} finally { stop(); await other.close(); }
});
it("splits many hints into payloads an older receiver still reads, within the per-topic byte cap", async () => {
const hints = Array.from({ length: 80 }, (_, index) => JSON.stringify({ companyId: `company-${String(index).padStart(4, "0")}` }));
const other = receiving(["many"]);
@@ -1,5 +1,5 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
import { installDatabaseWorkSignals, publishDatabaseWorkSettled } from "../../../packages/db/src/work-signals.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { beginIdleTrackedWork, idleWorkSnapshot, isIdleTaskDrainActive, readTaskDrain, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
@@ -399,3 +399,67 @@ it("invalidates owner reconciliation synchronously before awaiting an active sto
finish(); await worker.ready; await stopping;
expect(stopped).toBe(true);
});
describe("settled hints a task cannot apply", () => {
function settledFixture(onSettled: (hints?: ReadonlySet<string>) => void) {
const s = setup();
const t = { ...task(), onSettled: vi.fn(onSettled) };
const worker = s.coordinator.register(DELIVERY_QUEUES.feedback, t);
return { s, t, worker };
}
const received = (mock: ReturnType<typeof vi.fn>) => mock.mock.calls.map(([hints]) => hints === undefined ? null : [...hints as ReadonlySet<string>]);
it("still wakes the task, unscoped, and logs a warning when it rejects a hint", async () => {
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
try {
const { s, t, worker } = settledFixture(hints => {
if (hints && [...hints].some(hint => !hint.startsWith("company:"))) throw new Error("unknown hint shape");
});
await worker.ready;
t.run.mockClear();
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "{\"newShape\":1}");
await vi.advanceTimersByTimeAsync(1);
expect(received(t.onSettled)).toEqual([["{\"newShape\":1}"], null]);
expect(t.run).toHaveBeenCalledOnce();
expect(warn).toHaveBeenCalledOnce();
expect(String(warn.mock.calls[0]![0])).toContain(`Delivery settlement could not be applied for ${DELIVERY_QUEUES.feedback}`);
expect(s.onError).not.toHaveBeenCalled();
} finally { warn.mockRestore(); }
});
it("wakes the task once without retrying when even the unscoped settlement throws", async () => {
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
try {
const { s, t, worker } = settledFixture(() => { throw new Error("broken"); });
await worker.ready;
t.run.mockClear();
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1");
await vi.advanceTimersByTimeAsync(1);
expect(received(t.onSettled)).toEqual([["company:1"], null]);
expect(t.run).toHaveBeenCalledOnce();
expect(warn).toHaveBeenCalledTimes(2);
t.onSettled.mockClear(); warn.mockClear();
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback);
await vi.advanceTimersByTimeAsync(1);
expect(received(t.onSettled)).toEqual([null]);
expect(t.run).toHaveBeenCalledTimes(2);
expect(warn).toHaveBeenCalledOnce();
await vi.advanceTimersByTimeAsync(60_000);
expect(t.run).toHaveBeenCalledTimes(2);
} finally { warn.mockRestore(); }
});
it("passes recognised hints through once, without a warning", async () => {
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
try {
const { s, t, worker } = settledFixture(() => {});
await worker.ready;
t.run.mockClear();
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1");
await vi.advanceTimersByTimeAsync(1);
expect(received(t.onSettled)).toEqual([["company:1"]]);
expect(t.run).toHaveBeenCalledOnce();
expect(warn).not.toHaveBeenCalled();
} finally { warn.mockRestore(); }
});
});
@@ -110,6 +110,12 @@ export function createDeliveryWorkCoordinator(input: {
dirty = true;
if (!running) schedule(task.coalesceMs ?? 0);
}
function applySettled(hints?: ReadonlySet<string>) {
for (const attempt of hints === undefined ? [undefined] : [hints, undefined]) {
try { task.onSettled?.(attempt); return; }
catch (error) { process.emitWarning(`Delivery settlement could not be applied for ${queue}: ${String(error)}`); }
}
}
const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => {
if (workerStopped) return;
if (databaseWorkPending(input.owner, queue)) {
@@ -118,7 +124,7 @@ export function createDeliveryWorkCoordinator(input: {
// replaces this deadline with an immediate wake on the normal path.
if (!running) schedule(task.retryMs);
}
if (event === "settled") { task.onSettled?.(hints); wake(); }
if (event === "settled") { applySettled(hints); wake(); }
});
const worker = { wake, async stop() {
workerStopped = true;