mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for <queue>` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fb3b09aaef
commit
689d62661d
3 files changed
+88
-2
No files matched your search
@@ -472,6 +472,22 @@ describe("database work scope identity", () => {
|
||||
} finally { await other.close(); }
|
||||
});
|
||||
|
||||
it("scopes an unhinted signal alongside a hinted one to the hinted set, locally and in another process", async () => {
|
||||
const other = receiving(["alongside"]);
|
||||
const local: Array<string[] | null> = [];
|
||||
const stop = subscribeDatabaseWork(db, "alongside", (event, hints) => { if (event === "settled") local.push(hints ? [...hints] : null); });
|
||||
try {
|
||||
const result = await run(root => root.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, "alongside");
|
||||
await signalDatabaseWork(tx, "alongside", "company-1");
|
||||
}));
|
||||
expect(local).toEqual([["company-1"]]);
|
||||
expect(result.payloads.map(payload => JSON.parse(payload).hints)).toEqual([["company-1"]]);
|
||||
other.deliver(result.payloads);
|
||||
expect(other.wakes).toEqual([["alongside", ["company-1"]]]);
|
||||
} finally { stop(); await other.close(); }
|
||||
});
|
||||
|
||||
it("splits many hints into payloads an older receiver still reads, within the per-topic byte cap", async () => {
|
||||
const hints = Array.from({ length: 80 }, (_, index) => JSON.stringify({ companyId: `company-${String(index).padStart(4, "0")}` }));
|
||||
const other = receiving(["many"]);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
|
||||
import { installDatabaseWorkSignals, publishDatabaseWorkSettled } from "../../../packages/db/src/work-signals.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { notifyDeliveryWork, DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { beginIdleTrackedWork, idleWorkSnapshot, isIdleTaskDrainActive, readTaskDrain, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
||||
@@ -399,3 +399,67 @@ it("invalidates owner reconciliation synchronously before awaiting an active sto
|
||||
finish(); await worker.ready; await stopping;
|
||||
expect(stopped).toBe(true);
|
||||
});
|
||||
|
||||
describe("settled hints a task cannot apply", () => {
|
||||
function settledFixture(onSettled: (hints?: ReadonlySet<string>) => void) {
|
||||
const s = setup();
|
||||
const t = { ...task(), onSettled: vi.fn(onSettled) };
|
||||
const worker = s.coordinator.register(DELIVERY_QUEUES.feedback, t);
|
||||
return { s, t, worker };
|
||||
}
|
||||
const received = (mock: ReturnType<typeof vi.fn>) => mock.mock.calls.map(([hints]) => hints === undefined ? null : [...hints as ReadonlySet<string>]);
|
||||
|
||||
it("still wakes the task, unscoped, and logs a warning when it rejects a hint", async () => {
|
||||
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
|
||||
try {
|
||||
const { s, t, worker } = settledFixture(hints => {
|
||||
if (hints && [...hints].some(hint => !hint.startsWith("company:"))) throw new Error("unknown hint shape");
|
||||
});
|
||||
await worker.ready;
|
||||
t.run.mockClear();
|
||||
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "{\"newShape\":1}");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(received(t.onSettled)).toEqual([["{\"newShape\":1}"], null]);
|
||||
expect(t.run).toHaveBeenCalledOnce();
|
||||
expect(warn).toHaveBeenCalledOnce();
|
||||
expect(String(warn.mock.calls[0]![0])).toContain(`Delivery settlement could not be applied for ${DELIVERY_QUEUES.feedback}`);
|
||||
expect(s.onError).not.toHaveBeenCalled();
|
||||
} finally { warn.mockRestore(); }
|
||||
});
|
||||
|
||||
it("wakes the task once without retrying when even the unscoped settlement throws", async () => {
|
||||
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
|
||||
try {
|
||||
const { s, t, worker } = settledFixture(() => { throw new Error("broken"); });
|
||||
await worker.ready;
|
||||
t.run.mockClear();
|
||||
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(received(t.onSettled)).toEqual([["company:1"], null]);
|
||||
expect(t.run).toHaveBeenCalledOnce();
|
||||
expect(warn).toHaveBeenCalledTimes(2);
|
||||
t.onSettled.mockClear(); warn.mockClear();
|
||||
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(received(t.onSettled)).toEqual([null]);
|
||||
expect(t.run).toHaveBeenCalledTimes(2);
|
||||
expect(warn).toHaveBeenCalledOnce();
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(t.run).toHaveBeenCalledTimes(2);
|
||||
} finally { warn.mockRestore(); }
|
||||
});
|
||||
|
||||
it("passes recognised hints through once, without a warning", async () => {
|
||||
const warn = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
|
||||
try {
|
||||
const { s, t, worker } = settledFixture(() => {});
|
||||
await worker.ready;
|
||||
t.run.mockClear();
|
||||
publishDatabaseWorkSettled(s.owner, DELIVERY_QUEUES.feedback, "company:1");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(received(t.onSettled)).toEqual([["company:1"]]);
|
||||
expect(t.run).toHaveBeenCalledOnce();
|
||||
expect(warn).not.toHaveBeenCalled();
|
||||
} finally { warn.mockRestore(); }
|
||||
});
|
||||
});
|
||||
@@ -110,6 +110,12 @@ export function createDeliveryWorkCoordinator(input: {
|
||||
dirty = true;
|
||||
if (!running) schedule(task.coalesceMs ?? 0);
|
||||
}
|
||||
function applySettled(hints?: ReadonlySet<string>) {
|
||||
for (const attempt of hints === undefined ? [undefined] : [hints, undefined]) {
|
||||
try { task.onSettled?.(attempt); return; }
|
||||
catch (error) { process.emitWarning(`Delivery settlement could not be applied for ${queue}: ${String(error)}`); }
|
||||
}
|
||||
}
|
||||
const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => {
|
||||
if (workerStopped) return;
|
||||
if (databaseWorkPending(input.owner, queue)) {
|
||||
@@ -118,7 +124,7 @@ export function createDeliveryWorkCoordinator(input: {
|
||||
// replaces this deadline with an immediate wake on the normal path.
|
||||
if (!running) schedule(task.retryMs);
|
||||
}
|
||||
if (event === "settled") { task.onSettled?.(hints); wake(); }
|
||||
if (event === "settled") { applySettled(hints); wake(); }
|
||||
});
|
||||
const worker = { wake, async stop() {
|
||||
workerStopped = true;
|
||||
|
||||
Reference in new issue
Block a user