Files
PaperClipAI/server/src/services/delivery-work-coordinator.ts
T
DottaandClaude Opus 5.5 689d62661d fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The delivery work coordinator wakes background tasks when a database
work topic settles, and passes the settled scope hints to the task's
`onSettled`
> - The coordinator calls `onSettled(hints)` before `wake()`
> - If `onSettled` throws on a hint that it does not recognise, the task
is not woken. Only a generic listener warning is logged
> - Hints now cross server processes (#21681), so a hint shape from
another server version can arrive during a deploy overlap
> - This pull request makes the coordinator log a warning, apply the
settlement unscoped once, and always wake the task
> - The benefit is that a hint that a task cannot read never makes the
task miss committed work

## Linked Issues or Issue Description

Follow-up to #21681 (merged). This branch is based on master.

**What existing behavior does this improve?**
How the delivery work coordinator handles an `onSettled` callback that
throws.

**Subsystem affected**
`server/src/services/delivery-work-coordinator.ts`.

**Current behavior**
`subscribeDatabaseWork` calls the coordinator listener, which calls
`task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws,
`wake()` does not run. The work-signal publisher catches the error and
logs "Database work listener failed". The task stays asleep until a
different event or deadline wakes it.

**Proposed behavior**
If `onSettled(hints)` throws, the coordinator logs a warning and calls
`onSettled(undefined)` once, which is the unscoped settlement. If that
also throws, it logs a second warning. In all cases it then wakes the
task.

**Reason and benefit**
A settled event means committed work can be ready. A task must never
miss it because a hint could not be applied.

**Breaking changes**
None.

## What Changed

- `delivery-work-coordinator.ts`: a settled event goes through
`applySettled(hints)`. It tries `onSettled(hints)`. If that throws and
hints were given, it tries `onSettled(undefined)` one time. Each failure
logs `Delivery settlement could not be applied for <queue>` with
`process.emitWarning`. Then `wake()` runs.
- Unscoped means `onSettled(undefined)`. Each registered task already
defines what that means. For example, scoped deadline queues, dispatch
work, muse maintenance and the revocation index mark `*` dirty.
Counter-style tasks set their request flag. `chat-action-work` does
nothing. The fallback is bounded: it calls `onSettled` at most twice per
event, it does not recurse, and it does not schedule another settlement.
So it cannot loop.
- `work-signals.test.ts`: a new test pins one rule of #21681 for the
local path and the remote path. An unhinted `signalDatabaseWork` in the
same transaction as a hinted one gives a settlement that is scoped to
the hinted set. The owner of `work-signals.ts` confirmed that this is
deliberate. The repository does not allow code comments in pull
requests, so this sentence and the test document it.

## Verification

- `cd server && npx vitest run
src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests:
- An `onSettled` that throws on an unknown hint shape: the task gets
`[hint]` and then the unscoped settlement, it is woken once, and one
warning is logged. `onError` is not called.
- An `onSettled` that always throws: the task gets 2 calls and is woken
once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1
warning. No more runs happen after 60 seconds, so there is no loop.
- A normal `onSettled`: it gets the hints once, the task is woken once,
and no warning is logged.
- Red/green: with the coordinator from master, the 2 throwing tests
fail. `onSettled` is called only with the hints, and there is no
unscoped settlement and no wake. With this branch, all tests pass.
- `cd packages/db && npx vitest run src/work-signals.test.ts -t
alongside`: passes on real PostgreSQL. The local listener and the
receiving process both get `["company-1"]`.

## Risks

- Low risk. The change applies only when `onSettled` throws. Before,
that path skipped the wake. Now it wakes with an unscoped settlement.
For scoped deadline queues an unscoped settlement is a `*` reload. That
is the existing safe fallback, and the reconstruction interval of the
queue still throttles it.
- A task whose `onSettled` throws on every event now runs on every
settled event and logs a warning each time. Before, it did not run. The
warning makes such a defect visible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:02:54 -06:00

156 lines
7.5 KiB
TypeScript

import { subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "@paperclipai/db/work-signals";
import { withDatabaseOperation } from "@paperclipai/db";
import { createWorkScheduler } from "./work-scheduler.js";
import { type DeliveryQueue } from "./delivery-work-notifications.js";
import { beginIdleTrackedWork } from "./task-admission.js";
export function createDeliveryWorkCoordinator(input: {
owner: object;
canRun: () => boolean;
// Idle drain may reconcile admitted writes without dispatching new work.
// Warm standby can disable both operations.
canReconcile?: () => boolean;
onError: (error: unknown, queue: DeliveryQueue) => void;
}) {
const scheduler = createWorkScheduler();
const workers = new Map<DeliveryQueue, { wake: () => void; stop: () => Promise<void> }>();
let stopped = false;
return {
// Only these registered tasks are represented; not yet the whole instance.
nextWakeAt: scheduler.nextWakeAt,
register(queue: DeliveryQueue, task: {
retryMs: number;
// Bound bursts by the first notification's deadline; later notifications
// do not postpone the obligation indefinitely.
coalesceMs?: number;
run: (signal: AbortSignal) => Promise<unknown>;
hasPending: (runAllowed?: boolean) => Promise<boolean>;
// Scoped hints belong to the outer transaction, including uncertainty recovery.
onSettled?: (hints?: ReadonlySet<string>) => void;
onStop?: () => void;
// Optional maintenance/retry deadline restored from durable state.
nextRunAt?: () => number | null;
// Opt in only when run has recorded per-item failure backoffs. Other
// eligible items may already be due, unlike an unchanged failed deadline.
nextRunAtOnError?: () => number | null;
// An empty queue's maintenance may wait until dispatch admission reopens.
// Notifications and unresolved producer writes still reconcile in drain.
pausePassiveMaintenance?: boolean;
// Accounting and stop delivery finish already-admitted obligations;
// they must continue during admission drain, never warm standby.
runDuringDrain?: boolean;
}) {
if (stopped) throw new Error("Delivery coordinator is stopped");
if (workers.has(queue)) throw new Error(`Delivery worker already registered: ${queue}`);
let dirty = false;
let scheduledAt: number | null = null;
let running: Promise<void> | null = null;
let workerStopped = false;
let controller: AbortController | null = null;
let finishIntent: (() => void) | undefined;
let pendingWork = true;
function schedule(delay: number, deadline?: number) {
if (workerStopped) return;
const at = deadline ?? Date.now() + delay;
if (scheduledAt !== null && scheduledAt <= at) return;
scheduledAt = at;
scheduler.schedule(queue, at, start);
}
function start() {
if (workerStopped || running) return;
scheduler.cancel(queue);
scheduledAt = null;
if (!input.canRun() && !(input.canReconcile?.() ?? false)) { schedule(task.retryMs); return; }
if (!input.canRun() && task.pausePassiveMaintenance && !pendingWork && !dirty &&
!finishIntent && !databaseWorkPending(input.owner, queue)) {
// A timer alone must not change the idle generation during a slow
// backup. Retry without querying or admitting work; a notification
// still preempts this deadline and retains its normal intent hold.
schedule(task.retryMs);
return;
}
dirty = false;
const finish = beginIdleTrackedWork();
const attempt = new AbortController();
let runFailed = false;
controller = attempt;
running = withDatabaseOperation(`delivery.${queue}`, () => Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue))
.then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending(input.canRun() || Boolean(task.runDuringDrain && input.canReconcile?.()))).then(pending => {
pendingWork = pending || databaseWorkPending(input.owner, queue);
if (pendingWork) schedule(task.retryMs);
else { finishIntent?.(); finishIntent = undefined; }
const next = task.nextRunAt?.();
if (next != null) {
const delay = Math.max(0, next - Date.now());
schedule(delay === 0 && !input.canRun() ? task.retryMs : delay, delay > 0 ? next : undefined);
}
}).catch(error => {
schedule(task.retryMs);
let failure = error;
try {
const scopedRetry = runFailed && task.nextRunAtOnError;
const next = input.canRun() ? (scopedRetry ? scopedRetry() : task.nextRunAt?.()) : null;
if (next != null && (scopedRetry || next > Date.now())) schedule(0, Math.max(Date.now(), next));
} catch (deadlineError) {
failure = new AggregateError([error, deadlineError], "Delivery failure deadline could not be read");
}
// Logging must never turn a recoverable sweep failure into an
// unhandled rejection or prevent other queues from running.
try { if (!workerStopped) input.onError(failure, queue); } catch { /* Recovery remains scheduled. */ }
}).finally(() => {
running = null;
controller = null;
finish();
if (dirty) schedule(task.coalesceMs ?? 0);
}));
}
function wake() {
if (workerStopped) return;
dirty = true;
if (!running) schedule(task.coalesceMs ?? 0);
}
function applySettled(hints?: ReadonlySet<string>) {
for (const attempt of hints === undefined ? [undefined] : [hints, undefined]) {
try { task.onSettled?.(attempt); return; }
catch (error) { process.emitWarning(`Delivery settlement could not be applied for ${queue}: ${String(error)}`); }
}
}
const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => {
if (workerStopped) return;
if (databaseWorkPending(input.owner, queue)) {
finishIntent ??= beginIdleTrackedWork();
// Give the transaction time to settle. Its settlement notification
// replaces this deadline with an immediate wake on the normal path.
if (!running) schedule(task.retryMs);
}
if (event === "settled") { applySettled(hints); wake(); }
});
const worker = { wake, async stop() {
workerStopped = true;
task.onStop?.();
controller?.abort(new Error("Delivery worker stopped"));
unsubscribe();
scheduler.cancel(queue);
await running;
finishIntent?.(); finishIntent = undefined;
} };
workers.set(queue, worker);
const ready = Promise.resolve().then(() => { start(); return running; });
return { ready, ...worker };
},
// Stop dispatch queues first, keeping only already-admitted controller
// commands alive until their execution owners finish the handoff. A later
// stop() closes the retained workers and the shared scheduler as well.
async stop(keepQueues?: ReadonlySet<DeliveryQueue>) {
stopped = true;
if (!keepQueues?.size) scheduler.stop();
await Promise.all(Array.from(workers, async ([queue, worker]) => {
if (keepQueues?.has(queue)) return;
await worker.stop();
workers.delete(queue);
}));
},
};
}