mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for <queue>` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
7.5 KiB
TypeScript
156 lines
7.5 KiB
TypeScript
import { subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "@paperclipai/db/work-signals";
|
|
import { withDatabaseOperation } from "@paperclipai/db";
|
|
import { createWorkScheduler } from "./work-scheduler.js";
|
|
import { type DeliveryQueue } from "./delivery-work-notifications.js";
|
|
import { beginIdleTrackedWork } from "./task-admission.js";
|
|
|
|
export function createDeliveryWorkCoordinator(input: {
|
|
owner: object;
|
|
canRun: () => boolean;
|
|
// Idle drain may reconcile admitted writes without dispatching new work.
|
|
// Warm standby can disable both operations.
|
|
canReconcile?: () => boolean;
|
|
onError: (error: unknown, queue: DeliveryQueue) => void;
|
|
}) {
|
|
const scheduler = createWorkScheduler();
|
|
const workers = new Map<DeliveryQueue, { wake: () => void; stop: () => Promise<void> }>();
|
|
let stopped = false;
|
|
|
|
return {
|
|
// Only these registered tasks are represented; not yet the whole instance.
|
|
nextWakeAt: scheduler.nextWakeAt,
|
|
register(queue: DeliveryQueue, task: {
|
|
retryMs: number;
|
|
// Bound bursts by the first notification's deadline; later notifications
|
|
// do not postpone the obligation indefinitely.
|
|
coalesceMs?: number;
|
|
run: (signal: AbortSignal) => Promise<unknown>;
|
|
hasPending: (runAllowed?: boolean) => Promise<boolean>;
|
|
// Scoped hints belong to the outer transaction, including uncertainty recovery.
|
|
onSettled?: (hints?: ReadonlySet<string>) => void;
|
|
onStop?: () => void;
|
|
// Optional maintenance/retry deadline restored from durable state.
|
|
nextRunAt?: () => number | null;
|
|
// Opt in only when run has recorded per-item failure backoffs. Other
|
|
// eligible items may already be due, unlike an unchanged failed deadline.
|
|
nextRunAtOnError?: () => number | null;
|
|
// An empty queue's maintenance may wait until dispatch admission reopens.
|
|
// Notifications and unresolved producer writes still reconcile in drain.
|
|
pausePassiveMaintenance?: boolean;
|
|
// Accounting and stop delivery finish already-admitted obligations;
|
|
// they must continue during admission drain, never warm standby.
|
|
runDuringDrain?: boolean;
|
|
}) {
|
|
if (stopped) throw new Error("Delivery coordinator is stopped");
|
|
if (workers.has(queue)) throw new Error(`Delivery worker already registered: ${queue}`);
|
|
let dirty = false;
|
|
let scheduledAt: number | null = null;
|
|
let running: Promise<void> | null = null;
|
|
let workerStopped = false;
|
|
let controller: AbortController | null = null;
|
|
let finishIntent: (() => void) | undefined;
|
|
let pendingWork = true;
|
|
function schedule(delay: number, deadline?: number) {
|
|
if (workerStopped) return;
|
|
const at = deadline ?? Date.now() + delay;
|
|
if (scheduledAt !== null && scheduledAt <= at) return;
|
|
scheduledAt = at;
|
|
scheduler.schedule(queue, at, start);
|
|
}
|
|
function start() {
|
|
if (workerStopped || running) return;
|
|
scheduler.cancel(queue);
|
|
scheduledAt = null;
|
|
if (!input.canRun() && !(input.canReconcile?.() ?? false)) { schedule(task.retryMs); return; }
|
|
if (!input.canRun() && task.pausePassiveMaintenance && !pendingWork && !dirty &&
|
|
!finishIntent && !databaseWorkPending(input.owner, queue)) {
|
|
// A timer alone must not change the idle generation during a slow
|
|
// backup. Retry without querying or admitting work; a notification
|
|
// still preempts this deadline and retains its normal intent hold.
|
|
schedule(task.retryMs);
|
|
return;
|
|
}
|
|
dirty = false;
|
|
const finish = beginIdleTrackedWork();
|
|
const attempt = new AbortController();
|
|
let runFailed = false;
|
|
controller = attempt;
|
|
running = withDatabaseOperation(`delivery.${queue}`, () => Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue))
|
|
.then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending(input.canRun() || Boolean(task.runDuringDrain && input.canReconcile?.()))).then(pending => {
|
|
pendingWork = pending || databaseWorkPending(input.owner, queue);
|
|
if (pendingWork) schedule(task.retryMs);
|
|
else { finishIntent?.(); finishIntent = undefined; }
|
|
const next = task.nextRunAt?.();
|
|
if (next != null) {
|
|
const delay = Math.max(0, next - Date.now());
|
|
schedule(delay === 0 && !input.canRun() ? task.retryMs : delay, delay > 0 ? next : undefined);
|
|
}
|
|
}).catch(error => {
|
|
schedule(task.retryMs);
|
|
let failure = error;
|
|
try {
|
|
const scopedRetry = runFailed && task.nextRunAtOnError;
|
|
const next = input.canRun() ? (scopedRetry ? scopedRetry() : task.nextRunAt?.()) : null;
|
|
if (next != null && (scopedRetry || next > Date.now())) schedule(0, Math.max(Date.now(), next));
|
|
} catch (deadlineError) {
|
|
failure = new AggregateError([error, deadlineError], "Delivery failure deadline could not be read");
|
|
}
|
|
// Logging must never turn a recoverable sweep failure into an
|
|
// unhandled rejection or prevent other queues from running.
|
|
try { if (!workerStopped) input.onError(failure, queue); } catch { /* Recovery remains scheduled. */ }
|
|
}).finally(() => {
|
|
running = null;
|
|
controller = null;
|
|
finish();
|
|
if (dirty) schedule(task.coalesceMs ?? 0);
|
|
}));
|
|
}
|
|
function wake() {
|
|
if (workerStopped) return;
|
|
dirty = true;
|
|
if (!running) schedule(task.coalesceMs ?? 0);
|
|
}
|
|
function applySettled(hints?: ReadonlySet<string>) {
|
|
for (const attempt of hints === undefined ? [undefined] : [hints, undefined]) {
|
|
try { task.onSettled?.(attempt); return; }
|
|
catch (error) { process.emitWarning(`Delivery settlement could not be applied for ${queue}: ${String(error)}`); }
|
|
}
|
|
}
|
|
const unsubscribe = subscribeDatabaseWork(input.owner, queue, (event, hints) => {
|
|
if (workerStopped) return;
|
|
if (databaseWorkPending(input.owner, queue)) {
|
|
finishIntent ??= beginIdleTrackedWork();
|
|
// Give the transaction time to settle. Its settlement notification
|
|
// replaces this deadline with an immediate wake on the normal path.
|
|
if (!running) schedule(task.retryMs);
|
|
}
|
|
if (event === "settled") { applySettled(hints); wake(); }
|
|
});
|
|
const worker = { wake, async stop() {
|
|
workerStopped = true;
|
|
task.onStop?.();
|
|
controller?.abort(new Error("Delivery worker stopped"));
|
|
unsubscribe();
|
|
scheduler.cancel(queue);
|
|
await running;
|
|
finishIntent?.(); finishIntent = undefined;
|
|
} };
|
|
workers.set(queue, worker);
|
|
const ready = Promise.resolve().then(() => { start(); return running; });
|
|
return { ready, ...worker };
|
|
},
|
|
// Stop dispatch queues first, keeping only already-admitted controller
|
|
// commands alive until their execution owners finish the handoff. A later
|
|
// stop() closes the retained workers and the shared scheduler as well.
|
|
async stop(keepQueues?: ReadonlySet<DeliveryQueue>) {
|
|
stopped = true;
|
|
if (!keepQueues?.size) scheduler.stop();
|
|
await Promise.all(Array.from(workers, async ([queue, worker]) => {
|
|
if (keepQueues?.has(queue)) return;
|
|
await worker.stop();
|
|
workers.delete(queue);
|
|
}));
|
|
},
|
|
};
|
|
}
|