Files
PaperClipAI/server/src/__tests__/email-idle-work.test.ts
T
DottaandPaperclip ce024457f3 perf(server): wake email work from commits and deadlines (#15750)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Email connects agent tasks to durable incoming and outgoing
messages.
> - The email worker scans the database every second, even with no
inboxes.
> - These queries prevent idle database connections from closing.
> - This pull request wakes email work from committed writes and known
deadlines.
> - Empty email installations stop polling. Existing delivery and
catch-up guarantees remain.

## Linked Issues or Issue Description

Refs #15625. Related standby work: #15314.

**What existing behavior does this improve?**

The AgentMail background worker and its database traffic when idle.

**Current behavior**

Every instance queries active email endpoints each second. Active
inboxes also scan delivery queues each second even when no work is due.

**Proposed behavior**

Scan once at startup. Wake after endpoint and queue transactions settle.
Restore the next retry or catch-up deadline from existing rows. An
instance with no active inboxes has no recurring email queries or timer.

**Reason and benefit**

Thousands of empty instances should not keep their databases awake to
discover that email is still unconfigured.

## What Changed

- Use the shared transaction-aware coordinator for email work, including
changes made by separate service objects on the same database owner.
- Signal endpoint activation and control changes, incoming events, and
outgoing sends inside their transactions.
- Add an optional deadline to the coordinator. Email uses persisted
retry times, mailbox catch-up times, and socket reconnect times. Only
the first pending send in each conversation determines its send
deadline.
- Preserve startup recovery, lease fences, idempotency, standby
suppression, idle drain, and shutdown.
- Update the sleep inventory and add quiet-idle, commit, activation,
retry, and lifecycle tests.

## Verification

- Focused coordinator, email idle, and warm standby tests: 20 passed
locally.
- Database-backed email integration suite: 41 passed locally with real
PostgreSQL and a simulated AgentMail provider.
- After review, the updated delayed-reply regression passed separately:
1 selected test passed, 40 intentionally filtered out. It verifies no
scans before the first reply deadline and delivery of both replies in
order.
- `pnpm -r typecheck` and `pnpm build` passed locally.
- The local `pnpm test:run` was interrupted after its checkout was
removed externally. Completed local checks above are preserved. The
[final-commit hosted CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/37997072212)
passed: 54 successful checks, 2 expected Storybook skips, no failures.
It includes the full general and serialized test matrix, runner checks,
browser tests, build, typecheck, and canary packaging.
- Greptile: 5/5 on `ec98f3b86ee476d75e0edf3366b357377ea09d69`, with no
actionable findings.

## Risks

- New email producers must register work inside their write transaction.
No new tables or migrations are required.
- Signals are local to one database owner and its dedicated pools.
Independent processes and direct SQL do not emit them. Startup recovers
existing work; configured inboxes also retain their recovery deadline.
- Active inboxes still catch up each minute. WebSocket mode still renews
its lease every 20 seconds. This change does not provide external waking
or make active sockets sleep.
- Provider calls were simulated in tests. No live email was sent.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and test tools. The exact deployment model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 17:19:41 -05:00

82 lines
3.4 KiB
TypeScript

import { afterEach, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
import { emailChannelService } from "../services/email-channels.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../services/delivery-work-notifications.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const services: ReturnType<typeof emailChannelService>[] = [];
afterEach(async () => {
for (const service of services.splice(0)) await service.shutdown();
vi.useRealTimers();
});
function fixture() {
vi.useFakeTimers();
const select = vi.fn(() => ({ from: () => ({ where: async () => [] }) }));
const execute = vi.fn(async () => [{ xid: "42", status: "aborted" }]);
const db = installDatabaseWorkSignals({ select, execute,
async transaction(callback: (tx: any) => Promise<any>) {
return callback({ execute, transaction: db.transaction });
},
});
let enabled = true;
const service = emailChannelService(db as unknown as Db, {
heartbeat: { wakeup: vi.fn() },
isBackgroundWorkEnabled: () => enabled,
isReconciliationEnabled: () => true,
});
services.push(service);
return { db, select, service, enable: (value: boolean) => { enabled = value; } };
}
describe("email idle work", () => {
it("scans at startup, then leaves an unconfigured stack with zero timers and queries", async () => {
const f = fixture();
await f.service.start();
expect(f.select).toHaveBeenCalledTimes(1);
await vi.advanceTimersByTimeAsync(3_600_000);
expect(f.select).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
});
it("wakes on an outer transaction commit from a separate producer, never an inner savepoint", async () => {
const f = fixture();
await f.service.start();
await f.db.transaction(async tx => {
await tx.transaction(async (inner: object) => notifyDeliveryWork(inner, DELIVERY_QUEUES.email));
expect(f.select).toHaveBeenCalledTimes(1);
});
await vi.advanceTimersByTimeAsync(1);
expect(f.select).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
expect(idleWorkSnapshot().active).toBe(0);
});
it("retains startup and committed work across idle drain without SQL or dispatch", async () => {
const f = fixture();
f.enable(false);
await f.service.start();
await f.db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.email));
await vi.advanceTimersByTimeAsync(60_000);
expect(f.select).not.toHaveBeenCalled();
expect(idleWorkSnapshot().active).toBe(0);
f.enable(true);
await vi.advanceTimersByTimeAsync(1001);
expect(f.select).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
});
it("retries a failed startup scan and unsubscribes on shutdown", async () => {
const f = fixture();
const warning = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
f.select.mockImplementationOnce(() => { throw new Error("database unavailable"); });
await f.service.start();
await vi.advanceTimersByTimeAsync(1001);
expect(f.select).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
await f.service.shutdown();
await f.db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.email));
await vi.advanceTimersByTimeAsync(60_000);
expect(f.select).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
warning.mockRestore();
});
});