mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 09:41:04 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Email connects agent tasks to durable incoming and outgoing messages. > - The email worker scans the database every second, even with no inboxes. > - These queries prevent idle database connections from closing. > - This pull request wakes email work from committed writes and known deadlines. > - Empty email installations stop polling. Existing delivery and catch-up guarantees remain. ## Linked Issues or Issue Description Refs #15625. Related standby work: #15314. **What existing behavior does this improve?** The AgentMail background worker and its database traffic when idle. **Current behavior** Every instance queries active email endpoints each second. Active inboxes also scan delivery queues each second even when no work is due. **Proposed behavior** Scan once at startup. Wake after endpoint and queue transactions settle. Restore the next retry or catch-up deadline from existing rows. An instance with no active inboxes has no recurring email queries or timer. **Reason and benefit** Thousands of empty instances should not keep their databases awake to discover that email is still unconfigured. ## What Changed - Use the shared transaction-aware coordinator for email work, including changes made by separate service objects on the same database owner. - Signal endpoint activation and control changes, incoming events, and outgoing sends inside their transactions. - Add an optional deadline to the coordinator. Email uses persisted retry times, mailbox catch-up times, and socket reconnect times. Only the first pending send in each conversation determines its send deadline. - Preserve startup recovery, lease fences, idempotency, standby suppression, idle drain, and shutdown. - Update the sleep inventory and add quiet-idle, commit, activation, retry, and lifecycle tests. ## Verification - Focused coordinator, email idle, and warm standby tests: 20 passed locally. - Database-backed email integration suite: 41 passed locally with real PostgreSQL and a simulated AgentMail provider. - After review, the updated delayed-reply regression passed separately: 1 selected test passed, 40 intentionally filtered out. It verifies no scans before the first reply deadline and delivery of both replies in order. - `pnpm -r typecheck` and `pnpm build` passed locally. - The local `pnpm test:run` was interrupted after its checkout was removed externally. Completed local checks above are preserved. The [final-commit hosted CI matrix](https://github.com/paperclipai/paperclip/actions/runs/37997072212) passed: 54 successful checks, 2 expected Storybook skips, no failures. It includes the full general and serialized test matrix, runner checks, browser tests, build, typecheck, and canary packaging. - Greptile: 5/5 on `ec98f3b86ee476d75e0edf3366b357377ea09d69`, with no actionable findings. ## Risks - New email producers must register work inside their write transaction. No new tables or migrations are required. - Signals are local to one database owner and its dedicated pools. Independent processes and direct SQL do not emit them. Startup recovers existing work; configured inboxes also retain their recovery deadline. - Active inboxes still catch up each minute. WebSocket mode still renews its lease every 20 seconds. This change does not provide external waking or make active sockets sleep. - Provider calls were simulated in tests. No live email was sent. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and test tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
82 lines
3.4 KiB
TypeScript
82 lines
3.4 KiB
TypeScript
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
|
|
import { emailChannelService } from "../services/email-channels.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../services/delivery-work-notifications.js";
|
|
import { idleWorkSnapshot } from "../services/task-admission.js";
|
|
|
|
const services: ReturnType<typeof emailChannelService>[] = [];
|
|
afterEach(async () => {
|
|
for (const service of services.splice(0)) await service.shutdown();
|
|
vi.useRealTimers();
|
|
});
|
|
function fixture() {
|
|
vi.useFakeTimers();
|
|
const select = vi.fn(() => ({ from: () => ({ where: async () => [] }) }));
|
|
const execute = vi.fn(async () => [{ xid: "42", status: "aborted" }]);
|
|
const db = installDatabaseWorkSignals({ select, execute,
|
|
async transaction(callback: (tx: any) => Promise<any>) {
|
|
return callback({ execute, transaction: db.transaction });
|
|
},
|
|
});
|
|
let enabled = true;
|
|
const service = emailChannelService(db as unknown as Db, {
|
|
heartbeat: { wakeup: vi.fn() },
|
|
isBackgroundWorkEnabled: () => enabled,
|
|
isReconciliationEnabled: () => true,
|
|
});
|
|
services.push(service);
|
|
return { db, select, service, enable: (value: boolean) => { enabled = value; } };
|
|
}
|
|
|
|
describe("email idle work", () => {
|
|
it("scans at startup, then leaves an unconfigured stack with zero timers and queries", async () => {
|
|
const f = fixture();
|
|
await f.service.start();
|
|
expect(f.select).toHaveBeenCalledTimes(1);
|
|
await vi.advanceTimersByTimeAsync(3_600_000);
|
|
expect(f.select).toHaveBeenCalledTimes(1);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("wakes on an outer transaction commit from a separate producer, never an inner savepoint", async () => {
|
|
const f = fixture();
|
|
await f.service.start();
|
|
await f.db.transaction(async tx => {
|
|
await tx.transaction(async (inner: object) => notifyDeliveryWork(inner, DELIVERY_QUEUES.email));
|
|
expect(f.select).toHaveBeenCalledTimes(1);
|
|
});
|
|
await vi.advanceTimersByTimeAsync(1);
|
|
expect(f.select).toHaveBeenCalledTimes(2);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
it("retains startup and committed work across idle drain without SQL or dispatch", async () => {
|
|
const f = fixture();
|
|
f.enable(false);
|
|
await f.service.start();
|
|
await f.db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.email));
|
|
await vi.advanceTimersByTimeAsync(60_000);
|
|
expect(f.select).not.toHaveBeenCalled();
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
f.enable(true);
|
|
await vi.advanceTimersByTimeAsync(1001);
|
|
expect(f.select).toHaveBeenCalledTimes(1);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("retries a failed startup scan and unsubscribes on shutdown", async () => {
|
|
const f = fixture();
|
|
const warning = vi.spyOn(process, "emitWarning").mockImplementation(() => {});
|
|
f.select.mockImplementationOnce(() => { throw new Error("database unavailable"); });
|
|
await f.service.start();
|
|
await vi.advanceTimersByTimeAsync(1001);
|
|
expect(f.select).toHaveBeenCalledTimes(2);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
await f.service.shutdown();
|
|
await f.db.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.email));
|
|
await vi.advanceTimersByTimeAsync(60_000);
|
|
expect(f.select).toHaveBeenCalledTimes(2);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
warning.mockRestore();
|
|
});
|
|
});
|