Deliver durable resource lifecycle events to plugins without polling (#16203)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Plugins keep external resources aligned with Paperclip projects and
agents.
> - The durable resource journal stores changes, but its plugin
interface requires polling.
> - Project deletion has no journal entry, so consumers must scan for
missing projects.
> - This PR delivers committed journal events through a native plugin
callback and records deletion.
> - Plugins can recover unfinished work without periodic change scans.

## Linked Issues or Issue Description

Related: #15343 introduced the durable inbox. #15631 added required
agent lifecycle participation. This change keeps both contracts and adds
automatic delivery for the resource inbox.

**What happened?**

Plugins must schedule reads of the durable inbox. A deleted project has
no durable deletion event. Consumers must scan saved bindings to detect
its removal.

**Expected behavior**

Core should deliver committed resource events and replay unfinished
work. Empty queues should have no polling timer. Project removal should
leave a durable event.

**Steps to reproduce**

1. Install a plugin that consumes project lifecycle events.
2. Create or update a project. The plugin must call `listLifecycle` to
discover it.
3. Delete the project. The journal has no deletion entry.

## What Changed

- Add `resourceLifecycle` to the manifest and `onResourceLifecycle` to
the SDK. Require `events.subscribe` and a configured, enabled company
scope.
- Reuse transaction-aware work signals and the delivery coordinator.
Wake after commit, on startup and plugin activation, and after
configuration changes.
- Acknowledge only a matching successful result. Retry unfinished or
interrupted work. Retain deferred events without an idle retry timer.
- Wake again after successful worker configuration application, and
retain failed scans through idle drain (both Apex findings fixed).
- Keep ordering per resource and deliver past failed or deferred
resources. Keep the manual inbox methods for existing consumers.
- Record project deletion in the removal transaction. Extend the journal
constraint with a generated migration.

## Verification

- Apex follow-up: 129 focused worker, delivery, coordinator, and
application tests passed; server typecheck passed. Regression coverage
verifies configuration-application races and failed scans across idle
drain.

- Focused lifecycle, transaction coordinator, manifest, and SDK dispatch
suites: 105 tests passed, including single-connection saves, redacted
failure diagnostics, wakeups retained during idle drain, and per-event
retry deadlines.
- Complete SDK suite plus delivery database tests: 128 tests passed.
- Shared, database, and SDK typechecks passed. Direct server `tsc
--noEmit` passed against rebuilt SDK declarations.
- Full local `pnpm test:run` finished with 19,491 passed, 148 skipped,
and 6 failed tests across 5 files (plus two suite setup errors). This
run overlapped branch updates and is not final-head evidence: failures
included missing Cargo, local Git default-branch assumptions, and stale
in-process lifecycle modules. Final Apex-fix commit `75d028788` passed
all CI gates, including the full test shards, and received Apex 5/5 with
no unresolved comments.
- `pnpm -r typecheck` and `pnpm build` were attempted. Both stopped
because Cargo is unavailable locally. CI supplies the Rust toolchain.
- Local installation used the repository's documented lockfile fallback
because master has dependencies missing from its lockfile. No dependency
changes are included here.

## Risks

- Delivery is at least once. Plugins must re-read current state and make
external effects idempotent.
- Signals are process-local, like the other delivery queues. Startup
recovers committed rows. Independent database writers or multiple active
server processes require an explicit ownership and wake protocol.
- Apply migration `0340_open_warlock.sql` before running code that
records project deletion. Earlier deletions cannot be reconstructed.
- Automatic delivery selects plugins with company configuration rows.
`deferred` waits for a later startup, configuration change, or other
work notification.
- This does not replace required agent lifecycle participation or
establish that external mounts are unused.

## Model Used

OpenAI Codex (GPT-6), with tool-assisted code editing, code inspection,
and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-10-13 12:00:30 -07:00
1 parent 5a6d0d08a7
commit db65a0863e
27 files changed
+1252 -145

No files matched your search

@@ -20,7 +20,7 @@ rl.on("line", (line) => {
id: message.id,
result: {
ok: true,
supportedMethods: ["environmentExecute"],
supportedMethods: ["environmentExecute", "configChanged"],
},
});
return;
@@ -44,6 +44,16 @@ rl.on("line", (line) => {
return;
}
if (method === "configChanged") {
setTimeout(() => send({
jsonrpc: "2.0", id: message.id,
...(message.params.config.fail
? { error: { code: -32000, message: "configuration rejected" } }
: { result: {} }),
}), Number(message.params.config.delayMs ?? 0));
return;
}
if (method === "shutdown") {
send({
jsonrpc: "2.0",
@@ -0,0 +1,217 @@
import { randomUUID } from "node:crypto";
import { EventEmitter } from "node:events";
import { eq } from "drizzle-orm";
import { companies, createDb, pluginConfig, plugins, resourceLifecycleEvents, type Db } from "@paperclipai/db";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { logger } from "../middleware/logger.js";
import { registerPluginLifecycleDelivery } from "../services/plugin-lifecycle-delivery.js";
import { pluginLifecycleInbox } from "../services/plugin-lifecycle-inbox.js";
import { pluginRegistryService } from "../services/plugin-registry.js";
import { recordProjectLifecycleEvent, recordResourceCreationEvent } from "../services/resource-lifecycle-events.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
import type { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import type { PluginLifecycleManager } from "../services/plugin-lifecycle.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
if (!support.supported) console.warn(`Skipping lifecycle delivery database tests: ${support.reason}`);
describePostgres("durable plugin lifecycle delivery", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
let companyId: string;
let pluginId: string;
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("paperclip-plugin-delivery-"); db = createDb(database.connectionString); }, 90_000);
afterAll(async () => { await database?.cleanup(); });
afterEach(() => { vi.restoreAllMocks(); });
beforeEach(async () => {
await db.delete(plugins);
companyId = randomUUID(); pluginId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Delivery fixture", issuePrefix: `L${companyId.slice(0, 6)}` });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "delivery-fixture", version: "1.0.0", status: "ready",
manifestJson: { resourceLifecycle: ["project"], capabilities: ["events.subscribe"] } as never });
await db.insert(pluginConfig).values({ pluginId, companyId, configJson: {} });
});
async function event(resourceId = randomUUID(), action: "create" | "update" | "delete" = "create") {
await db.transaction(async tx => {
if (action === "create") await recordResourceCreationEvent(tx as unknown as Db, companyId, "project", resourceId);
else await recordProjectLifecycleEvent(tx as unknown as Db, companyId, resourceId, action);
});
return resourceId;
}
function delivery() {
let task!: Parameters<ReturnType<typeof createDeliveryWorkCoordinator>["register"]>[1];
const wake = vi.fn();
const coordinator = { register: vi.fn((_queue, input) => { task = input; return { wake, ready: Promise.resolve() }; }) };
const call = vi.fn(async (_pluginId, _method, input) => ({ eventId: input.id, status: "acknowledged" }));
const running = vi.fn(() => true);
const lifecycle = new EventEmitter();
const configApplied = new Set<() => void>();
const onConfigApplied = (listener: () => void) => { configApplied.add(listener); return () => { configApplied.delete(listener); }; };
const worker = registerPluginLifecycleDelivery(coordinator as never, db, { call, isRunning: running, onConfigApplied } as unknown as PluginWorkerManager, lifecycle as unknown as PluginLifecycleManager);
return { call, running, wake, lifecycle, worker, applied: () => { for (const listener of configApplied) listener(); }, run: () => task.run(new AbortController().signal), pending: () => task.hasPending(), nextRunAt: () => task.nextRunAt?.(), settled: () => task.onSettled?.() };
}
it("drains creation before transitions, recovers acknowledgments across dispatcher replacement, and ignores agent events", async () => {
const id = await event(); await event(id, "update"); await event(id, "delete");
await db.transaction(tx => recordResourceCreationEvent(tx as unknown as Db, companyId, "agent", randomUUID()));
const first = delivery(); await first.run(); first.worker.dispose();
expect(first.call.mock.calls.map(c => c[2].action)).toEqual(["create"]);
const restarted = delivery(); await restarted.run(); await restarted.run(); await restarted.run();
expect(restarted.call.mock.calls.map(c => c[2].action)).toEqual(["update", "delete"]);
expect(await restarted.pending()).toBe(false);
expect((await pluginLifecycleInbox(db, pluginId).list(companyId)).map(e => e.resourceType)).toEqual(["agent"]);
restarted.worker.dispose();
});
it("replays failures and invalid acknowledgments without blocking independent resources", async () => {
let now = Date.now(); vi.spyOn(Date, "now").mockImplementation(() => now);
const failed = await event(); await event(failed, "delete"); await event();
const d = delivery();
d.call.mockImplementationOnce(async () => { throw new Error("worker exited after provider effect"); });
await d.run();
expect(d.call).toHaveBeenCalledTimes(2); expect(await d.pending()).toBe(true);
d.call.mockImplementationOnce(async () => ({ eventId: "wrong", status: "acknowledged" }));
now += 30_000; await d.run(); expect(await d.pending()).toBe(true);
now += 30_000; await d.run(); await d.run();
expect(d.call.mock.calls.map(c => c[2].action)).toEqual(["create", "create", "create", "create", "delete"]);
expect(await pluginLifecycleInbox(db, pluginId).list(companyId)).toEqual([]);
d.worker.dispose();
});
it("healthy acknowledgments and new events do not bypass a failed event's retry deadline", async () => {
let now = Date.now(); vi.spyOn(Date, "now").mockImplementation(() => now);
const failed = await event(), healthy = await event();
await event(healthy, "update"); await event(healthy, "delete");
const d = delivery();
d.call.mockImplementation(async (_p, _m, e) => ({ eventId: e.id, status: e.resourceId === failed ? "retry" : "acknowledged" }));
await d.run();
const deadline = now + 30_000;
expect(d.nextRunAt()).toBe(deadline);
now += 1000;
await d.run(); await d.run(); await event(); d.settled(); await d.run();
expect(d.call.mock.calls.filter(c => c[2].resourceId === healthy).map(c => c[2].action)).toEqual(["create", "update", "delete"]);
expect(d.call.mock.calls.filter(c => c[2].resourceId === failed)).toHaveLength(1);
expect(d.nextRunAt()).toBe(deadline);
now = deadline;
expect(d.nextRunAt()).toBeNull(); // Paused dispatch waits for the normal retry, not a zero-delay loop.
await d.run();
expect(d.call.mock.calls.filter(c => c[2].resourceId === failed)).toHaveLength(2);
expect(d.nextRunAt()).toBe(now + 30_000);
d.worker.dispose();
});
it("defers without a retry deadline, retries unfinished work, and wakes on plugin startup", async () => {
await event(); const d = delivery();
d.call.mockImplementationOnce(async (_p, _m, e) => ({ eventId: e.id, status: "deferred" }));
await d.run(); expect(await d.pending()).toBe(false); expect(d.wake).not.toHaveBeenCalled();
d.lifecycle.emit("plugin.worker_started"); expect(d.wake).toHaveBeenCalledTimes(1);
d.call.mockImplementationOnce(async (_p, _m, e) => ({ eventId: e.id, status: "retry" }));
await d.run(); expect(await d.pending()).toBe(true);
d.running.mockReturnValue(false); await d.run(); expect(await d.pending()).toBe(true);
expect(d.call).toHaveBeenCalledTimes(2); d.worker.dispose();
d.lifecycle.emit("plugin.worker_started"); expect(d.wake).toHaveBeenCalledTimes(1);
});
it("requires ready, configured, company-enabled plugins and rechecks disabling within a batch", async () => {
await event(); await event(); const d = delivery(); const registry = pluginRegistryService(db);
await registry.upsertCompanySettings(pluginId, companyId, { enabled: false, settingsJson: {} });
await d.run(); expect(d.call).not.toHaveBeenCalled();
await registry.upsertCompanySettings(pluginId, companyId, { enabled: true, settingsJson: {} });
d.call.mockImplementationOnce(async (_p, _m, e) => {
await registry.upsertCompanySettings(pluginId, companyId, { enabled: false, settingsJson: {} });
return { eventId: e.id, status: "acknowledged" };
});
await d.run(); expect(d.call).toHaveBeenCalledTimes(1);
await registry.upsertCompanySettings(pluginId, companyId, { enabled: true, settingsJson: {} });
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, pluginId));
await d.run(); expect(d.call).toHaveBeenCalledTimes(1);
await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, pluginId));
await db.delete(pluginConfig).where(eq(pluginConfig.pluginId, pluginId));
await d.run(); expect(d.call).toHaveBeenCalledTimes(1); d.worker.dispose();
});
it("notifies only after the outer transaction settles and wakes deferred work for config changes", async () => {
const wake = vi.fn(); const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.pluginLifecycle, wake);
try {
await db.transaction(async tx => {
await eventInTransaction(tx as unknown as Db);
expect(wake).not.toHaveBeenCalled();
});
expect(wake).toHaveBeenCalledTimes(1);
const registry = pluginRegistryService(db);
await registry.upsertConfig(pluginId, companyId, { configJson: { enabled: true } });
await registry.patchConfig(pluginId, companyId, { configJson: { enabled: false } });
await registry.upsertCompanySettings(pluginId, companyId, { enabled: true, settingsJson: {} });
expect(wake).toHaveBeenCalledTimes(4);
} finally { unsubscribe(); }
async function eventInTransaction(tx: Db) {
await tx.transaction(inner => recordResourceCreationEvent(inner as unknown as Db, companyId, "project", randomUUID()));
}
});
it("retains startup and settled work while idle drain prevents dispatch", async () => {
const d = delivery();
// The coordinator can call hasPending without run when admission is paused.
expect(await d.pending()).toBe(true);
await d.run(); expect(await d.pending()).toBe(false);
await event(); d.settled();
expect(await d.pending()).toBe(true);
expect(d.call).not.toHaveBeenCalled();
await d.run();
expect(d.call).toHaveBeenCalledTimes(1);
expect(await d.pending()).toBe(true); // An acknowledgment must drain its successor.
await d.run(); expect(await d.pending()).toBe(false);
d.lifecycle.emit("plugin.enabled"); expect(await d.pending()).toBe(true);
await d.run(); expect(await d.pending()).toBe(false);
d.worker.wake(); expect(await d.pending()).toBe(true);
d.worker.dispose();
});
it("wakes deferred events after live configuration finishes applying", async () => {
await event(); const d = delivery();
let configured = false;
d.call.mockImplementation(async (_p, _m, e) => ({ eventId: e.id, status: configured ? "acknowledged" : "deferred" }));
await d.run(); expect(await d.pending()).toBe(false);
// Commit notification races ahead of the asynchronous worker callback.
d.settled(); await d.run(); expect(await d.pending()).toBe(false);
configured = true; d.applied();
expect(await d.pending()).toBe(true); expect(d.wake).toHaveBeenCalledTimes(1);
await d.run(); expect(await pluginLifecycleInbox(db, pluginId).list(companyId)).toEqual([]);
d.worker.dispose(); d.wake.mockClear(); d.applied(); expect(d.wake).not.toHaveBeenCalled();
});
it("retains a failed scan across a retry skipped during idle drain", async () => {
await event(); const d = delivery();
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("database unavailable"); });
await expect(d.run()).rejects.toThrow("database unavailable");
// The coordinator's drain path checks hasPending without calling run.
expect(await d.pending()).toBe(true); expect(d.call).not.toHaveBeenCalled();
select.mockRestore(); await d.run(); await d.run();
expect(await d.pending()).toBe(false);
expect(await pluginLifecycleInbox(db, pluginId).list(companyId)).toEqual([]);
d.worker.dispose();
});
it("saves config and settings without borrowing another pool connection", async () => {
const single = createDb(database.connectionString, { maxConnections: 1 });
const registry = pluginRegistryService(single);
await registry.upsertConfig(pluginId, companyId, { configJson: { first: true } });
await registry.patchConfig(pluginId, companyId, { configJson: { second: true } });
await registry.upsertCompanySettings(pluginId, companyId, { enabled: true, settingsJson: {} });
expect((await registry.getConfig(pluginId, companyId))?.configJson).toEqual({ first: true, second: true });
}, 5000);
it("records failed delivery scope and stage without provider secrets", async () => {
await event(); const d = delivery();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
try {
d.call.mockImplementationOnce(async () => { throw Object.assign(new Error("provider-secret-fixture"), { code: -32001 }); });
await d.run();
expect(warn).toHaveBeenCalledWith(expect.objectContaining({ pluginId, companyId, eventId: expect.any(String), stage: "callback", rpcCode: -32001 }), expect.any(String));
expect(JSON.stringify(warn.mock.calls)).not.toContain("provider-secret-fixture");
expect(await d.pending()).toBe(true);
} finally { warn.mockRestore(); d.worker.dispose(); }
});
it("rollbacks never expose queue rows even when settlement wakes the dispatcher", async () => {
const d = delivery();
await expect(db.transaction(async tx => {
await recordResourceCreationEvent(tx as unknown as Db, companyId, "project", randomUUID());
throw new Error("rollback");
})).rejects.toThrow("rollback");
await d.run(); expect(d.call).not.toHaveBeenCalled(); expect(await d.pending()).toBe(false);
expect(await db.select().from(resourceLifecycleEvents).where(eq(resourceLifecycleEvents.companyId, companyId))).toEqual([]);
d.worker.dispose();
});
});
@@ -31,6 +31,7 @@ import {
appendStderrExcerpt,
createDuplexRouteSlotController,
createPluginWorkerHandle,
createPluginWorkerManager,
formatWorkerFailureMessage,
resolveRpcCallTimeoutMs,
} from "../services/plugin-worker-manager.js";
@@ -56,6 +57,28 @@ const TEST_MANIFEST: PaperclipPluginManifestV1 = {
entrypoints: { worker: "dist/worker.js" },
};
describe("worker configuration completion", () => {
it("notifies only after a successful config callback and releases subscriptions", async () => {
const manager = createPluginWorkerManager();
const applied = vi.fn(), unsubscribe = manager.onConfigApplied(applied);
try {
await manager.startWorker("test.plugin", {
entrypointPath: DELAYED_WORKER_ENTRYPOINT, manifest: TEST_MANIFEST, config: {},
instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {},
});
expect(applied).not.toHaveBeenCalled();
const updating = manager.call("test.plugin", "configChanged", { companyId: "company-1", config: { delayMs: 20 } });
expect(applied).not.toHaveBeenCalled();
await updating; expect(applied).toHaveBeenCalledTimes(1);
await expect(manager.call("test.plugin", "configChanged", { companyId: "company-1", config: { fail: true } })).rejects.toThrow("configuration rejected");
expect(applied).toHaveBeenCalledTimes(1);
unsubscribe();
await manager.call("test.plugin", "configChanged", { companyId: "company-1", config: {} });
expect(applied).toHaveBeenCalledTimes(1);
} finally { unsubscribe(); await manager.stopAll(); }
});
});
describe("resolveRpcCallTimeoutMs", () => {
const MAX_RPC_TIMEOUT_MS = 15 * 60 * 1_000;
const MAX_NODE_TIMER_TIMEOUT_MS = 2_147_483_647;
@@ -53,7 +53,7 @@ describePostgres("Resource lifecycle events", () => {
it("records direct and approval-free hires once, including concurrent duplicate submissions", async () => {
const agent = await createAgent();
await Promise.all(Array.from({ length: 5 }, () => recordResourceCreationEvent(db, companyId, "agent", agent.id)));
await Promise.all(Array.from({ length: 5 }, () => db.transaction(tx => recordResourceCreationEvent(tx as unknown as Db, companyId, "agent", agent.id))));
expect(await events()).toEqual([expect.objectContaining({ resourceType: "agent", resourceId: agent.id, action: "create" })]);
});
@@ -235,11 +235,11 @@ describePostgres("Resource lifecycle events", () => {
vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined);
vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "fixture-token");
const id = randomUUID();
await recordResourceCreationEvent(db, companyId, "agent", id);
await recordResourceCreationEvent(db, companyId, "project", id);
await db.transaction(tx => recordResourceCreationEvent(tx as unknown as Db, companyId, "agent", id));
await db.transaction(tx => recordResourceCreationEvent(tx as unknown as Db, companyId, "project", id));
const otherCompanyId = randomUUID();
await db.insert(companies).values({ id: otherCompanyId, name: "Other company", issuePrefix: "OTHER" });
await recordResourceCreationEvent(db, otherCompanyId, "agent", id);
await db.transaction(tx => recordResourceCreationEvent(tx as unknown as Db, otherCompanyId, "agent", id));
expect(await events()).toHaveLength(2);
expect(await db.select().from(resourceLifecycleEvents).where(eq(resourceLifecycleEvents.companyId, otherCompanyId))).toHaveLength(1);
await db.delete(companies).where(eq(companies.id, otherCompanyId));
@@ -267,6 +267,23 @@ describePostgres("Resource lifecycle events", () => {
expect(beforeArchive.sort((a, b) => a.id - b.id).map(row => row.action)).toEqual(["create", "update", "update", "update", "update", "update"]);
});
it("records deletion atomically and retains the event after removing the project", async () => {
const service = projectService(db);
const project = await service.create(companyId, { name: "Delete fixture" });
await db.execute(sql`ALTER TABLE resource_lifecycle_events ADD CONSTRAINT fixture_reject_delete CHECK (action <> 'delete') NOT VALID`);
try {
await expect(service.remove(project.id)).rejects.toThrow();
expect(await service.getById(project.id)).not.toBeNull();
expect((await events()).map(row => row.action)).toEqual(["create"]);
} finally {
await db.execute(sql`ALTER TABLE resource_lifecycle_events DROP CONSTRAINT fixture_reject_delete`);
}
await service.remove(project.id);
await service.remove(project.id);
expect(await service.getById(project.id)).toBeNull();
expect((await events()).map(row => row.action)).toEqual(["create", "delete"]);
});
it("records archive once per transition and update on restore, retaining repositories", async () => {
const service = projectService(db);
const project = await service.createWithRepositories(companyId, { name: "Archive fixture" }, [