mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins must refresh runtime preparation when tool definitions change. > - Standing assignments alone do not cover catalogs, profile entries, or policies. > - New journal hooks must preserve the order used by setup finalization. > - This pull request adds tool-definition invalidation and its required writer guards. > - It leaves credentials and provider access changes for the next independent capability. ## Linked Issues or Issue Description **What happened?** Catalog, profile-entry, policy, gateway, and connection-configuration changes do not request agent runtime synchronization. A new profile-entry trigger also exposes a deletion/finalization deadlock if the child row is changed before its parent is locked. **Expected behavior** Committed tool-definition changes advance runtime revisions without polling. Bookkeeping does not. Entry changes and profile replacement use a consistent parent/reference lock order and commit atomically. **Steps to reproduce** 1. Change a catalog descriptor, profile entry, policy, or connection configuration. 2. Expect committed runtime events for the affected agents. 3. Run profile-entry deletion concurrently with setup finalization while both wait for an agent row. 4. Expect both operations to complete without PostgreSQL deadlock 40P01. **Additional context** Extracted from #16503. Catalog publication #16602 and standing assignments #16603 are merged. This PR is rebased onto master and is the active review. It addresses the original Apex profile-entry deletion finding. Credential/access hooks #16605 remain a draft without review requests. ## What Changed - Add migration 0355 for application/connection configuration, catalog, profile-entry, policy, and MCP gateway invalidation. - Preserve transaction coalescing and ignore health bookkeeping and trust-rule usage counters. - Keep credential-only fields outside this migration's coverage. - Preserve merged recipient scoping for bindings and installations. Profile-entry edits resolve old/new standing profile bindings. Policy edits resolve old/new agent selectors with their intersection/list semantics. - Lock old and new profile-entry references before parent profiles and child mutations. Recheck the reference set under the parent lock, retry changed snapshots up to three times, and merge unspecified fields from the locked profile. - Make entry add/edit/delete and profile replacement transactional, including validation and profile reads on the reserved connection. - Lock policy rows in ID order and change priorities in one statement so scoped agent locks are ordered across the full set. - Lock the managed profile before connection removal deletes its entries in the same transaction. - Update assignment tests whose old absence assertions are superseded by expanded configuration coverage. - Document the supported capability and the remaining credential/access boundary. ## Verification - Final head: `809fcb11794bfb12c545af0a2f96ab9bc2b570a2`. It is rebased onto master with merged #16623. GitHub reports no merge conflicts. [CI run 39293925136](https://github.com/paperclipai/paperclip/actions/runs/39293925136) has 55 successful checks and two optional Storybook skips. - Standard Greptile returned an independent 5/5. The final Apex review returned 5/5 on this exact head at 17:47 UTC on 2026-10-19. Its summary explicitly clears this PR for merge independently of draft #16605. The security scan passed. All review threads are resolved. - Two real-PostgreSQL negative controls reproduce `40P01` without the old-reference guard or the concurrent-snapshot check. Both regression tests pass with the final correction. Server typecheck, migration safety and numbering, five-snapshot retention, journal history, feature maps, and diff checks pass. The actual PR diff was scanned for secrets and personal data before push. - The final 15-suite local run passed 821/822 tests. The one failure was in the unchanged runtime-delivery test: a startup callback changed its global callback count during an independent transaction. The test-only correction is isolated in #16871. That PR passes all 17 runtime-hook tests, server types, hosted CI, and standard Greptile 5/5. Its notification-disabled control fails at the independent event-delivery assertion. No changes from #16871 are included here; this feature head passes hosted CI independently. - Earlier affected-suite runs passed 814 tests after rebase and 820 tests after recipient and whitespace corrections. Four affected suites passed 514 tests before the second reference-contention regression was added. Single-connection tests cover entry add, edit, delete, and replacement. Validation and profile reads use the reserved transaction connection. - Review fixes have negative controls: the old code hangs with a one-connection pool; parent-lock omissions deadlock entry deletion or app removal against setup finalization; per-row policy reorder deadlocks under contention; unscoped triggers journal unrelated peers; default SQL trimming misses JavaScript whitespace. Corrected cases pass. Selector tests cover all 25 ECMAScript whitespace characters. Recipient tests cover repeated edits, old/new targets, ignored bindings, rollback, coalescing, and company scope. The definition suite now has 28 cases. - Earlier standard scores that depended on #16605 were not accepted as independent clearance. The first Apex review returned 4/5 for replacement versus application deletion. The final correction locks old and new references, checks the reference set again under the parent lock, and retries before any mutation. The final independent reviews clear that correction. - Preserve prior failures for acceptance: draft run 38975432017 had failures whose causes were not re-investigated. Run 38990400765 exposed the unrelated durable-resume watchdog race, fixed in merged #16623. In run 39287657148, two jobs reported cancellation without a failed-test summary; the cause remains unknown. Both passed on retry without source changes. Initial local fixtures and SQL bindings also needed correction: an internal helper name was replaced with public `archiveConnection`, unchanged priorities were changed to exercise reorder, a driver timestamp was bound as ISO text, and a superseded company-fanout assertion was updated. These recovered checks do not establish a cause for the unexplained CI failures. - Local checks target the affected services and real database boundaries. A full local build and repository-wide test run were not performed. Hosted CI supplies the broader checks. Provider responses in tests are controlled; live provider behavior was not qualified. ## Risks Migration 0355 adds statement triggers without changing table schema. Apply it through the normal migration flow. A tool-definition event can advance a runtime revision without changing the bounded assignment snapshot. Consumers still need authorized readers for definitions. Credential-only changes, grants, secrets, and endpoints remain for the access-hook migration. Profile replacement now rolls back as a unit if any entry fails. Binding and install notifications retain scoped recipients. Profile-entry changes use standing bindings, and policies use old/new agent selectors. Unscoped policies, company-bound profiles, and other conservative definition changes retain synchronous transaction-atomic company-wide fanout: O(eligible company agents) locks and journal storage, without a fanout ceiling or mutation quota. This retains the accepted tradeoff from #16603 for company-wide changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local checks; results and the isolated test failure are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
522 lines
39 KiB
TypeScript
522 lines
39 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { agents, companies, createDb, plugins, toolApplications, toolConnections, toolCatalogEntries, toolProfiles, toolProfileEntries, toolProfileBindings, toolPolicies, resourceLifecycleEvents, type Db } from "@paperclipai/db";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { toolAccessService } from "../services/tool-access.js";
|
|
import { toolAccessPolicyService } from "../services/tool-access-policy.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const describeDatabase = support.supported ? describe : describe.skip;
|
|
|
|
describeDatabase("tool definition runtime synchronization", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
let companyId: string;
|
|
let agentId: string;
|
|
let peerId: string;
|
|
let pluginId: string;
|
|
let applicationId: string;
|
|
let connectionId: string;
|
|
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-definitions-"); db = createDb(database.connectionString); }, 90_000);
|
|
afterAll(async () => { await database?.cleanup(); });
|
|
afterEach(() => vi.unstubAllEnvs());
|
|
beforeEach(async () => {
|
|
companyId = randomUUID(); agentId = randomUUID(); peerId = randomUUID(); pluginId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Connector runtime fixture", issuePrefix: `C${companyId.slice(0, 6)}` });
|
|
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "runtime-fixture", version: "1.0.0", status: "ready", manifestJson: { resourceLifecycle: ["agent_runtime"], capabilities: ["events.subscribe", "agents.lifecycle.manage"] } as never });
|
|
await db.insert(agents).values([agentId, peerId].map(id => ({ id, companyId, name: id, adapterType: "paperclip_runner", lifecycleRequiredPluginIds: [pluginId] })));
|
|
const [application] = await db.insert(toolApplications).values({ companyId, name: "Fixture MCP", type: "mcp_http" }).returning(); applicationId = application.id;
|
|
const [connection] = await db.insert(toolConnections).values({ companyId, applicationId, name: "Fixture connection", uid: randomUUID(), transport: "mcp_remote", authKind: "none", enabled: true, status: "active" }).returning(); connectionId = connection.id;
|
|
});
|
|
const events = () => db.select().from(resourceLifecycleEvents).where(and(eq(resourceLifecycleEvents.companyId, companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"))).orderBy(resourceLifecycleEvents.id);
|
|
|
|
it("captures settings and removal while ignoring health bookkeeping and empty writes", async () => {
|
|
const before = (await events()).length;
|
|
await db.update(toolConnections).set({ healthCheckedAt: new Date(), lastHealthAt: new Date(), lastError: "fixture", updatedAt: new Date() }).where(eq(toolConnections.id, connectionId));
|
|
await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, randomUUID()));
|
|
expect(await events()).toHaveLength(before);
|
|
await db.update(toolConnections).set({ config: { endpoint: "https://mcp.example.test/mcp" }, enabled: false }).where(eq(toolConnections.id, connectionId));
|
|
expect((await events()).slice(before).map(row => row.resourceId).sort()).toEqual([agentId, peerId].sort());
|
|
const changed = (await events()).length;
|
|
await db.delete(toolConnections).where(eq(toolConnections.id, connectionId));
|
|
expect(await events()).toHaveLength(changed + 2);
|
|
});
|
|
it("batches catalog refresh fanout once per company per statement", async () => {
|
|
const before = (await events()).length;
|
|
await db.insert(toolCatalogEntries).values(Array.from({ length: 250 }, (_, index) => ({ companyId, applicationId, connectionId, name: `tool-${index}`, toolName: `tool-${index}`, versionHash: `fixture-${index}` })));
|
|
expect(await events()).toHaveLength(before + 2);
|
|
await db.update(toolCatalogEntries).set({ status: "quarantined" }).where(eq(toolCatalogEntries.connectionId, connectionId));
|
|
expect(await events()).toHaveLength(before + 4);
|
|
});
|
|
it("does not invalidate runtime inputs when a real health probe records failure", async () => {
|
|
const before = await events();
|
|
await expect(toolAccessService(db).checkHealth(connectionId)).rejects.toThrow();
|
|
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, connectionId));
|
|
expect(connection.healthStatus).not.toBe("unknown");
|
|
expect(connection.healthCheckedAt).not.toBeNull();
|
|
expect(await events()).toEqual(before);
|
|
});
|
|
it.each([250, 2500])("batches a real %i-tool catalog refresh and ignores unchanged rediscovery", async count => {
|
|
const tools = Array.from({ length: count }, (_, index) => ({ name: `search_${index}`, description: "Search fixture records", inputSchema: { type: "object", properties: {} }, annotations: { readOnlyHint: true } }));
|
|
await db.update(toolConnections).set({ config: { url: "https://mcp.example.test/mcp" } }).where(eq(toolConnections.id, connectionId));
|
|
const service = toolAccessService(db, {
|
|
remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 }],
|
|
remoteHttpRequest: async (_url, init) => {
|
|
const body = JSON.parse(String(init?.body));
|
|
if (body.method === "notifications/initialized") return new Response(null, { status: 202 });
|
|
return new Response(JSON.stringify({ jsonrpc: "2.0", id: body.id, result: body.method === "initialize" ? { protocolVersion: "2025-06-18" } : { tools } }), { headers: { "content-type": "application/json" } });
|
|
},
|
|
});
|
|
const before = (await events()).length;
|
|
expect((await service.refreshCatalog(connectionId)).catalog).toHaveLength(count);
|
|
const after = (await events()).length;
|
|
expect(after - before).toBeGreaterThanOrEqual(2);
|
|
expect(after - before).toBeLessThanOrEqual(count === 250 ? 12 : 24);
|
|
const first = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, connectionId)).orderBy(toolCatalogEntries.toolName);
|
|
expect((await service.refreshCatalog(connectionId)).catalog).toHaveLength(count);
|
|
expect(await events()).toHaveLength(after);
|
|
const second = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, connectionId)).orderBy(toolCatalogEntries.toolName);
|
|
expect(second.map(row => [row.id, row.firstSeenAt, row.status])).toEqual(first.map(row => [row.id, row.firstSeenAt, row.status]));
|
|
});
|
|
it.each(["catalog refresh", "profile edit", "policy edit"] as const)("serializes %s with action finalization before runtime locks", async change => {
|
|
await db.update(toolConnections).set({ config: { url: "https://mcp.example.test/mcp" } }).where(eq(toolConnections.id, connectionId));
|
|
let description = "Initial search";
|
|
const options = {
|
|
remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 as const }],
|
|
remoteHttpRequest: async (_url: string | URL | Request, init?: RequestInit) => {
|
|
const body = JSON.parse(String(init?.body));
|
|
if (body.method === "notifications/initialized") return new Response(null, { status: 202 });
|
|
return new Response(JSON.stringify({ jsonrpc: "2.0", id: body.id, result: body.method === "initialize" ? { protocolVersion: "2025-06-18" } : { tools: [{ name: "search", description, inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }] } }), { headers: { "content-type": "application/json" } });
|
|
},
|
|
};
|
|
await toolAccessService(db, options).refreshCatalog(connectionId);
|
|
const [entry] = await db.update(toolCatalogEntries).set({ status: "quarantined" }).where(eq(toolCatalogEntries.connectionId, connectionId)).returning();
|
|
const [profile] = await db.select().from(toolProfiles).where(and(eq(toolProfiles.companyId, companyId), eq(toolProfiles.profileKey, `app:${connectionId}`)));
|
|
const [policy] = change === "policy edit" ? await db.insert(toolPolicies).values({ companyId, name: "Managed search review", policyType: "require_approval", selectors: { catalogEntryId: entry.id }, config: { source: "app_gallery_finish", connectionId, catalogEntryId: entry.id } }).returning() : [];
|
|
if (change === "catalog refresh") description = "Changed search";
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, [agentId, peerId].sort()[0])).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
const pids: number[] = [];
|
|
const run = (work: (service: ReturnType<typeof toolAccessService>, policies: ReturnType<typeof toolAccessPolicyService>) => Promise<unknown>) => db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
pids.push(Number(row.pid));
|
|
return work(toolAccessService(tx as unknown as Db, options), toolAccessPolicyService(tx as unknown as Db));
|
|
});
|
|
const waitForLock = (index: number) => vi.waitFor(async () => {
|
|
expect(pids.length).toBeGreaterThan(index);
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pids[index]}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
const finish = run(writer => writer.finishGalleryAppConnection(companyId, connectionId, { enabledCatalogEntryIds: [entry.id], askFirstCatalogEntryIds: change === "policy edit" ? [entry.id] : [], reviewedCatalogEntryIds: [entry.id], access: "all_agents" }));
|
|
let refresh: Promise<unknown> | undefined;
|
|
try {
|
|
await waitForLock(0);
|
|
refresh = run((writer, policies) => change === "catalog refresh" ? writer.refreshCatalog(connectionId)
|
|
: change === "profile edit" ? writer.updateProfile(profile.id, { description: "Edited concurrently" })
|
|
: policies.updatePolicy({ companyId, policyId: policy!.id, body: { description: "Edited concurrently", config: null } }));
|
|
await waitForLock(1);
|
|
} finally { release(); await blocker; }
|
|
const results = await Promise.allSettled([finish, refresh]);
|
|
for (const result of results) if (result.status === "rejected") throw result.reason;
|
|
expect(results.map(result => result.status)).toEqual(["fulfilled", "fulfilled"]);
|
|
const [updated] = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.id, entry.id));
|
|
expect(updated).toMatchObject({ status: "active", description });
|
|
if (change === "profile edit") expect((await db.select().from(toolProfiles).where(eq(toolProfiles.id, profile.id)))[0].description).toBe("Edited concurrently");
|
|
if (change === "policy edit") expect((await db.select().from(toolPolicies).where(eq(toolPolicies.id, policy!.id)))[0].description).toBe("Edited concurrently");
|
|
}, 15000);
|
|
it("orders managed policy reorder with install saves by policy ID", async () => {
|
|
await db.update(toolConnections).set({ config: { mcpAgentAccessConfigured: true } }).where(eq(toolConnections.id, connectionId));
|
|
await db.insert(toolProfiles).values({ companyId, profileKey: `app:${connectionId}`, name: "Install profile" });
|
|
const [lower, higher] = [randomUUID(), randomUUID()].sort();
|
|
await db.insert(toolPolicies).values([higher, lower].map((id, index) => ({ id, companyId, name: `Managed policy ${index}`, policyType: "require_approval", priority: (index + 1) * 100, config: { source: "app_gallery_finish", connectionId } })));
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(toolPolicies).where(eq(toolPolicies.id, lower)).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
const pids: number[] = [];
|
|
const run = (work: (client: Db) => Promise<unknown>, sequentialScan = false) => db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
if (sequentialScan) {
|
|
await tx.execute(sql`SET LOCAL enable_indexscan = off`);
|
|
await tx.execute(sql`SET LOCAL enable_bitmapscan = off`);
|
|
}
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
pids.push(Number(row.pid));
|
|
return work(tx as unknown as Db);
|
|
});
|
|
const waitForLock = (index: number) => vi.waitFor(async () => {
|
|
expect(pids.length).toBeGreaterThan(index);
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pids[index]}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
const install = run(client => toolAccessService(client).putConnectionInstalls(connectionId, { installs: [{ targetType: "agent", targetId: peerId }] }));
|
|
let reorder: Promise<unknown> | undefined;
|
|
try {
|
|
await waitForLock(0);
|
|
reorder = run(client => toolAccessPolicyService(client).reorderPolicies(companyId, { policyIds: [lower, higher] }), true);
|
|
await waitForLock(1);
|
|
} finally { release(); await blocker; }
|
|
const results = await Promise.allSettled([install, reorder]);
|
|
for (const result of results) if (result.status === "rejected") throw result.reason;
|
|
expect(results.map(result => result.status)).toEqual(["fulfilled", "fulfilled"]);
|
|
expect((await toolAccessPolicyService(db).listPolicies(companyId)).map(policy => [policy.id, policy.priority])).toEqual([[lower, 100], [higher, 200]]);
|
|
expect((await toolAccessService(db).listConnectionInstalls(connectionId, companyId)).map(install => install.targetId)).toEqual([peerId]);
|
|
}, 15000);
|
|
it("ignores real trust-rule hits while retaining meaningful rule changes", async () => {
|
|
const [entry] = await db.insert(toolCatalogEntries).values({ companyId, applicationId, connectionId, name: "Search", toolName: "search", versionHash: "fixture" }).returning();
|
|
const [rule] = await db.insert(toolPolicies).values({ companyId, name: "Trusted search", policyType: "trust_rule", selectors: { agentId, connectionId, toolName: "search" }, config: { trustRule: { hitCount: 0 } } }).returning();
|
|
const before = await events();
|
|
const decision = await toolAccessPolicyService(db).decide({ companyId, actor: { actorType: "agent", actorId: agentId, agentId }, request: { connectionId, catalogEntryId: entry.id, toolName: "search", arguments: {} }, consumeRateLimit: true });
|
|
expect(decision).toMatchObject({ allowed: true, reasonCode: "allow_trust_rule" });
|
|
const [used] = await db.select().from(toolPolicies).where(eq(toolPolicies.id, rule.id));
|
|
expect(used.config).toMatchObject({ trustRule: { hitCount: 1, lastHitAt: expect.any(String) } });
|
|
expect(await events()).toEqual(before);
|
|
await db.update(toolPolicies).set({ config: { ...used.config, trustRule: { ...(used.config?.trustRule as object), revokedAt: new Date().toISOString() } } }).where(eq(toolPolicies.id, rule.id));
|
|
expect((await events()).slice(before.length).map(row => row.resourceId)).toEqual([agentId]);
|
|
});
|
|
it.each(["profile entry add", "profile entry edit", "profile entry delete", "profile replacement"] as const)("orders %s before action finalization and runtime fanout", async change => {
|
|
await db.update(toolConnections).set({ config: { url: "https://mcp.example.test/mcp" } }).where(eq(toolConnections.id, connectionId));
|
|
let description = "Initial search";
|
|
const options = {
|
|
remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 as const }],
|
|
remoteHttpRequest: async (_url: string | URL | Request, init?: RequestInit) => {
|
|
const body = JSON.parse(String(init?.body));
|
|
if (body.method === "notifications/initialized") return new Response(null, { status: 202 });
|
|
return new Response(JSON.stringify({ jsonrpc: "2.0", id: body.id, result: body.method === "initialize" ? { protocolVersion: "2025-06-18" } : { tools: [{ name: "search", description, inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }] } }), { headers: { "content-type": "application/json" } });
|
|
},
|
|
};
|
|
await toolAccessService(db, options).refreshCatalog(connectionId);
|
|
const [entry] = await db.update(toolCatalogEntries).set({ status: "quarantined" }).where(eq(toolCatalogEntries.connectionId, connectionId)).returning();
|
|
const [profile] = await db.select().from(toolProfiles).where(and(eq(toolProfiles.companyId, companyId), eq(toolProfiles.profileKey, `app:${connectionId}`)));
|
|
const [profileEntry] = await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.profileId, profile.id));
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, [agentId, peerId].sort()[0])).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
const pids: number[] = [];
|
|
const run = (work: (service: ReturnType<typeof toolAccessService>, policies: ReturnType<typeof toolAccessPolicyService>) => Promise<unknown>) => db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
pids.push(Number(row.pid));
|
|
return work(toolAccessService(tx as unknown as Db, options), toolAccessPolicyService(tx as unknown as Db));
|
|
});
|
|
const waitForLock = (index: number) => vi.waitFor(async () => {
|
|
expect(pids.length).toBeGreaterThan(index);
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pids[index]}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
const mutation = run(writer => change === "profile entry add"
|
|
? writer.addProfileEntry(profile.id, { selectorType: "catalog_entry", catalogEntryId: entry.id, connectionId, applicationId })
|
|
: change === "profile entry edit" ? writer.updateProfileEntry(profileEntry.id, { effect: "exclude" })
|
|
: change === "profile entry delete" ? writer.deleteProfileEntry(profileEntry.id)
|
|
: writer.updateProfile(profile.id, { entries: [{ selectorType: "catalog_entry", catalogEntryId: entry.id, connectionId, applicationId }] }));
|
|
const mutationResult = Promise.allSettled([mutation]);
|
|
let finishResult: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
try {
|
|
await waitForLock(0);
|
|
finishResult = Promise.allSettled([run(writer => writer.finishGalleryAppConnection(companyId, connectionId, { enabledCatalogEntryIds: [entry.id], askFirstCatalogEntryIds: [], reviewedCatalogEntryIds: [entry.id], access: "all_agents" }))]);
|
|
await waitForLock(1);
|
|
} finally { release(); await blocker; }
|
|
for (const result of [...await mutationResult, ...await finishResult!]) if (result.status === "rejected") throw result.reason;
|
|
const [updated] = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.id, entry.id));
|
|
expect(updated).toMatchObject({ status: "active", description });
|
|
}, 15000);
|
|
|
|
it("orders connection removal before setup finalization and runtime fanout", async () => {
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: `app:${connectionId}`, name: "Removal fixture" }).returning();
|
|
await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, selectorType: "connection", connectionId, applicationId });
|
|
const single = createDb(database.connectionString, { maxConnections: 1 });
|
|
await single.execute(sql`SET lock_timeout = '8s'`);
|
|
const [removalPid] = await single.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, [agentId, peerId].sort()[0])).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
let removal: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
const waitForLock = (pid: number) => vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pid}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
let finishPid: number | undefined;
|
|
let finish: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
try {
|
|
try {
|
|
finish = Promise.allSettled([db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
finishPid = Number(row.pid);
|
|
return toolAccessService(tx as unknown as Db).finishGalleryAppConnection(companyId, connectionId, { enabledCatalogEntryIds: [], askFirstCatalogEntryIds: [], access: "all_agents" });
|
|
})]);
|
|
await vi.waitFor(() => expect(finishPid).toBeDefined(), { timeout: 5000 });
|
|
await waitForLock(finishPid!);
|
|
removal = Promise.allSettled([toolAccessService(single).archiveConnection(connectionId, companyId)]);
|
|
await waitForLock(Number(removalPid.pid));
|
|
} finally { release(); await blocker; }
|
|
for (const result of [...await removal!, ...await finish!]) if (result.status === "rejected") throw result.reason;
|
|
expect((await db.select().from(toolConnections).where(eq(toolConnections.id, connectionId)))[0].status).toBe("archived");
|
|
} finally { release(); await blocker; await finish; await removal; await single.$client.end({ timeout: 1 }); }
|
|
}, 15000);
|
|
|
|
it.each(["add", "edit", "delete", "replace"] as const)("completes profile entry %s with a single database connection", async change => {
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "single-entry", name: "Single entry" }).returning();
|
|
const input = { selectorType: "connection" as const, applicationId, connectionId };
|
|
const [entry] = await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, ...input }).returning();
|
|
const single = createDb(database.connectionString, { maxConnections: 1 });
|
|
const service = toolAccessService(single);
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
const work = change === "add" ? service.addProfileEntry(profile.id, input)
|
|
: change === "edit" ? service.updateProfileEntry(entry.id, { effect: "exclude" })
|
|
: change === "delete" ? service.deleteProfileEntry(entry.id)
|
|
: service.updateProfile(profile.id, { entries: [{ ...input, effect: "exclude" }] });
|
|
const settled = Promise.allSettled([work]);
|
|
try {
|
|
await Promise.race([work, new Promise<never>((_, reject) => { timer = setTimeout(() => reject(new Error("Profile entry waited for another pooled connection")), 3000); })]);
|
|
const rows = await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.profileId, profile.id));
|
|
expect(rows).toHaveLength(change === "delete" ? 0 : change === "add" ? 2 : 1);
|
|
if (change === "edit" || change === "replace") expect(rows[0].effect).toBe("exclude");
|
|
} finally { clearTimeout(timer); await single.$client.end({ timeout: 1 }); await settled; }
|
|
}, 10000);
|
|
|
|
it.each(["entry", "policy"] as const)("keeps repeated single-agent %s changes scoped and atomic", async change => {
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "scoped", name: "Scoped" }).returning();
|
|
await db.insert(toolProfileBindings).values({ companyId, profileId: profile.id, targetType: "agent", targetId: agentId });
|
|
const [entry] = await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, selectorType: "connection", connectionId }).returning();
|
|
const [policy] = await db.insert(toolPolicies).values({ companyId, name: "Scoped", policyType: "deny", selectors: { agentId } }).returning();
|
|
for (let index = 0; index < 4; index++) {
|
|
const before = (await events()).length;
|
|
if (change === "entry") await db.update(toolProfileEntries).set({ effect: index % 2 ? "include" : "exclude" }).where(eq(toolProfileEntries.id, entry.id));
|
|
else await db.update(toolPolicies).set({ description: `Edit ${index}` }).where(eq(toolPolicies.id, policy.id));
|
|
expect((await events()).slice(before).map(row => row.resourceId)).toEqual([agentId]);
|
|
}
|
|
const before = await events();
|
|
await expect(db.transaction(async tx => {
|
|
await tx.update(toolPolicies).set({ description: "Rolled back" }).where(eq(toolPolicies.id, policy.id));
|
|
await tx.update(toolProfileEntries).set({ effect: "exclude" }).where(eq(toolProfileEntries.id, entry.id));
|
|
throw new Error("rollback");
|
|
})).rejects.toThrow("rollback");
|
|
expect(await events()).toEqual(before);
|
|
});
|
|
|
|
it("uses old and new policy selectors with the execution match semantics", async () => {
|
|
const [policy] = await db.insert(toolPolicies).values({ companyId, name: "Retarget", policyType: "deny", selectors: { agentId } }).returning();
|
|
const write = async (selectors: Record<string, unknown>, expected: string[]) => {
|
|
const before = (await events()).length;
|
|
await db.update(toolPolicies).set({ selectors }).where(eq(toolPolicies.id, policy.id));
|
|
expect((await events()).slice(before).map(row => row.resourceId).sort()).toEqual(expected.sort());
|
|
};
|
|
await write({ agentId: peerId }, [agentId, peerId]);
|
|
await write({ agentIds: [agentId] }, [agentId, peerId]);
|
|
await write({ agentId, agentIds: [peerId] }, [agentId]);
|
|
await write({ agentIds: ` ${peerId} ` }, [peerId]);
|
|
await write({ agentIds: [" ", null, agentId] }, [agentId, peerId]);
|
|
const whitespace = "\t\n\v\f\r \u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000\ufeff";
|
|
await write({ agentIds: `${whitespace}${peerId}${whitespace}` }, [agentId, peerId]);
|
|
await write({ agentIds: [whitespace] }, [agentId, peerId]);
|
|
await write({ agentIds: [`${whitespace}${peerId}${whitespace}`] }, [agentId, peerId]);
|
|
await write({ agentIds: [agentId] }, [agentId]);
|
|
await write({ agentId: randomUUID() }, [agentId]);
|
|
await write({ agentIds: [] }, [agentId, peerId]);
|
|
await write({ agentId: "invalid" }, [agentId, peerId]);
|
|
const before = (await events()).length;
|
|
await db.delete(toolPolicies).where(eq(toolPolicies.id, policy.id));
|
|
expect(await events()).toHaveLength(before);
|
|
});
|
|
|
|
it("uses both profile-entry parents and skips profiles outside standing assignments", async () => {
|
|
const profiles = await db.insert(toolProfiles).values(["first", "second", "unassigned"].map(profileKey => ({ companyId, profileKey, name: profileKey }))).returning();
|
|
await db.insert(toolProfileBindings).values([
|
|
{ companyId, profileId: profiles[0].id, targetType: "agent", targetId: agentId },
|
|
{ companyId, profileId: profiles[1].id, targetType: "agent", targetId: peerId },
|
|
{ companyId, profileId: profiles[2].id, targetType: "project", targetId: randomUUID() },
|
|
]);
|
|
const [entry] = await db.insert(toolProfileEntries).values({ companyId, profileId: profiles[0].id, selectorType: "connection", connectionId }).returning();
|
|
let before = (await events()).length;
|
|
await db.update(toolProfileEntries).set({ profileId: profiles[1].id }).where(eq(toolProfileEntries.id, entry.id));
|
|
expect((await events()).slice(before).map(row => row.resourceId).sort()).toEqual([agentId, peerId].sort());
|
|
before = (await events()).length;
|
|
await db.update(toolProfileEntries).set({ profileId: profiles[2].id }).where(eq(toolProfileEntries.id, entry.id));
|
|
expect((await events()).slice(before).map(row => row.resourceId)).toEqual([peerId]);
|
|
before = (await events()).length;
|
|
await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, entry.id));
|
|
expect(await events()).toHaveLength(before);
|
|
});
|
|
|
|
it("coalesces scoped definitions and preserves explicit company recipients", async () => {
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "coalesced", name: "Coalesced" }).returning();
|
|
await db.insert(toolProfileBindings).values({ companyId, profileId: profile.id, targetType: "agent", targetId: agentId });
|
|
const [entry] = await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, selectorType: "connection", connectionId }).returning();
|
|
const [policy] = await db.insert(toolPolicies).values({ companyId, name: "Coalesced", policyType: "deny", selectors: { agentId } }).returning();
|
|
let before = (await events()).length;
|
|
await db.transaction(async tx => {
|
|
await tx.update(toolProfileEntries).set({ effect: "exclude" }).where(eq(toolProfileEntries.id, entry.id));
|
|
await tx.update(toolPolicies).set({ description: "Scoped" }).where(eq(toolPolicies.id, policy.id));
|
|
});
|
|
expect((await events()).slice(before).map(row => row.resourceId)).toEqual([agentId]);
|
|
await db.insert(toolProfileBindings).values({ companyId, profileId: profile.id, targetType: "company", targetId: companyId });
|
|
before = (await events()).length;
|
|
await db.transaction(async tx => {
|
|
await tx.update(toolProfileEntries).set({ effect: "include" }).where(eq(toolProfileEntries.id, entry.id));
|
|
await tx.update(toolPolicies).set({ description: "Company" }).where(eq(toolPolicies.id, policy.id));
|
|
});
|
|
expect((await events()).slice(before).map(row => row.resourceId).sort()).toEqual([agentId, peerId].sort());
|
|
});
|
|
|
|
it("orders scoped policy reorder before company agent fanout", async () => {
|
|
const [lower, higher] = [agentId, peerId].sort();
|
|
const policies = await db.insert(toolPolicies).values([higher, lower].map((target, index) => ({ companyId, name: `Scoped ${index}`, policyType: "deny", priority: (index + 5) * 100, selectors: { agentId: target } }))).returning();
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, lower)).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
const pids: number[] = [];
|
|
const run = (work: (client: Db) => Promise<unknown>) => db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
pids.push(Number(row.pid));
|
|
return work(tx as unknown as Db);
|
|
});
|
|
const waitForLock = (index: number) => vi.waitFor(async () => {
|
|
expect(pids.length).toBeGreaterThan(index);
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pids[index]}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
const settings = Promise.allSettled([run(client => client.update(toolConnections).set({ name: "Settings race" }).where(eq(toolConnections.id, connectionId)))]);
|
|
let reorder: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
try {
|
|
await waitForLock(0);
|
|
reorder = Promise.allSettled([run(client => toolAccessPolicyService(client).reorderPolicies(companyId, { policyIds: policies.map(row => row.id) }))]);
|
|
await waitForLock(1);
|
|
} finally { release(); await blocker; }
|
|
for (const result of [...await settings, ...await reorder!]) if (result.status === "rejected") throw result.reason;
|
|
expect((await toolAccessPolicyService(db).listPolicies(companyId)).map(row => row.id)).toEqual(policies.map(row => row.id));
|
|
}, 15000);
|
|
|
|
it("locks old profile references before replacement and application deletion", async () => {
|
|
const [application] = await db.insert(toolApplications).values({ companyId, name: "Old application", type: "mcp_http" }).returning();
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "old-reference", name: "Old reference" }).returning();
|
|
await db.insert(toolProfileBindings).values({ companyId, profileId: profile.id, targetType: "agent", targetId: agentId });
|
|
await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, selectorType: "application", applicationId: application.id });
|
|
let locked!: () => void; let release!: () => void;
|
|
const held = new Promise<void>(resolve => { locked = resolve; });
|
|
const resume = new Promise<void>(resolve => { release = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, [agentId, peerId].sort()[0])).for("no key update");
|
|
locked(); await resume;
|
|
});
|
|
await held;
|
|
const pids: number[] = [];
|
|
const run = (work: (service: ReturnType<typeof toolAccessService>) => Promise<unknown>) => db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
pids.push(Number(row.pid));
|
|
return work(toolAccessService(tx as unknown as Db));
|
|
});
|
|
const waitForLock = (index: number) => vi.waitFor(async () => {
|
|
expect(pids.length).toBeGreaterThan(index);
|
|
const [row] = await db.execute(sql`SELECT wait_event_type FROM pg_stat_activity WHERE pid = ${pids[index]}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
}, { timeout: 5000 });
|
|
const replacement = Promise.allSettled([run(service => service.updateProfile(profile.id, { name: "Replacement", entries: [] }))]);
|
|
let deletion: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
try {
|
|
await waitForLock(0);
|
|
deletion = Promise.allSettled([run(service => service.deleteApplication(application.id))]);
|
|
await waitForLock(1);
|
|
} finally { release(); await blocker; }
|
|
for (const result of [...await replacement, ...await deletion!]) if (result.status === "rejected") throw result.reason;
|
|
expect(await db.select().from(toolApplications).where(eq(toolApplications.id, application.id))).toHaveLength(0);
|
|
expect(await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.profileId, profile.id))).toHaveLength(0);
|
|
expect((await db.select().from(toolProfiles).where(eq(toolProfiles.id, profile.id)))[0].name).toBe("Replacement");
|
|
}, 15000);
|
|
|
|
it("retries changed old references before taking profile runtime locks", async () => {
|
|
const [oldApp, newApp] = await db.insert(toolApplications).values(["Old retry app", "New retry app"].map(name => ({ companyId, name, type: "mcp_http" }))).returning();
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "retry-references", name: "Retry references" }).returning();
|
|
await db.insert(toolProfileBindings).values({ companyId, profileId: profile.id, targetType: "agent", targetId: agentId });
|
|
const [entry] = await db.insert(toolProfileEntries).values({ companyId, profileId: profile.id, selectorType: "application", applicationId: oldApp.id }).returning();
|
|
const single = createDb(database.connectionString, { maxConnections: 1 });
|
|
await single.execute(sql`SET lock_timeout = '8s'`);
|
|
const [replacementPid] = await single.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
let refLocked!: () => void; let releaseRef!: () => void;
|
|
const refsHeld = new Promise<void>(resolve => { refLocked = resolve; });
|
|
const refsResume = new Promise<void>(resolve => { releaseRef = resolve; });
|
|
const refBlocker = db.transaction(async tx => {
|
|
await tx.select().from(toolApplications).where(eq(toolApplications.id, oldApp.id)).for("update");
|
|
refLocked(); await refsResume;
|
|
});
|
|
await refsHeld;
|
|
const replacement = Promise.allSettled([toolAccessService(single).updateProfile(profile.id, { name: "Retried replacement", entries: [] })]);
|
|
const waitForLock = (pid: number, query?: string) => vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`SELECT wait_event_type, query FROM pg_stat_activity WHERE pid = ${pid}`);
|
|
expect(row.wait_event_type).toBe("Lock");
|
|
if (query) expect(row.query).toContain(query);
|
|
}, { timeout: 5000 });
|
|
let agentBlocker: Promise<unknown> | undefined; let releaseAgent: (() => void) | undefined;
|
|
let deletion: Promise<PromiseSettledResult<unknown>[]> | undefined;
|
|
let deletionPid: number | undefined;
|
|
try {
|
|
await waitForLock(Number(replacementPid.pid));
|
|
await toolAccessService(db).updateProfileEntry(entry.id, { applicationId: newApp.id });
|
|
await toolAccessService(db).updateProfile(profile.id, { description: "Concurrent description" });
|
|
let agentLocked!: () => void;
|
|
const agentHeld = new Promise<void>(resolve => { agentLocked = resolve; });
|
|
const agentResume = new Promise<void>(resolve => { releaseAgent = resolve; });
|
|
agentBlocker = db.transaction(async tx => {
|
|
await tx.select().from(agents).where(eq(agents.id, [agentId, peerId].sort()[0])).for("no key update");
|
|
agentLocked(); await agentResume;
|
|
});
|
|
await agentHeld;
|
|
releaseRef(); await refBlocker;
|
|
await waitForLock(Number(replacementPid.pid), 'update "tool_profiles"');
|
|
deletion = Promise.allSettled([db.transaction(async tx => {
|
|
await tx.execute(sql`SET LOCAL lock_timeout = '8s'`);
|
|
const [row] = await tx.execute(sql`SELECT pg_backend_pid() AS pid`);
|
|
deletionPid = Number(row.pid);
|
|
return toolAccessService(tx as unknown as Db).deleteApplication(newApp.id);
|
|
})]);
|
|
await vi.waitFor(() => expect(deletionPid).toBeDefined(), { timeout: 5000 });
|
|
await waitForLock(deletionPid!);
|
|
releaseAgent!(); await agentBlocker;
|
|
for (const result of [...await replacement, ...await deletion]) if (result.status === "rejected") throw result.reason;
|
|
expect((await db.select().from(toolProfiles).where(eq(toolProfiles.id, profile.id)))[0]).toMatchObject({ name: "Retried replacement", description: "Concurrent description" });
|
|
expect(await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.profileId, profile.id))).toHaveLength(0);
|
|
expect(await db.select().from(toolApplications).where(eq(toolApplications.id, newApp.id))).toHaveLength(0);
|
|
} finally {
|
|
releaseRef(); releaseAgent?.(); await refBlocker; await agentBlocker; await replacement; await deletion;
|
|
await single.$client.end({ timeout: 1 });
|
|
}
|
|
}, 20000);
|
|
|
|
it("journals application and connection configuration", async () => {
|
|
const before = (await events()).length;
|
|
await db.update(toolApplications).set({ name: "Changed application" }).where(eq(toolApplications.id, applicationId));
|
|
await db.update(toolConnections).set({ config: { endpoint: "fixture" } }).where(eq(toolConnections.id, connectionId));
|
|
expect(await events()).toHaveLength(before + 4);
|
|
});
|
|
|
|
});
|