mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
perf(server): list skills without a folder transaction when bundled folders exist (#21715)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI polls `GET /api/companies/:companyId/skills`, and the server runs all requests on one Node main thread > - The cost on that thread is per database statement, so a polled read must stay small > - Each skills list read reconciles the bundled skill folders, and that opened two transactions with a company advisory lock, even when all folders already existed > - A write-path guard must not run a lock and a transaction on every poll > - This pull request checks the current folders with a plain read first and takes the lock only when a folder must change > - The benefit is a skills list read with no transaction and no advisory lock, and 13 fewer statements (58 to 45) ## Linked Issues or Issue Description **What existing behavior does this improve?** `folderService.ensureBundledCategory` and `pruneEmptyBundledCategories` always open a transaction and take `pg_advisory_xact_lock` for the company folders. **Subsystem affected** Server: skill folders service, used by the company skills list. **Current behavior** Every `GET /api/companies/:companyId/skills` sends 58 statements, including 2 transactions (BEGIN, lock, reads, COMMIT), even when nothing changes. **Proposed behavior** `ensureBundledCategory` reads the company skill folders once. When the bundled root exists and the category folder exists with the expected name, it returns that folder. `pruneEmptyBundledCategories` computes the empty categories with plain reads and returns when there is nothing to delete. In all other cases, both functions take the lock and run the old code, which reads the folders again under the lock. The same request sends 45 statements with no transaction. **Reason and benefit** The polled read stops taking a company lock, and it sends fewer statements. **Breaking changes** None. ## What Changed - `ensureBundledCategory`: a plain read fast path when the category folder and the bundled root exist and the name matches. Create and rename still run under the company folder lock. - `pruneEmptyBundledCategories`: a plain read pre-check. The delete still runs under the lock, and it computes the empty set again under the lock. - New test `company-skills-list-statements.test.ts`: a warm skills list read has no transaction and no advisory lock and at most 45 statements; two concurrent first ensures create exactly one folder; two concurrent first list requests create no duplicate folder; a renamed category is repaired and an empty category is pruned through the locked path. ## Verification - Red: before the change, the warm read had 2 transactions (`expected 2 to be +0`) and 58 statements. - Green: after the change, the warm read has 0 transactions, no `pg_advisory` statement, and 45 statements. The response body is equal to the first read. - Concurrency: two `ensureBundledCategory` calls at once return the same folder, and there is exactly one row for the system key. - `pnpm exec vitest run` on `company-skills-list-statements`, `company-skills-service`, `company-skills-routes`, `company-skills-runtime-concurrency`, `company-skills-catalog-service`, `company-skills` and the folder tests: 215 tests pass. Two `company-skills.test.ts` tests in "project workspace skill discovery" fail on my machine with and without this change. They test file system discovery and do not use the folders service. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The fast path returns only when the locked path would return the same folder without a write. Every write still happens under the lock, after a fresh read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
30c4d2f4a1
commit
89de6cbe3e
2 files changed
+133
-15
No files matched your search
@@ -0,0 +1,104 @@
|
||||
import request from "supertest";
|
||||
import { and, eq, like } from "drizzle-orm";
|
||||
import { expect, it } from "vitest";
|
||||
import { folders } from "@paperclipai/db";
|
||||
import { companySkillRoutes } from "../routes/company-skills.js";
|
||||
import { folderService } from "../services/folders.js";
|
||||
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
|
||||
import { describeEmbeddedPostgres, routeApp, seedCompanyWithBoardAccess, useEmbeddedPostgres } from "./helpers/route-test-harness.js";
|
||||
|
||||
describeEmbeddedPostgres("GET /api/companies/:companyId/skills statements per request", () => {
|
||||
const ctx = useEmbeddedPostgres("company-skills-list-statements-");
|
||||
|
||||
function meter() {
|
||||
const { host, port } = (ctx.db.$client as unknown as { options: { host: string[]; port: number[] } }).options;
|
||||
return meterDatabaseStatements(ctx.db, `postgres://${host[0]}:${port[0]}/skills`);
|
||||
}
|
||||
|
||||
const bundledFolders = (companyId: string) => ctx.db.select().from(folders)
|
||||
.where(and(eq(folders.companyId, companyId), like(folders.systemKey, "bundled%")));
|
||||
|
||||
it("lists skills with plain reads once the bundled folders exist", async () => {
|
||||
const statements = meter();
|
||||
try {
|
||||
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills list");
|
||||
const app = routeApp(ctx.db, company.actor, (db) => companySkillRoutes(db));
|
||||
const first = await request(app).get(`/api/companies/${company.companyId}/skills`).expect(200);
|
||||
expect(first.body.length).toBeGreaterThan(0);
|
||||
const foldersAfterFirst = await bundledFolders(company.companyId);
|
||||
|
||||
const before = statements.read();
|
||||
statements.startLog();
|
||||
const second = await request(app).get(`/api/companies/${company.companyId}/skills`).expect(200);
|
||||
const log = statements.stopLog();
|
||||
const after = statements.read();
|
||||
|
||||
expect(second.body).toEqual(first.body);
|
||||
expect(after.transactions - before.transactions).toBe(0);
|
||||
expect(log.filter((entry) => /pg_advisory/.test(entry.sql))).toEqual([]);
|
||||
expect(after.statements - before.statements).toBeLessThanOrEqual(45);
|
||||
expect(await bundledFolders(company.companyId)).toEqual(foldersAfterFirst);
|
||||
} finally {
|
||||
statements.uninstall();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("creates exactly one folder when two first requests ensure the same category at once", async () => {
|
||||
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills race");
|
||||
const [left, right] = await Promise.all([
|
||||
folderService(ctx.db).ensureBundledCategory(company.companyId, "Race Category"),
|
||||
folderService(ctx.db).ensureBundledCategory(company.companyId, "Race Category"),
|
||||
]);
|
||||
expect(left.id).toBe(right.id);
|
||||
const rows = await bundledFolders(company.companyId);
|
||||
expect(rows.filter((row) => row.systemKey === "bundled:race-category")).toHaveLength(1);
|
||||
expect(rows.filter((row) => row.systemKey === "bundled")).toHaveLength(1);
|
||||
|
||||
const app = routeApp(ctx.db, company.actor, (db) => companySkillRoutes(db));
|
||||
await Promise.all([
|
||||
request(app).get(`/api/companies/${company.companyId}/skills`).expect(200),
|
||||
request(app).get(`/api/companies/${company.companyId}/skills`).expect(200),
|
||||
]);
|
||||
const systemKeys = (await bundledFolders(company.companyId)).map((row) => row.systemKey);
|
||||
expect(new Set(systemKeys).size).toBe(systemKeys.length);
|
||||
}, 60_000);
|
||||
|
||||
it("still renames a changed category and prunes an empty one under the lock", async () => {
|
||||
const statements = meter();
|
||||
try {
|
||||
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills repair");
|
||||
const svc = folderService(ctx.db);
|
||||
const locked = async (write: () => Promise<unknown>) => {
|
||||
const before = statements.read();
|
||||
statements.startLog();
|
||||
await write();
|
||||
const log = statements.stopLog();
|
||||
const lockIndex = log.findIndex((entry) => /pg_advisory_xact_lock/.test(entry.sql));
|
||||
const writes = log.filter((entry) => /^\s*(insert|update|delete)\b/i.test(entry.sql));
|
||||
return {
|
||||
transactions: statements.read().transactions - before.transactions,
|
||||
lockedBeforeWrite: lockIndex >= 0 && writes.every((entry) => entry.inTransaction && log.indexOf(entry) > lockIndex),
|
||||
writes: writes.length,
|
||||
};
|
||||
};
|
||||
|
||||
const created = await locked(() => svc.ensureBundledCategory(company.companyId, "Repair Category"));
|
||||
expect(created).toMatchObject({ transactions: 1, lockedBeforeWrite: true });
|
||||
expect(created.writes).toBeGreaterThan(0);
|
||||
const [category] = (await bundledFolders(company.companyId)).filter((row) => row.systemKey === "bundled:repair-category");
|
||||
await ctx.db.update(folders).set({ name: "Renamed by hand" }).where(eq(folders.id, category!.id));
|
||||
const renamed = await locked(() => svc.ensureBundledCategory(company.companyId, "Repair Category"));
|
||||
expect(renamed).toMatchObject({ transactions: 1, lockedBeforeWrite: true, writes: 1 });
|
||||
expect((await bundledFolders(company.companyId)).find((row) => row.id === category!.id)?.name).toBe("Repair Category");
|
||||
|
||||
expect(await locked(() => svc.pruneEmptyBundledCategories(company.companyId, ["Repair Category"])))
|
||||
.toEqual({ transactions: 0, lockedBeforeWrite: false, writes: 0 });
|
||||
expect((await bundledFolders(company.companyId)).map((row) => row.id)).toContain(category!.id);
|
||||
expect(await locked(() => svc.pruneEmptyBundledCategories(company.companyId, [])))
|
||||
.toEqual({ transactions: 1, lockedBeforeWrite: true, writes: 1 });
|
||||
expect((await bundledFolders(company.companyId)).map((row) => row.id)).not.toContain(category!.id);
|
||||
} finally {
|
||||
statements.uninstall();
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -544,14 +544,23 @@ export function folderService(db: Db, mutationLockHeld = false) {
|
||||
throw conflict("Could not create project skill folder");
|
||||
}
|
||||
|
||||
async function findCurrentBundledCategory(companyId: string, systemKey: string, name: string) {
|
||||
const rows = await getRows(companyId, "skill");
|
||||
const existing = rows.find((row) => row.systemKey === systemKey);
|
||||
if (!existing || existing.name !== name || !rows.some((row) => row.systemKey === "bundled")) return null;
|
||||
return buildFolderViews(rows).get(existing.id) ?? null;
|
||||
}
|
||||
|
||||
async function ensureBundledCategory(companyId: string, category: string): Promise<Folder> {
|
||||
if (!mutationLockHeld) {
|
||||
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).ensureBundledCategory(companyId, category));
|
||||
}
|
||||
const root = await ensureContainer(companyId, "bundled", "Bundled");
|
||||
const name = normalizeName(category);
|
||||
const slug = normalizeFolderSlug(category);
|
||||
const systemKey = `bundled:${slug}`;
|
||||
if (!mutationLockHeld) {
|
||||
const current = await findCurrentBundledCategory(companyId, systemKey, name);
|
||||
if (current) return current;
|
||||
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).ensureBundledCategory(companyId, category));
|
||||
}
|
||||
const root = await ensureContainer(companyId, "bundled", "Bundled");
|
||||
for (let attempt = 0; attempt < 3; attempt += 1) {
|
||||
const resolved = await uniqueSystemSlug(companyId, root.id, slug, systemKey, "bundled");
|
||||
if (resolved.id) {
|
||||
@@ -576,16 +585,15 @@ export function folderService(db: Db, mutationLockHeld = false) {
|
||||
throw conflict("Could not create bundled skill folder");
|
||||
}
|
||||
|
||||
async function pruneEmptyBundledCategories(companyId: string, retainedCategories: string[]): Promise<void> {
|
||||
if (!mutationLockHeld) {
|
||||
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).pruneEmptyBundledCategories(companyId, retainedCategories));
|
||||
}
|
||||
const root = await findSystemFolder(companyId, "bundled");
|
||||
if (!root) return;
|
||||
const rows = await getRows(companyId, "skill");
|
||||
async function emptyBundledCategoryIds(companyId: string, retainedCategories: string[], rows: FolderRow[]) {
|
||||
const root = rows.find((row) => row.kind === "skill" && row.systemKey === "bundled");
|
||||
if (!root) return [];
|
||||
const retainedSystemKeys = new Set(
|
||||
retainedCategories.map((category) => `bundled:${normalizeFolderSlug(category)}`),
|
||||
);
|
||||
const candidates = rows.filter((row) => row.parentId === root.id && row.systemKey?.startsWith("bundled:")
|
||||
&& !retainedSystemKeys.has(row.systemKey));
|
||||
if (candidates.length === 0) return [];
|
||||
const usedFolderIds = new Set(
|
||||
await db
|
||||
.select({ folderId: companySkills.folderId })
|
||||
@@ -594,10 +602,16 @@ export function folderService(db: Db, mutationLockHeld = false) {
|
||||
.then((skills) => skills.flatMap((skill) => skill.folderId ? [skill.folderId] : [])),
|
||||
);
|
||||
const parentIds = new Set(rows.flatMap((row) => row.parentId ? [row.parentId] : []));
|
||||
for (const row of rows) {
|
||||
if (row.parentId !== root.id || !row.systemKey?.startsWith("bundled:")) continue;
|
||||
if (retainedSystemKeys.has(row.systemKey) || usedFolderIds.has(row.id) || parentIds.has(row.id)) continue;
|
||||
await db.delete(folders).where(and(eq(folders.companyId, companyId), eq(folders.id, row.id)));
|
||||
return candidates.filter((row) => !usedFolderIds.has(row.id) && !parentIds.has(row.id)).map((row) => row.id);
|
||||
}
|
||||
|
||||
async function pruneEmptyBundledCategories(companyId: string, retainedCategories: string[]): Promise<void> {
|
||||
if (!mutationLockHeld) {
|
||||
if ((await emptyBundledCategoryIds(companyId, retainedCategories, await getRows(companyId, "skill"))).length === 0) return;
|
||||
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).pruneEmptyBundledCategories(companyId, retainedCategories));
|
||||
}
|
||||
for (const id of await emptyBundledCategoryIds(companyId, retainedCategories, await getRows(companyId, "skill"))) {
|
||||
await db.delete(folders).where(and(eq(folders.companyId, companyId), eq(folders.id, id)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user