perf(server): list skills without a folder transaction when bundled folders exist (#21715)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI polls `GET /api/companies/:companyId/skills`, and the
server runs all requests on one Node main thread
> - The cost on that thread is per database statement, so a polled read
must stay small
> - Each skills list read reconciles the bundled skill folders, and that
opened two transactions with a company advisory lock, even when all
folders already existed
> - A write-path guard must not run a lock and a transaction on every
poll
> - This pull request checks the current folders with a plain read first
and takes the lock only when a folder must change
> - The benefit is a skills list read with no transaction and no
advisory lock, and 13 fewer statements (58 to 45)

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`folderService.ensureBundledCategory` and `pruneEmptyBundledCategories`
always open a transaction and take `pg_advisory_xact_lock` for the
company folders.

**Subsystem affected**
Server: skill folders service, used by the company skills list.

**Current behavior**
Every `GET /api/companies/:companyId/skills` sends 58 statements,
including 2 transactions (BEGIN, lock, reads, COMMIT), even when nothing
changes.

**Proposed behavior**
`ensureBundledCategory` reads the company skill folders once. When the
bundled root exists and the category folder exists with the expected
name, it returns that folder. `pruneEmptyBundledCategories` computes the
empty categories with plain reads and returns when there is nothing to
delete. In all other cases, both functions take the lock and run the old
code, which reads the folders again under the lock. The same request
sends 45 statements with no transaction.

**Reason and benefit**
The polled read stops taking a company lock, and it sends fewer
statements.

**Breaking changes**
None.

## What Changed

- `ensureBundledCategory`: a plain read fast path when the category
folder and the bundled root exist and the name matches. Create and
rename still run under the company folder lock.
- `pruneEmptyBundledCategories`: a plain read pre-check. The delete
still runs under the lock, and it computes the empty set again under the
lock.
- New test `company-skills-list-statements.test.ts`: a warm skills list
read has no transaction and no advisory lock and at most 45 statements;
two concurrent first ensures create exactly one folder; two concurrent
first list requests create no duplicate folder; a renamed category is
repaired and an empty category is pruned through the locked path.

## Verification

- Red: before the change, the warm read had 2 transactions (`expected 2
to be +0`) and 58 statements.
- Green: after the change, the warm read has 0 transactions, no
`pg_advisory` statement, and 45 statements. The response body is equal
to the first read.
- Concurrency: two `ensureBundledCategory` calls at once return the same
folder, and there is exactly one row for the system key.
- `pnpm exec vitest run` on `company-skills-list-statements`,
`company-skills-service`, `company-skills-routes`,
`company-skills-runtime-concurrency`, `company-skills-catalog-service`,
`company-skills` and the folder tests: 215 tests pass. Two
`company-skills.test.ts` tests in "project workspace skill discovery"
fail on my machine with and without this change. They test file system
discovery and do not use the folders service.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. The fast path returns only when the locked path would return
the same folder without a write. Every write still happens under the
lock, after a fresh read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 21:42:10 -06:00
1 parent 30c4d2f4a1
commit 89de6cbe3e
2 files changed
+133 -15

No files matched your search

@@ -0,0 +1,104 @@
import request from "supertest";
import { and, eq, like } from "drizzle-orm";
import { expect, it } from "vitest";
import { folders } from "@paperclipai/db";
import { companySkillRoutes } from "../routes/company-skills.js";
import { folderService } from "../services/folders.js";
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
import { describeEmbeddedPostgres, routeApp, seedCompanyWithBoardAccess, useEmbeddedPostgres } from "./helpers/route-test-harness.js";
describeEmbeddedPostgres("GET /api/companies/:companyId/skills statements per request", () => {
const ctx = useEmbeddedPostgres("company-skills-list-statements-");
function meter() {
const { host, port } = (ctx.db.$client as unknown as { options: { host: string[]; port: number[] } }).options;
return meterDatabaseStatements(ctx.db, `postgres://${host[0]}:${port[0]}/skills`);
}
const bundledFolders = (companyId: string) => ctx.db.select().from(folders)
.where(and(eq(folders.companyId, companyId), like(folders.systemKey, "bundled%")));
it("lists skills with plain reads once the bundled folders exist", async () => {
const statements = meter();
try {
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills list");
const app = routeApp(ctx.db, company.actor, (db) => companySkillRoutes(db));
const first = await request(app).get(`/api/companies/${company.companyId}/skills`).expect(200);
expect(first.body.length).toBeGreaterThan(0);
const foldersAfterFirst = await bundledFolders(company.companyId);
const before = statements.read();
statements.startLog();
const second = await request(app).get(`/api/companies/${company.companyId}/skills`).expect(200);
const log = statements.stopLog();
const after = statements.read();
expect(second.body).toEqual(first.body);
expect(after.transactions - before.transactions).toBe(0);
expect(log.filter((entry) => /pg_advisory/.test(entry.sql))).toEqual([]);
expect(after.statements - before.statements).toBeLessThanOrEqual(45);
expect(await bundledFolders(company.companyId)).toEqual(foldersAfterFirst);
} finally {
statements.uninstall();
}
}, 60_000);
it("creates exactly one folder when two first requests ensure the same category at once", async () => {
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills race");
const [left, right] = await Promise.all([
folderService(ctx.db).ensureBundledCategory(company.companyId, "Race Category"),
folderService(ctx.db).ensureBundledCategory(company.companyId, "Race Category"),
]);
expect(left.id).toBe(right.id);
const rows = await bundledFolders(company.companyId);
expect(rows.filter((row) => row.systemKey === "bundled:race-category")).toHaveLength(1);
expect(rows.filter((row) => row.systemKey === "bundled")).toHaveLength(1);
const app = routeApp(ctx.db, company.actor, (db) => companySkillRoutes(db));
await Promise.all([
request(app).get(`/api/companies/${company.companyId}/skills`).expect(200),
request(app).get(`/api/companies/${company.companyId}/skills`).expect(200),
]);
const systemKeys = (await bundledFolders(company.companyId)).map((row) => row.systemKey);
expect(new Set(systemKeys).size).toBe(systemKeys.length);
}, 60_000);
it("still renames a changed category and prunes an empty one under the lock", async () => {
const statements = meter();
try {
const company = await seedCompanyWithBoardAccess(ctx.db, "Skills repair");
const svc = folderService(ctx.db);
const locked = async (write: () => Promise<unknown>) => {
const before = statements.read();
statements.startLog();
await write();
const log = statements.stopLog();
const lockIndex = log.findIndex((entry) => /pg_advisory_xact_lock/.test(entry.sql));
const writes = log.filter((entry) => /^\s*(insert|update|delete)\b/i.test(entry.sql));
return {
transactions: statements.read().transactions - before.transactions,
lockedBeforeWrite: lockIndex >= 0 && writes.every((entry) => entry.inTransaction && log.indexOf(entry) > lockIndex),
writes: writes.length,
};
};
const created = await locked(() => svc.ensureBundledCategory(company.companyId, "Repair Category"));
expect(created).toMatchObject({ transactions: 1, lockedBeforeWrite: true });
expect(created.writes).toBeGreaterThan(0);
const [category] = (await bundledFolders(company.companyId)).filter((row) => row.systemKey === "bundled:repair-category");
await ctx.db.update(folders).set({ name: "Renamed by hand" }).where(eq(folders.id, category!.id));
const renamed = await locked(() => svc.ensureBundledCategory(company.companyId, "Repair Category"));
expect(renamed).toMatchObject({ transactions: 1, lockedBeforeWrite: true, writes: 1 });
expect((await bundledFolders(company.companyId)).find((row) => row.id === category!.id)?.name).toBe("Repair Category");
expect(await locked(() => svc.pruneEmptyBundledCategories(company.companyId, ["Repair Category"])))
.toEqual({ transactions: 0, lockedBeforeWrite: false, writes: 0 });
expect((await bundledFolders(company.companyId)).map((row) => row.id)).toContain(category!.id);
expect(await locked(() => svc.pruneEmptyBundledCategories(company.companyId, [])))
.toEqual({ transactions: 1, lockedBeforeWrite: true, writes: 1 });
expect((await bundledFolders(company.companyId)).map((row) => row.id)).not.toContain(category!.id);
} finally {
statements.uninstall();
}
}, 60_000);
});
+29 -15
View File
@@ -544,14 +544,23 @@ export function folderService(db: Db, mutationLockHeld = false) {
throw conflict("Could not create project skill folder");
}
async function findCurrentBundledCategory(companyId: string, systemKey: string, name: string) {
const rows = await getRows(companyId, "skill");
const existing = rows.find((row) => row.systemKey === systemKey);
if (!existing || existing.name !== name || !rows.some((row) => row.systemKey === "bundled")) return null;
return buildFolderViews(rows).get(existing.id) ?? null;
}
async function ensureBundledCategory(companyId: string, category: string): Promise<Folder> {
if (!mutationLockHeld) {
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).ensureBundledCategory(companyId, category));
}
const root = await ensureContainer(companyId, "bundled", "Bundled");
const name = normalizeName(category);
const slug = normalizeFolderSlug(category);
const systemKey = `bundled:${slug}`;
if (!mutationLockHeld) {
const current = await findCurrentBundledCategory(companyId, systemKey, name);
if (current) return current;
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).ensureBundledCategory(companyId, category));
}
const root = await ensureContainer(companyId, "bundled", "Bundled");
for (let attempt = 0; attempt < 3; attempt += 1) {
const resolved = await uniqueSystemSlug(companyId, root.id, slug, systemKey, "bundled");
if (resolved.id) {
@@ -576,16 +585,15 @@ export function folderService(db: Db, mutationLockHeld = false) {
throw conflict("Could not create bundled skill folder");
}
async function pruneEmptyBundledCategories(companyId: string, retainedCategories: string[]): Promise<void> {
if (!mutationLockHeld) {
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).pruneEmptyBundledCategories(companyId, retainedCategories));
}
const root = await findSystemFolder(companyId, "bundled");
if (!root) return;
const rows = await getRows(companyId, "skill");
async function emptyBundledCategoryIds(companyId: string, retainedCategories: string[], rows: FolderRow[]) {
const root = rows.find((row) => row.kind === "skill" && row.systemKey === "bundled");
if (!root) return [];
const retainedSystemKeys = new Set(
retainedCategories.map((category) => `bundled:${normalizeFolderSlug(category)}`),
);
const candidates = rows.filter((row) => row.parentId === root.id && row.systemKey?.startsWith("bundled:")
&& !retainedSystemKeys.has(row.systemKey));
if (candidates.length === 0) return [];
const usedFolderIds = new Set(
await db
.select({ folderId: companySkills.folderId })
@@ -594,10 +602,16 @@ export function folderService(db: Db, mutationLockHeld = false) {
.then((skills) => skills.flatMap((skill) => skill.folderId ? [skill.folderId] : [])),
);
const parentIds = new Set(rows.flatMap((row) => row.parentId ? [row.parentId] : []));
for (const row of rows) {
if (row.parentId !== root.id || !row.systemKey?.startsWith("bundled:")) continue;
if (retainedSystemKeys.has(row.systemKey) || usedFolderIds.has(row.id) || parentIds.has(row.id)) continue;
await db.delete(folders).where(and(eq(folders.companyId, companyId), eq(folders.id, row.id)));
return candidates.filter((row) => !usedFolderIds.has(row.id) && !parentIds.has(row.id)).map((row) => row.id);
}
async function pruneEmptyBundledCategories(companyId: string, retainedCategories: string[]): Promise<void> {
if (!mutationLockHeld) {
if ((await emptyBundledCategoryIds(companyId, retainedCategories, await getRows(companyId, "skill"))).length === 0) return;
return withCompanyFolderLock(companyId, (lockedDb) => folderService(lockedDb, true).pruneEmptyBundledCategories(companyId, retainedCategories));
}
for (const id of await emptyBundledCategoryIds(companyId, retainedCategories, await getRows(companyId, "skill"))) {
await db.delete(folders).where(and(eq(folders.companyId, companyId), eq(folders.id, id)));
}
}