Files
PaperClipAI/server/src/__tests__/agent-runtime-skills.test.ts
T
DottaandClaude Opus 5.5 8f1d39da79 perf(server): cache the instance settings row per process (#21768)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load

## Linked Issues or Issue Description

**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.

**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.

**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.

## What Changed

- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.

Hot callers of the cached row:

- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
  - 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
  - A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.

## Risks

- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:50:19 -06:00

162 lines
13 KiB
TypeScript

import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq } from "drizzle-orm";
import { agents, companies, companyMemberships, companySkills, companySkillVersions, companySkillSources, projectSkillSources, projects, projectAccessMembers, createDb, instanceSettings, pluginConfig, plugins, resourceLifecycleEvents, type Db } from "@paperclipai/db";
import { LOW_TRUST_REVIEW_PRESET } from "@paperclipai/shared";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { pluginAgentRuntimeState } from "../services/plugin-agent-runtime-state.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js";
const support = await getEmbeddedPostgresTestSupport();
const describeDatabase = support.supported ? describe : describe.skip;
describeDatabase("skill runtime invalidation", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
let home: string;
let companyId: string;
let pluginId: string;
let skillId: string;
let key: string;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-skills-"); db = createDb(database.connectionString);
home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-skills-home-"));
}, 90_000);
afterAll(async () => { await database?.cleanup(); await fs.rm(home, { recursive: true, force: true }); });
afterEach(() => vi.unstubAllEnvs());
beforeEach(async () => {
vi.stubEnv("PAPERCLIP_HOME", home);
companyId = randomUUID(); pluginId = randomUUID(); skillId = randomUUID(); key = `company/${companyId}/fixture`;
await db.insert(companies).values({ id: companyId, name: "Skill runtime fixture", issuePrefix: `S${companyId.slice(0, 6)}` });
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "runtime-fixture", version: "1.0.0", status: "ready", manifestJson: { resourceLifecycle: ["agent_runtime"], capabilities: ["events.subscribe", "agents.lifecycle.manage"] } as never });
await db.insert(pluginConfig).values({ companyId, pluginId, configJson: {} });
await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "fixture", name: "Fixture", markdown: "old", sourceType: "github", sourceLocator: "https://github.com/fixture/skills" });
});
async function agent(selection: unknown[] = [key], company = companyId) {
const [created] = await db.insert(agents).values({ companyId: company, name: randomUUID(), adapterType: "paperclip_runner", lifecycleRequiredPluginIds: [pluginId], adapterConfig: { paperclipSkillSync: { desiredSkills: selection } } }).returning();
return created;
}
const events = () => db.select().from(resourceLifecycleEvents).where(and(eq(resourceLifecycleEvents.companyId, companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"))).orderBy(resourceLifecycleEvents.id);
async function publish(content: string, revision: number) {
return db.transaction(async tx => {
const [version] = await tx.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: revision, fileInventory: [{ path: "SKILL.md", kind: "skill", content }] }).returning();
await tx.update(companySkills).set({ markdown: content, currentVersionId: version.id }).where(eq(companySkills.id, skillId));
return version;
});
}
it("invalidates string, structured, and legacy alias assignments while leaving other agents untouched", async () => {
const first = await agent(); const second = await agent([{ key, versionId: randomUUID() }]); const alias = await agent(["fixture"]);
await agent([]); await agent(["unrelated"]);
const otherCompany = randomUUID();
await db.insert(companies).values({ id: otherCompany, name: "Other company", issuePrefix: `X${otherCompany.slice(0, 6)}` });
await agent([key], otherCompany);
const before = (await events()).length;
await db.update(companySkills).set({ markdown: "changed" }).where(eq(companySkills.id, skillId));
expect((await events()).slice(before).map(row => row.resourceId).sort()).toEqual([first.id, second.id, alias.id].sort());
const after = (await events()).length;
await db.update(companySkills).set({ starCount: 9, updatedAt: new Date() }).where(eq(companySkills.id, skillId));
expect(await events()).toHaveLength(after);
});
it.each(["paperclip", "complain", "suggestion-box"])("invalidates implicit legacy %s skills with an empty selection", async slug => {
const legacy = await agent([]);
const native = await agent([]);
await db.update(agents).set({ adapterType: "process" }).where(eq(agents.id, legacy.id));
const [skill] = await db.insert(companySkills).values({ companyId, key: `paperclipai/paperclip/${slug}`, slug, name: slug, markdown: "implicit" }).returning();
const before = (await events()).length;
await db.update(companySkills).set({ markdown: "changed implicit" }).where(eq(companySkills.id, skill.id));
expect((await events()).slice(before)).toEqual([expect.objectContaining({ resourceId: legacy.id })]);
await db.delete(companySkills).where(eq(companySkills.id, skill.id));
expect((await events()).slice(before).map(row => row.resourceId)).toEqual([legacy.id, legacy.id]);
expect((await events()).slice(before).some(row => row.resourceId === native.id)).toBe(false);
});
it("captures companion-file changes, rename, and deletion using both old and new references", async () => {
const old = await agent(); const next = await agent([`${key}-next`]);
const before = (await events()).length;
await db.update(companySkills).set({ fileInventory: [{ path: "references/guide.md", kind: "reference", size: 5 }] }).where(eq(companySkills.id, skillId));
expect((await events()).slice(before)).toEqual([expect.objectContaining({ resourceId: old.id })]);
const renamed = (await events()).length;
await db.update(companySkills).set({ key: `${key}-next`, slug: "next" }).where(eq(companySkills.id, skillId));
expect((await events()).slice(renamed).map(row => row.resourceId).sort()).toEqual([old.id, next.id].sort());
const deleted = (await events()).length;
await db.delete(companySkills).where(eq(companySkills.id, skillId));
expect((await events()).slice(deleted)).toEqual([expect.objectContaining({ resourceId: next.id })]);
});
it("invalidates private-source access changes while ignoring source polling bookkeeping", async () => {
const assigned = await agent();
const [project] = await db.insert(projects).values({ companyId, name: "Private source", visibility: "private" }).returning();
const [source] = await db.insert(companySkillSources).values({ companyId, repositoryUrl: "https://github.com/fixture/private", fullName: "fixture/private", trackingRef: "HEAD", accessScope: "projects" }).returning();
await db.update(companySkills).set({ accessSourceId: source.id }).where(eq(companySkills.id, skillId));
const [binding] = await db.insert(projectSkillSources).values({ companyId, projectId: project.id, sourceId: source.id, repositoryUrl: source.repositoryUrl, trackingRef: "HEAD" }).returning();
await db.insert(agents).values(Array.from({ length: 250 }, () => ({ companyId, name: randomUUID(), adapterType: "paperclip_runner", adapterConfig: { paperclipSkillSync: { desiredSkills: ["unrelated"] } } })));
const before = (await events()).length;
await db.update(projectSkillSources).set({ nextSyncAt: new Date(), failureCount: 1 }).where(eq(projectSkillSources.id, binding.id));
expect(await events()).toHaveLength(before);
const [access] = await db.insert(projectAccessMembers).values({ companyId, projectId: project.id, subjectType: "agent", subjectId: assigned.id }).returning();
expect(await events()).toHaveLength(before + 1);
await db.delete(projectAccessMembers).where(eq(projectAccessMembers.id, access.id));
expect(await events()).toHaveLength(before + 2);
await db.update(projects).set({ visibility: "open" }).where(eq(projects.id, project.id));
expect(await events()).toHaveLength(before + 3);
expect((await events()).slice(before).map(row => row.resourceId)).toEqual([assigned.id, assigned.id, assigned.id]);
const [unrelatedProject] = await db.insert(projects).values({ companyId, name: "Unrelated", visibility: "private" }).returning();
await db.insert(projectAccessMembers).values({ companyId, projectId: unrelatedProject.id, subjectType: "agent", subjectId: assigned.id });
await db.update(projects).set({ visibility: "open" }).where(eq(projects.id, unrelatedProject.id));
expect(await events()).toHaveLength(before + 3);
await db.update(projectSkillSources).set({ sourceId: null }).where(eq(projectSkillSources.id, binding.id));
expect((await events()).slice(before + 3)).toEqual([expect.objectContaining({ resourceId: assigned.id })]);
});
it("keeps publication and invalidation in the same transaction", async () => {
await agent(); const before = await events();
await expect(db.transaction(async tx => {
await tx.update(companySkills).set({ markdown: "rolled back" }).where(eq(companySkills.id, skillId));
throw new Error("rollback");
})).rejects.toThrow("rollback");
expect(await events()).toEqual(before);
expect((await db.select().from(companySkills).where(eq(companySkills.id, skillId)))[0].markdown).toBe("old");
});
it("invalidates policy-only project access changes and removes or restores the selected skill", async () => {
const assigned = await agent();
await agent([]);
await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: assigned.id, status: "active" });
const [project] = await db.insert(projects).values({ companyId, name: "Policy-bound source", visibility: "open" }).returning();
const [source] = await db.insert(companySkillSources).values({ companyId, repositoryUrl: "https://github.com/fixture/policy", fullName: "fixture/policy", trackingRef: "HEAD", accessScope: "projects" }).returning();
await db.insert(projectSkillSources).values({ companyId, projectId: project.id, sourceId: source.id, repositoryUrl: source.repositoryUrl, trackingRef: "HEAD" });
await db.update(companySkills).set({ accessSourceId: source.id }).where(eq(companySkills.id, skillId));
await publish("policy-scoped content", 1);
const state = () => pluginAgentRuntimeState(db, pluginId, companyId, assigned.id);
expect((await state())!.skills.map(skill => skill.key)).toEqual([key]);
const before = (await events()).length;
const restricted = { authorizationPolicy: { trustBoundary: { mode: LOW_TRUST_REVIEW_PRESET, projectIds: [] } } };
await db.update(projects).set({ executionWorkspacePolicy: restricted }).where(eq(projects.id, project.id));
expect((await events()).slice(before).map(event => event.resourceId)).toEqual([assigned.id]);
expect((await state())!.skills).toEqual([]);
await db.update(projects).set({ executionWorkspacePolicy: null }).where(eq(projects.id, project.id));
expect((await events()).slice(before).map(event => event.resourceId)).toEqual([assigned.id, assigned.id]);
expect((await state())!.skills.map(skill => skill.key)).toEqual([key]);
const restored = (await events()).length;
await db.update(projects).set({ executionWorkspacePolicy: null }).where(eq(projects.id, project.id));
expect(await events()).toHaveLength(restored);
});
it("resolves immutable pinned and latest skill sources and removes unassigned skills", async () => {
await instanceSettingsService(db).getExperimental();
await db.update(instanceSettings).set({ experimental: { enableBetaSkills: true } });
invalidateInstanceSettingsCache();
const first = await publish("first version", 1);
const pinned = await agent([{ key, versionId: first.id }]);
const latest = await agent();
const second = await publish("second version", 2);
const pinnedState = await pluginAgentRuntimeState(db, pluginId, companyId, pinned.id);
const latestState = await pluginAgentRuntimeState(db, pluginId, companyId, latest.id);
expect(pinnedState!.skills).toEqual([expect.objectContaining({ key, versionId: first.id })]);
expect(latestState!.skills).toEqual([expect.objectContaining({ key, versionId: second.id })]);
expect(await fs.readFile(path.join(pinnedState!.skills[0].source, "SKILL.md"), "utf8")).toBe("first version");
expect(await fs.readFile(path.join(latestState!.skills[0].source, "SKILL.md"), "utf8")).toBe("second version");
await db.update(agents).set({ adapterConfig: { paperclipSkillSync: { desiredSkills: [] } } }).where(eq(agents.id, latest.id));
expect((await pluginAgentRuntimeState(db, pluginId, companyId, latest.id))!.skills).toEqual([]);
});
});