From 8f1d39da79d989e7b64aa3d8296153e71b196699 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 09:50:19 -0600 Subject: [PATCH] perf(server): cache the instance settings row per process (#21768) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process runs every agent on one Node main thread, and each database statement costs main-thread time > - The `default` instance settings row is read on hot paths: live run event delivery, run log redaction, the stranded-task recovery check, runner tool calls, adapter routes and run preparation > - A staging measurement with 7 live runs showed the full-row settings read at 10.24 statements per second, and 3.5 to 5.4 per second when idle > - The row changes only when an admin or a boot step writes it, so almost every read returns the same value > - This pull request adds a per-process read-through cache for that row, with in-process invalidation on every write and a 2 second age bound for changes made by another process > - The benefit is at most one settings read per 2 seconds per server process, whatever the load ## Linked Issues or Issue Description **Is your feature request related to a problem?** The full `instance_settings` row read (`select id, singleton_key, default_environment_id, general, experimental ...`) runs at 10.24 statements per second on staging with 7 live runs, and 3.5 to 5.4 per second at idle. It scales with run events and recovery checks, not with settings changes. **Describe the solution you would like** Cache the row per process. Drop the cached row when a settings write commits, in this process and, through the existing work-signal notification, in the other process during a deploy overlap. Bound the age of a cached row to 2 seconds, so any other change (a direct database edit, a second process outside an overlap) takes effect within 2 seconds. **Describe alternatives you have considered** A database trigger with LISTEN/NOTIFY would also catch direct edits, but it needs a migration, and work notifications cross processes only during a deploy overlap today. The existing signal plus a 2 second bound keeps the change small. ## What Changed - New `server/src/services/instance-settings-cache.ts`: a read-through cache keyed by the database client. Concurrent misses share one read. A cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old, measured from the start of the read. Every caller gets its own copy. Reads through a transaction handle skip the cache. - `instanceSettingsService` `get`, `getGeneral` and `getExperimental` use the cache. Write paths (`update`, `updateGeneral`, `updateExperimental`, and the feedback vote save) still read the row fresh. - Every settings write publishes an `instance-settings-changed` work signal through the existing work-signal channel (`@paperclipai/db/work-signals`). Inside a transaction it is the first signal of the write, so its `pg_notify` payload goes out before the other wakes of the same commit (for example `status-card-evaluation`). This process drops the cached row when the write commits. During a deploy overlap, when work notifications cross processes, the other process drops its cached row on receipt, before a worker woken by the same write reads the settings. A reconnect of the notification listener also drops the row. - `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its signal is sent when its transaction commits. - A read that starts before an invalidation never stores its result, so an old value can not come back after a write. - The test setup clears the cache before each test, because many tests write the row directly. Hot callers of the cached row: - Live run event delivery: `live-run-event-redaction.ts` (`censorUsernameInLogs`), once per delivery batch. - Run output and run lifecycle redaction: `heartbeat.ts` `getCurrentUserRedactionOptions`, used by `run-output.ts`, `run-lifecycle.ts`, `run-completion.ts`. - The stranded-task recovery check: `recovery/service.ts` reads `getExperimental().enableAgentChat` once per checked conversation issue. - Runner tool calls: `paperclip-runner-tool-authority.ts` (`enableIsolatedWorkspaces`, `enableExternalObjects`). - Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per request. - Run preparation: `heartbeat.ts` (`instance_settings` preparation step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`, `run-control.ts`. - Issue and comment paths in `issues.ts` (`censorUsernameInLogs`, `enableAgentChat`). ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL): - 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service objects) after one warm read: 0 statements. Without the cache the same test measures 150 statements (red run). - 20 concurrent misses: 1 statement. - `updateExperimental` and `updateGeneral` are visible at once to a new read in the same process. - `fillUnsetInstanceDefault` (transaction write of the default environment) is visible at once after it returns. - A change written by a second client (another process) becomes visible within 2000 ms plus a 500 ms test margin. - A caller that changes the returned object does not change the cached row. - A read through a transaction handle sees the transaction's uncommitted write, two such reads issue 2 statements, and the cache does not keep the write after rollback. - A read that started before an invalidation is not stored. - A received `instance-settings-changed` notification from another process drops the cached row (next read issues 1 statement). - `updateExperimental({ enableStatusCards: true })` sends `instance-settings-changed` as its first notification, before `status-card-evaluation`; `updateGeneral` sends it alone. - The server tests that use instance settings or write the row directly (230 files): see the run notes in the first comment. ## Risks - A direct database edit, or a change from a second process outside a deploy overlap, takes up to 2 seconds to apply in this process. Before this change it applied at the next read. This includes access flags such as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the default environment. - A change made through the settings service applies at once in this process, and during a deploy overlap at once in the other process. - A new code path that writes the `default` row must call `announceInstanceSettingsChange`, or its change applies within 2 seconds. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no user-facing setting or command changed) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../agent-runtime-assignments.test.ts | 2 + .../__tests__/agent-runtime-skills.test.ts | 2 + server/src/__tests__/ai-connections.test.ts | 2 + .../__tests__/instance-settings-cache.test.ts | 168 ++++++++++++++++++ .../__tests__/project-environments.test.ts | 15 +- server/src/__tests__/setup-supertest.ts | 6 + server/src/__tests__/smoke-lab.test.ts | 2 + .../src/middleware/cloud-tenant-actor.test.ts | 4 +- .../cloud-tenant-sync-postgres.test.ts | 3 + server/src/services/feedback.ts | 2 + .../src/services/instance-settings-cache.ts | 63 +++++++ server/src/services/instance-settings.ts | 14 +- .../native-run-log-watermark.test.ts | 3 +- 13 files changed, 278 insertions(+), 8 deletions(-) create mode 100644 server/src/__tests__/instance-settings-cache.test.ts create mode 100644 server/src/services/instance-settings-cache.ts diff --git a/server/src/__tests__/agent-runtime-assignments.test.ts b/server/src/__tests__/agent-runtime-assignments.test.ts index 6c9b9bc2550..a049dd6b92f 100644 --- a/server/src/__tests__/agent-runtime-assignments.test.ts +++ b/server/src/__tests__/agent-runtime-assignments.test.ts @@ -15,6 +15,7 @@ import { toolAccessService } from "../services/tool-access.js"; import { createToolGatewayService } from "../services/tool-gateway.js"; import { connectionIntentService } from "../services/connection-intents.js"; import { aiConnectionService } from "../services/ai-connections.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; vi.mock("../services/subscription-refresh.js", () => ({ refreshSubscriptionConnection: vi.fn().mockResolvedValue(undefined) })); @@ -170,6 +171,7 @@ describeDatabase("connection assignment synchronization", () => { await pluginAgentRuntimeState(db, pluginId, companyId, agentId); const before = (await events()).length; await db.update(instanceSettings).set({ experimental: { enableChatConnectors: true } }); + invalidateInstanceSettingsCache(); expect(await events()).toHaveLength(before + 2); expect((await pluginAgentRuntimeState(db, pluginId, companyId, agentId))!.connectionAssignments.features).toEqual({ publicMcp: false, chatConnectors: true }); }); diff --git a/server/src/__tests__/agent-runtime-skills.test.ts b/server/src/__tests__/agent-runtime-skills.test.ts index 0f2f5d1f5eb..26207f985de 100644 --- a/server/src/__tests__/agent-runtime-skills.test.ts +++ b/server/src/__tests__/agent-runtime-skills.test.ts @@ -9,6 +9,7 @@ import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { pluginAgentRuntimeState } from "../services/plugin-agent-runtime-state.js"; import { instanceSettingsService } from "../services/instance-settings.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; const support = await getEmbeddedPostgresTestSupport(); const describeDatabase = support.supported ? describe : describe.skip; @@ -143,6 +144,7 @@ describeDatabase("skill runtime invalidation", () => { it("resolves immutable pinned and latest skill sources and removes unassigned skills", async () => { await instanceSettingsService(db).getExperimental(); await db.update(instanceSettings).set({ experimental: { enableBetaSkills: true } }); + invalidateInstanceSettingsCache(); const first = await publish("first version", 1); const pinned = await agent([{ key, versionId: first.id }]); const latest = await agent(); diff --git a/server/src/__tests__/ai-connections.test.ts b/server/src/__tests__/ai-connections.test.ts index ae747a42c69..3bb34223966 100644 --- a/server/src/__tests__/ai-connections.test.ts +++ b/server/src/__tests__/ai-connections.test.ts @@ -49,6 +49,7 @@ import * as copilotProbe from "../services/copilot-connection-probe.js"; import * as settingsModule from "../services/instance-settings.js"; import { agentRoutes } from "../routes/agents.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; vi.mock("../services/local-ai-browser-login.js", () => ({ startLocalBrowserLogin: () => ({ authorizationUrl: "https://auth.openai.com/codex/device", code: "ABCD-EFGHJ", abort: () => {} }), })); @@ -2519,6 +2520,7 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello" await db.insert(toolConnections).values(ids.map(id => ({ ...connection, id, uid: id, name: `Cost ${id}` }))); await db.insert(connectionGrants).values(ids.map(connectionId => ({ ...grant, id: randomUUID(), connectionId }))); selects.mockClear(); + invalidateInstanceSettingsCache(); const large = await request(app).get(url); largeQueries = selects.mock.calls.length; expect(large.status).toBe(200); diff --git a/server/src/__tests__/instance-settings-cache.test.ts b/server/src/__tests__/instance-settings-cache.test.ts new file mode 100644 index 00000000000..ce433d4e1fc --- /dev/null +++ b/server/src/__tests__/instance-settings-cache.test.ts @@ -0,0 +1,168 @@ +import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { createDb, environments, companies, instanceSettings, type Db } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createStatementCensus } from "./helpers/statement-census.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { INSTANCE_SETTINGS_CACHE_TTL_MS, INSTANCE_SETTINGS_CHANGED_TOPIC, invalidateInstanceSettingsCache, readThroughInstanceSettings } from "../services/instance-settings-cache.js"; +import { DATABASE_WORK_CHANNEL, enableDatabaseWorkNotifications, receiveDatabaseWorkNotification } from "@paperclipai/db/work-signals"; +import { fillUnsetInstanceDefault } from "../services/managed-instance-default.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +async function waitUntil(check: () => Promise, timeoutMs: number) { + const started = Date.now(); + while (Date.now() - started < timeoutMs) { + if (await check()) return Date.now() - started; + await new Promise(resolve => setTimeout(resolve, 50)); + } + throw new Error(`condition not met within ${timeoutMs} ms`); +} + +(support.supported ? describe : describe.skip)("instance settings read-through cache", () => { + let database: Awaited>; + let db: Db; + let otherProcess: Db; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("instance-settings-cache-"); + db = createDb(database.connectionString); + otherProcess = createDb(database.connectionString); + await db.insert(companies).values({ name: "Settings cache", issuePrefix: "SCC" }); + }, 90_000); + + afterAll(async () => { + await Promise.all([db?.$client.end({ timeout: 0 }), otherProcess?.$client.end({ timeout: 0 })]); + await database?.cleanup(); + }, 60_000); + + beforeEach(async () => { + await db.delete(instanceSettings); + invalidateInstanceSettingsCache(); + }); + + it("serves repeated hot reads from one statement", async () => { + const census = createStatementCensus(db); + try { + const settings = instanceSettingsService(db); + await settings.get(); + const measured = await census.measure(async () => { + for (let index = 0; index < 50; index++) { + await Promise.all([settings.getGeneral(), settings.getExperimental(), instanceSettingsService(db).get()]); + } + }); + expect(measured.statements).toBe(0); + invalidateInstanceSettingsCache(); + const concurrent = await census.measure(() => Promise.all(Array.from({ length: 20 }, () => settings.getExperimental()))); + expect(concurrent.statements).toBe(1); + } finally { + census.uninstall(); + } + }); + + it("reads its own writes in this process at once", async () => { + const settings = instanceSettingsService(db); + expect((await settings.getExperimental()).enableIsolatedWorkspaces).toBe(false); + await settings.updateExperimental({ enableIsolatedWorkspaces: true }); + expect((await instanceSettingsService(db).getExperimental()).enableIsolatedWorkspaces).toBe(true); + expect((await settings.getGeneral()).censorUsernameInLogs).toBe(false); + await settings.updateGeneral({ censorUsernameInLogs: true }); + expect((await settings.getGeneral()).censorUsernameInLogs).toBe(true); + }); + + it("sees a default environment committed in a caller's transaction once it commits", async () => { + const settings = instanceSettingsService(db); + expect((await settings.get()).defaultEnvironmentId).toBeNull(); + const [environment] = await db.insert(environments).values({ + name: "Managed Sandbox", driver: "sandbox", status: "active", config: {}, + }).returning({ id: environments.id }); + expect((await fillUnsetInstanceDefault(db, environment!.id)).filled).toBe(true); + expect((await settings.get()).defaultEnvironmentId).toBe(environment!.id); + }); + + it("takes a change made by another process within the cache bound", async () => { + const settings = instanceSettingsService(db); + expect((await settings.getExperimental()).enableIsolatedWorkspaces).toBe(false); + const changedAt = Date.now(); + await otherProcess.update(instanceSettings) + .set({ experimental: sql`coalesce(${instanceSettings.experimental}, '{}'::jsonb) || '{"enableIsolatedWorkspaces":true}'::jsonb` }) + .where(eq(instanceSettings.singletonKey, "default")); + await waitUntil(async () => (await settings.getExperimental()).enableIsolatedWorkspaces === true, + INSTANCE_SETTINGS_CACHE_TTL_MS + 1_000); + expect(Date.now() - changedAt).toBeLessThanOrEqual(INSTANCE_SETTINGS_CACHE_TTL_MS + 500); + }); + + it("never shares cached rows with callers that mutate them", async () => { + const settings = instanceSettingsService(db); + const first = await settings.get(); + (first.experimental as unknown as Record).enableIsolatedWorkspaces = true; + expect((await settings.get()).experimental.enableIsolatedWorkspaces).toBe(false); + }); + + it("reads inside a transaction bypass the cache", async () => { + const settings = instanceSettingsService(db); + expect((await settings.getExperimental()).enableIsolatedWorkspaces).toBe(false); + await db.transaction(async tx => { + await tx.update(instanceSettings).set({ experimental: { enableIsolatedWorkspaces: true } }) + .where(eq(instanceSettings.singletonKey, "default")); + const inside = instanceSettingsService(tx as unknown as Db); + expect((await inside.getExperimental()).enableIsolatedWorkspaces).toBe(true); + const census = createStatementCensus(db); + try { + const reads = await census.measure(async () => { await inside.getExperimental(); await inside.getGeneral(); }); + expect(reads.statements).toBe(2); + } finally { + census.uninstall(); + } + throw new Error("rollback"); + }).catch(() => undefined); + expect((await settings.getExperimental()).enableIsolatedWorkspaces).toBe(false); + }); + + it("never stores a read that started before an invalidation", async () => { + const owner = {}; + let resolveFirst!: (value: string) => void; + const first = readThroughInstanceSettings(owner, () => new Promise(resolve => { resolveFirst = resolve; })); + invalidateInstanceSettingsCache(); + resolveFirst("before the write"); + expect(await first).toBe("before the write"); + let loads = 0; + expect(await readThroughInstanceSettings(owner, async () => { loads += 1; return "after the write"; })).toBe("after the write"); + expect(await readThroughInstanceSettings(owner, async () => { loads += 1; return "unexpected"; })).toBe("after the write"); + expect(loads).toBe(1); + }); + + it("drops the cached row when another process announces a settings change", async () => { + const settings = instanceSettingsService(db); + await settings.getExperimental(); + const census = createStatementCensus(db); + try { + const cached = await census.measure(() => settings.getExperimental()); + expect(cached.statements).toBe(0); + receiveDatabaseWorkNotification(db, JSON.stringify({ version: 1, origin: "another-process", topic: INSTANCE_SETTINGS_CHANGED_TOPIC })); + const refreshed = await census.measure(() => settings.getExperimental()); + expect(refreshed.statements).toBe(1); + } finally { + census.uninstall(); + } + }); + + it("announces a settings change before the other wakes of the same write", async () => { + const payloads: string[] = []; + const listener = await otherProcess.$client.listen(DATABASE_WORK_CHANNEL, payload => { payloads.push(payload); }); + enableDatabaseWorkNotifications(db); + try { + await instanceSettingsService(db).updateExperimental({ enableStatusCards: true }); + await waitUntil(async () => payloads.length > 1, 5_000); + const topics = payloads.map(payload => (JSON.parse(payload) as { topic: string }).topic); + expect(topics[0]).toBe(INSTANCE_SETTINGS_CHANGED_TOPIC); + expect(topics).toContain("status-card-evaluation"); + payloads.length = 0; + await instanceSettingsService(db).updateGeneral({ censorUsernameInLogs: true }); + await waitUntil(async () => payloads.length > 0, 5_000); + expect(payloads.map(payload => (JSON.parse(payload) as { topic: string }).topic)).toEqual([INSTANCE_SETTINGS_CHANGED_TOPIC]); + } finally { + await listener.unlisten(); + } + }); +}); diff --git a/server/src/__tests__/project-environments.test.ts b/server/src/__tests__/project-environments.test.ts index db951894ed6..9521ef45baa 100644 --- a/server/src/__tests__/project-environments.test.ts +++ b/server/src/__tests__/project-environments.test.ts @@ -15,6 +15,7 @@ import { projectRoutes } from "../routes/projects.js"; import { errorHandler } from "../middleware/index.js"; import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; const support = await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("project environment compatibility", () => { @@ -43,6 +44,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { { name: "Build host", driver: "ssh", config: { host: "build.example.com" } }, ]).returning(); await db.insert(instanceSettings).values({ defaultEnvironmentId: sandbox.id }); + invalidateInstanceSettingsCache(); return { ...f, plugin, computer, local, sandbox, ssh }; } @@ -85,6 +87,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { await db.update(agents).set({ defaultEnvironmentId: null }).where(eq(agents.id, f.input.agentId)); expect((await projectEnvironmentOptions(db, f.companyId, { userId: "person" })).defaultEnvironmentId).toBe(f.sandbox.id); await db.update(instanceSettings).set({ defaultEnvironmentId: null }); + invalidateInstanceSettingsCache(); expect((await projectEnvironmentOptions(db, f.companyId)).defaultEnvironmentId).toBe(f.local.id); }); @@ -101,6 +104,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { expect(await withDefaultProjectStorage(db, f.companyId, explicit)).toEqual(explicit); await expect(withDefaultProjectStorage(db, f.companyId, explicit, { environmentId: f.computer.id })).rejects.toMatchObject({ status: 409 }); await db.update(instanceSettings).set({ defaultEnvironmentId: f.computer.id }); + invalidateInstanceSettingsCache(); expect((await svc.getById(sandboxProject.id))?.workspaceProvider?.pluginId).toBeNull(); }); @@ -175,6 +179,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { it("keeps a cold worker's storage identity, and uses the only storage left after its default is archived", async () => { const f = await fixture(); await db.update(instanceSettings).set({ defaultEnvironmentId: f.computer.id }); + invalidateInstanceSettingsCache(); await db.update(plugins).set({ status: "error" }).where(eq(plugins.id, f.pluginId)); expect(await withDefaultProjectStorage(db, f.companyId, null)).toMatchObject({ storageProviderPluginId: f.pluginId }); await db.update(environments).set({ status: "archived" }).where(eq(environments.id, f.computer.id)); @@ -186,6 +191,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { const f = await fixture(); const [fake] = await db.insert(environments).values({ name: "Fake", driver: "sandbox", config: { provider: "fake" } }).returning(); await db.update(instanceSettings).set({ defaultEnvironmentId: fake.id }); + invalidateInstanceSettingsCache(); const options = await projectEnvironmentOptions(db, f.companyId); expect(options.groups).toHaveLength(2); expect(options.defaultEnvironmentId).toBeNull(); @@ -197,6 +203,7 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { const f = await fixture(); await db.update(environments).set({ status: "archived" }).where(inArray(environments.id, [f.sandbox.id, f.ssh.id])); await db.update(instanceSettings).set({ experimental: { enableManagedSandboxOnly: true } }); + invalidateInstanceSettingsCache(); const options = await projectEnvironmentOptions(db, f.companyId); expect(options).toEqual({ groups: [{ storageProviderPluginId: f.pluginId, environments: [{ id: f.computer.id, name: f.computer.name }] }], @@ -214,8 +221,10 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { await db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "operator" }); await db.update(environments).set({ status: "archived" }).where(eq(environments.id, f.computer.id)); - if (source === "instance") await db.update(instanceSettings).set({ defaultEnvironmentId: f.computer.id }); - else { + if (source === "instance") { + await db.update(instanceSettings).set({ defaultEnvironmentId: f.computer.id }); + invalidateInstanceSettingsCache(); + } else { await db.update(agents).set({ defaultEnvironmentId: f.computer.id }).where(eq(agents.id, f.input.agentId)); await db.insert(userCompanyPreferences).values({ companyId: f.companyId, userId, primaryAgentId: f.input.agentId }); } @@ -231,10 +240,12 @@ describe.skipIf(!support.supported)("project environment compatibility", () => { const f = await fixture(); await db.update(environments).set({ metadata: { managedByPaperclip: true, managedKubernetesSandbox: true } }).where(eq(environments.id, f.sandbox.id)); await db.update(instanceSettings).set({ defaultEnvironmentId: f.local.id, experimental: { enableManagedSandboxOnly: true } }); + invalidateInstanceSettingsCache(); const managed = await projectEnvironmentOptions(db, f.companyId); expect(managed.defaultEnvironmentId).toBe(f.sandbox.id); expect(managed.groups.flatMap(group => group.environments).map(value => value.id)).not.toContain(f.local.id); await db.update(instanceSettings).set({ general: { executionMode: "kubernetes" } }); + invalidateInstanceSettingsCache(); const forced = await projectEnvironmentOptions(db, f.companyId); expect(forced.groups).toEqual([{ storageProviderPluginId: null, environments: [{ id: f.sandbox.id, name: f.sandbox.name }] }]); await expect(withDefaultProjectStorage(db, f.companyId, null, { environmentId: f.computer.id })).rejects.toMatchObject({ status: 409 }); diff --git a/server/src/__tests__/setup-supertest.ts b/server/src/__tests__/setup-supertest.ts index be189a6abe0..adab4b32f77 100644 --- a/server/src/__tests__/setup-supertest.ts +++ b/server/src/__tests__/setup-supertest.ts @@ -1,5 +1,6 @@ import fs from "node:fs"; import { createIsolatedTestHome } from "./helpers/isolated-test-home.js"; +import { beforeEach } from "vitest"; import { createRequire } from "node:module"; import type { AddressInfo, Server as NetServer } from "node:net"; import os from "node:os"; @@ -76,3 +77,8 @@ if (!SupertestTest.prototype.__paperclipLoopbackPatched) { SupertestTest.prototype.__paperclipLoopbackPatched = true; } + +beforeEach(async () => { + const { invalidateInstanceSettingsCache } = await import("../services/instance-settings-cache.js"); + invalidateInstanceSettingsCache(); +}); diff --git a/server/src/__tests__/smoke-lab.test.ts b/server/src/__tests__/smoke-lab.test.ts index 165ca21fc51..2fe45a6f942 100644 --- a/server/src/__tests__/smoke-lab.test.ts +++ b/server/src/__tests__/smoke-lab.test.ts @@ -28,6 +28,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { smokeLabRoutes } from "../routes/smoke-lab.js"; import { SMOKE_LAB_OAUTH_SCOPE } from "../services/smoke-lab.js"; import { errorHandler } from "../middleware/index.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SMOKE_LAB_TEST_DATABASE_URL"); const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); @@ -50,6 +51,7 @@ async function enableSmokeLab(db: TestDb) { target: [instanceSettings.singletonKey], set: { experimental: { enableSmokeLab: true }, updatedAt: new Date() }, }); + invalidateInstanceSettingsCache(); } async function createAgent(db: TestDb, companyId: string) { diff --git a/server/src/middleware/cloud-tenant-actor.test.ts b/server/src/middleware/cloud-tenant-actor.test.ts index 61929549d0d..f13be1579d1 100644 --- a/server/src/middleware/cloud-tenant-actor.test.ts +++ b/server/src/middleware/cloud-tenant-actor.test.ts @@ -279,7 +279,7 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => { expect(insertedTables.filter((table) => table === companyMemberships)).toHaveLength(3); }); - it("coalesces concurrent identical syncs while keeping authority reads and stale-admin purges per request", async () => { + it("coalesces concurrent identical syncs and settings reads while keeping membership reads and stale-admin purges per request", async () => { let release!: () => void; let entered!: () => void; const blocked = new Promise(resolve => { release = resolve; }); @@ -294,7 +294,7 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => { const actors = await Promise.all(requests); expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1); expect(deletedTables.filter(table => table === instanceUserRoles)).toHaveLength(20); - expect(selectWheres.filter(entry => entry.table === instanceSettings)).toHaveLength(20); + expect(selectWheres.filter(entry => entry.table === instanceSettings)).toHaveLength(1); expect(selectWheres.filter(entry => entry.table === companyMemberships && !entry.locked)).toHaveLength(20); expect(selectWheres.filter(entry => entry.table === companyMemberships && entry.locked)).toHaveLength(1); expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true); diff --git a/server/src/middleware/cloud-tenant-sync-postgres.test.ts b/server/src/middleware/cloud-tenant-sync-postgres.test.ts index 46fbf351f41..ce7ce28f464 100644 --- a/server/src/middleware/cloud-tenant-sync-postgres.test.ts +++ b/server/src/middleware/cloud-tenant-sync-postgres.test.ts @@ -11,6 +11,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js"; import { errorHandler } from "./error-handler.js"; import { userProfileRoutes } from "../routes/user-profiles.js"; +import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js"; const support = await getEmbeddedPostgresTestSupport(); const aliasFields = ["name", "email-local", "email", "principal"] as const; @@ -172,11 +173,13 @@ const aliasFields = ["name", "email-local", "email", "principal"] as const; await db.update(companyMemberships).set({ status: "inactive" }).where(and(eq(companyMemberships.companyId, importedCompanyId), eq(companyMemberships.principalId, userId))); await db.insert(instanceSettings).values({ singletonKey: "default", experimental: { enableOwnerInstanceAdmin: true } }) .onConflictDoUpdate({ target: instanceSettings.singletonKey, set: { experimental: { enableOwnerInstanceAdmin: true } } }); + invalidateInstanceSettingsCache(); const elevated = await resolve(db, value); expect(elevated?.isInstanceAdmin).toBe(true); expect(elevated?.companyIds).toEqual(actor!.companyIds); expect(await db.select().from(instanceUserRoles).where(eq(instanceUserRoles.userId, userId))).toEqual([]); await db.update(instanceSettings).set({ experimental: { enableOwnerInstanceAdmin: false } }); + invalidateInstanceSettingsCache(); expect((await resolve(db, value))?.isInstanceAdmin).toBe(false); }); diff --git a/server/src/services/feedback.ts b/server/src/services/feedback.ts index 5a202a84860..aa4e13af3c0 100644 --- a/server/src/services/feedback.ts +++ b/server/src/services/feedback.ts @@ -51,6 +51,7 @@ import { getRunLogStore, isRunLogStoreType } from "./run-log-store.js"; import { isObjectJournalBusyError } from "./object-byte-journal.js"; import { recoverSettledNativeRunLog } from "./native-runtime/native-run-log-recovery.js"; import { getOperatorSettingDefaults } from "./setting-defaults.js"; +import { announceInstanceSettingsChange } from "./instance-settings-cache.js"; const FEEDBACK_SCHEMA_VERSION = "paperclip-feedback-envelope-v2"; const FEEDBACK_BUNDLE_VERSION = "paperclip-feedback-bundle-v2"; @@ -2027,6 +2028,7 @@ export function feedbackService(db: Db, options: FeedbackServiceOptions = {}) { getOperatorSettingDefaults(), ); if (currentInstanceSettings && currentGeneral.feedbackDataSharingPreference === "prompt") { + await announceInstanceSettingsChange(tx); const nextSharingPreference = sharedWithLabs ? "allowed" : "not_allowed"; const currentGeneralRaw = asRecord(currentInstanceSettings.general) ?? {}; await tx diff --git a/server/src/services/instance-settings-cache.ts b/server/src/services/instance-settings-cache.ts new file mode 100644 index 00000000000..08ad61e73e3 --- /dev/null +++ b/server/src/services/instance-settings-cache.ts @@ -0,0 +1,63 @@ +import { is } from "drizzle-orm"; +import { PgTransaction } from "drizzle-orm/pg-core"; +import { publishDatabaseWorkHint, publishDatabaseWorkSettled, subscribeDatabaseWork } from "@paperclipai/db/work-signals"; + +export const INSTANCE_SETTINGS_CACHE_TTL_MS = 2_000; +export const INSTANCE_SETTINGS_CHANGED_TOPIC = "instance-settings-changed"; + +type Entry = { generation: number; at: number; value?: unknown; pending?: Promise }; + +let generation = 0; +const entries = new WeakMap(); +const subscribed = new WeakSet(); + +export function invalidateInstanceSettingsCache() { + generation += 1; +} + +export async function announceInstanceSettingsChange(connection: object) { + invalidateInstanceSettingsCache(); + publishDatabaseWorkSettled(connection, INSTANCE_SETTINGS_CHANGED_TOPIC); + if (!is(connection, PgTransaction)) { + await publishDatabaseWorkHint(connection, INSTANCE_SETTINGS_CHANGED_TOPIC).catch(() => undefined); + } +} + +function subscribeToChanges(db: object) { + if (subscribed.has(db)) return; + subscribed.add(db); + try { + subscribeDatabaseWork(db, INSTANCE_SETTINGS_CHANGED_TOPIC, invalidateInstanceSettingsCache); + } catch { + return; + } +} + +export async function readThroughInstanceSettings(db: object, load: () => Promise): Promise { + if (is(db, PgTransaction)) return load(); + subscribeToChanges(db); + const current = entries.get(db); + if (current && current.generation === generation) { + if ("value" in current && Date.now() - current.at < INSTANCE_SETTINGS_CACHE_TTL_MS) { + return structuredClone(current.value as T); + } + if (current.pending) return structuredClone(await (current.pending as Promise)); + } + const startedAt = Date.now(); + const startedGeneration = generation; + const entry: Entry = { generation: startedGeneration, at: startedAt }; + const pending = load(); + entry.pending = pending; + entries.set(db, entry); + try { + const value = await pending; + if (entries.get(db) === entry) { + if (generation === startedGeneration) entries.set(db, { generation: startedGeneration, at: startedAt, value }); + else entries.delete(db); + } + return structuredClone(value); + } catch (error) { + if (entries.get(db) === entry) entries.delete(db); + throw error; + } +} diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index d440ce2ae25..ece871a0cf1 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -59,6 +59,7 @@ import { getManagedInstanceConfig, type ManagedInstanceConfig } from "./managed- import { getOperatorSettingDefaults } from "./setting-defaults.js"; import { isCloudStagingExperimentalSetting, isCloudStagingInstance } from "./cloud-instance.js"; import { forbidden } from "../errors.js"; +import { announceInstanceSettingsChange, readThroughInstanceSettings } from "./instance-settings-cache.js"; const DEFAULT_SINGLETON_KEY = "default"; const instanceGeneralSettingsStorageSchema = instanceGeneralSettingsSchema.strip(); @@ -552,8 +553,12 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService throw new Error("Failed to initialize instance settings row"); } + function getCachedRow(runner: InstanceSettingsWriteDb = db) { + return readThroughInstanceSettings(runner, () => getOrCreateRow(runner)); + } + return { - get: async (): Promise => toInstanceSettings(await getOrCreateRow()), + get: async (): Promise => toInstanceSettings(await getCachedRow()), update: async ( patch: PatchInstanceSettings, @@ -576,13 +581,14 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService }) .where(eq(instanceSettings.id, current.id)) .returning(); + await announceInstanceSettingsChange(runner); return toInstanceSettings(updated ?? current); }, getGeneral: async ( readOptions?: { db?: InstanceSettingsWriteDb }, ): Promise => { - const row = await getOrCreateRow(readOptions?.db); + const row = await getCachedRow(readOptions?.db); return toGeneralView(row.general); }, @@ -593,7 +599,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService }, getExperimental: async (): Promise => { - const row = await getOrCreateRow(); + const row = await getCachedRow(); return toExperimentalView(row.experimental); }, @@ -617,6 +623,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService }) .where(eq(instanceSettings.id, current.id)) .returning(); + await announceInstanceSettingsChange(db); return toInstanceSettings(updated ?? current); }, @@ -651,6 +658,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService && toExperimentalView(nextExperimental).enableAgentChat; const now = new Date(); const [updated] = await db.transaction(async tx => { + await announceInstanceSettingsChange(tx); if (patch.enableStatusCards !== undefined) await notifyStatusCardEvaluation(tx); if (chatBecameEnabled) await notifyDeferredExecutionEligibility(tx); if(toExperimentalView(current.experimental).enablePublicMcp!==toExperimentalView(nextExperimental).enablePublicMcp)await prepareMcpClientMirrorPolicy(tx as unknown as Db,toExperimentalView(nextExperimental).enablePublicMcp); diff --git a/server/src/services/native-runtime/native-run-log-watermark.test.ts b/server/src/services/native-runtime/native-run-log-watermark.test.ts index e0896fd7e79..77c24f1d8af 100644 --- a/server/src/services/native-runtime/native-run-log-watermark.test.ts +++ b/server/src/services/native-runtime/native-run-log-watermark.test.ts @@ -18,6 +18,7 @@ import { createObjectRunLogStore } from "../object-run-log-store.js"; import { nativeSha256 } from "./canonical.js"; import { createNativeRunLogCoverageVerifier, formatNativeRunLogEvent } from "./native-run-log-export.js"; import { readRecoverableRunLog, recoverSettledNativeRunLog } from "./native-run-log-recovery.js"; +import { invalidateInstanceSettingsCache } from "../instance-settings-cache.js"; // Use an explicit guarded test database or a new owned embedded fixture. const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_LOG_WATERMARK_DATABASE_URL"); @@ -348,7 +349,7 @@ describe.skipIf(!support.supported)("native run log source watermarks", () => { const f = await fixture(); const setMasking = async (value: boolean) => db.execute(sql`insert into instance_settings(general) values(${JSON.stringify({ censorUsernameInLogs: value })}::jsonb) - on conflict(singleton_key) do update set general=excluded.general`); + on conflict(singleton_key) do update set general=excluded.general`).finally(invalidateInstanceSettingsCache); const redaction = async () => ({ enabled: (await instanceSettingsService(db).getGeneral()).censorUsernameInLogs, userNames: ["fixture-operator"], homeDirs: ["/home/fixture-operator"] }); await setMasking(enabled);