Files
PaperClipAI/server/src/__tests__/smoke-lab.test.ts
T
DottaandClaude Opus 5.5 8f1d39da79 perf(server): cache the instance settings row per process (#21768)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load

## Linked Issues or Issue Description

**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.

**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.

**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.

## What Changed

- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.

Hot callers of the cached row:

- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
  - 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
  - A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.

## Risks

- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:50:19 -06:00

426 lines
16 KiB
TypeScript

import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
applyPendingMigrations,
activityLog,
agents,
authUsers,
connectionGrants,
companies,
companyMemberships,
createDb,
heartbeatRuns,
instanceSettings,
smokeRuns,
smokeRunSteps,
toolApplications,
toolCatalogEntries,
toolConnections,
toolProfileBindings,
toolProfileEntries,
toolProfiles,
} from "@paperclipai/db";
import { eq } from "drizzle-orm";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { smokeLabRoutes } from "../routes/smoke-lab.js";
import { SMOKE_LAB_OAUTH_SCOPE } from "../services/smoke-lab.js";
import { errorHandler } from "../middleware/index.js";
import { invalidateInstanceSettingsCache } from "../services/instance-settings-cache.js";
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SMOKE_LAB_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
type TestDb = ReturnType<typeof createDb>;
async function createCompany(db: TestDb) {
return db.insert(companies).values({
name: `Smoke Lab ${randomUUID()}`,
issuePrefix: `SL${randomUUID().slice(0, 6).toUpperCase()}`,
}).returning().then((rows) => rows[0]!);
}
async function enableSmokeLab(db: TestDb) {
await db.insert(instanceSettings).values({
singletonKey: "default",
experimental: { enableSmokeLab: true },
}).onConflictDoUpdate({
target: [instanceSettings.singletonKey],
set: { experimental: { enableSmokeLab: true }, updatedAt: new Date() },
});
invalidateInstanceSettingsCache();
}
async function createAgent(db: TestDb, companyId: string) {
return db.insert(agents).values({
companyId,
name: `Smoke Agent ${randomUUID()}`,
role: "qa",
status: "active",
adapterType: "process",
adapterConfig: {},
runtimeConfig: {},
}).returning().then((rows) => rows[0]!);
}
function boardActor(companyId?: string): Express.Request["actor"] {
return {
type: "board",
userId: "board-user",
userName: "Board User",
userEmail: null,
isInstanceAdmin: true,
source: "local_implicit",
companyIds: companyId ? [companyId] : [],
};
}
function agentActor(companyId: string, agentId: string, runId: string): Express.Request["actor"] {
return {
type: "agent",
companyId,
agentId,
runId,
source: "agent_jwt",
};
}
function createRouteApp(
db: TestDb,
actor?: Express.Request["actor"],
options: Parameters<typeof smokeLabRoutes>[1] = { nodeEnv: "test" },
) {
const app = express();
app.use(express.json());
app.use((req, _res, next) => {
req.actor = actor ?? { type: "none", source: "none" };
next();
});
app.use("/api", smokeLabRoutes(db, { nodeEnv: "test", ...options }));
app.use(errorHandler);
return app;
}
describeEmbeddedPostgres("smoke lab service pack and results API", () => {
let db: TestDb;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-smoke-lab-");
if (externalTestDatabaseUrl) await applyPendingMigrations(externalTestDatabaseUrl);
db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
vi.unstubAllEnvs();
await db.delete(activityLog);
await db.delete(smokeRunSteps);
await db.delete(smokeRuns);
await db.delete(toolProfileBindings);
await db.delete(toolProfileEntries);
await db.delete(toolProfiles);
await db.delete(toolCatalogEntries);
await db.delete(toolConnections);
await db.delete(toolApplications);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companyMemberships);
await db.delete(companies);
await db.delete(authUsers);
await db.delete(instanceSettings);
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
it("gates smoke lab behind the experimental flag and public exposure, not auth mode or NODE_ENV", async () => {
const company = await createCompany(db);
// Flag off -> hidden (404) regardless of deployment.
await request(createRouteApp(db, boardActor(company.id)))
.get(`/api/companies/${company.id}/smoke-lab/services`)
.expect(404);
await enableSmokeLab(db);
// Public exposure is the only disallowed deployment -> 403.
await request(createRouteApp(db, boardActor(company.id), { deploymentMode: "authenticated", deploymentExposure: "public" }))
.get(`/api/companies/${company.id}/smoke-lab/services`)
.expect(403);
// Authenticated + private (e.g. a Tailscale dev box) is allowed, even when the
// instance runs a production Node build.
await request(createRouteApp(db, boardActor(company.id), { deploymentMode: "authenticated", deploymentExposure: "private", nodeEnv: "production" }))
.get(`/api/companies/${company.id}/smoke-lab/services`)
.expect(200);
});
it("runs the deterministic fake OAuth code, refresh, userinfo, and revoke flow", async () => {
const company = await createCompany(db);
await enableSmokeLab(db);
const app = createRouteApp(db);
const redirectUri = "http://127.0.0.1/callback";
const page = await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.query({ client_id: "smoke-client", redirect_uri: redirectUri, state: "state-1", response_type: "code" })
.expect(200);
expect(page.text).toContain("SMOKE TEST - not a real provider");
expect(page.text).toContain("smoke@paperclip.test");
expect(page.text).toContain(SMOKE_LAB_OAUTH_SCOPE);
const authorizeBody = {
client_id: "smoke-client",
redirect_uri: redirectUri,
state: "state-1",
response_type: "code",
scope: SMOKE_LAB_OAUTH_SCOPE,
email: "smoke@paperclip.test",
password: "smoke-password",
};
const authorize = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send(authorizeBody)
.expect(302);
const redirected = new URL(authorize.headers.location);
const code = redirected.searchParams.get("code");
expect(code).toMatch(/^smoke_code_/);
expect(redirected.searchParams.get("state")).toBe("state-1");
const token = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/token`)
.type("form")
.send({ grant_type: "authorization_code", code, client_id: "smoke-client", redirect_uri: redirectUri })
.expect(200);
expect(token.body.access_token).toMatch(/^smoke_access_/);
expect(token.body.refresh_token).toMatch(/^smoke_refresh_/);
expect(token.body.scope).toBe(SMOKE_LAB_OAUTH_SCOPE);
const repeatAuthorize = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send(authorizeBody)
.expect(302);
const repeatCode = new URL(repeatAuthorize.headers.location).searchParams.get("code");
expect(repeatCode).toBe(code);
const repeatToken = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/token`)
.type("form")
.send({ grant_type: "authorization_code", code: repeatCode, client_id: "smoke-client", redirect_uri: redirectUri })
.expect(200);
expect(repeatToken.body.access_token).toBe(token.body.access_token);
expect(repeatToken.body.refresh_token).toBe(token.body.refresh_token);
const refreshed = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/token`)
.type("form")
.send({ grant_type: "refresh_token", refresh_token: token.body.refresh_token })
.expect(200);
expect(refreshed.body.access_token).toMatch(/^smoke_access_/);
expect(refreshed.body.scope).toBe(SMOKE_LAB_OAUTH_SCOPE);
const userinfo = await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/userinfo`)
.set("Authorization", `Bearer ${refreshed.body.access_token}`)
.expect(200);
expect(userinfo.body).toMatchObject({ sub: "smoke-user-1", email: "smoke@paperclip.test" });
await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/revoke`)
.type("form")
.send({ token: refreshed.body.access_token })
.expect(200);
await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/userinfo`)
.set("Authorization", `Bearer ${refreshed.body.access_token}`)
.expect(403);
});
it("rejects real-looking vendor scopes at the fake OAuth provider", async () => {
const company = await createCompany(db);
await enableSmokeLab(db);
const app = createRouteApp(db);
const redirectUri = "http://127.0.0.1/callback";
await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.query({ client_id: "smoke-client", redirect_uri: redirectUri, scope: "repo user:email offline_access", response_type: "code" })
.expect(400);
await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send({
client_id: "smoke-client",
redirect_uri: redirectUri,
scope: "repo user:email offline_access",
email: "smoke@paperclip.test",
password: "smoke-password",
})
.expect(400);
});
it("requires a loopback or same-origin HTTP(S) redirect URI before rendering or completing consent", async () => {
const company = await createCompany(db);
await enableSmokeLab(db);
const app = createRouteApp(db);
vi.stubEnv("PAPERCLIP_PUBLIC_URL", "http://paperclip-dev:45439");
// A redirect host that is neither loopback nor the instance's own origin
// could leak fixture authorization codes off the gated deployment.
await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.query({ client_id: "smoke-client", redirect_uri: "http://other-host:45439/callback", response_type: "code" })
.expect(403);
// The instance's own (non-loopback) origin is fine — the smoke lab runs on
// any private instance, e.g. an authenticated Tailscale host.
await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.query({ client_id: "smoke-client", redirect_uri: "http://paperclip-dev:45439/callback", response_type: "code" })
.expect(200);
await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send({
client_id: "smoke-client",
redirect_uri: "http://paperclip-dev:45439/api/tools/oauth/callback",
email: "smoke@paperclip.test",
password: "smoke-password",
})
.expect(302);
await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send({
client_id: "smoke-client",
redirect_uri: "http://127.0.0.2/callback",
email: "smoke@paperclip.test",
password: "smoke-password",
})
.expect(302);
await request(app)
.post(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.type("form")
.send({
client_id: "smoke-client",
redirect_uri: "ftp://localhost/callback",
email: "smoke@paperclip.test",
password: "smoke-password",
})
.expect(400);
});
it("does not trust the Host header as the OAuth redirect origin", async () => {
const company = await createCompany(db);
await enableSmokeLab(db);
const app = createRouteApp(db);
vi.stubEnv("PAPERCLIP_PUBLIC_URL", "");
vi.stubEnv("PAPERCLIP_AUTH_PUBLIC_BASE_URL", "");
vi.stubEnv("BETTER_AUTH_URL", "");
vi.stubEnv("BETTER_AUTH_BASE_URL", "");
await request(app)
.get(`/api/companies/${company.id}/smoke-lab/oauth/authorize`)
.set("Host", "attacker.example")
.query({
client_id: "smoke-client",
redirect_uri: "http://attacker.example/callback",
response_type: "code",
})
.expect(403);
});
it("installs smoke fixtures idempotently into tool access tables", async () => {
const company = await createCompany(db);
await enableSmokeLab(db);
const app = createRouteApp(db, boardActor(company.id));
const first = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/install-fixtures`)
.expect(201);
const second = await request(app)
.post(`/api/companies/${company.id}/smoke-lab/install-fixtures`)
.expect(200);
expect(first.body.created).toBe(true);
expect(second.body.created).toBe(false);
expect(first.body.applications).toHaveLength(2);
expect(first.body.connections).toHaveLength(2);
expect(first.body.catalog.length).toBeGreaterThanOrEqual(6);
expect(first.body.profileEntries.every((entry: { toolName: string }) => entry.toolName.includes(".") || entry.toolName)).toBe(true);
const applications = await db.select().from(toolApplications).where(eq(toolApplications.companyId, company.id));
const connections = await db.select().from(toolConnections).where(eq(toolConnections.companyId, company.id));
const grants = await db.select().from(connectionGrants).where(eq(connectionGrants.companyId, company.id));
const catalog = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.companyId, company.id));
const profiles = await db.select().from(toolProfiles).where(eq(toolProfiles.companyId, company.id));
expect(applications).toHaveLength(2);
expect(connections).toHaveLength(2);
expect(grants).toHaveLength(2);
expect(grants.every((grant) => grant.kind === "organization" && grant.isDefault)).toBe(true);
expect(catalog.some((entry) => entry.toolName === "todo.add" && entry.riskLevel === "write")).toBe(true);
expect(catalog.some((entry) => entry.toolName === "time.now" && entry.riskLevel === "read")).toBe(true);
expect(profiles).toHaveLength(1);
});
it("creates runs and lets an agent JWT actor record step results", async () => {
const company = await createCompany(db);
const agent = await createAgent(db, company.id);
await enableSmokeLab(db);
const boardApp = createRouteApp(db, boardActor(company.id));
const created = await request(boardApp)
.post(`/api/companies/${company.id}/smoke-lab/runs`)
.send({ trigger: "manual", summary: { scenario: "P1" } })
.expect(201);
const runId = created.body.run.id;
const [heartbeatRun] = await db.insert(heartbeatRuns).values({
companyId: company.id,
agentId: agent.id,
invocationSource: "manual",
status: "running",
}).returning();
const agentApp = createRouteApp(db, agentActor(company.id, agent.id, heartbeatRun!.id));
const step = await request(agentApp)
.post(`/api/companies/${company.id}/smoke-lab/runs/${runId}/steps`)
.send({
path: "P1",
scenarioStep: "oauth-login",
status: "pass",
detail: "OAuth login completed",
screenshotArtifactRef: { provider: "paperclip", attachmentId: randomUUID() },
durationMs: 42,
})
.expect(201);
expect(step.body.step).toMatchObject({ path: "P1", scenarioStep: "oauth-login", status: "pass" });
expect(step.body.summary).toMatchObject({ totalSteps: 1, passedSteps: 1, failedSteps: 0 });
const fetched = await request(boardApp)
.get(`/api/companies/${company.id}/smoke-lab/runs/${runId}`)
.expect(200);
expect(fetched.body.steps).toHaveLength(1);
await request(boardApp)
.patch(`/api/companies/${company.id}/smoke-lab/runs/${runId}`)
.send({ status: "passed", summary: { totalSteps: 1, passedSteps: 1 } })
.expect(200);
await request(agentApp)
.post(`/api/companies/${company.id}/smoke-lab/runs/${runId}/steps`)
.send({ path: "P1", scenarioStep: "late", status: "pass" })
.expect(409);
});
});