mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load
## Linked Issues or Issue Description
**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.
**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.
**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.
## What Changed
- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.
Hot callers of the cached row:
- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
- 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
- A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.
## Risks
- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
594 lines
28 KiB
TypeScript
594 lines
28 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import express, { type Request } from "express";
|
|
import request from "supertest";
|
|
import { and, eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
authUsers,
|
|
companies,
|
|
companyMemberships,
|
|
instanceSettings,
|
|
instanceUserRoles,
|
|
principalPermissionGrants,
|
|
} from "@paperclipai/db";
|
|
import { actorMiddleware, cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js";
|
|
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./idle-admission.js";
|
|
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
|
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
|
|
|
// Minimal fake Drizzle Db: records every table passed to .insert() / .delete() and
|
|
// supports the chained call shapes used by resolveCloudTenantActor (values /
|
|
// onConflictDo* / returning().then() / delete().where()), plus the
|
|
// select().from(table).where() reads: instanceSettings for the owner-elevation
|
|
// flag resolution through instanceSettingsService, and companyMemberships for
|
|
// the user's own membership rows (rows configurable via membershipQueryRows,
|
|
// where-conditions captured in selectWheres). The chain is awaitable so
|
|
// directly-awaited statements resolve.
|
|
function createFakeDb(options: {
|
|
membershipRow?: { companyId: string; membershipRole: string; status: string };
|
|
membershipQueryRows?: Array<{ companyId: string; membershipRole: string | null; status: string }>;
|
|
settingsRow?: Record<string, unknown> | null;
|
|
selectThrows?: boolean;
|
|
beforeWrite?: (table: unknown) => Promise<void>;
|
|
beforeRead?: (table: unknown) => Promise<void>;
|
|
unchangedMembership?: boolean;
|
|
} = {}) {
|
|
const settingsRow =
|
|
options.settingsRow === undefined
|
|
? {
|
|
id: "00000000-0000-0000-0000-000000000001",
|
|
singletonKey: "default",
|
|
defaultEnvironmentId: null,
|
|
general: {},
|
|
experimental: {},
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}
|
|
: options.settingsRow;
|
|
const insertedTables: unknown[] = [];
|
|
const deletedTables: unknown[] = [];
|
|
const selectWheres: Array<{ table: unknown; condition: unknown; locked?: boolean }> = [];
|
|
function writeChain(table: unknown) {
|
|
let values: unknown;
|
|
const chain: Record<string, unknown> = {};
|
|
chain.values = (value: unknown) => { values = value; return chain; };
|
|
chain.onConflictDoUpdate = () => chain;
|
|
chain.onConflictDoNothing = () => chain;
|
|
chain.where = () => chain;
|
|
chain.returning = async () => {
|
|
await options.beforeWrite?.(table);
|
|
return options.unchangedMembership && table === companyMemberships ? [] : [options.membershipRow ?? values];
|
|
};
|
|
chain.then = (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) =>
|
|
Promise.resolve().then(() => options.beforeWrite?.(table)).then(resolve, reject);
|
|
return chain;
|
|
}
|
|
const db = {
|
|
transaction: async (work: (tx: Db) => Promise<unknown>) => work(db),
|
|
insert: (table: unknown) => {
|
|
insertedTables.push(table);
|
|
return writeChain(table);
|
|
},
|
|
delete: (table: unknown) => {
|
|
deletedTables.push(table);
|
|
return writeChain(table);
|
|
},
|
|
select: () => {
|
|
return {
|
|
from: (table: unknown) => ({
|
|
// Pending Muse bindings use a joined read; this fixture has none.
|
|
innerJoin() { return this; },
|
|
where: (condition: unknown) => {
|
|
// Only settings lookup is degradable. Membership locking and
|
|
// permission seeding must still work in the fail-closed flag test.
|
|
if (options.selectThrows && table === instanceSettings) {
|
|
throw new Error("select unavailable");
|
|
}
|
|
const selected: (typeof selectWheres)[number] = { table, condition };
|
|
selectWheres.push(selected);
|
|
const rows =
|
|
table === instanceSettings && settingsRow
|
|
? [settingsRow]
|
|
: table === companyMemberships
|
|
? (options.membershipQueryRows ?? [])
|
|
: [];
|
|
let locked = false;
|
|
const result = {
|
|
orderBy: () => ({ limit: () => result }),
|
|
for: () => { locked = true; selected.locked = true; return result; },
|
|
then: (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) => {
|
|
// The optional cross-company read can fail independently of
|
|
// the primary membership transaction, which must succeed.
|
|
const query = options.selectThrows && table === companyMemberships && !locked
|
|
? Promise.reject(new Error("select unavailable"))
|
|
: Promise.resolve(rows);
|
|
return Promise.resolve().then(() => options.beforeRead?.(table)).then(() => query).then(resolve, reject);
|
|
},
|
|
};
|
|
return result;
|
|
},
|
|
}),
|
|
};
|
|
},
|
|
} as unknown as Db;
|
|
return { db, insertedTables, deletedTables, selectWheres };
|
|
}
|
|
|
|
function settingsRowWith(experimental: Record<string, unknown>) {
|
|
return {
|
|
id: "00000000-0000-0000-0000-000000000001",
|
|
singletonKey: "default",
|
|
defaultEnvironmentId: null,
|
|
general: {},
|
|
experimental,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
};
|
|
}
|
|
|
|
function fakeReq(headers: Record<string, string>): Request {
|
|
const lower: Record<string, string> = {};
|
|
for (const [k, v] of Object.entries(headers)) lower[k.toLowerCase()] = v;
|
|
return { header: (name: string) => lower[name.toLowerCase()] } as unknown as Request;
|
|
}
|
|
|
|
const VALID_HEADERS = {
|
|
"x-paperclip-cloud-tenant-token": "test-server-token",
|
|
"x-paperclip-cloud-user-id": "user-123",
|
|
"x-paperclip-cloud-user-email": "Owner@Example.com",
|
|
"x-paperclip-cloud-stack-id": "stack-abc",
|
|
"x-paperclip-cloud-stack-role": "owner",
|
|
};
|
|
|
|
const MANAGED_CONFIG_FLAG_ON = JSON.stringify({
|
|
v: 1,
|
|
mode: "cloud",
|
|
catalogVersion: "test",
|
|
features: { enableOwnerInstanceAdmin: true },
|
|
plugins: { autoInstall: [] },
|
|
});
|
|
|
|
const MANAGED_CONFIG_FLAG_OFF = JSON.stringify({
|
|
v: 1,
|
|
mode: "cloud",
|
|
catalogVersion: "test",
|
|
features: { enableOwnerInstanceAdmin: false },
|
|
plugins: { autoInstall: [] },
|
|
});
|
|
|
|
describe("resolveCloudTenantActor (shared-pool hardening)", () => {
|
|
beforeEach(() => {
|
|
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "test-server-token";
|
|
});
|
|
afterEach(() => {
|
|
stopTaskDrain();
|
|
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
|
delete process.env.PAPERCLIP_MANAGED_CONFIG;
|
|
});
|
|
|
|
it("does not grant instance admin by default (flag off)", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("stays scoped to exactly the stack's company when the user has no other membership rows", async () => {
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
expect(actor!.memberships).toHaveLength(1);
|
|
expect(actor?.memberships?.[0]?.companyId).toBe(actor?.companyIds?.[0]);
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("owner");
|
|
expect(actor!.source).toBe("cloud_tenant");
|
|
});
|
|
|
|
it("settles its concurrent reads before a failed purge leaves authentication", async () => {
|
|
let settingsReadStarted = false;
|
|
let settingsReadSettled = false;
|
|
const { db } = createFakeDb({
|
|
beforeWrite: async (table) => {
|
|
if (table === instanceUserRoles) throw new Error("purge failed");
|
|
},
|
|
beforeRead: async (table) => {
|
|
if (table !== instanceSettings) return;
|
|
settingsReadStarted = true;
|
|
await new Promise((resolve) => setTimeout(resolve, 50));
|
|
settingsReadSettled = true;
|
|
},
|
|
});
|
|
await expect(resolveCloudTenantActor(db, fakeReq(VALID_HEADERS))).rejects.toThrow("purge failed");
|
|
// No started read may still be running once the error surfaces.
|
|
expect(settingsReadStarted).toBe(true);
|
|
expect(settingsReadSettled).toBe(true);
|
|
});
|
|
|
|
it("purges stale instance_admin rows left by pre-hardening deployments", async () => {
|
|
const { db, deletedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("still upserts the user, company, and membership", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables).toContain(authUsers);
|
|
expect(insertedTables).toContain(companies);
|
|
expect(insertedTables).toContain(companyMemberships);
|
|
});
|
|
|
|
it.each(["/api/health", "/api/instance/task-drain", "/api/instance/task-drain/status"])("counts unfinished authentication on exempt %s requests", async (path) => {
|
|
let enterWrite!: () => void, finishWrite!: () => void;
|
|
const entered = new Promise<void>(resolve => { enterWrite = resolve; });
|
|
const pending = new Promise<void>(resolve => { finishWrite = resolve; });
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async () => { enterWrite(); await pending; } });
|
|
const server = express();
|
|
server.use(idleAdmissionMiddleware);
|
|
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
|
const hold = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
|
|
run({ select: () => ({ from: () => ({ where: async () => [] }) }), execute: async () => [{ blocked: false }] }) } as unknown as Db;
|
|
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
|
server.get("/api/instance/task-drain", async (_req, res) => {
|
|
res.json(await readIdleSleepSafety(emptyDb, () => ({ ...hold, draining: true, activeRuns: 0, pendingWakes: 0 }),
|
|
Date.now, hold.ownerId, async () => "none"));
|
|
});
|
|
server.post("/api/instance/task-drain/status", (_req, res) => res.sendStatus(204));
|
|
trackIdleRequestHandlers(server);
|
|
const accepted = (path.endsWith("/status") ? request(server).post(path) : request(server).get(path)).set(VALID_HEADERS).then(response => response);
|
|
try {
|
|
await entered;
|
|
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
|
|
const during = await request(server).get("/api/instance/task-drain").expect(200);
|
|
expect(during.body.backgroundWork).toBe("unknown");
|
|
} finally {
|
|
finishWrite();
|
|
await accepted;
|
|
}
|
|
expect(insertedTables).toEqual(expect.arrayContaining([authUsers, companies, companyMemberships]));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
const after = await request(server).get("/api/instance/task-drain").expect(200);
|
|
expect(after.body.backgroundWork).toBe("none");
|
|
});
|
|
|
|
it("releases failed control authentication and preserves Express error handling", async () => {
|
|
const { db } = createFakeDb({ beforeWrite: async () => { throw new Error("fixture write failed"); } });
|
|
const server = express();
|
|
server.use(idleAdmissionMiddleware);
|
|
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
|
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
|
server.use((_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => res.sendStatus(500));
|
|
trackIdleRequestHandlers(server);
|
|
await request(server).get("/api/health").set(VALID_HEADERS).expect(500);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
await request(server).get("/api/health").expect(204);
|
|
});
|
|
|
|
it("resyncs an A to B to A context transition inside the debounce window", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const contextA = VALID_HEADERS;
|
|
const contextB = { ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" };
|
|
|
|
await resolveCloudTenantActor(db, fakeReq(contextA));
|
|
await resolveCloudTenantActor(db, fakeReq(contextB));
|
|
await resolveCloudTenantActor(db, fakeReq(contextA));
|
|
|
|
expect(insertedTables.filter((table) => table === authUsers)).toHaveLength(3);
|
|
expect(insertedTables.filter((table) => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("coalesces concurrent identical syncs and settings reads while keeping membership reads and stale-admin purges per request", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const { db, insertedTables, deletedTables, selectWheres } = createFakeDb({
|
|
beforeWrite: async table => { if (table === authUsers) { entered(); await blocked; } },
|
|
});
|
|
const requests = Array.from({ length: 20 }, () => resolveCloudTenantActor(db, fakeReq(VALID_HEADERS)));
|
|
await started;
|
|
expect(insertedTables.filter(table => table === authUsers)).toHaveLength(1);
|
|
release();
|
|
const actors = await Promise.all(requests);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
expect(deletedTables.filter(table => table === instanceUserRoles)).toHaveLength(20);
|
|
expect(selectWheres.filter(entry => entry.table === instanceSettings)).toHaveLength(1);
|
|
expect(selectWheres.filter(entry => entry.table === companyMemberships && !entry.locked)).toHaveLength(20);
|
|
expect(selectWheres.filter(entry => entry.table === companyMemberships && entry.locked)).toHaveLength(1);
|
|
expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true);
|
|
});
|
|
|
|
it("serializes concurrent A to B to A without sharing the wrong role", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
let writes = 0;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (table === authUsers && ++writes === 1) { entered(); await blocked; }
|
|
} });
|
|
const first = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await started;
|
|
const second = resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }));
|
|
const third = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
release();
|
|
const actors = await Promise.all([first, second, third]);
|
|
expect(actors.map(actor => actor?.memberships?.[0]?.membershipRole)).toEqual(["owner", "member", "owner"]);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("does not block another user behind a slow user's synchronization", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
let firstWrite = true;
|
|
const { db } = createFakeDb({ beforeWrite: async table => {
|
|
if (table === authUsers && firstWrite) { firstWrite = false; entered(); await blocked; }
|
|
} });
|
|
const slow = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await started;
|
|
try {
|
|
const other = await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-user-id": "other-user" }));
|
|
expect(other?.userId).toBe("other-user");
|
|
} finally {
|
|
release();
|
|
await slow;
|
|
}
|
|
});
|
|
|
|
it("does not debounce a failed partial context transition or its repair", async () => {
|
|
let fail = false;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (fail && table === companyMemberships) throw new Error("partial sync failed");
|
|
} });
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
fail = true;
|
|
await expect(resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" })))
|
|
.rejects.toThrow("partial sync failed");
|
|
fail = false;
|
|
const repaired = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(repaired?.memberships?.[0]?.membershipRole).toBe("owner");
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("retries a shared transient failure and completes default grants once", async () => {
|
|
let failed = false;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (!failed && table === companyMemberships) {
|
|
failed = true;
|
|
throw Object.assign(new Error("synthetic connection loss"), { code: "CONNECTION_CLOSED" });
|
|
}
|
|
} });
|
|
const actors = await Promise.all(Array.from({ length: 10 }, () => resolveCloudTenantActor(db, fakeReq(VALID_HEADERS))));
|
|
expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(insertedTables.filter(table => table === principalPermissionGrants)).toHaveLength(1);
|
|
});
|
|
|
|
it("keeps default-grant repair and the input role when a guarded upsert returns no row", async () => {
|
|
const { db, insertedTables } = createFakeDb({ unchangedMembership: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }));
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("member");
|
|
expect(insertedTables).toContain(principalPermissionGrants);
|
|
});
|
|
|
|
it("does not share completed syncs across databases or companies", async () => {
|
|
const first = createFakeDb();
|
|
const second = createFakeDb();
|
|
await Promise.all([resolveCloudTenantActor(first.db, fakeReq(VALID_HEADERS)), resolveCloudTenantActor(second.db, fakeReq(VALID_HEADERS))]);
|
|
await resolveCloudTenantActor(first.db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-id": "stack-other" }));
|
|
expect(first.insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(second.insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
});
|
|
|
|
it("resyncs when the requested company name changes", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-paperclip-company-name": "First" }));
|
|
await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-paperclip-company-name": "Second" }));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
});
|
|
|
|
it("resumes membership and grant repair after the existing five-second debounce expires", async () => {
|
|
const now = Date.now();
|
|
const clock = vi.spyOn(Date, "now").mockReturnValue(now);
|
|
try {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
clock.mockReturnValue(now + 5_001);
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(insertedTables.filter(table => table === principalPermissionGrants)).toHaveLength(2);
|
|
} finally { clock.mockRestore(); }
|
|
});
|
|
|
|
it("returns null when the server token is unset", async () => {
|
|
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).toBeNull();
|
|
});
|
|
|
|
it("resolves identically from a raw upgrade-request header map via the shim", async () => {
|
|
// Websocket upgrades hand us IncomingMessage.headers (lowercased keys,
|
|
// possibly string[] values), not an Express Request. The shim must feed
|
|
// resolveCloudTenantActor the same way Express header() does.
|
|
const { db } = createFakeDb();
|
|
const rawHeaders: Record<string, string | string[] | undefined> = {};
|
|
for (const [k, v] of Object.entries(VALID_HEADERS)) rawHeaders[k.toLowerCase()] = v;
|
|
rawHeaders["x-paperclip-cloud-user-name"] = ["Cloud Owner", "ignored-duplicate"];
|
|
const actor = await resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders(rawHeaders));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.userId).toBe("user-123");
|
|
expect(actor!.userName).toBe("Cloud Owner");
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
});
|
|
|
|
it("maps a non-owner stack role through to the membership without elevating", async () => {
|
|
const { db } = createFakeDb({
|
|
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
|
|
});
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }),
|
|
);
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("member");
|
|
});
|
|
|
|
it.each(["admin", "member", "support"])(
|
|
"grants experimental settings access only to the admin stack role (%s)",
|
|
async (stackRole) => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": stackRole }),
|
|
);
|
|
expect(actor?.canManageExperimentalSettings).toBe(stackRole === "admin");
|
|
expect(actor?.isInstanceAdmin).toBe(false);
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
},
|
|
);
|
|
|
|
it("removes experimental settings access on the next request after a stack admin is demoted", async () => {
|
|
const { db } = createFakeDb();
|
|
const admin = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "admin" }),
|
|
);
|
|
const member = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }),
|
|
);
|
|
expect(admin?.canManageExperimentalSettings).toBe(true);
|
|
expect(member?.canManageExperimentalSettings).toBe(false);
|
|
});
|
|
|
|
describe("company membership union", () => {
|
|
// The primary company id is derived from the stack id; resolve it once
|
|
// through the same code path instead of duplicating the hash here.
|
|
async function resolvePrimaryCompanyId() {
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
return actor!.companyIds![0]!;
|
|
}
|
|
|
|
it("unions the pinned primary with the user's own active membership rows, primary first and deduped", async () => {
|
|
const primaryCompanyId = await resolvePrimaryCompanyId();
|
|
const { db } = createFakeDb({
|
|
membershipQueryRows: [
|
|
// The user's own row for the primary company comes back from the
|
|
// query too — it must not appear twice.
|
|
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
|
|
{ companyId: "company-imported", membershipRole: "member", status: "active" },
|
|
],
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.companyIds).toEqual([primaryCompanyId, "company-imported"]);
|
|
expect(actor!.memberships).toEqual([
|
|
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
|
|
{ companyId: "company-imported", membershipRole: "member", status: "active" },
|
|
]);
|
|
});
|
|
|
|
it("loads memberships with the session path's exact own-user active-status filter", async () => {
|
|
const { db, selectWheres } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
const membershipSelects = selectWheres.filter((entry) => entry.table === companyMemberships);
|
|
expect(membershipSelects).toHaveLength(2);
|
|
// One read locks the primary membership before its upsert; the other,
|
|
// which runs concurrently, loads the user's active memberships for
|
|
// request authorization. Rows for other users and rows in any
|
|
// non-active status are excluded by the query itself: the filter binds
|
|
// exactly this user id and the "active" status — the same condition
|
|
// the session path applies.
|
|
const authorizationReads = membershipSelects.filter((entry) => !entry.locked);
|
|
expect(authorizationReads).toHaveLength(1);
|
|
expect(authorizationReads[0]!.condition).toEqual(
|
|
and(
|
|
eq(companyMemberships.principalType, "user"),
|
|
eq(companyMemberships.principalId, "user-123"),
|
|
eq(companyMemberships.status, "active"),
|
|
),
|
|
);
|
|
});
|
|
|
|
it("degrades to the primary company when the membership read fails", async () => {
|
|
const { db } = createFakeDb({ selectThrows: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
expect(actor!.memberships).toHaveLength(1);
|
|
});
|
|
});
|
|
|
|
describe("owner instance-admin elevation (enableOwnerInstanceAdmin)", () => {
|
|
it("elevates the owner while the flag is enabled, still without any role row", async () => {
|
|
const { db, insertedTables, deletedTables } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(true);
|
|
expect(actor!.source).toBe("cloud_tenant");
|
|
// Elevation is computed, never persisted: no instance_user_roles insert,
|
|
// and the stale-row purge still runs on every authentication.
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("does not elevate the owner while the flag is disabled", async () => {
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
});
|
|
|
|
it.each(["member", "admin", "support"] as const)(
|
|
"never elevates the %s stack role even with the flag enabled",
|
|
async (stackRole) => {
|
|
const { db } = createFakeDb({
|
|
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": stackRole }),
|
|
);
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
},
|
|
);
|
|
|
|
it("resolves the flag through the managed overlay: overlay on elevates over a DB value of off", async () => {
|
|
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_ON;
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(true);
|
|
});
|
|
|
|
it("resolves the flag through the managed overlay: overlay off wins over a DB value of on", async () => {
|
|
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_OFF;
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
});
|
|
|
|
it("fails closed when the settings read errors: actor resolves without elevation", async () => {
|
|
const { db, deletedTables } = createFakeDb({ selectThrows: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
});
|
|
});
|