mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
## Thinking Path > - Paperclip manages AI agents and their work under company access rules. > - Muse readiness uses a saved connection, receiver contact and an authenticated challenge reply. > - A failed challenge write could leave a saved contact with no later recovery trigger. > - A server restart or restored access must recover that contact under current authority. > - This pull request uses the existing delivery coordinator and saved Muse state for recovery. > - It keeps the normal receiver schedule, authorization checks and real deadlines. > - It lets eligible verification recover and lets idle verification stop work. ## Linked Issues or Issue Description **What happened?** The first Muse receiver contact could persist while readiness challenge creation failed. The contact then had no durable retry or startup recovery path. Later membership, lifecycle or feature restoration could also leave pending verification without a notice. **Expected behavior** Retry a temporary failure through the existing worker. Recover pending contact after restart. Recheck the current company, agent, operator, credential and binding generation before issuing a challenge. Stop verification work while it is ineligible or complete. Do not revive revoked credentials. **Steps to reproduce** 1. Use the normal invitation and OAuth registration, PKCE and token flow. 2. Send one receiver contact and let its normal persistence schedule run. 3. Fail the first readiness challenge operation before it commits. 4. Observe whether the existing worker retries without another contact or a manual Verify action. 5. Restart the App with a new database owner. Check that it creates one usable challenge from the saved contact. **Paperclip version or commit** The causal baseline test used the unchanged production implementation at `90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`. The initial published head was `fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its added Cloud actor fixture correction passed the full normal owning program and all 35 tests. Its current normal build, lifecycle owner controls, healthy-wait and restoration measurements are described below. Earlier results below are bound to their stated revisions. Related public changes: Refs #19819 for original-principal authentication. Refs #19750 for the transaction-owned external feature read. These are related foundations, not duplicate fixes. The current Muse title and head-branch searches found no duplicate contact-verification recovery PR. Related maintenance work is #17000 and terminal assignment settlement is #17793. ## What Changed - Add a typed, exact-binding verification notice through the existing Muse delivery queue. - Register the first-contact notice before its durable write in the same transaction. - Recover pending contact from saved state at startup. Retry temporary failures through the existing scheduler. - Recheck current feature, lifecycle, credential and access state after commit. Park unavailable authority without recurring verification scans. - Add narrowly scoped notices to the existing access, lifecycle, configuration, company, budget and settings write owners. - Save a healthy lifecycle verification wait in the existing operation JSONB. Keep generic unresolved-work and error retries unchanged. - Commit authenticated confirmation and its lifecycle recheck in one transaction. Retain a raced recheck under the active lease, and recover saved waits once at startup. - Retain a committed lifecycle recheck while an expired attempt still owns the row, so its late wait cannot park accepted confirmation. Replacement claims keep their existing owner fence and fresh fact read. - Keep the receiver gap, persistence and per-segment contact limits unchanged. - Add real App OAuth, failure, retry, restart, challenge and quiet evidence. Add two normal-route restoration tests. - Document recovery, reconnect requirements and the existing connected-receiver sleep restriction. ## Verification - Causal baseline: the real invitation and OAuth flow saved one contact. The first challenge operation failed. The normal test failed because it observed one verification attempt instead of a retry. No setup or cleanup failure was counted as this regression. - At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets, Runner TypeScript core and surfaces, module boundaries, and all seven normal App programs passed. One new redundant role comparison first failed typecheck; that diagnostic remains recorded and the corrected full App gate passed. - At `9d383041`: all 17 maintenance component tests passed with zero skips. All original deadline, rollback, drain, terminal settlement and startup controls remain. - At `9d383041`: the one normal App case passed. Real OAuth authorized one receiver signal. A retry occurred about 5.01 seconds after the first failure. A new Db/App owner created one challenge, advanced revision 2 to 3, and accepted authenticated confirmation and mailbox ACK. Query instrumentation had an active positive control. All measured query, callback, session and verifier counters stayed equal across the unchanged scheduled 15-second quiet window. Token revocation and old-token/signal rejection passed. - Known App, worker, HTTP and database owners closed without cleanup errors. The tests used bounded local PostgreSQL and HTTP fixtures. They did not call a live Muse model. - Current integrated semantics: the full normal App driver completed all programs. Only a new test callback had two undefined-value diagnostics. An explicit defined-value guard corrected that test. The full 416-root owning program then passed with empty diagnostics. A separate collector incorrectly required an explicit child `noEmit` field; the normal inherited setting was verified without repeating the compiler. The original failures remain recorded. - The first restoration run failed both cases. Permission parking recorded 120 main-pool queries without more verifier calls. The lifecycle counter recorded two denials without their exact scope. These observations remain unexplained at the individual-query/call level. Source separately shows a lifecycle readiness retry every two seconds while the real external reply is absent. - At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all seven full App programs, and all 33 pure diagnostic controls passed. The lifecycle owner file passed all 19 cases, including both confirmation/park orders, rollback, duplicate notices, startup recovery and authority controls. - At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction case passed. Its new healthy-wait window measured 15,002 ms with unchanged total activity before genuine confirmation; the existing ready quiet and revocation checks also passed. - At `fe518c9b`: both normal restoration tests passed in Vitest. Ordinary viewer-to-operator restoration and Pause/Resume recovered the same binding, grant and single contact; each committed one challenge and advanced revision 2 to 3. All six parked, restored and revoked/disabled quiet windows measured at least 15 seconds with active positive read controls and unchanged total queries, callbacks, sessions and verifier calls. Both cleanups passed. - The restoration wrapper still recorded failure: it required lifetime lifecycle denial count zero, whereas the measured count was one before the parked quiet window. The reviewed fixture explicitly permits a current-authority denial from an earlier observation and requires no committed challenge before restoration. Precise denial timing relative to Pause was not serialized. The original wrapper failure is retained; no rerun or retrospective explanation of the earlier failures is claimed. - Review exposed an expired-lease race. The added real-store baseline test failed after the late defer restored `waitingForSignal`; no setup or timeout was counted as the regression. On `cfd515a`, the whole lifecycle owner file passed all 20 tests, including that case. The full normal production declaration program (1067 roots), verification program (662), and auth owning program (405) passed; all 1942 declarations remained byte-identical. - The original remote auth-session shard failed three tests because their partial query adapters lacked the new binding query's join/order/limit chain. A fixture-only correction preserved the real notifier and existing authorization assertions. All 12 auth-session tests passed locally. - Complete remote inspection also found 33 failures in the 35-test Cloud actor fixture with the same missing `innerJoin` method and downstream incomplete-auth assertions. Its original full log and exact synthetic Source comparison are retained. At `6efe722e`, the three-line fixture adapter repair passed the complete normal 662-root test program and the whole 35-test file with zero failures or skips. All existing lock, authorization, settings, debounce and unfinished-auth assertions remain. - The earlier healthy/restoration quiet measurements remain bound to `fe518c9b`. Those steady 15-second windows did not exercise a still-owned attempt passing its 120-second lease. The targeted expired-owner regression qualifies that new boundary; it is not a fresh measurement of the earlier windows. - The initial `fe518c9b` remote gates failed the auth fixtures and reported a Greptile 4/5 P1. The aggregate failure reported the underlying shard. Those actual causes were fixed and qualified above. - Current `6efe722e` remote gates completed with **57 successful checks and three explicit skips**. The full latest edited Greptile summary is **5/5**, explicitly reviews this exact head, and has no new finding. The original P1 thread is resolved. All general, inline and nested review pages were read. - One current-head CI attempt timed out after 30 seconds in the Runner evidence/privacy file, with 6,864 passes and one failure. That file and its transitive inputs are byte-identical to the CI base. A separate local diagnostic preserved every assertion and the original timeout, ran the full normal Runner test project and the entire 41-case file, and passed all 41 with zero skips. Its private bounded timing records measured the target body at 74 ms. The diagnostic was then removed, restoring the exact original test. This is local recovery evidence; the original CI stalled await was not recorded and its cause remains **unproven**. - One retry of only the failed jobs on the unchanged head passed the Runner support suite and dependent aggregate verification. The complete second attempt and original failure logs are preserved. No test, timeout, privacy guard or product code was weakened. This result does not turn the original timeout into an explained failure or a diagnosed flake. - Full recursive typecheck, full `test:run` and a full release build were not repeated locally. The normal full App programs, focused producers and tests above passed, and the required current-head remote gates passed. Final review is against the exact unchanged head; local quiet remains separate from deployed suspension. Reviewer commands use the normal project entry points: `node server/scripts/typecheck-layers.mjs`, the whole `server/src/services/muse-maintenance.test.ts` file, and both normal App PostgreSQL files. PostgreSQL fixtures require explicit owned fixture/admin URLs and normal migrations before imports. Run the files serially with one worker. The private controller retains exact Source, output, worker and owner-closure receipts. ## Risks - Missing or early producer notices could lose a restoration trigger. Notices stay in the transaction that commits the write. The worker checks current authority after commit. - A stale notice must not grant access. Binding generation, current membership, company, agent and credential checks remain authoritative. Revoked credentials require reconnecting. - The change adds no table, migration, alternate task queue or periodic verification scan. It adds two optional fields to existing lifecycle operation JSONB. Generic pending/error deadlines, setup holds and manual repair remain obligations. - Confirmation can race with an active verification attempt. The park compare-and-set must retain the durable recheck. Startup recovery uses a fixed cutoff and keyset pages; it must not become another recurring scan. Wrong company, stale version, pause and revoked credentials must stay fenced. - The connected receiver still has its documented tenant-sleep restriction. Worker quiet is not proof of deployed suspension or external wake. - The normal challenge reply is sent by an authenticated protocol fixture. Live Muse background replies, cost/usage, full Muse lifecycle and full hosted Dot behavior remain unqualified. Partial Dot profile ACK and unsigned-device cutover stay withheld. - Restoration coverage for the additional company, budget, configuration and feature producers remains a separate qualification obligation. This pull request does not mark the whole hosted area complete. ## Model Used OpenAI `gpt-6.1-sol` authored the implementation and tests with local code and tool use. A GPT-6-based Codex coordinator reviewed the changes and owns integration. Its exact underlying model ID is not exposed. Context window sizes are unavailable. No context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
569 lines
27 KiB
TypeScript
569 lines
27 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import express, { type Request } from "express";
|
|
import request from "supertest";
|
|
import { and, eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
authUsers,
|
|
companies,
|
|
companyMemberships,
|
|
instanceSettings,
|
|
instanceUserRoles,
|
|
principalPermissionGrants,
|
|
} from "@paperclipai/db";
|
|
import { actorMiddleware, cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js";
|
|
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./idle-admission.js";
|
|
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
|
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
|
|
|
// Minimal fake Drizzle Db: records every table passed to .insert() / .delete() and
|
|
// supports the chained call shapes used by resolveCloudTenantActor (values /
|
|
// onConflictDo* / returning().then() / delete().where()), plus the
|
|
// select().from(table).where() reads: instanceSettings for the owner-elevation
|
|
// flag resolution through instanceSettingsService, and companyMemberships for
|
|
// the user's own membership rows (rows configurable via membershipQueryRows,
|
|
// where-conditions captured in selectWheres). The chain is awaitable so
|
|
// directly-awaited statements resolve.
|
|
function createFakeDb(options: {
|
|
membershipRow?: { companyId: string; membershipRole: string; status: string };
|
|
membershipQueryRows?: Array<{ companyId: string; membershipRole: string | null; status: string }>;
|
|
settingsRow?: Record<string, unknown> | null;
|
|
selectThrows?: boolean;
|
|
beforeWrite?: (table: unknown) => Promise<void>;
|
|
unchangedMembership?: boolean;
|
|
} = {}) {
|
|
const settingsRow =
|
|
options.settingsRow === undefined
|
|
? {
|
|
id: "00000000-0000-0000-0000-000000000001",
|
|
singletonKey: "default",
|
|
defaultEnvironmentId: null,
|
|
general: {},
|
|
experimental: {},
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}
|
|
: options.settingsRow;
|
|
const insertedTables: unknown[] = [];
|
|
const deletedTables: unknown[] = [];
|
|
const selectWheres: Array<{ table: unknown; condition: unknown; locked?: boolean }> = [];
|
|
function writeChain(table: unknown) {
|
|
let values: unknown;
|
|
const chain: Record<string, unknown> = {};
|
|
chain.values = (value: unknown) => { values = value; return chain; };
|
|
chain.onConflictDoUpdate = () => chain;
|
|
chain.onConflictDoNothing = () => chain;
|
|
chain.where = () => chain;
|
|
chain.returning = async () => {
|
|
await options.beforeWrite?.(table);
|
|
return options.unchangedMembership && table === companyMemberships ? [] : [options.membershipRow ?? values];
|
|
};
|
|
chain.then = (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) =>
|
|
Promise.resolve().then(() => options.beforeWrite?.(table)).then(resolve, reject);
|
|
return chain;
|
|
}
|
|
const db = {
|
|
transaction: async (work: (tx: Db) => Promise<unknown>) => work(db),
|
|
insert: (table: unknown) => {
|
|
insertedTables.push(table);
|
|
return writeChain(table);
|
|
},
|
|
delete: (table: unknown) => {
|
|
deletedTables.push(table);
|
|
return writeChain(table);
|
|
},
|
|
select: () => {
|
|
return {
|
|
from: (table: unknown) => ({
|
|
// Pending Muse bindings use a joined read; this fixture has none.
|
|
innerJoin() { return this; },
|
|
where: (condition: unknown) => {
|
|
// Only settings lookup is degradable. Membership locking and
|
|
// permission seeding must still work in the fail-closed flag test.
|
|
if (options.selectThrows && table === instanceSettings) {
|
|
throw new Error("select unavailable");
|
|
}
|
|
const selected: (typeof selectWheres)[number] = { table, condition };
|
|
selectWheres.push(selected);
|
|
const rows =
|
|
table === instanceSettings && settingsRow
|
|
? [settingsRow]
|
|
: table === companyMemberships
|
|
? (options.membershipQueryRows ?? [])
|
|
: [];
|
|
let locked = false;
|
|
const result = {
|
|
orderBy: () => ({ limit: () => result }),
|
|
for: () => { locked = true; selected.locked = true; return result; },
|
|
then: (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) => {
|
|
// The optional cross-company read can fail independently of
|
|
// the primary membership transaction, which must succeed.
|
|
const query = options.selectThrows && table === companyMemberships && !locked
|
|
? Promise.reject(new Error("select unavailable"))
|
|
: Promise.resolve(rows);
|
|
return query.then(resolve, reject);
|
|
},
|
|
};
|
|
return result;
|
|
},
|
|
}),
|
|
};
|
|
},
|
|
} as unknown as Db;
|
|
return { db, insertedTables, deletedTables, selectWheres };
|
|
}
|
|
|
|
function settingsRowWith(experimental: Record<string, unknown>) {
|
|
return {
|
|
id: "00000000-0000-0000-0000-000000000001",
|
|
singletonKey: "default",
|
|
defaultEnvironmentId: null,
|
|
general: {},
|
|
experimental,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
};
|
|
}
|
|
|
|
function fakeReq(headers: Record<string, string>): Request {
|
|
const lower: Record<string, string> = {};
|
|
for (const [k, v] of Object.entries(headers)) lower[k.toLowerCase()] = v;
|
|
return { header: (name: string) => lower[name.toLowerCase()] } as unknown as Request;
|
|
}
|
|
|
|
const VALID_HEADERS = {
|
|
"x-paperclip-cloud-tenant-token": "test-server-token",
|
|
"x-paperclip-cloud-user-id": "user-123",
|
|
"x-paperclip-cloud-user-email": "Owner@Example.com",
|
|
"x-paperclip-cloud-stack-id": "stack-abc",
|
|
"x-paperclip-cloud-stack-role": "owner",
|
|
};
|
|
|
|
const MANAGED_CONFIG_FLAG_ON = JSON.stringify({
|
|
v: 1,
|
|
mode: "cloud",
|
|
catalogVersion: "test",
|
|
features: { enableOwnerInstanceAdmin: true },
|
|
plugins: { autoInstall: [] },
|
|
});
|
|
|
|
const MANAGED_CONFIG_FLAG_OFF = JSON.stringify({
|
|
v: 1,
|
|
mode: "cloud",
|
|
catalogVersion: "test",
|
|
features: { enableOwnerInstanceAdmin: false },
|
|
plugins: { autoInstall: [] },
|
|
});
|
|
|
|
describe("resolveCloudTenantActor (shared-pool hardening)", () => {
|
|
beforeEach(() => {
|
|
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "test-server-token";
|
|
});
|
|
afterEach(() => {
|
|
stopTaskDrain();
|
|
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
|
delete process.env.PAPERCLIP_MANAGED_CONFIG;
|
|
});
|
|
|
|
it("does not grant instance admin by default (flag off)", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("stays scoped to exactly the stack's company when the user has no other membership rows", async () => {
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
expect(actor!.memberships).toHaveLength(1);
|
|
expect(actor?.memberships?.[0]?.companyId).toBe(actor?.companyIds?.[0]);
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("owner");
|
|
expect(actor!.source).toBe("cloud_tenant");
|
|
});
|
|
|
|
it("purges stale instance_admin rows left by pre-hardening deployments", async () => {
|
|
const { db, deletedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("still upserts the user, company, and membership", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables).toContain(authUsers);
|
|
expect(insertedTables).toContain(companies);
|
|
expect(insertedTables).toContain(companyMemberships);
|
|
});
|
|
|
|
it.each(["/api/health", "/api/instance/task-drain"])("counts unfinished authentication on exempt %s requests", async (path) => {
|
|
let enterWrite!: () => void, finishWrite!: () => void;
|
|
const entered = new Promise<void>(resolve => { enterWrite = resolve; });
|
|
const pending = new Promise<void>(resolve => { finishWrite = resolve; });
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async () => { enterWrite(); await pending; } });
|
|
const server = express();
|
|
server.use(idleAdmissionMiddleware);
|
|
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
|
const hold = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
|
|
run({ execute: async () => [{ blocked: false }] }) } as unknown as Db;
|
|
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
|
server.get("/api/instance/task-drain", async (_req, res) => {
|
|
res.json(await readIdleSleepSafety(emptyDb, () => ({ ...hold, draining: true, activeRuns: 0, pendingWakes: 0 }),
|
|
Date.now, hold.ownerId, async () => "none"));
|
|
});
|
|
trackIdleRequestHandlers(server);
|
|
const accepted = request(server).get(path).set(VALID_HEADERS).then(response => response);
|
|
try {
|
|
await entered;
|
|
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
|
|
const during = await request(server).get("/api/instance/task-drain").expect(200);
|
|
expect(during.body.backgroundWork).toBe("unknown");
|
|
} finally {
|
|
finishWrite();
|
|
await accepted;
|
|
}
|
|
expect(insertedTables).toEqual(expect.arrayContaining([authUsers, companies, companyMemberships]));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
const after = await request(server).get("/api/instance/task-drain").expect(200);
|
|
expect(after.body.backgroundWork).toBe("none");
|
|
});
|
|
|
|
it("releases failed control authentication and preserves Express error handling", async () => {
|
|
const { db } = createFakeDb({ beforeWrite: async () => { throw new Error("fixture write failed"); } });
|
|
const server = express();
|
|
server.use(idleAdmissionMiddleware);
|
|
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
|
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
|
server.use((_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => res.sendStatus(500));
|
|
trackIdleRequestHandlers(server);
|
|
await request(server).get("/api/health").set(VALID_HEADERS).expect(500);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
await request(server).get("/api/health").expect(204);
|
|
});
|
|
|
|
it("resyncs an A to B to A context transition inside the debounce window", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const contextA = VALID_HEADERS;
|
|
const contextB = { ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" };
|
|
|
|
await resolveCloudTenantActor(db, fakeReq(contextA));
|
|
await resolveCloudTenantActor(db, fakeReq(contextB));
|
|
await resolveCloudTenantActor(db, fakeReq(contextA));
|
|
|
|
expect(insertedTables.filter((table) => table === authUsers)).toHaveLength(3);
|
|
expect(insertedTables.filter((table) => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("coalesces concurrent identical syncs while keeping authority reads and stale-admin purges per request", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const { db, insertedTables, deletedTables, selectWheres } = createFakeDb({
|
|
beforeWrite: async table => { if (table === authUsers) { entered(); await blocked; } },
|
|
});
|
|
const requests = Array.from({ length: 20 }, () => resolveCloudTenantActor(db, fakeReq(VALID_HEADERS)));
|
|
await started;
|
|
expect(insertedTables.filter(table => table === authUsers)).toHaveLength(1);
|
|
release();
|
|
const actors = await Promise.all(requests);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
expect(deletedTables.filter(table => table === instanceUserRoles)).toHaveLength(20);
|
|
expect(selectWheres.filter(entry => entry.table === instanceSettings)).toHaveLength(20);
|
|
expect(selectWheres.filter(entry => entry.table === companyMemberships && !entry.locked)).toHaveLength(20);
|
|
expect(selectWheres.filter(entry => entry.table === companyMemberships && entry.locked)).toHaveLength(1);
|
|
expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true);
|
|
});
|
|
|
|
it("serializes concurrent A to B to A without sharing the wrong role", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
let writes = 0;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (table === authUsers && ++writes === 1) { entered(); await blocked; }
|
|
} });
|
|
const first = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await started;
|
|
const second = resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }));
|
|
const third = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
release();
|
|
const actors = await Promise.all([first, second, third]);
|
|
expect(actors.map(actor => actor?.memberships?.[0]?.membershipRole)).toEqual(["owner", "member", "owner"]);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("does not block another user behind a slow user's synchronization", async () => {
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const blocked = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
let firstWrite = true;
|
|
const { db } = createFakeDb({ beforeWrite: async table => {
|
|
if (table === authUsers && firstWrite) { firstWrite = false; entered(); await blocked; }
|
|
} });
|
|
const slow = resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await started;
|
|
try {
|
|
const other = await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-user-id": "other-user" }));
|
|
expect(other?.userId).toBe("other-user");
|
|
} finally {
|
|
release();
|
|
await slow;
|
|
}
|
|
});
|
|
|
|
it("does not debounce a failed partial context transition or its repair", async () => {
|
|
let fail = false;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (fail && table === companyMemberships) throw new Error("partial sync failed");
|
|
} });
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
fail = true;
|
|
await expect(resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" })))
|
|
.rejects.toThrow("partial sync failed");
|
|
fail = false;
|
|
const repaired = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(repaired?.memberships?.[0]?.membershipRole).toBe("owner");
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(3);
|
|
});
|
|
|
|
it("retries a shared transient failure and completes default grants once", async () => {
|
|
let failed = false;
|
|
const { db, insertedTables } = createFakeDb({ beforeWrite: async table => {
|
|
if (!failed && table === companyMemberships) {
|
|
failed = true;
|
|
throw Object.assign(new Error("synthetic connection loss"), { code: "CONNECTION_CLOSED" });
|
|
}
|
|
} });
|
|
const actors = await Promise.all(Array.from({ length: 10 }, () => resolveCloudTenantActor(db, fakeReq(VALID_HEADERS))));
|
|
expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true);
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(insertedTables.filter(table => table === principalPermissionGrants)).toHaveLength(1);
|
|
});
|
|
|
|
it("keeps default-grant repair and the input role when a guarded upsert returns no row", async () => {
|
|
const { db, insertedTables } = createFakeDb({ unchangedMembership: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }));
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("member");
|
|
expect(insertedTables).toContain(principalPermissionGrants);
|
|
});
|
|
|
|
it("does not share completed syncs across databases or companies", async () => {
|
|
const first = createFakeDb();
|
|
const second = createFakeDb();
|
|
await Promise.all([resolveCloudTenantActor(first.db, fakeReq(VALID_HEADERS)), resolveCloudTenantActor(second.db, fakeReq(VALID_HEADERS))]);
|
|
await resolveCloudTenantActor(first.db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-id": "stack-other" }));
|
|
expect(first.insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(second.insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
});
|
|
|
|
it("resyncs when the requested company name changes", async () => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-paperclip-company-name": "First" }));
|
|
await resolveCloudTenantActor(db, fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-paperclip-company-name": "Second" }));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
});
|
|
|
|
it("resumes membership and grant repair after the existing five-second debounce expires", async () => {
|
|
const now = Date.now();
|
|
const clock = vi.spyOn(Date, "now").mockReturnValue(now);
|
|
try {
|
|
const { db, insertedTables } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
|
|
clock.mockReturnValue(now + 5_001);
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(2);
|
|
expect(insertedTables.filter(table => table === principalPermissionGrants)).toHaveLength(2);
|
|
} finally { clock.mockRestore(); }
|
|
});
|
|
|
|
it("returns null when the server token is unset", async () => {
|
|
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).toBeNull();
|
|
});
|
|
|
|
it("resolves identically from a raw upgrade-request header map via the shim", async () => {
|
|
// Websocket upgrades hand us IncomingMessage.headers (lowercased keys,
|
|
// possibly string[] values), not an Express Request. The shim must feed
|
|
// resolveCloudTenantActor the same way Express header() does.
|
|
const { db } = createFakeDb();
|
|
const rawHeaders: Record<string, string | string[] | undefined> = {};
|
|
for (const [k, v] of Object.entries(VALID_HEADERS)) rawHeaders[k.toLowerCase()] = v;
|
|
rawHeaders["x-paperclip-cloud-user-name"] = ["Cloud Owner", "ignored-duplicate"];
|
|
const actor = await resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders(rawHeaders));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.userId).toBe("user-123");
|
|
expect(actor!.userName).toBe("Cloud Owner");
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
});
|
|
|
|
it("maps a non-owner stack role through to the membership without elevating", async () => {
|
|
const { db } = createFakeDb({
|
|
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
|
|
});
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }),
|
|
);
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(actor?.memberships?.[0]?.membershipRole).toBe("member");
|
|
});
|
|
|
|
it.each(["admin", "member", "support"])(
|
|
"grants experimental settings access only to the admin stack role (%s)",
|
|
async (stackRole) => {
|
|
const { db, insertedTables } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": stackRole }),
|
|
);
|
|
expect(actor?.canManageExperimentalSettings).toBe(stackRole === "admin");
|
|
expect(actor?.isInstanceAdmin).toBe(false);
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
},
|
|
);
|
|
|
|
it("removes experimental settings access on the next request after a stack admin is demoted", async () => {
|
|
const { db } = createFakeDb();
|
|
const admin = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "admin" }),
|
|
);
|
|
const member = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }),
|
|
);
|
|
expect(admin?.canManageExperimentalSettings).toBe(true);
|
|
expect(member?.canManageExperimentalSettings).toBe(false);
|
|
});
|
|
|
|
describe("company membership union", () => {
|
|
// The primary company id is derived from the stack id; resolve it once
|
|
// through the same code path instead of duplicating the hash here.
|
|
async function resolvePrimaryCompanyId() {
|
|
const { db } = createFakeDb();
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
return actor!.companyIds![0]!;
|
|
}
|
|
|
|
it("unions the pinned primary with the user's own active membership rows, primary first and deduped", async () => {
|
|
const primaryCompanyId = await resolvePrimaryCompanyId();
|
|
const { db } = createFakeDb({
|
|
membershipQueryRows: [
|
|
// The user's own row for the primary company comes back from the
|
|
// query too — it must not appear twice.
|
|
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
|
|
{ companyId: "company-imported", membershipRole: "member", status: "active" },
|
|
],
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.companyIds).toEqual([primaryCompanyId, "company-imported"]);
|
|
expect(actor!.memberships).toEqual([
|
|
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
|
|
{ companyId: "company-imported", membershipRole: "member", status: "active" },
|
|
]);
|
|
});
|
|
|
|
it("loads memberships with the session path's exact own-user active-status filter", async () => {
|
|
const { db, selectWheres } = createFakeDb();
|
|
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
const membershipSelects = selectWheres.filter((entry) => entry.table === companyMemberships);
|
|
expect(membershipSelects).toHaveLength(2);
|
|
// The first read locks the primary membership before its upsert; the
|
|
// second loads the user's active memberships for request authorization.
|
|
// Rows for other users and rows in any non-active status are excluded
|
|
// by the query itself: the filter binds exactly this user id and the
|
|
// "active" status — the same condition the session path applies.
|
|
expect(membershipSelects[1]!.condition).toEqual(
|
|
and(
|
|
eq(companyMemberships.principalType, "user"),
|
|
eq(companyMemberships.principalId, "user-123"),
|
|
eq(companyMemberships.status, "active"),
|
|
),
|
|
);
|
|
});
|
|
|
|
it("degrades to the primary company when the membership read fails", async () => {
|
|
const { db } = createFakeDb({ selectThrows: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.companyIds).toHaveLength(1);
|
|
expect(actor!.memberships).toHaveLength(1);
|
|
});
|
|
});
|
|
|
|
describe("owner instance-admin elevation (enableOwnerInstanceAdmin)", () => {
|
|
it("elevates the owner while the flag is enabled, still without any role row", async () => {
|
|
const { db, insertedTables, deletedTables } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(true);
|
|
expect(actor!.source).toBe("cloud_tenant");
|
|
// Elevation is computed, never persisted: no instance_user_roles insert,
|
|
// and the stale-row purge still runs on every authentication.
|
|
expect(insertedTables).not.toContain(instanceUserRoles);
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
|
|
it("does not elevate the owner while the flag is disabled", async () => {
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
});
|
|
|
|
it.each(["member", "admin", "support"] as const)(
|
|
"never elevates the %s stack role even with the flag enabled",
|
|
async (stackRole) => {
|
|
const { db } = createFakeDb({
|
|
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(
|
|
db,
|
|
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": stackRole }),
|
|
);
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
},
|
|
);
|
|
|
|
it("resolves the flag through the managed overlay: overlay on elevates over a DB value of off", async () => {
|
|
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_ON;
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(true);
|
|
});
|
|
|
|
it("resolves the flag through the managed overlay: overlay off wins over a DB value of on", async () => {
|
|
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_OFF;
|
|
const { db } = createFakeDb({
|
|
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
|
|
});
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
});
|
|
|
|
it("fails closed when the settings read errors: actor resolves without elevation", async () => {
|
|
const { db, deletedTables } = createFakeDb({ selectThrows: true });
|
|
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
|
|
expect(actor).not.toBeNull();
|
|
expect(actor!.isInstanceAdmin).toBe(false);
|
|
expect(deletedTables).toContain(instanceUserRoles);
|
|
});
|
|
});
|
|
});
|