mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-17 20:09:05 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A host can stop an idle instance to reduce unused compute. > - Zero active runs do not prove that requests, saved work, or cleanup are complete. > - A client can disconnect while its request still writes, and failed cleanup can remain only in memory or on disk. > - This pull request holds new admission and checks accepted work, cleanup, and persisted work under an owned drain. > - The host gets an empty report only while those checks remain valid. ## Linked Issues or Issue Description Refs #13413. That companion PR uses the same owned-hold vocabulary for runtime services. This PR covers HTTP requests, scheduler work, accounting, cleanup, and the instance work inventory. It does not include the preview gateway changes. **What existing behavior does this improve?** The instance task-drain API reports process counters. It does not prove that a host can safely stop the instance for idle sleep. **Current behavior** A quiet run set can coexist with an unfinished request, cleanup after a completed run, future work, or a failed accounting write. **Proposed behavior** Provide a bounded owned idle hold. Block new ingress, track accepted handler promises, inspect durable and local work, and return `none` only when the same hold stays quiet through the checks. Keep normal deployment drains compatible. **Reason and benefit** Hosts can identify eligible idle instances without treating a disconnect or failed cleanup write as completed work. ## What Changed - Add `purpose: "idle"`, a bounded TTL, unique owners, and owner-checked release to task drain. Existing holds cannot be replaced by another API request. - Gate HTTP ingress before parsers, auth, webhooks, and MCP. Gate new WebSocket upgrades. Track async handlers in nested Express routers and error middleware until they settle, even after the response or client disconnect. - Count accepted live-event WebSocket authentication through settlement, even after disconnects. Count detached built-in agent, managed-home and runtime-service startup reconciliation after readiness. - Keep health and control mutations tracked. Count control-request authentication separately from the read-only report, including concurrent user/company/membership writes. - Pause new scheduler admissions during idle holds. Count work already in flight, including database backups, and reject scans whose work generation changes. Periodic backups block sleep without a host wake schedule. - Inspect accounting and orphan-cleanup spool directories without skipping temporary or malformed entries. Retain orphan tokens through queue splices, flush failures, and buffer overflow. Keep failed usage capture counted until its database failure fence is written. - Check persisted work across companies in a bounded read-only transaction. Include deferred agent-file cleanup and saved watchdogs whose watched issues are complete. Enabled plugins and unsupported retained work remain blockers. - Require the exact idle owner and completed startup tracking before returning an empty report. Keep reports free of tenant details. - Document the hosting protocol, retry behavior, conservative blockers, and the remaining external provider-stop race. ## Verification - Current head: `8d9a599b732c2047b8c671d4799065d7c90a3567`, rebased on master `941a3fa991aeb97eb1ac390c65b7973b5f6de1ad`. Both heartbeat helper extractions are preserved. GitHub confirms no merge conflicts. - Focused heartbeat renderer/run-log, drain, control-auth, admission, route and PostgreSQL inventory checks: 234 tests passed in 10 suites after the rebase. - The POST task-drain contract includes the expected `409` conflict response. The 84 OpenAPI and instance-settings route tests and server typecheck passed after that final documentation fix. - Final accepted-upgrade/startup regression run: 104 tests passed in five suites, including success and failure after readiness or disconnect. Final server typecheck also passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm install --frozen-lockfile` and the PR diff secret scan passed. No dependency or lockfile changes are added by this PR. - The prior verification covered HTTP disconnects and early responses, async error handlers, concurrent authentication writes, signed bootstrap, saved watchdogs, backup promises, orphan cleanup, spools and failed accounting fences. Those tests passed. - The last full local `pnpm test:run` stopped in the general-server phase with 16,658 passing tests and five skill/connector fixture failures caused by an ancestor workspace skill directory. That full local run preceded this rebase and has not been repeated for the import conflict. Full current-head CI passed: 53 successful checks and two conditional skips, with no failures. - All eight review findings are fixed and their threads resolved, including accepted upgrade authentication, detached startup writes and the POST conflict contract. Current-head Apex review is 5/5 with no new findings; all eight review threads remain resolved. - No live provider stop or production change was performed. ## Risks - The hosting controller must use the owned protocol and hold external admission through its final validation and provider stop. A legacy drain cannot authorize idle sleep. During an idle hold, new requests receive 503 with `Retry-After: 1`; the host must handle queueing or retry before enabling this path. - This is a single-process protocol. An unexpected restart after the last validation can race an external provider stop. The host must serialize deploy/wake/sleep operations and bind the validation to the instance it stops. Multiple replicas need shared fencing. - Some retained state conservatively prevents sleep, including every enabled plugin. This PR does not promise that every inactive instance becomes eligible. - The HTTP adapter uses Express 5 router layers. Real Express tests cover nested routes, errors and disconnects. New routes must register before tracking is installed. Detached work must have durable state or explicit work tracking. - Unrecoverable in-memory cleanup debt keeps the instance awake until reconciliation. This change does not make such debt survive an unplanned process crash. No schema migration or provider configuration change is included. ## Model Used OpenAI GPT-6 via Codex, with reasoning, repository inspection, tool use and code execution. The exact deployment variant/model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run focused tests locally and they pass; the full local fixture limitation and clean full CI result are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1023 lines
35 KiB
TypeScript
1023 lines
35 KiB
TypeScript
import { createHash, timingSafeEqual } from "node:crypto";
|
|
import type { Request, RequestHandler } from "express";
|
|
import { and, eq, isNull } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
activityLog,
|
|
agentApiKeys,
|
|
agents,
|
|
authUsers,
|
|
companies,
|
|
companyMemberships,
|
|
heartbeatRuns,
|
|
instanceUserRoles,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
MAX_ISSUE_PREFIX_ATTEMPTS,
|
|
deriveIssuePrefixBase,
|
|
isIssuePrefixConflict,
|
|
issuePrefixSuffixForAttempt,
|
|
pickAvailableIssuePrefix,
|
|
rekeyCompanyIssueIdentifiers,
|
|
} from "../services/issue-prefix.js";
|
|
import { verifyLocalAgentJwt } from "../agent-auth-jwt.js";
|
|
import { agentRunWritesRevoked } from "../agent-run-cancellation.js";
|
|
import { isUuidLike, normalizeAgentApiKeyScope, type DeploymentMode } from "@paperclipai/shared";
|
|
import type { BetterAuthSessionResult } from "../auth/better-auth.js";
|
|
import { logger } from "./logger.js";
|
|
import { captureRunIdentity } from "../services/run-identity.js";
|
|
import { boardAuthService } from "../services/board-auth.js";
|
|
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
|
|
import { beginIdleTrackedWork } from "../services/task-admission.js";
|
|
|
|
export {
|
|
isTransientDbConnectionError,
|
|
retryIdempotentDatabaseOperation as retryOnTransientDbConnectionError,
|
|
} from "../database-retry.js";
|
|
|
|
const CLOUD_TENANT_WRITE_DEBOUNCE_MS = 5_000;
|
|
const CLOUD_TENANT_WRITE_DEBOUNCE_MAX = 1_000;
|
|
const cloudTenantWriteDebounces = new WeakMap<Db, Map<string, { fingerprint: string; syncedAt: number }>>();
|
|
|
|
function cloudTenantWriteDebounceFor(db: Db) {
|
|
let debounce = cloudTenantWriteDebounces.get(db);
|
|
if (!debounce) {
|
|
debounce = new Map();
|
|
cloudTenantWriteDebounces.set(db, debounce);
|
|
}
|
|
return debounce;
|
|
}
|
|
|
|
function pruneCloudTenantWriteDebounce(
|
|
debounce: Map<string, { fingerprint: string; syncedAt: number }>,
|
|
nowMs: number,
|
|
) {
|
|
for (const [subject, entry] of debounce) {
|
|
if (entry.syncedAt <= nowMs - CLOUD_TENANT_WRITE_DEBOUNCE_MS) debounce.delete(subject);
|
|
}
|
|
while (debounce.size > CLOUD_TENANT_WRITE_DEBOUNCE_MAX) {
|
|
const oldestSubject = debounce.keys().next().value;
|
|
if (!oldestSubject) break;
|
|
debounce.delete(oldestSubject);
|
|
}
|
|
}
|
|
import { instanceSettingsService } from "../services/instance-settings.js";
|
|
import { ensureHumanRoleDefaultGrants } from "../services/principal-access-compatibility.js";
|
|
import { forbidden, unauthorized, unprocessable } from "../errors.js";
|
|
|
|
export { isCloudManagedInstance } from "../services/cloud-instance.js";
|
|
import { cloudTenantPrimaryCompanyId } from "../services/cloud-instance.js";
|
|
|
|
function hashToken(token: string) {
|
|
return createHash("sha256").update(token).digest("hex");
|
|
}
|
|
|
|
function normalizeOptionalString(value: string | null | undefined) {
|
|
return value?.trim() || null;
|
|
}
|
|
|
|
function invalidAgentTokenMessage(token: string) {
|
|
try {
|
|
const payload = JSON.parse(Buffer.from(token.split(".")[1] ?? "", "base64url").toString("utf8")) as {
|
|
exp?: unknown;
|
|
};
|
|
if (typeof payload.exp === "number" && payload.exp <= Math.floor(Date.now() / 1000)) {
|
|
return "Expired agent token; obtain fresh credentials and retry";
|
|
}
|
|
} catch {
|
|
// Malformed and incorrectly signed tokens share the generic failure below.
|
|
}
|
|
return "Agent token did not verify; obtain fresh credentials and retry";
|
|
}
|
|
|
|
async function resolveLegacyRunResponsibleUserId(
|
|
db: Db,
|
|
input: { companyId: string; agentId: string; runId: string },
|
|
) {
|
|
if (!isUuidLike(input.runId)) return null;
|
|
const run = await db
|
|
.select({ responsibleUserId: heartbeatRuns.responsibleUserId })
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
return normalizeOptionalString(run?.responsibleUserId);
|
|
}
|
|
|
|
async function loadResponsibleUserMemberships(
|
|
db: Db,
|
|
input: { companyId: string; userId: string | null },
|
|
) {
|
|
if (!input.userId) return [];
|
|
const [user, memberships] = await Promise.all([
|
|
db
|
|
.select({ id: authUsers.id })
|
|
.from(authUsers)
|
|
.where(eq(authUsers.id, input.userId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db
|
|
.select({
|
|
companyId: companyMemberships.companyId,
|
|
membershipRole: companyMemberships.membershipRole,
|
|
status: companyMemberships.status,
|
|
})
|
|
.from(companyMemberships)
|
|
.where(
|
|
and(
|
|
eq(companyMemberships.companyId, input.companyId),
|
|
eq(companyMemberships.principalType, "user"),
|
|
eq(companyMemberships.principalId, input.userId),
|
|
eq(companyMemberships.status, "active"),
|
|
),
|
|
),
|
|
]);
|
|
return user ? memberships : [];
|
|
}
|
|
|
|
/**
|
|
* The user's own active company memberships — the exact company scope a
|
|
* locally authenticated session actor carries. Shared by the session path
|
|
* and the Cloud trusted-header path so both resolve the same access set.
|
|
*/
|
|
async function loadActiveUserCompanyMemberships(db: Db, userId: string) {
|
|
return db
|
|
.select({
|
|
companyId: companyMemberships.companyId,
|
|
membershipRole: companyMemberships.membershipRole,
|
|
status: companyMemberships.status,
|
|
})
|
|
.from(companyMemberships)
|
|
.where(
|
|
and(
|
|
eq(companyMemberships.principalType, "user"),
|
|
eq(companyMemberships.principalId, userId),
|
|
eq(companyMemberships.status, "active"),
|
|
),
|
|
);
|
|
}
|
|
|
|
async function auditAgentJwtRunHeaderMismatch(
|
|
db: Db,
|
|
input: { companyId: string; agentId: string; claimRunId: string; headerRunId: string; method: string; url: string },
|
|
) {
|
|
try {
|
|
await db.insert(activityLog).values({
|
|
companyId: input.companyId,
|
|
actorType: "agent",
|
|
actorId: input.agentId,
|
|
action: "auth.agent_jwt_run_header_mismatch",
|
|
entityType: "heartbeat_run",
|
|
entityId: input.claimRunId,
|
|
...(isUuidLike(input.agentId) ? { agentId: input.agentId } : {}),
|
|
...(isUuidLike(input.claimRunId) ? { runId: input.claimRunId } : {}),
|
|
details: {
|
|
claimRunId: input.claimRunId,
|
|
headerRunId: input.headerRunId,
|
|
method: input.method,
|
|
url: input.url,
|
|
},
|
|
});
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, companyId: input.companyId, agentId: input.agentId, claimRunId: input.claimRunId },
|
|
"Failed to audit rejected agent JWT run header mismatch",
|
|
);
|
|
}
|
|
}
|
|
|
|
async function auditAgentKeyMissingResponsibleUser(
|
|
db: Db,
|
|
input: { companyId: string; agentId: string; keyId: string; method: string; url: string },
|
|
) {
|
|
try {
|
|
await db.insert(activityLog).values({
|
|
companyId: input.companyId,
|
|
actorType: "agent",
|
|
actorId: input.agentId,
|
|
action: "auth.agent_key_missing_responsible_user",
|
|
entityType: "agent_api_key",
|
|
entityId: input.keyId,
|
|
...(isUuidLike(input.agentId) ? { agentId: input.agentId } : {}),
|
|
details: {
|
|
method: input.method,
|
|
url: input.url,
|
|
},
|
|
});
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, companyId: input.companyId, agentId: input.agentId, keyId: input.keyId },
|
|
"Failed to audit rejected agent key without responsible user binding",
|
|
);
|
|
}
|
|
}
|
|
|
|
interface ActorMiddlewareOptions {
|
|
deploymentMode: DeploymentMode;
|
|
resolveSession?: (req: Request) => Promise<BetterAuthSessionResult | null>;
|
|
}
|
|
|
|
const publicRoutineWebhookPath = /^\/api\/routine-triggers\/public\/[a-f0-9]{24}\/fire\/?$/i;
|
|
|
|
const publicMcpGatewayProtocolPath = /^\/mcp\/gateways\/gw_[a-f0-9]{32}\/?$/i;
|
|
|
|
export function actorMiddleware(db: Db, opts: ActorMiddlewareOptions): RequestHandler {
|
|
const boardAuth = boardAuthService(db);
|
|
const authenticate: RequestHandler = async (req, _res, next) => {
|
|
req.actor =
|
|
opts.deploymentMode === "local_trusted"
|
|
? {
|
|
type: "board",
|
|
userId: "local-board",
|
|
userName: "Local Board",
|
|
userEmail: null,
|
|
isInstanceAdmin: true,
|
|
source: "local_implicit",
|
|
}
|
|
: { type: "none", source: "none" };
|
|
|
|
// Routine ingress authenticates its own bearer/signature. Never interpret
|
|
// webhook credentials as agent keys or attach an ambient browser session.
|
|
if (req.method === "POST" && publicRoutineWebhookPath.test(req.path)) {
|
|
req.actor = { type: "none", source: "none" };
|
|
next();
|
|
return;
|
|
}
|
|
|
|
const runIdHeader = req.header("x-paperclip-run-id");
|
|
|
|
const authHeader = req.header("authorization");
|
|
const hasBearerCredentials = /^bearer(?:\s|$)/i.test(authHeader ?? "");
|
|
|
|
// Public MCP gateway protocol requests carry a pcgw_* bearer that is
|
|
// validated by the gateway service itself. Do not interpret that bearer as
|
|
// a board key or agent JWT here: doing so rejects the MCP handshake before
|
|
// the protocol route can verify its run-scoped credential. Keep this bypass
|
|
// restricted to the unguessable public gateway path; all /api routes retain
|
|
// the normal actor authentication path below.
|
|
if (hasBearerCredentials && publicMcpGatewayProtocolPath.test(req.path)) {
|
|
if (runIdHeader) req.actor.runId = runIdHeader;
|
|
next();
|
|
return;
|
|
}
|
|
|
|
if (!hasBearerCredentials) {
|
|
if (opts.deploymentMode === "authenticated" && opts.resolveSession) {
|
|
const cloudTenantActor = await resolveCloudTenantActor(db, req);
|
|
if (cloudTenantActor) {
|
|
req.actor = {
|
|
...cloudTenantActor,
|
|
runId: runIdHeader ?? undefined,
|
|
};
|
|
next();
|
|
return;
|
|
}
|
|
|
|
let session: BetterAuthSessionResult | null = null;
|
|
try {
|
|
session = await opts.resolveSession(req);
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, method: req.method, url: req.originalUrl },
|
|
"Failed to resolve auth session from request headers",
|
|
);
|
|
}
|
|
if (session?.user?.id && session.session?.id) {
|
|
const userId = session.user.id;
|
|
const [roleRow, memberships] = await Promise.all([
|
|
db
|
|
.select({ id: instanceUserRoles.id })
|
|
.from(instanceUserRoles)
|
|
.where(and(eq(instanceUserRoles.userId, userId), eq(instanceUserRoles.role, "instance_admin")))
|
|
.then((rows) => rows[0] ?? null),
|
|
loadActiveUserCompanyMemberships(db, userId),
|
|
]);
|
|
req.actor = {
|
|
type: "board",
|
|
userId,
|
|
sessionId: session.session.id,
|
|
userName: session.user.name ?? null,
|
|
userEmail: session.user.email ?? null,
|
|
companyIds: memberships.map((row) => row.companyId),
|
|
memberships,
|
|
isInstanceAdmin: Boolean(roleRow),
|
|
runId: runIdHeader ?? undefined,
|
|
source: "session",
|
|
};
|
|
next();
|
|
return;
|
|
}
|
|
}
|
|
if (runIdHeader) req.actor.runId = runIdHeader;
|
|
next();
|
|
return;
|
|
}
|
|
|
|
const token = authHeader!.slice("bearer".length).trim();
|
|
if (!token) {
|
|
next(unauthorized("Empty bearer token; provide valid agent credentials and retry"));
|
|
return;
|
|
}
|
|
|
|
const boardKey = await boardAuth.findBoardApiKeyByToken(token);
|
|
if (boardKey) {
|
|
const access = await boardAuth.resolveBoardAccess(boardKey.userId);
|
|
if (access.user) {
|
|
await boardAuth.touchBoardApiKey(boardKey.id);
|
|
req.actor = {
|
|
type: "board",
|
|
userId: boardKey.userId,
|
|
userName: access.user?.name ?? null,
|
|
userEmail: access.user?.email ?? null,
|
|
companyIds: access.companyIds,
|
|
memberships: access.memberships,
|
|
isInstanceAdmin: access.isInstanceAdmin,
|
|
keyId: boardKey.id,
|
|
runId: runIdHeader || undefined,
|
|
source: "board_key",
|
|
};
|
|
next();
|
|
return;
|
|
}
|
|
}
|
|
|
|
const tokenHash = hashToken(token);
|
|
const key = await db
|
|
.select()
|
|
.from(agentApiKeys)
|
|
.where(and(eq(agentApiKeys.keyHash, tokenHash), isNull(agentApiKeys.revokedAt)))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!key) {
|
|
const claims = verifyLocalAgentJwt(token);
|
|
if (!claims) {
|
|
next(unauthorized(invalidAgentTokenMessage(token)));
|
|
return;
|
|
}
|
|
|
|
const agentRecord = await db
|
|
.select()
|
|
.from(agents)
|
|
.where(eq(agents.id, claims.sub))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!agentRecord || agentRecord.companyId !== claims.company_id) {
|
|
next(unauthorized("Agent record is missing or belongs to another company; obtain fresh credentials and retry"));
|
|
return;
|
|
}
|
|
|
|
if (agentRecord.status === "terminated") {
|
|
next(unauthorized("Agent is terminated and cannot authenticate"));
|
|
return;
|
|
}
|
|
if (agentRecord.status === "pending_approval") {
|
|
next(unauthorized("Agent is pending approval and cannot authenticate"));
|
|
return;
|
|
}
|
|
|
|
const normalizedRunIdHeader = normalizeOptionalString(runIdHeader);
|
|
if (normalizedRunIdHeader && normalizedRunIdHeader !== claims.run_id) {
|
|
await auditAgentJwtRunHeaderMismatch(db, {
|
|
companyId: claims.company_id,
|
|
agentId: claims.sub,
|
|
claimRunId: claims.run_id,
|
|
headerRunId: normalizedRunIdHeader,
|
|
method: req.method,
|
|
url: req.originalUrl,
|
|
});
|
|
next(
|
|
unprocessable("X-Paperclip-Run-Id does not match signed agent JWT run_id", {
|
|
code: "agent_jwt_run_id_mismatch",
|
|
claimRunId: claims.run_id,
|
|
headerRunId: normalizedRunIdHeader,
|
|
}),
|
|
);
|
|
return;
|
|
}
|
|
|
|
const [identityRun] = await db.select({ activeIdentityContextId: heartbeatRuns.activeIdentityContextId,
|
|
responsibleUserId: heartbeatRuns.responsibleUserId, status: heartbeatRuns.status, resultJson: heartbeatRuns.resultJson,
|
|
contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, claims.run_id), eq(heartbeatRuns.companyId, claims.company_id), eq(heartbeatRuns.agentId, claims.sub),
|
|
));
|
|
if (agentRunWritesRevoked(identityRun)
|
|
&& !["GET", "HEAD", "OPTIONS"].includes(req.method)) {
|
|
const conversation = identityRun?.contextSnapshot?.conversationMode === true;
|
|
_res.status(403).json({ error: conversation ? "This conversation turn was cancelled" : "This run was cancelled",
|
|
code: conversation ? "conversation_turn_cancelled" : "agent_run_cancelled" });
|
|
return;
|
|
}
|
|
if (identityRun?.activeIdentityContextId && identityRun.status === "running") {
|
|
const captured = await captureRunIdentity(db, { companyId: claims.company_id, agentId: claims.sub, runId: claims.run_id });
|
|
identityRun.activeIdentityContextId = captured.context?.id ?? null;
|
|
identityRun.responsibleUserId = captured.context?.responsibleUserId ?? null;
|
|
}
|
|
const onBehalfOfUserId = identityRun?.activeIdentityContextId
|
|
? identityRun.responsibleUserId
|
|
: claims.responsible_user_id !== undefined
|
|
? normalizeOptionalString(claims.responsible_user_id)
|
|
: await resolveLegacyRunResponsibleUserId(db, {
|
|
companyId: claims.company_id,
|
|
agentId: claims.sub,
|
|
runId: claims.run_id,
|
|
});
|
|
const onBehalfOfMemberships = await loadResponsibleUserMemberships(db, {
|
|
companyId: claims.company_id,
|
|
userId: onBehalfOfUserId,
|
|
});
|
|
|
|
req.actor = {
|
|
type: "agent",
|
|
agentId: claims.sub,
|
|
companyId: claims.company_id,
|
|
keyId: undefined,
|
|
keyScope: normalizeAgentApiKeyScope(claims.key_scope),
|
|
runId: claims.run_id,
|
|
onBehalfOfUserId,
|
|
identityContextId: identityRun?.activeIdentityContextId ?? null,
|
|
onBehalfOfMemberships,
|
|
source: "agent_jwt",
|
|
};
|
|
next();
|
|
return;
|
|
}
|
|
|
|
await db
|
|
.update(agentApiKeys)
|
|
.set({ lastUsedAt: new Date() })
|
|
.where(eq(agentApiKeys.id, key.id));
|
|
|
|
const agentRecord = await db
|
|
.select()
|
|
.from(agents)
|
|
.where(eq(agents.id, key.agentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!agentRecord || agentRecord.companyId !== key.companyId) {
|
|
next(unauthorized("Agent record is missing or belongs to another company; obtain fresh credentials and retry"));
|
|
return;
|
|
}
|
|
if (agentRecord.status === "terminated") {
|
|
next(unauthorized("Agent is terminated and cannot authenticate"));
|
|
return;
|
|
}
|
|
if (agentRecord.status === "pending_approval") {
|
|
next(unauthorized("Agent is pending approval and cannot authenticate"));
|
|
return;
|
|
}
|
|
|
|
const responsibleUserId = normalizeOptionalString(key.responsibleUserId);
|
|
if (!responsibleUserId) {
|
|
await auditAgentKeyMissingResponsibleUser(db, {
|
|
companyId: key.companyId,
|
|
agentId: key.agentId,
|
|
keyId: key.id,
|
|
method: req.method,
|
|
url: req.originalUrl,
|
|
});
|
|
next(forbidden("Responsible user is unavailable for this agent key", {
|
|
code: "RESPONSIBLE_USER_UNAVAILABLE",
|
|
}));
|
|
return;
|
|
}
|
|
|
|
req.actor = {
|
|
type: "agent",
|
|
agentId: key.agentId,
|
|
companyId: key.companyId,
|
|
keyId: key.id,
|
|
keyScope: normalizeAgentApiKeyScope(key.scopeConfig),
|
|
onBehalfOfUserId: responsibleUserId,
|
|
onBehalfOfMemberships: await loadResponsibleUserMemberships(db, {
|
|
companyId: key.companyId,
|
|
userId: responsibleUserId,
|
|
}),
|
|
runId: runIdHeader || undefined,
|
|
source: "agent_key",
|
|
};
|
|
|
|
next();
|
|
};
|
|
return async (req, res, next) => {
|
|
// Health and task-drain requests bypass tenant admission, but resolving
|
|
// their actor can still write users, companies, memberships and key usage.
|
|
// Finish all authentication before entering the next handler: the report
|
|
// must count concurrent authentication, without counting its own auth.
|
|
const finish = beginIdleTrackedWork();
|
|
let continueRequest: (() => void) | undefined;
|
|
try {
|
|
await authenticate(req, res, (error?: unknown) => {
|
|
continueRequest = () => next(error);
|
|
});
|
|
} finally {
|
|
finish();
|
|
}
|
|
continueRequest?.();
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Whether the trusted-header actor being resolved should carry computed
|
|
* instance-admin elevation: only the stack `owner` role elevates, and only
|
|
* while `enableOwnerInstanceAdmin` is enabled. The flag is resolved through
|
|
* the instance-settings service so the cloud managed-config overlay applies
|
|
* (the harness can turn elevation off fleet-wide without touching tenant
|
|
* DBs). Fails closed: a settings read error means no elevation.
|
|
*/
|
|
async function resolveOwnerInstanceAdmin(
|
|
db: Db,
|
|
stackRole: "owner" | "admin" | "member" | "support",
|
|
): Promise<boolean> {
|
|
if (stackRole !== "owner") return false;
|
|
try {
|
|
const experimental = await instanceSettingsService(db).getExperimental();
|
|
return experimental.enableOwnerInstanceAdmin === true;
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err },
|
|
"Failed to resolve enableOwnerInstanceAdmin for cloud tenant owner; treating elevation as disabled",
|
|
);
|
|
return false;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Minimal header accessor `resolveCloudTenantActor` needs. Express `Request`
|
|
* satisfies it directly; websocket upgrade paths adapt a raw
|
|
* `IncomingMessage` with {@link cloudActorHeaderSourceFromHeaders} since
|
|
* trusted-header authentication must work identically for upgrades — a
|
|
* cloud-proxied browser has no local Better Auth session to fall back on.
|
|
*/
|
|
export interface CloudActorHeaderSource {
|
|
header(name: string): string | undefined;
|
|
}
|
|
|
|
/** Adapts a raw header map (e.g. `IncomingMessage.headers`) to {@link CloudActorHeaderSource}. */
|
|
export function cloudActorHeaderSourceFromHeaders(
|
|
headers: Record<string, string | string[] | undefined>,
|
|
): CloudActorHeaderSource {
|
|
return {
|
|
header(name: string) {
|
|
const value = headers[name.toLowerCase()];
|
|
return Array.isArray(value) ? value[0] : value;
|
|
},
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Trusted-header actor resolution with bounded transient-connection retries.
|
|
* The tenant sync inside is idempotent end to end — every write is an
|
|
* upsert/on-conflict/delete and the write debounce records only after the
|
|
* whole sync succeeds — so replaying it after a dropped connection is safe,
|
|
* and turns a golden-path authentication 500 into a served request.
|
|
*/
|
|
export async function resolveCloudTenantActor(
|
|
db: Db,
|
|
req: CloudActorHeaderSource,
|
|
): Promise<Express.Request["actor"] | null> {
|
|
return retryIdempotentDatabaseOperation(() => resolveCloudTenantActorOnce(db, req));
|
|
}
|
|
|
|
async function resolveCloudTenantActorOnce(
|
|
db: Db,
|
|
req: CloudActorHeaderSource,
|
|
): Promise<Express.Request["actor"] | null> {
|
|
const expectedToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
|
|
if (!expectedToken) return null;
|
|
|
|
const token = req.header("x-paperclip-cloud-tenant-token")?.trim();
|
|
if (!token || !constantTimeStringEqual(token, expectedToken)) return null;
|
|
|
|
const userId = requiredCloudHeader(req, "x-paperclip-cloud-user-id");
|
|
const userEmail = requiredCloudHeader(req, "x-paperclip-cloud-user-email").toLowerCase();
|
|
const stackId = requiredCloudHeader(req, "x-paperclip-cloud-stack-id");
|
|
const stackRole = stackMembershipRole(req.header("x-paperclip-cloud-stack-role"));
|
|
const userName = req.header("x-paperclip-cloud-user-name")?.trim() || userEmail;
|
|
const paperclipCompanyId = req.header("x-paperclip-cloud-paperclip-company-id")?.trim();
|
|
const paperclipCompanyName = req
|
|
.header("x-paperclip-cloud-paperclip-company-name")
|
|
?.trim();
|
|
const companyId = cloudTenantCompanyId(stackId);
|
|
const companyName = paperclipCompanyName || humanizeCloudStackSlug(stackId);
|
|
const now = new Date();
|
|
const membershipRole = stackRole === "owner" || stackRole === "admin" ? "owner" : stackRole;
|
|
const syncFingerprint = [userEmail, userName, stackId, stackRole, paperclipCompanyId ?? ""].join(":");
|
|
const cloudTenantWriteDebounce = cloudTenantWriteDebounceFor(db);
|
|
pruneCloudTenantWriteDebounce(cloudTenantWriteDebounce, now.getTime());
|
|
const previousSync = cloudTenantWriteDebounce.get(userId);
|
|
const shouldSync = previousSync?.fingerprint !== syncFingerprint
|
|
|| previousSync.syncedAt <= now.getTime() - CLOUD_TENANT_WRITE_DEBOUNCE_MS;
|
|
let effectiveMembership: { companyId: string; membershipRole: string | null; status: string } = {
|
|
companyId,
|
|
membershipRole,
|
|
status: "active",
|
|
};
|
|
|
|
if (shouldSync) await db
|
|
.insert(authUsers)
|
|
.values({
|
|
id: userId,
|
|
name: userName,
|
|
email: userEmail,
|
|
emailVerified: true,
|
|
image: null,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoUpdate({
|
|
target: authUsers.id,
|
|
set: {
|
|
name: userName,
|
|
email: userEmail,
|
|
emailVerified: true,
|
|
updatedAt: now,
|
|
},
|
|
});
|
|
|
|
// Earlier cloud_tenant builds granted every tenant user `instance_admin`.
|
|
// Stale rows from those deployments would still elevate this user through
|
|
// the BetterAuth session path, board API keys, and the authorization
|
|
// service's own instanceUserRoles lookup — so actively purge them on every
|
|
// trusted-header authentication instead of merely no longer inserting them.
|
|
await db
|
|
.delete(instanceUserRoles)
|
|
.where(and(eq(instanceUserRoles.userId, userId), eq(instanceUserRoles.role, "instance_admin")));
|
|
|
|
if (shouldSync) await insertCloudTenantCompany(db, { companyId, companyName, now });
|
|
|
|
if (shouldSync && paperclipCompanyName) {
|
|
await repairCloudTenantCompanyName(db, {
|
|
companyId,
|
|
paperclipCompanyId,
|
|
paperclipCompanyName,
|
|
now,
|
|
});
|
|
}
|
|
|
|
// Runs after the name repair so the prefix derives from the repaired name.
|
|
// The helper self-gates on the legacy markers, so it is a no-op once the
|
|
// company has been repaired or was claimed by a current build.
|
|
if (shouldSync) {
|
|
await repairCloudTenantCompanyProvisionDefaults(db, { companyId, stackId, now });
|
|
}
|
|
|
|
effectiveMembership = shouldSync ? await db
|
|
.insert(companyMemberships)
|
|
.values({
|
|
companyId,
|
|
principalType: "user",
|
|
principalId: userId,
|
|
status: "active",
|
|
membershipRole,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoUpdate({
|
|
target: [
|
|
companyMemberships.companyId,
|
|
companyMemberships.principalType,
|
|
companyMemberships.principalId,
|
|
],
|
|
set: {
|
|
status: "active",
|
|
membershipRole,
|
|
updatedAt: now,
|
|
},
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? {
|
|
companyId,
|
|
membershipRole,
|
|
status: "active",
|
|
}) : { companyId, membershipRole, status: "active" as const };
|
|
|
|
// Without instance-admin elevation, cloud tenant users are authorized purely
|
|
// through company-scoped permission grants — seed the same role defaults the
|
|
// regular membership flows create.
|
|
if (shouldSync) await ensureHumanRoleDefaultGrants(db, {
|
|
companyId,
|
|
principalId: userId,
|
|
membershipRole: effectiveMembership.membershipRole ?? membershipRole,
|
|
grantedByUserId: null,
|
|
});
|
|
if (shouldSync) {
|
|
cloudTenantWriteDebounce.delete(userId);
|
|
cloudTenantWriteDebounce.set(userId, { fingerprint: syncFingerprint, syncedAt: Date.now() });
|
|
pruneCloudTenantWriteDebounce(cloudTenantWriteDebounce, Date.now());
|
|
}
|
|
|
|
// The stack's seeded company is only where Cloud provisioned this user.
|
|
// Companies created afterwards on the instance (imports, in-app company
|
|
// creation) attach real membership rows for the user, so union those with
|
|
// the pinned primary — the same active-membership scope a locally
|
|
// authenticated session actor carries. Strictly this user's own rows; the
|
|
// membership-creating flows seed their own permission grants, so nothing
|
|
// needs seeding per request here. A read failure degrades to the pinned
|
|
// primary instead of blocking authentication, mirroring the fail-closed
|
|
// owner-elevation resolution below.
|
|
let additionalMemberships: { companyId: string; membershipRole: string | null; status: string }[] =
|
|
[];
|
|
try {
|
|
additionalMemberships = (await loadActiveUserCompanyMemberships(db, userId)).filter(
|
|
(row) => row.companyId !== companyId,
|
|
);
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, userId, stackId },
|
|
"Failed to load cloud tenant user's company memberships; scoping actor to the stack's primary company",
|
|
);
|
|
}
|
|
|
|
return {
|
|
type: "board",
|
|
userId,
|
|
userName,
|
|
userEmail,
|
|
companyIds: [companyId, ...additionalMemberships.map((row) => row.companyId)],
|
|
memberships: [
|
|
{
|
|
companyId,
|
|
membershipRole: effectiveMembership.membershipRole ?? membershipRole,
|
|
status: effectiveMembership.status,
|
|
},
|
|
...additionalMemberships,
|
|
],
|
|
// Computed per request, never persisted: the stack owner is elevated to
|
|
// instance admin of their own dedicated instance only while the
|
|
// `enableOwnerInstanceAdmin` flag is on. Non-owner stack roles stay
|
|
// company-scoped. Turning the flag off de-elevates on the next request —
|
|
// there is no role row to clean up.
|
|
isInstanceAdmin: await resolveOwnerInstanceAdmin(db, stackRole),
|
|
source: "cloud_tenant",
|
|
};
|
|
}
|
|
|
|
function requiredCloudHeader(req: CloudActorHeaderSource, name: string): string {
|
|
const value = req.header(name)?.trim();
|
|
if (!value) {
|
|
throw new Error(`Missing trusted Cloud tenant header ${name}`);
|
|
}
|
|
return value;
|
|
}
|
|
|
|
function stackMembershipRole(value: string | undefined): "owner" | "admin" | "member" | "support" {
|
|
if (value === "owner" || value === "admin" || value === "member" || value === "support") {
|
|
return value;
|
|
}
|
|
throw new Error("Invalid trusted Cloud tenant stack role");
|
|
}
|
|
|
|
function constantTimeStringEqual(left: string, right: string): boolean {
|
|
const leftBuffer = Buffer.from(left);
|
|
const rightBuffer = Buffer.from(right);
|
|
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer);
|
|
}
|
|
|
|
function cloudTenantCompanyId(stackId: string): string {
|
|
return cloudTenantPrimaryCompanyId(stackId);
|
|
}
|
|
|
|
export function humanizeCloudStackSlug(stackId: string): string {
|
|
const slug = stackId
|
|
.trim()
|
|
.replace(/^paperclip-stack-/i, "")
|
|
.replace(/^stack-/i, "");
|
|
const displayName = slug
|
|
.split(/[-_]+/)
|
|
.filter(Boolean)
|
|
.map((part) => `${part[0]?.toUpperCase() ?? ""}${part.slice(1)}`)
|
|
.join(" ");
|
|
return displayName || "Workspace";
|
|
}
|
|
|
|
export function isKnownBadCloudCompanyName(
|
|
name: string,
|
|
ids: { companyId: string; paperclipCompanyId?: string },
|
|
): boolean {
|
|
const normalized = name.trim();
|
|
return (
|
|
/^paperclip-stack-.+/i.test(normalized) ||
|
|
/^stack-.+\s+paperclip$/i.test(normalized) ||
|
|
normalized === ids.companyId ||
|
|
(ids.paperclipCompanyId !== undefined &&
|
|
normalized === ids.paperclipCompanyId)
|
|
);
|
|
}
|
|
|
|
async function repairCloudTenantCompanyName(
|
|
db: Db,
|
|
input: {
|
|
companyId: string;
|
|
paperclipCompanyId?: string;
|
|
paperclipCompanyName: string;
|
|
now: Date;
|
|
},
|
|
): Promise<void> {
|
|
try {
|
|
const existing = await db
|
|
.select({ name: companies.name })
|
|
.from(companies)
|
|
.where(eq(companies.id, input.companyId))
|
|
.then((rows) => rows[0]);
|
|
if (
|
|
!existing ||
|
|
!isKnownBadCloudCompanyName(existing.name, {
|
|
companyId: input.companyId,
|
|
paperclipCompanyId: input.paperclipCompanyId,
|
|
})
|
|
) {
|
|
return;
|
|
}
|
|
await db.transaction(async (tx) => {
|
|
const [updated] = await tx
|
|
.update(companies)
|
|
.set({ name: input.paperclipCompanyName, updatedAt: input.now })
|
|
.where(
|
|
and(
|
|
eq(companies.id, input.companyId),
|
|
// A user may rename the company between the read above and this
|
|
// repair. Match the exact observed machine name so that concurrent
|
|
// genuine renames always win.
|
|
eq(companies.name, existing.name),
|
|
),
|
|
)
|
|
.returning({ id: companies.id });
|
|
if (!updated) return;
|
|
|
|
await tx.insert(activityLog).values({
|
|
companyId: input.companyId,
|
|
actorType: "system",
|
|
actorId: "cloud-tenant-auth",
|
|
action: "company.updated",
|
|
entityType: "company",
|
|
entityId: input.companyId,
|
|
details: {
|
|
source: "cloud_tenant_auth",
|
|
reason: "legacy_machine_name_repair",
|
|
previousName: existing.name,
|
|
name: input.paperclipCompanyName,
|
|
},
|
|
});
|
|
});
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, companyId: input.companyId },
|
|
"Failed to repair legacy Cloud tenant company name",
|
|
);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Claims the tenant company row for this stack.
|
|
*
|
|
* The prefix derives from the company name, exactly as it does for a
|
|
* self-hosted company. Each attempt is a standalone INSERT, so a failed
|
|
* attempt is its own implicit transaction and cannot poison a surrounding
|
|
* one. `onConflictDoNothing` only absorbs the `companies.id` conflict — a
|
|
* prefix that another company already holds still raises `23505`, so the loop
|
|
* moves on to the next suffix.
|
|
*/
|
|
async function insertCloudTenantCompany(
|
|
db: Db,
|
|
input: { companyId: string; companyName: string; now: Date },
|
|
): Promise<void> {
|
|
const base = deriveIssuePrefixBase(input.companyName);
|
|
for (let attempt = 1; attempt <= MAX_ISSUE_PREFIX_ATTEMPTS; attempt += 1) {
|
|
try {
|
|
await db
|
|
.insert(companies)
|
|
.values({
|
|
id: input.companyId,
|
|
name: input.companyName,
|
|
description: null,
|
|
status: "active",
|
|
issuePrefix: `${base}${issuePrefixSuffixForAttempt(attempt)}`,
|
|
updatedAt: input.now,
|
|
})
|
|
.onConflictDoNothing({
|
|
target: companies.id,
|
|
});
|
|
return;
|
|
} catch (error) {
|
|
if (!isIssuePrefixConflict(error)) throw error;
|
|
}
|
|
}
|
|
throw new Error("Unable to allocate a unique issue prefix for the tenant company");
|
|
}
|
|
|
|
/**
|
|
* The issue prefix that pre-name-derivation builds gave a tenant company.
|
|
*
|
|
* This derivation survives only as the detector for the one-time repair
|
|
* below. Nothing mints a prefix this way any more.
|
|
*/
|
|
function legacyProvisionedIssuePrefix(stackId: string): string {
|
|
const hash = createHash("sha256").update(stackId).digest("hex").slice(0, 4).toUpperCase();
|
|
return `PC${hash}`;
|
|
}
|
|
|
|
/** The placeholder description that pre-name-derivation builds wrote. */
|
|
const LEGACY_PROVISIONED_DESCRIPTION_PREFIX = "Provisioned by Paperclip Cloud for stack ";
|
|
|
|
/**
|
|
* One-time repair for companies claimed by a pre-name-derivation build.
|
|
*
|
|
* Those companies carry an opaque hash prefix and a placeholder description
|
|
* that the operator never chose. Re-derive the prefix from the company's
|
|
* current name, re-key the stored issue and case identifiers onto it, and drop
|
|
* the placeholder. Both guards stop matching once the repair lands, so a later
|
|
* pass is a no-op.
|
|
*/
|
|
async function repairCloudTenantCompanyProvisionDefaults(
|
|
db: Db,
|
|
input: { companyId: string; stackId: string; now: Date },
|
|
): Promise<void> {
|
|
try {
|
|
const existing = await db
|
|
.select({
|
|
name: companies.name,
|
|
issuePrefix: companies.issuePrefix,
|
|
description: companies.description,
|
|
})
|
|
.from(companies)
|
|
.where(eq(companies.id, input.companyId))
|
|
.then((rows) => rows[0]);
|
|
if (!existing) return;
|
|
|
|
const legacyPrefix = legacyProvisionedIssuePrefix(input.stackId);
|
|
const legacyDescription = existing.description?.startsWith(LEGACY_PROVISIONED_DESCRIPTION_PREFIX)
|
|
? existing.description
|
|
: null;
|
|
if (existing.issuePrefix !== legacyPrefix) {
|
|
if (!legacyDescription) return;
|
|
// The prefix was already re-derived, so only the placeholder is left.
|
|
await db
|
|
.update(companies)
|
|
.set({ description: null, updatedAt: input.now })
|
|
.where(and(
|
|
eq(companies.id, input.companyId),
|
|
eq(companies.description, legacyDescription),
|
|
));
|
|
return;
|
|
}
|
|
|
|
await db.transaction(async (tx) => {
|
|
const candidate = await pickAvailableIssuePrefix(tx, deriveIssuePrefixBase(existing.name));
|
|
if (!candidate || candidate === legacyPrefix) return;
|
|
|
|
const [updated] = await tx
|
|
.update(companies)
|
|
.set({
|
|
issuePrefix: candidate,
|
|
...(legacyDescription ? { description: null } : {}),
|
|
updatedAt: input.now,
|
|
})
|
|
.where(and(
|
|
eq(companies.id, input.companyId),
|
|
// A user may rename the company between the read above and this
|
|
// repair. Match the exact observed legacy prefix so that concurrent
|
|
// genuine renames always win.
|
|
eq(companies.issuePrefix, legacyPrefix),
|
|
))
|
|
.returning({ id: companies.id });
|
|
if (!updated) return;
|
|
|
|
const rekeyed = await rekeyCompanyIssueIdentifiers(tx, {
|
|
companyId: input.companyId,
|
|
fromPrefix: legacyPrefix,
|
|
toPrefix: candidate,
|
|
});
|
|
|
|
await tx.insert(activityLog).values({
|
|
companyId: input.companyId,
|
|
actorType: "system",
|
|
actorId: "cloud-tenant-auth",
|
|
action: "company.updated",
|
|
entityType: "company",
|
|
entityId: input.companyId,
|
|
details: {
|
|
source: "cloud_tenant_auth",
|
|
reason: "legacy_provision_defaults_repair",
|
|
previousIssuePrefix: legacyPrefix,
|
|
issuePrefix: candidate,
|
|
descriptionCleared: legacyDescription !== null,
|
|
issuesRekeyed: rekeyed.issues,
|
|
casesRekeyed: rekeyed.cases,
|
|
},
|
|
});
|
|
});
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, companyId: input.companyId },
|
|
"Failed to repair legacy tenant company provisioning defaults",
|
|
);
|
|
}
|
|
}
|
|
|
|
export function requireBoard(req: Express.Request) {
|
|
return req.actor.type === "board";
|
|
}
|