mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use outside services through connections. The control plane holds each connection's credentials and makes the calls. > - Some clouds, such as AWS, do not need stored keys. They trust a signed token from an identity provider that they can verify against its public keys. > - Paperclip has no signing key whose public half it can publish. Agent identity keys are Ed25519, which AWS does not accept, and the agent's run receives the private key. Agent run tokens use a shared secret. > - This pull request gives each company its own signing key and publishes the public half as a standard OIDC issuer. > - The benefit is that a connection, such as the planned AWS connector, can get its own short-lived cloud credentials with no stored cloud keys. The connection's role in the customer's account sets its permissions. ## Linked Issues or Issue Description Refs #19414 and #10411. Those pull requests add OIDC *sign-in* to Paperclip. This pull request is the other direction: Paperclip *issues* tokens that outside clouds verify. **Subsystem affected** Connections (tool access) and the database. **Problem or motivation** An AWS connector should not store a long-lived AWS access key. AWS supports OIDC federation (`AssumeRoleWithWebIdentity`). AWS fetches the issuer's public keys over HTTPS and checks each token. Paperclip has no key that it can publish for this, so this method is not available. **Proposed solution** Each company gets an ES256 key that Paperclip creates on first use. The vault master key encrypts the private half. The issuer is `https://{instance}/oidc/companies/{companyId}`. Paperclip publishes a discovery document and a JWKS there. Tokens last 5 minutes and have `sub = connection:{id}`. Only the control plane signs them. The board can see key state and run planned rotation or emergency rotation. The design is in `doc/plans/2026-11-17-workload-identity-issuer.md`. The AWS connector plan that uses it is in `doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`. **Alternatives considered** - Reuse agent identity keys. Rejected: AWS does not accept Ed25519, and the agent's run receives the private key. - Derive the key from the master key, with no table. Rejected: the master key alone would then sign as production. Worktree environments copy the master key on purpose. A master key rotation would also change the signing key and break every customer trust. - Use one issuer for the instance. Rejected: if a customer's role checks the issuer but not `sub`, any company on a shared instance could assume that role. ## What Changed - `packages/db`: new table `company_workload_identity_keys` and migration `0498`. The migration only creates a new table, so the previous release keeps working. Partial unique indexes allow one `active` key and one `next` key per company. - `server/src/services/workload-identity.ts`: lazy key creation with concurrent convergence, a private-key cache with a fail-closed key match check, the discovery document, the JWKS, `signConnectionToken`, and rotation steps. A key that is no longer active stays published for 24 hours. Paperclip calculates this when it reads the keys, so no background job is necessary. - `server/src/routes/workload-identity.ts`: public `GET /oidc/companies/{companyId}/.well-known/openid-configuration` and `GET /oidc/companies/{companyId}/jwks.json` (404 until a key exists). Board `GET /api/companies/{companyId}/workload-identity` and `POST .../workload-identity/rotation`, with an activity log entry for each rotation. - Token issuance writes a tool-access audit event. The event never contains the token. - `cli`: worktree seeds exclude the new table in the minimal mode and the full mode. This is the same as for `agent_identity_keys`, so a development copy never holds production's signing key. - Docs: `docs/api/workload-identity.md`, an entry in the API overview, and the two plan docs. - No connector uses `signConnectionToken` yet. The AWS connector pull request will use it. ## Verification - `cd server && npx vitest run src/__tests__/workload-identity.test.ts`: 8 tests pass. The tests cover these items: - The issuer requires https with no port and no loopback host. - The endpoints return 404 before a key exists. - Six concurrent first uses create one key. - The JWKS contains only `kty crv x y kid alg use`. - A token verifies against the published key and has the expected claims. - A company cannot sign for another company's connection, and its keys do not verify that company's tokens. - Signing stops when the public address is not usable or has changed. - Two-step rotation and emergency rotation work. - Board-only access is enforced, and each rotation writes an activity log entry. - Red/green: I put back a JWK that leaks `d` and removed the company check on the connection. Three tests failed. I restored the code, and all 8 tests passed. - `cd cli && npx vitest run src/__tests__/worktree.test.ts`: 63 tests pass. - `npx tsc --noEmit` passes for `packages/db`, `server` and `cli`. - I started this branch on a real dev server. Before a key existed, the public routes returned JSON 404 (not the UI page). After an emergency rotation from the board route, the JWKS returned one public P-256 key. On the http dev instance, discovery stayed 404 and the board route reported `not_https`. - `pnpm verify` exited 2 because of the migration. The required CI gates supply the broad coverage. - Not run: a real AWS `AssumeRoleWithWebIdentity` call. That needs a public HTTPS instance. It is the first step of the AWS connector work and will run against staging. ## Risks - The migration is low risk. It creates a new table with no backfill, and the compatibility checker classifies it as compatible with the running release. - The public routes are unauthenticated. They show only public keys, return 404 for companies without a key, and use random company UUIDs. Each request is one small indexed read. - The issuer URL shape is permanent after customers trust it. A change to the instance's public address breaks those trusts until the customer updates the identity provider. `signConnectionToken` stops on that mismatch when the caller supplies the recorded issuer. - Anyone who holds the production database and the master key could sign for that instance's companies. That is the same exposure as every other vault secret. - Edge routing on Paperclip Cloud must pass `/oidc/...` to the stack. I will confirm this on staging with the AWS connector work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, web fetch of AWS documentation, a local dev server). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
1537 lines
72 KiB
TypeScript
1537 lines
72 KiB
TypeScript
import { authorizationReadScope } from "./services/authorization-read-scope.js";
|
|
import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js";
|
|
import { startMcpClientMirrorSync } from "./services/public-mcp/client-mirror-sync.js";
|
|
import { registerIdleMcpOwners } from "./services/public-mcp/idle-sleep-proof.js";
|
|
import { startMcpRevocationIndexSync } from "./services/public-mcp/revocation-index-sync.js";
|
|
import { startMcpIngressHandoffExpiry } from "./services/public-mcp/ingress-handoff-expiry.js";
|
|
import { runnerAttachmentDownloadRoutes } from "./services/native-runtime/runner-attachment-download.js";
|
|
import { nativeRunMcpRoutes } from "./routes/native-run-mcp.js";
|
|
import { connectionEventWakeRoutes } from "./routes/connection-event-wake.js";
|
|
import { registerGitHubReviewProjectionWork } from "./services/chat-channels/github/review-projection-work.js";
|
|
import { finishServiceShutdown } from "./shutdown.js";
|
|
import { promptInjectionPreventionRoutes } from "./routes/prompt-injection-prevention.js";
|
|
import { voiceSessionRoutes, voiceWebhookRoutes } from "./routes/voice-sessions.js";
|
|
import { eq } from "drizzle-orm";
|
|
import { chatEndpoints, dedicatedDbConnectionStats } from "@paperclipai/db";
|
|
import { fastResponseRoutes } from "./routes/fast-responses.js";
|
|
import { fastResponseService } from "./services/fast-responses.js";
|
|
import { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js";
|
|
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
|
|
import { createLifecycleDriver } from "./services/agent-lifecycle-driver.js";
|
|
import { startAgentLifecycle } from "./services/agent-lifecycle.js";
|
|
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
|
|
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
|
|
import { staticUiFiles } from "./middleware/static-ui.js";
|
|
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
|
|
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
|
|
import { loadStorageConfig } from "./storage/config.js";
|
|
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
|
|
import { customerSuccessRoutes } from "./routes/customer-success.js";
|
|
import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js";
|
|
import type { CloudWarmStandby } from "./services/cloud-warm-standby.js";
|
|
import { browserUseRoutes } from "./routes/browser-use.js";
|
|
import { mcpAppsRoutes } from "./routes/mcp-apps.js";
|
|
import { mcpSkillsRoutes } from "./routes/mcp-skills.js";
|
|
import { registerChatActionWork } from "./services/chat-action-work.js";
|
|
import { registerChatDeliveryWork } from "./services/chat-delivery-work.js";
|
|
import { registerBrowserUseCleanup } from "./services/browser-use-work.js";
|
|
import { browserUseService } from "./services/browser-use.js";
|
|
import { slackToolRoutes } from "./routes/slack-tools.js";
|
|
import { createPublicMcpOAuth, publicMcpConfig } from "./services/public-mcp/oauth.js";
|
|
import { createDotPrincipalHttp } from "./modules/external-agents/index.js";
|
|
import { createDotRunnerMcpTools, dotRunnerBroker } from "./services/dot-runner-broker.js";
|
|
import { agentProfileAvatarRoutes } from "./routes/agent-profile-avatar.js";
|
|
import { museTransportRoutes, museBoardRoutes } from "./routes/muse.js";
|
|
import { museQualificationRoutes } from "./routes/muse-qualification.js";
|
|
import { startMuseMaintenance } from "./services/muse-maintenance.js";
|
|
import { grokBotRoutes } from "./routes/grok-bot.js";
|
|
import { createGrokRoutineDispatcher } from "./services/grok-bot/dispatcher.js";
|
|
import { dotRunnerRoutes } from "./routes/dot-runner.js";
|
|
import { createPublicMcpTransfers } from "./services/public-mcp/file-transfers.js";
|
|
import { createMcpApiDispatch, createPublicMcpExecutor } from "./services/public-mcp/capabilities.js";
|
|
import { createPublicMcpEvents, type PublicMcpEvents } from "./services/public-mcp/events.js";
|
|
import { publicMcpIngressRoutes, publicMcpManagementRoutes } from "./routes/public-mcp.js";
|
|
import { agentAvatarRoutes } from "./routes/agent-avatars.js";
|
|
import { decisionModelRoutes } from "./routes/decision-models.js";
|
|
import { agentObservabilityRoutes } from "./routes/agent-observability.js";
|
|
import { aiConnectionRoutes } from "./routes/ai-connections.js";
|
|
import { taskWorkspaceRoutes } from "./routes/task-workspaces.js";
|
|
import { projectToolRoutes } from "./routes/project-tools.js";
|
|
import { emailChannelService } from "./services/email-channels.js";
|
|
import { emailRoutes, emailWebhookRoutes } from "./routes/email.js";
|
|
import { toolActionDeliveryService } from "./services/tool-action-delivery.js";
|
|
import { registerAssignedMcpGateway } from "./services/native-runtime/assigned-mcp-tools.js";
|
|
import express, { Router, type Request as ExpressRequest } from "express";
|
|
import {
|
|
createServer as createHttpServer,
|
|
type Server as HttpServer,
|
|
} from "node:http";
|
|
import path from "node:path";
|
|
import fs from "node:fs";
|
|
import { fileURLToPath } from "node:url";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
derivePaperclipViteHmrPort,
|
|
type DeploymentExposure,
|
|
type DeploymentMode,
|
|
} from "@paperclipai/shared";
|
|
import type { InspectDatabaseBackupHealthOptions } from "./services/database-backup-health.js";
|
|
import type { StorageService } from "./storage/types.js";
|
|
import { httpLogger, errorHandler } from "./middleware/index.js";
|
|
import { skillAccessRequestContext } from "./services/skill-access.js";
|
|
import { actorMiddleware } from "./middleware/auth.js";
|
|
import { boardMutationGuard } from "./middleware/board-mutation-guard.js";
|
|
import {
|
|
privateHostnameGuard,
|
|
resolvePrivateHostnameAllowSet,
|
|
} from "./middleware/private-hostname-guard.js";
|
|
import {
|
|
applyTrustProxy,
|
|
parseTrustProxyEnv,
|
|
} from "./middleware/trust-proxy.js";
|
|
import {
|
|
resolveDefaultImportTransferSpoolRoot,
|
|
} from "./services/company-import-transfers.js";
|
|
import { registerImportSpoolCleanupWork } from "./services/import-spool-cleanup-work.js";
|
|
import { companyTransferRunService } from "./services/company-transfer-runs.js";
|
|
import { healthRoutes } from "./routes/health.js";
|
|
import { readinessRoutes } from "./routes/ready.js";
|
|
import { livenessRoutes } from "./routes/liveness.js";
|
|
import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js";
|
|
import { cloudControlMiddleware } from "./middleware/cloud-control.js";
|
|
import { runnerCallOutcomeMiddleware } from "./middleware/runner-call-outcome.js";
|
|
import { cloudRoutes } from "./routes/cloud.js";
|
|
import { companyRoutes } from "./routes/companies.js";
|
|
import { companySkillRoutes } from "./routes/company-skills.js";
|
|
import { companySkillPolicyRoutes } from "./routes/company-skill-policy.js";
|
|
import { inboxAgentPolicyRoutes } from "./routes/inbox-agent-policy.js";
|
|
import { builtInAgentRoutes } from "./routes/built-in-agents.js";
|
|
import { folderRoutes } from "./routes/folders.js";
|
|
import { summarySlotRoutes } from "./routes/summary-slots.js";
|
|
import { statusCardRoutes } from "./routes/status-cards.js";
|
|
import { teamsCatalogRoutes } from "./routes/teams-catalog.js";
|
|
import { agentRoutes } from "./routes/agents.js";
|
|
import { agentInteractionPermissionRoutes } from "./routes/agent-interaction-permissions.js";
|
|
import { agentPermissionGrantRoutes } from "./routes/agent-permission-grants.js";
|
|
import { resolveAgentRunnerTargetForCompany } from "./services/agent-runner-selection.js";
|
|
import type { SetupTokenSessionService } from "./services/setup-token-session.js";
|
|
import {
|
|
buildSetupTokenLoginTransport,
|
|
createProductionSetupTokenSandboxProvider,
|
|
createProductionSetupTokenCleanupStore,
|
|
createSetupTokenSecretWriter,
|
|
createWorkerBoundLoginPtyOpener,
|
|
} from "./services/setup-token-transport-binding.js";
|
|
import { environmentService } from "./services/environments.js";
|
|
import { environmentRuntimeService } from "./services/environment-runtime.js";
|
|
import { projectRoutes } from "./routes/projects.js";
|
|
import { issueRoutes } from "./routes/issues.js";
|
|
import { linkPreviewRoutes } from "./routes/link-previews.js";
|
|
import { issueTreeControlRoutes } from "./routes/issue-tree-control.js";
|
|
import { caseRoutes } from "./routes/cases.js";
|
|
import { caseHandoffRoutes } from "./routes/case-handoffs.js";
|
|
import { workflowRoutes } from "./routes/workflows.js";
|
|
import { fileResourceRoutes } from "./routes/file-resources.js";
|
|
import { createRemoteWorkspaceFiles } from "./services/remote-workspace-files.js";
|
|
import { createSavedWorkspaceLoader } from "./services/remote-workspaces.js";
|
|
import { createSavedWorkspaceArchiveOpener } from "./services/saved-workspace-archive.js";
|
|
import { routineRoutes } from "./routes/routines.js";
|
|
import { routineSuggestionRoutes } from "./routes/routine-suggestions.js";
|
|
import { environmentRoutes } from "./routes/environments.js";
|
|
import { tailscaleConnectionRoutes } from "./routes/tailscale-connections.js";
|
|
import { workloadIdentityPublicRoutes, workloadIdentityRoutes } from "./routes/workload-identity.js";
|
|
import { setWorkloadIdentityConfiguredPublicOrigin } from "./services/workload-identity.js";
|
|
import { executionWorkspaceRoutes } from "./routes/execution-workspaces.js";
|
|
import { goalRoutes } from "./routes/goals.js";
|
|
import { responsibilityRoutes } from "./routes/responsibilities.js";
|
|
import { responsibilityAreaRoutes } from "./routes/responsibility-areas.js";
|
|
import { responsibilityBarRoutes } from "./routes/responsibility-bars.js";
|
|
import { responsibilityDashboardRoutes } from "./routes/responsibility-dashboard.js";
|
|
import { responsibilitySummaryRoutes } from "./routes/responsibility-summaries.js";
|
|
import { responsibilityAutomationRoutes } from "./routes/responsibility-automation.js";
|
|
import { responsibilityStatusReportRoutes } from "./routes/responsibility-status-reports.js";
|
|
import { responsibilityMetricRoutes } from "./routes/responsibility-metrics.js";
|
|
import { responsibilityRunRoutes } from "./routes/responsibility-runs.js";
|
|
import { onboardingSeedRoutes } from "./routes/onboarding-seed.js";
|
|
import { onboardingChatRoutes } from "./routes/onboarding-chat.js";
|
|
import { boardChatRoutes } from "./routes/board-chat.js";
|
|
import { approvalRoutes } from "./routes/approvals.js";
|
|
import { secretRoutes } from "./routes/secrets.js";
|
|
import { toolAccessRoutes } from "./routes/tool-access.js";
|
|
import {
|
|
chatChannelRoutes,
|
|
chatWebhookRoutes,
|
|
} from "./routes/chat-channels.js";
|
|
import { smokeLabRoutes } from "./routes/smoke-lab.js";
|
|
import { costRoutes } from "./routes/costs.js";
|
|
import { agentCommentaryRoutes } from "./routes/agent-commentary.js";
|
|
import { agentChatMessageRoutes } from "./routes/agent-chat-messages.js";
|
|
import { activityRoutes } from "./routes/activity.js";
|
|
import { dashboardRoutes } from "./routes/dashboard.js";
|
|
import { attentionRoutes } from "./routes/attention.js";
|
|
import { decisionTrainingRoutes } from "./routes/decision-training.js";
|
|
import { decisionRoutes } from "./routes/decisions.js";
|
|
import { decisionQueueRoutes } from "./routes/decision-queues.js";
|
|
import type { DecisionServiceOptions } from "./services/decisions.js";
|
|
import { userProfileRoutes } from "./routes/user-profiles.js";
|
|
import { sidebarBadgeRoutes } from "./routes/sidebar-badges.js";
|
|
import { sidebarPreferenceRoutes } from "./routes/sidebar-preferences.js";
|
|
import { announcementRoutes } from "./routes/announcements.js";
|
|
import { serverVersion } from "./version.js";
|
|
import { workspacePreferenceRoutes } from "./routes/workspace-preferences.js";
|
|
import { primaryAgentRoutes } from "./routes/primary-agent.js";
|
|
import { resourceMembershipRoutes } from "./routes/resource-memberships.js";
|
|
import { inboxDismissalRoutes } from "./routes/inbox-dismissals.js";
|
|
import { instanceSettingsRoutes } from "./routes/instance-settings.js";
|
|
import { ephemeralEnrollmentRoutes } from "./routes/ephemeral-enrollment.js";
|
|
import { instanceSettingsService } from "./services/instance-settings.js";
|
|
import { openApiRoutes } from "./routes/openapi.js";
|
|
import {
|
|
instanceDatabaseBackupRoutes,
|
|
type InstanceDatabaseBackupService,
|
|
} from "./routes/instance-database-backups.js";
|
|
import { llmRoutes } from "./routes/llms.js";
|
|
import { authRoutes } from "./routes/auth.js";
|
|
import { resolvePostHogProxyMount } from "./routes/posthog-proxy.js";
|
|
import { assetRoutes } from "./routes/assets.js";
|
|
import { accessRoutes } from "./routes/access.js";
|
|
import { pluginRoutes } from "./routes/plugins.js";
|
|
import {
|
|
mcpGatewayProtocolRoutes,
|
|
toolGatewayRoutes,
|
|
} from "./routes/tool-gateway.js";
|
|
import {
|
|
connectionIntentBoardRoutes,
|
|
runtimeConnectionIntentRoutes,
|
|
} from "./routes/connection-intents.js";
|
|
import { adapterRoutes } from "./routes/adapters.js";
|
|
import { managedAgentProfileRoutes } from "./routes/managed-agent-profiles.js";
|
|
import { remoteAgentProfileRoutes } from "./routes/remote-agent-profiles.js";
|
|
import { pluginUiStaticRoutes } from "./routes/plugin-ui-static.js";
|
|
import { readBrandedStaticIndexHtml } from "./static-index-html.js";
|
|
import { staticUiCacheControl } from "./static-ui-cache.js";
|
|
import { staticUiCompression } from "./middleware/static-ui-compression.js";
|
|
import { applyUiBranding } from "./ui-branding.js";
|
|
import { logger } from "./middleware/logger.js";
|
|
import {
|
|
DEFAULT_LOCAL_PLUGIN_DIR,
|
|
pluginLoader,
|
|
type PluginLoader,
|
|
} from "./services/plugin-loader.js";
|
|
import {
|
|
SELF_HOSTED_AUTO_INSTALL_KEYS,
|
|
BUNDLED_PLUGIN_CATALOG,
|
|
ensureBundledPlugins,
|
|
resolveBundledCatalogRoot,
|
|
resolveBundledPluginInstalls,
|
|
resolveManagedBundledPluginKeys,
|
|
} from "./services/bundled-plugins.js";
|
|
import { createBundledPluginPackageResolver } from "./services/bundled-plugin-package.js";
|
|
import { readDistributionPluginCatalog, distributionPluginActivationGuard } from "./services/distribution-plugin-catalog.js";
|
|
import {
|
|
createPluginWorkerManager,
|
|
type PluginWorkerManager,
|
|
} from "./services/plugin-worker-manager.js";
|
|
import { createPluginJobScheduler } from "./services/plugin-job-scheduler.js";
|
|
import { pluginJobStore } from "./services/plugin-job-store.js";
|
|
import { createPluginToolDispatcher } from "./services/plugin-tool-dispatcher.js";
|
|
import { createToolGatewayService } from "./services/tool-gateway.js";
|
|
import { toolAccessService } from "./services/tool-access.js";
|
|
import { chatChannelService } from "./services/chat-channels.js";
|
|
import type { BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
|
|
import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js";
|
|
import { nativeChatProgressWork } from "./services/chat-native-progress-work.js";
|
|
import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-work.js";
|
|
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
|
|
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
|
|
import { heartbeatService } from "./services/heartbeat.js";
|
|
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
|
|
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
|
|
import { loadPluginAppCatalog } from "./services/plugin-app-catalog.js";
|
|
import { setPluginConnectorWorkers } from "./services/plugin-connector-runtime.js";
|
|
import { completeLegacyPluginToolAccess } from "./services/plugin-connector-legacy-access.js";
|
|
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
|
|
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
|
|
import {
|
|
buildHostServices,
|
|
flushPluginLogBuffer,
|
|
} from "./services/plugin-host-services.js";
|
|
import { createPluginEventBus } from "./services/plugin-event-bus.js";
|
|
import { setPluginEventBus } from "./services/activity-log.js";
|
|
import { createPluginDevWatcher } from "./services/plugin-dev-watcher.js";
|
|
import { createPluginHostServiceCleanup } from "./services/plugin-host-service-cleanup.js";
|
|
import { pluginRegistryService } from "./services/plugin-registry.js";
|
|
import { createHostClientHandlers } from "@paperclipai/plugin-sdk";
|
|
import type { BetterAuthSessionResult } from "./auth/better-auth.js";
|
|
import { createCachedViteHtmlRenderer } from "./vite-html-renderer.js";
|
|
import {
|
|
DEFAULT_JSON_BODY_LIMIT,
|
|
PORTABLE_JSON_BODY_LIMIT,
|
|
} from "./http/body-limits.js";
|
|
import { COMPANY_IMPORT_API_PATH } from "./routes/company-import-paths.js";
|
|
import { apiCompression } from "./middleware/api-compression.js";
|
|
import { apiServerTiming } from "./middleware/api-server-timing.js";
|
|
import { instanceServerMetricsRoutes } from "./routes/instance-server-metrics.js";
|
|
import {
|
|
serverMetricsAgentPreHandler,
|
|
serverMetricsArrival,
|
|
serverMetricsEnabledFromEnv,
|
|
startServerMetrics,
|
|
stopServerMetrics,
|
|
} from "./services/server-metrics.js";
|
|
import { chatWebhookBodyParser } from "./middleware/chat-webhook-body.js";
|
|
import { postHogRequestContextMiddlewareForEnv } from "./services/posthog-direct/request-context.js";
|
|
import { createChatWebhookDiagnostics } from "./services/chat-webhook-diagnostics.js";
|
|
|
|
type UiMode = "none" | "static" | "vite-dev";
|
|
const FEEDBACK_EXPORT_FLUSH_INTERVAL_MS = 5_000;
|
|
const VITE_DEV_ASSET_PREFIXES = [
|
|
"/@fs/",
|
|
"/@id/",
|
|
"/@react-refresh",
|
|
"/@vite/",
|
|
"/assets/",
|
|
"/node_modules/",
|
|
"/src/",
|
|
];
|
|
const VITE_DEV_STATIC_PATHS = new Set([
|
|
"/apple-touch-icon.png",
|
|
"/favicon-16x16.png",
|
|
"/favicon-32x32.png",
|
|
"/favicon.ico",
|
|
"/favicon.svg",
|
|
"/site.webmanifest",
|
|
"/sw.js",
|
|
]);
|
|
|
|
export function isDatabaseConnectionUnavailableError(err: unknown): boolean {
|
|
const error = err as { code?: unknown; message?: unknown; cause?: unknown };
|
|
if (error?.code === "ECONNREFUSED") return true;
|
|
return Boolean(
|
|
error?.cause && isDatabaseConnectionUnavailableError(error.cause),
|
|
);
|
|
}
|
|
|
|
export function resolveViteHmrPort(serverPort: number): number {
|
|
return derivePaperclipViteHmrPort(serverPort);
|
|
}
|
|
|
|
export function resolveViteHmrHost(bindHost: string): string | undefined {
|
|
const normalized = bindHost.trim().toLowerCase();
|
|
if (
|
|
normalized === "0.0.0.0" ||
|
|
normalized === "::" ||
|
|
normalized === "127.0.0.1" ||
|
|
normalized === "::1" ||
|
|
normalized === "localhost"
|
|
)
|
|
return undefined;
|
|
return bindHost;
|
|
}
|
|
|
|
export function resolveViteHmrProtocol(
|
|
value: string | undefined,
|
|
): "ws" | "wss" | undefined {
|
|
if (!value) return undefined;
|
|
if (value === "ws" || value === "wss") return value;
|
|
throw new Error("PAPERCLIP_VITE_HMR_PROTOCOL must be ws or wss");
|
|
}
|
|
|
|
export function listenViteHmrServer(
|
|
server: HttpServer,
|
|
port: number,
|
|
bindHost: string,
|
|
): Promise<void> {
|
|
return new Promise((resolve, reject) => {
|
|
const onError = (error: Error) => {
|
|
server.off("listening", onListening);
|
|
reject(error);
|
|
};
|
|
const onListening = () => {
|
|
server.off("error", onError);
|
|
resolve();
|
|
};
|
|
server.once("error", onError);
|
|
server.once("listening", onListening);
|
|
server.listen(port, bindHost);
|
|
});
|
|
}
|
|
|
|
export function shouldServeViteDevHtml(req: ExpressRequest): boolean {
|
|
const pathname = req.path;
|
|
if (VITE_DEV_STATIC_PATHS.has(pathname)) return false;
|
|
if (VITE_DEV_ASSET_PREFIXES.some((prefix) => pathname.startsWith(prefix)))
|
|
return false;
|
|
return req.accepts(["html"]) === "html";
|
|
}
|
|
|
|
export function shouldEnablePrivateHostnameGuard(opts: {
|
|
deploymentMode: DeploymentMode;
|
|
deploymentExposure: DeploymentExposure;
|
|
}): boolean {
|
|
return (
|
|
opts.deploymentExposure === "private" &&
|
|
(opts.deploymentMode === "local_trusted" ||
|
|
opts.deploymentMode === "authenticated")
|
|
);
|
|
}
|
|
|
|
export function createManagedBundledPluginWorkerRecovery(input: {
|
|
managedBundledPluginKeys: readonly string[];
|
|
workerManager: Pick<
|
|
PluginWorkerManager,
|
|
"getWorker" | "isRunning" | "stopWorker"
|
|
>;
|
|
getLoader: () => Pick<PluginLoader, "loadSingle"> | null;
|
|
}): (plugin: { id: string; pluginKey: string }) => Promise<boolean> {
|
|
const recoverablePluginKeys = new Set(input.managedBundledPluginKeys);
|
|
const inFlightStarts = new Map<string, Promise<boolean>>();
|
|
|
|
// A failed attempt can leave behind the dead handle it registered (e.g. the
|
|
// worker process died during initialize, which kills the process without
|
|
// scheduling a restart). No pre-existing handle survives to a recovery
|
|
// attempt — recovery only starts when getWorker() was empty — so discarding
|
|
// the dead handle lets a later capability request retry instead of being
|
|
// blocked by the handle-presence gate until the process restarts. Handles
|
|
// in starting/running/backoff states belong to the worker manager's own
|
|
// lifecycle and are left alone.
|
|
const discardDeadRecoveryHandle = async (plugin: {
|
|
id: string;
|
|
pluginKey: string;
|
|
}) => {
|
|
const handle = input.workerManager.getWorker(plugin.id);
|
|
if (!handle || (handle.status !== "crashed" && handle.status !== "stopped"))
|
|
return;
|
|
try {
|
|
await input.workerManager.stopWorker(plugin.id);
|
|
} catch (err) {
|
|
logger.warn(
|
|
{
|
|
pluginId: plugin.id,
|
|
pluginKey: plugin.pluginKey,
|
|
err: err instanceof Error ? err.message : String(err),
|
|
},
|
|
"failed to discard dead plugin worker handle after recovery failure",
|
|
);
|
|
}
|
|
};
|
|
|
|
return async (plugin) => {
|
|
if (!recoverablePluginKeys.has(plugin.pluginKey)) return false;
|
|
|
|
const inFlight = inFlightStarts.get(plugin.id);
|
|
if (inFlight) return inFlight;
|
|
|
|
const startPromise = (async () => {
|
|
if (input.workerManager.getWorker(plugin.id)) {
|
|
return input.workerManager.isRunning(plugin.id);
|
|
}
|
|
|
|
const loader = input.getLoader();
|
|
if (!loader) return false;
|
|
|
|
try {
|
|
const result = await loader.loadSingle(plugin.id, {
|
|
markErrorOnFailure: false,
|
|
});
|
|
if (
|
|
result.success === true ||
|
|
input.workerManager.isRunning(plugin.id)
|
|
) {
|
|
return true;
|
|
}
|
|
await discardDeadRecoveryHandle(plugin);
|
|
return false;
|
|
} catch (err) {
|
|
logger.warn(
|
|
{
|
|
pluginId: plugin.id,
|
|
pluginKey: plugin.pluginKey,
|
|
err: err instanceof Error ? err.message : String(err),
|
|
},
|
|
"managed bundled plugin lazy worker recovery failed",
|
|
);
|
|
await discardDeadRecoveryHandle(plugin);
|
|
throw err;
|
|
}
|
|
})();
|
|
|
|
inFlightStarts.set(plugin.id, startPromise);
|
|
try {
|
|
return await startPromise;
|
|
} finally {
|
|
if (inFlightStarts.get(plugin.id) === startPromise) {
|
|
inFlightStarts.delete(plugin.id);
|
|
}
|
|
}
|
|
};
|
|
}
|
|
|
|
export async function createApp(
|
|
db: Db,
|
|
opts: {
|
|
cloudWarmStandby?: CloudWarmStandby;
|
|
uiMode: UiMode;
|
|
serverPort: number;
|
|
storageService: StorageService;
|
|
feedbackExportService?: {
|
|
hasPendingFeedbackTraces(): Promise<boolean>;
|
|
flushPendingFeedbackTraces(input?: {
|
|
companyId?: string;
|
|
traceId?: string;
|
|
limit?: number;
|
|
now?: Date;
|
|
signal?: AbortSignal;
|
|
}): Promise<unknown>;
|
|
};
|
|
databaseBackupService?: InstanceDatabaseBackupService;
|
|
prepareIdleDatabaseBackup?: () => Promise<boolean>;
|
|
idleDeadlineOwners?: IdleDeadlineOwners;
|
|
/** Filled with the chat service's Retry callback for background budget-resume delivery. */
|
|
budgetResumeDelivery?: BudgetResumeDelivery;
|
|
databaseBackupHealth?: InspectDatabaseBackupHealthOptions;
|
|
deploymentMode: DeploymentMode;
|
|
deploymentExposure: DeploymentExposure;
|
|
allowedHostnames: string[];
|
|
bindHost: string;
|
|
authPublicBaseUrl?: string;
|
|
chatWebhookPublicBaseUrl?: string;
|
|
authReady: boolean;
|
|
companyDeletionEnabled: boolean;
|
|
announcements?: { enabled: boolean; feedUrl: string };
|
|
instanceId?: string;
|
|
hostVersion?: string;
|
|
localPluginDir?: string;
|
|
pluginMigrationDb?: Db;
|
|
pluginWorkerManager?: PluginWorkerManager;
|
|
decisionServiceOptions: DecisionServiceOptions;
|
|
betterAuthHandler?: express.RequestHandler;
|
|
resolveSession?: (
|
|
req: ExpressRequest,
|
|
) => Promise<BetterAuthSessionResult | null>;
|
|
/**
|
|
* `plugins.autoInstall` from the managed config (PAPERCLIP_MANAGED_CONFIG).
|
|
* `null`/absent ⇒ self-hosted: only the built-in kubernetes bundle is
|
|
* ensured, exactly as before. A managed list is resolved against the
|
|
* bundled catalog fail-to-start (see services/bundled-plugins.ts).
|
|
*/
|
|
managedPluginAutoInstall?: readonly string[] | null;
|
|
/** Test override for the bundled plugin catalog root. */
|
|
bundledPluginCatalogRoot?: string;
|
|
serverMetricsEnabled?: boolean;
|
|
},
|
|
) {
|
|
const app = express();
|
|
const serverMetrics = (opts.serverMetricsEnabled ?? serverMetricsEnabledFromEnv(process.env)) ? startServerMetrics(() => dedicatedDbConnectionStats(db)) : null;
|
|
if (serverMetrics) app.use("/api", serverMetricsArrival());
|
|
// First, so the reported server time includes admission and database waits.
|
|
app.use("/api", apiServerTiming());
|
|
app.use(databaseOperationMiddleware);
|
|
app.use(idleAdmissionMiddleware);
|
|
const staticUi = express.Router();
|
|
app.locals.paperclipDb = db;
|
|
const isWarmStandby = opts.cloudWarmStandby ?? (() => false);
|
|
const health = healthRoutes(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
authReady: opts.authReady,
|
|
companyDeletionEnabled: opts.companyDeletionEnabled,
|
|
databaseBackupHealth: opts.databaseBackupHealth,
|
|
isWarmStandby,
|
|
});
|
|
const captureRawBody = (
|
|
req: express.Request,
|
|
_res: express.Response,
|
|
buf: Buffer,
|
|
) => {
|
|
(req as unknown as { rawBody: Buffer }).rawBody = buf;
|
|
};
|
|
|
|
// Respect the operator's `TRUST_PROXY` env var (see middleware/trust-proxy.ts).
|
|
// Default is unset → Express trusts nothing, which is the only safe choice
|
|
// when the server may be reachable without a known reverse proxy in front.
|
|
applyTrustProxy(app, parseTrustProxyEnv(process.env.TRUST_PROXY));
|
|
// Browser analytics on Paperclip Cloud only. Mounted before every body
|
|
// parser so PostHog payloads stream through unread; see routes/posthog-proxy.ts.
|
|
const posthogProxy = resolvePostHogProxyMount(process.env);
|
|
if (posthogProxy) app.use(posthogProxy.path, posthogProxy.router);
|
|
|
|
app.use(
|
|
COMPANY_IMPORT_API_PATH,
|
|
express.json({
|
|
limit: PORTABLE_JSON_BODY_LIMIT,
|
|
verify: captureRawBody,
|
|
}),
|
|
);
|
|
// Chat providers sign the exact request bytes. Capture every webhook media
|
|
// type before the global JSON parser so JSON events and form-encoded action
|
|
// callbacks are verified against the provider's original body.
|
|
app.use(
|
|
"/api/chat-webhooks",
|
|
createChatWebhookDiagnostics(),
|
|
chatWebhookBodyParser,
|
|
);
|
|
let dotPrincipalOAuth: ReturnType<typeof createPublicMcpOAuth> | null = null;
|
|
const dotPrincipalHttp = createDotPrincipalHttp(() => dotPrincipalOAuth);
|
|
app.use(dotPrincipalHttp.bodyParser);
|
|
const jsonBodyParser = express.json({ limit: DEFAULT_JSON_BODY_LIMIT, verify: captureRawBody });
|
|
// File tickets authenticate before parsing bytes; JSON attachments must remain bytes too.
|
|
app.use((req, res, next) => (dotPrincipalHttp.matchesPost(req) || ["/mcp/files/upload", "/mcp/runner/files/upload"].includes(req.path)) ? next() : jsonBodyParser(req, res, next));
|
|
app.use("/api", apiCompression());
|
|
app.use(httpLogger);
|
|
// Paperclip Cloud only: link server analytics events to the browser session
|
|
// that caused them. Self-hosted servers never mount this, so posthog-js
|
|
// tracing headers are ignored there.
|
|
const postHogRequestContext = postHogRequestContextMiddlewareForEnv();
|
|
if (postHogRequestContext) app.use(postHogRequestContext);
|
|
const privateHostnameGateEnabled = shouldEnablePrivateHostnameGuard({
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
});
|
|
const privateHostnameAllowSet = resolvePrivateHostnameAllowSet({
|
|
allowedHostnames: opts.allowedHostnames,
|
|
bindHost: opts.bindHost,
|
|
});
|
|
app.use(
|
|
privateHostnameGuard({
|
|
enabled: privateHostnameGateEnabled,
|
|
allowedHostnames: opts.allowedHostnames,
|
|
bindHost: opts.bindHost,
|
|
}),
|
|
);
|
|
let mcpConfig: ReturnType<typeof publicMcpConfig> = null;
|
|
try { mcpConfig = publicMcpConfig(process.env, opts.authPublicBaseUrl); }
|
|
catch { logger.warn("Assistant connections require an HTTPS public URL (HTTP loopback is allowed for development)."); }
|
|
const publicMcpOAuth = mcpConfig ? createPublicMcpOAuth(db, mcpConfig) : null;
|
|
const publicMcpIngress = Router();
|
|
publicMcpIngress.use(dotPrincipalHttp.router);
|
|
|
|
app.use(cloudRuntimeIdentityMiddleware(db));
|
|
// Routine probes stop before session/actor resolution, even after claim.
|
|
// Keep hostname validation and the signed-bootstrap endpoint guard above.
|
|
app.use("/api/liveness", livenessRoutes());
|
|
app.use("/api/ready", readinessRoutes());
|
|
// A signed claim above commits identity before any normal request can seed
|
|
// company data. Unclaimed probes bypass session resolution as well as SQL.
|
|
app.use(cloudWarmStandbyMiddleware(isWarmStandby, health, staticUi));
|
|
const workerManager = opts.pluginWorkerManager ?? createPluginWorkerManager();
|
|
app.use("/api/customer-success/v1", customerSuccessRoutes(db, opts.storageService, {
|
|
pluginWorkerManager: workerManager,
|
|
}));
|
|
app.use(museTransportRoutes(db, opts.authPublicBaseUrl));
|
|
app.use(publicMcpIngress);
|
|
// Outside clouds fetch these anonymously to verify connection identity tokens.
|
|
setWorkloadIdentityConfiguredPublicOrigin(opts.authPublicBaseUrl);
|
|
app.use(workloadIdentityPublicRoutes(db));
|
|
// Connection-intent tools carry their own short-lived, run-bound bearer and
|
|
// must be reachable by remote adapters that intentionally do not receive an
|
|
// agent API key. Every request revalidates the active heartbeat row.
|
|
app.use(runtimeConnectionIntentRoutes(db));
|
|
app.use(nativeRunMcpRoutes());
|
|
app.use(runnerAttachmentDownloadRoutes(db, opts.storageService));
|
|
const hostServicesDisposers = new Map<string, () => void>();
|
|
let lifecyclePluginsReady = false;
|
|
const agentLifecycle = startAgentLifecycle(db, createLifecycleDriver(db, workerManager), () => lifecyclePluginsReady && !isWarmStandby() && !isIdleTaskDrainActive());
|
|
const connectionIntentHeartbeat = heartbeatService(db, {
|
|
pluginWorkerManager: workerManager,
|
|
});
|
|
const chatChannels = chatChannelService(db, {
|
|
allowLocalVoiceBoard: opts.deploymentMode === "local_trusted",
|
|
deferWebhookProcessing: true,
|
|
heartbeat: connectionIntentHeartbeat,
|
|
publicBaseUrl: opts.authPublicBaseUrl,
|
|
githubWizardOrigin: opts.deploymentMode === "local_trusted"
|
|
&& ["127.0.0.1", "localhost", "::1"].includes(opts.bindHost ?? "")
|
|
&& Number.isInteger(opts.serverPort) && opts.serverPort! > 0
|
|
? `http://${opts.bindHost === "::1" ? "[::1]" : opts.bindHost}:${opts.serverPort}` : null,
|
|
webhookPublicBaseUrl: opts.chatWebhookPublicBaseUrl,
|
|
resolveNativeQuestion: (interaction) =>
|
|
deliverNativeQuestionResponse(db, interaction),
|
|
storage: opts.storageService,
|
|
});
|
|
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
|
|
// A restart's automatic Retry of a connected conversation needs the same callback.
|
|
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
|
|
// Voice capabilities are verified by the signed callback route, not by
|
|
// board/agent bearer authentication. Mount before actorMiddleware so its
|
|
// per-session Bearer cannot be mistaken for an agent API key.
|
|
app.use(voiceWebhookRoutes(db, chatChannels.voice));
|
|
app.use(
|
|
actorMiddleware(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
resolveSession: opts.resolveSession,
|
|
}),
|
|
);
|
|
// After the actor middleware on purpose: a valid Cloud control assertion
|
|
// REPLACES whatever actor the request otherwise resolved to, and only on
|
|
// the one endpoint it authorizes (see the middleware for the contract).
|
|
app.use(cloudControlMiddleware());
|
|
app.use(authorizationReadScope);
|
|
app.use(runnerCallOutcomeMiddleware(db));
|
|
app.use(connectionEventWakeRoutes(db));
|
|
app.use(skillAccessRequestContext);
|
|
app.use("/api/auth", authRoutes(db));
|
|
if (opts.betterAuthHandler) {
|
|
app.all("/api/auth/{*authPath}", opts.betterAuthHandler);
|
|
}
|
|
app.use(llmRoutes(db));
|
|
|
|
// Provider-authenticated ingress is intentionally outside the board
|
|
// mutation guard. The Chat SDK adapter verifies the provider signature
|
|
// before Paperclip persists or acts on any event.
|
|
const emailChannels = emailChannelService(db, {
|
|
isReconciliationEnabled: () => !isWarmStandby(),
|
|
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
|
heartbeat: connectionIntentHeartbeat,
|
|
storage: opts.storageService,
|
|
publicBaseUrl: opts.chatWebhookPublicBaseUrl ?? opts.authPublicBaseUrl,
|
|
});
|
|
app.use(emailWebhookRoutes(emailChannels));
|
|
app.locals.fastResponses = fastResponseService(db, {
|
|
authorizeExternal: async (tx, request) => {
|
|
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
|
|
return endpoint?.provider === "agentmail" ? emailChannels.authorizeFastResponse(tx, request) : chatChannels.authorizeFastResponse(tx, request);
|
|
},
|
|
publishExternal: async (tx, request, commentId, text) => {
|
|
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
|
|
if (endpoint?.provider !== "agentmail") return false;
|
|
await emailChannels.publishFastResponse(tx, request, commentId, text);
|
|
return true;
|
|
},
|
|
budgetHooks: { cancelWorkForScope: connectionIntentHeartbeat.cancelBudgetScopeWork },
|
|
});
|
|
app.use(chatWebhookRoutes(chatChannels));
|
|
// The instance validates single-use registration state and its trusted
|
|
// current origin. These exact callback routes are the public setup returns.
|
|
app.get("/api/chat-github/manifest/callback", async (req, res) => {
|
|
res.set("Cache-Control", "no-store");
|
|
res.set("Referrer-Policy", "no-referrer");
|
|
const redirect = await chatChannels.githubWizard.directCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor);
|
|
res.redirect(303, redirect);
|
|
});
|
|
app.get("/api/chat-github/cloud/callback", async (req, res) => {
|
|
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
|
|
res.redirect(303, await chatChannels.githubWizard.cloudCallback(String(req.query.state ?? ""), String(req.query.registration ?? ""), typeof req.query.claim === "string" ? req.query.claim : undefined, req.actor));
|
|
});
|
|
app.get("/api/chat-github/identity/callback", async (req, res) => {
|
|
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
|
|
res.redirect(303, await chatChannels.githubWizard.identityCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor));
|
|
});
|
|
const managedAutoInstallKeys = resolveManagedBundledPluginKeys(opts.managedPluginAutoInstall);
|
|
const bundledCatalogRoot =
|
|
opts.bundledPluginCatalogRoot ?? resolveBundledCatalogRoot(process.env);
|
|
const distributionPlugins = readDistributionPluginCatalog(bundledCatalogRoot, BUNDLED_PLUGIN_CATALOG);
|
|
const bundledPluginInstalls = resolveBundledPluginInstalls(
|
|
managedAutoInstallKeys ?? SELF_HOSTED_AUTO_INSTALL_KEYS,
|
|
{
|
|
catalogRoot: bundledCatalogRoot,
|
|
env: process.env,
|
|
enforceCatalogRoot: managedAutoInstallKeys !== null,
|
|
distributionPlugins,
|
|
},
|
|
);
|
|
const bundledPackageResolver = managedAutoInstallKeys === null ? undefined : createBundledPluginPackageResolver(bundledPluginInstalls);
|
|
const managedBundledPluginKeys =
|
|
managedAutoInstallKeys !== null
|
|
? bundledPluginInstalls.map((install) => install.pluginKey)
|
|
: [];
|
|
let runtimePluginLoader: Pick<PluginLoader, "loadSingle"> | null = null;
|
|
// A sibling process can install a managed bundled plugin while this process
|
|
// skips the mid-install row, then finish the row after this process's
|
|
// loadAll() pass. The capabilities route may recover only those managed
|
|
// bundles by starting their ready-but-unstarted worker lazily.
|
|
const recoverManagedBundledPluginWorker =
|
|
managedAutoInstallKeys !== null
|
|
? createManagedBundledPluginWorkerRecovery({
|
|
managedBundledPluginKeys,
|
|
workerManager,
|
|
getLoader: () => runtimePluginLoader,
|
|
})
|
|
: undefined;
|
|
|
|
// Mount API routes
|
|
const api = Router();
|
|
const agentAvatars = agentAvatarRoutes();
|
|
api.use(agentAvatars.router);
|
|
api.use(boardMutationGuard());
|
|
api.use("/health", health);
|
|
api.use(openApiRoutes());
|
|
api.use("/cloud", cloudRoutes());
|
|
api.use("/companies", companyRoutes(db, opts.storageService));
|
|
api.use(llmRoutes(db));
|
|
api.use(folderRoutes(db));
|
|
api.use(companySkillRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(companySkillPolicyRoutes(db));
|
|
api.use(inboxAgentPolicyRoutes(db));
|
|
api.use(agentCommentaryRoutes(db));
|
|
api.use(agentChatMessageRoutes(db));
|
|
api.use(builtInAgentRoutes(db));
|
|
api.use(summarySlotRoutes(db));
|
|
api.use(statusCardRoutes(db));
|
|
api.use(teamsCatalogRoutes(db));
|
|
// The setup-token login session service. The router builds it and hands it
|
|
// back through the callback below, so the shutdown hook can cancel every live
|
|
// session (SR-4).
|
|
let setupTokenLoginService: SetupTokenSessionService | null = null;
|
|
// The dedicated proxy IP or CIDR allowlist for the confidential setup-token
|
|
// login responses (SR-7). The global `TRUST_PROXY` setting does not satisfy
|
|
// the guard; an operator sets this allowlist to the real TLS-terminating
|
|
// proxy addresses. An empty value keeps the confidential responses on direct
|
|
// TLS (or a `local_trusted` loopback peer) only.
|
|
const setupTokenLoginProxyAllowlist = (
|
|
process.env.CLAUDE_LOGIN_TRUSTED_PROXIES ?? ""
|
|
)
|
|
.split(",")
|
|
.map((entry) => entry.trim())
|
|
.filter((entry) => entry.length > 0);
|
|
// The explicit operator declaration that a platform edge terminates TLS for
|
|
// every client request (SR-7). This complements the allowlist for managed
|
|
// platforms (Railway, Render, Fly, and the like) where the app socket is
|
|
// always plain HTTP and the edge-proxy peer addresses are not stable or
|
|
// documented, so `CLAUDE_LOGIN_TRUSTED_PROXIES` cannot express them. It is a
|
|
// dedicated, single-purpose setting; the guard still never reads the global
|
|
// `TRUST_PROXY` value.
|
|
const setupTokenLoginEdgeTlsTerminated = /^(1|true|yes|on)$/i.test(
|
|
(process.env.CLAUDE_LOGIN_EDGE_TLS_TERMINATED ?? "").trim(),
|
|
);
|
|
// Bind the production setup-token login transport. It carries the live lease
|
|
// manager, the login-process factory over the sandbox pseudo-terminal, and the
|
|
// durable cleanup store. The factory passes only the fixed command
|
|
// `CLAUDE_SETUP_TOKEN_COMMAND`; it never reads a command from a
|
|
// route, a request body, or an adapter configuration. The durable store and the
|
|
// startup reaper are live now, so a restart reaps a leftover lease.
|
|
//
|
|
// The live sandbox pseudo-terminal opener binds inside the sandbox provider
|
|
// worker, so the server process does not hold the raw sandbox process. The
|
|
// opener drives the worker through the plugin worker manager route gate.
|
|
// The manager mints a host-owned route identifier, permits one
|
|
// active credential pseudo-terminal per worker, binds the worker session
|
|
// identifier one time for output only, and terminalizes the route on every open
|
|
// failure path. With the opener supplied, the provider acquires a lease and the
|
|
// start route drives a live login instead of the fixed 503.
|
|
const setupTokenLoginTransport = buildSetupTokenLoginTransport({
|
|
sandbox: createProductionSetupTokenSandboxProvider({
|
|
db,
|
|
environments: environmentService(db),
|
|
environmentRuntime: environmentRuntimeService(db, {
|
|
pluginWorkerManager: workerManager,
|
|
}),
|
|
openLivePtySession: createWorkerBoundLoginPtyOpener({
|
|
workerManager,
|
|
environments: environmentService(db),
|
|
log: (line) => logger.info(line),
|
|
}),
|
|
log: (line) => logger.info(line),
|
|
}),
|
|
store: createProductionSetupTokenCleanupStore(db),
|
|
// Bind the atomic credential-claim writer, so a completed login transitions
|
|
// the durable row to `stored` and stores the minted token in one control-plane
|
|
// transaction. The writer reads the company and the owner only from the
|
|
// immutable session scope. The confirm-replacement flow owns rotation. Without
|
|
// this writer the router falls back to the deferred, fail-closed 503 and never
|
|
// stores the token.
|
|
completeCredential: createSetupTokenSecretWriter({ db }),
|
|
// Forward the login runner diagnostic lines to the server logger. The
|
|
// runner is the sole producer, and every line is a fixed, non-secret
|
|
// literal. Without this sink the diagnostics fall back to a no-op in
|
|
// production, so a failed login leaves no log trail.
|
|
log: (line) => logger.info(line),
|
|
});
|
|
api.use(
|
|
agentRoutes(db, {
|
|
chatRunRetries: chatChannels,
|
|
pluginWorkerManager: workerManager,
|
|
deploymentMode: opts.deploymentMode,
|
|
confidentialProxyAllowlist: setupTokenLoginProxyAllowlist,
|
|
confidentialEdgeTlsTerminated: setupTokenLoginEdgeTlsTerminated,
|
|
setupTokenLogin: setupTokenLoginTransport,
|
|
onSetupTokenLoginService: (service) => {
|
|
// Capture the service, so the graceful-shutdown hook cancels every live
|
|
// session and releases each lease. The standalone scheduled reaper owns
|
|
// the startup and interval lease cleanup now (SR-4).
|
|
setupTokenLoginService = service;
|
|
},
|
|
}),
|
|
);
|
|
api.use(agentPermissionGrantRoutes(db));
|
|
api.use(agentInteractionPermissionRoutes(db));
|
|
api.use(assetRoutes(db, opts.storageService));
|
|
api.use(agentProfileAvatarRoutes(db, opts.storageService));
|
|
api.use(projectToolRoutes(db));
|
|
api.use(taskWorkspaceRoutes(db));
|
|
api.use(projectRoutes(db));
|
|
api.use(caseRoutes(db, opts.storageService));
|
|
api.use(caseHandoffRoutes(db));
|
|
api.use(workflowRoutes(db));
|
|
api.use(issueTreeControlRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(fileResourceRoutes(db, { remoteFiles: createRemoteWorkspaceFiles(db, {
|
|
environmentRuntime: environmentRuntimeService(db, { pluginWorkerManager: workerManager }),
|
|
loadSavedWorkspace: createSavedWorkspaceLoader(db, opts.storageService),
|
|
openSavedWorkspaceArchive: createSavedWorkspaceArchiveOpener(db, opts.storageService),
|
|
}) }));
|
|
api.use(routineRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(tailscaleConnectionRoutes(db));
|
|
api.use(workloadIdentityRoutes(db));
|
|
api.use(routineSuggestionRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(
|
|
environmentRoutes(db, {
|
|
pluginWorkerManager: workerManager,
|
|
retryPendingEnvironmentCleanup: input => connectionIntentHeartbeat.sweepPendingCleanupLeases({ environmentRetry: input }),
|
|
recoverMissingPluginWorker: recoverManagedBundledPluginWorker
|
|
? {
|
|
pluginKeys: managedBundledPluginKeys,
|
|
startWorker: recoverManagedBundledPluginWorker,
|
|
}
|
|
: undefined,
|
|
}),
|
|
);
|
|
api.use(executionWorkspaceRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(emailRoutes(db, emailChannels));
|
|
api.use(goalRoutes(db));
|
|
api.use(responsibilityRoutes(db));
|
|
api.use(responsibilityAreaRoutes(db));
|
|
api.use(responsibilityBarRoutes(db));
|
|
api.use(responsibilityDashboardRoutes(db));
|
|
api.use(responsibilitySummaryRoutes(db));
|
|
api.use(responsibilityAutomationRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(responsibilityStatusReportRoutes(db));
|
|
api.use(responsibilityMetricRoutes(db));
|
|
api.use(responsibilityRunRoutes(db));
|
|
api.use(onboardingSeedRoutes(db));
|
|
api.use(onboardingChatRoutes(db, connectionIntentHeartbeat));
|
|
api.use(boardChatRoutes(db, { deploymentMode: opts.deploymentMode }));
|
|
api.use(voiceSessionRoutes(db, chatChannels.voice));
|
|
api.use(approvalRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(secretRoutes(db));
|
|
api.use(managedAgentProfileRoutes(db));
|
|
api.use(remoteAgentProfileRoutes(db));
|
|
api.use(
|
|
chatChannelRoutes(db, {
|
|
heartbeat: connectionIntentHeartbeat,
|
|
publicBaseUrl: opts.authPublicBaseUrl,
|
|
storage: opts.storageService,
|
|
service: chatChannels,
|
|
}),
|
|
);
|
|
const trustedLocalStdioRuntimeHost =
|
|
process.env.PAPERCLIP_TRUSTED_MCP_RUNTIME_HOST ??
|
|
process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ??
|
|
null;
|
|
api.use(costRoutes(db, { pluginWorkerManager: workerManager }));
|
|
api.use(activityRoutes(db));
|
|
api.use(dashboardRoutes(db));
|
|
api.use(attentionRoutes(db, {
|
|
pluginWorkerManager: workerManager,
|
|
decisionServiceOptions: opts.decisionServiceOptions,
|
|
}));
|
|
api.use(decisionTrainingRoutes(db));
|
|
api.use(decisionRoutes(db, opts.decisionServiceOptions));
|
|
api.use(decisionQueueRoutes(db));
|
|
api.use(userProfileRoutes(db));
|
|
api.use(sidebarBadgeRoutes(db));
|
|
api.use(sidebarPreferenceRoutes(db));
|
|
api.use(announcementRoutes(db, { ...opts.announcements, version: opts.hostVersion ?? serverVersion }));
|
|
api.use(resourceMembershipRoutes(db));
|
|
api.use(primaryAgentRoutes(db));
|
|
api.use(workspacePreferenceRoutes(db));
|
|
api.use(inboxDismissalRoutes(db));
|
|
api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup, opts.idleDeadlineOwners,
|
|
input => resourceLifecycleDelivery.prepareForIdleSleep(input)));
|
|
api.use(ephemeralEnrollmentRoutes(db));
|
|
if (opts.databaseBackupService) {
|
|
api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService));
|
|
}
|
|
if (serverMetrics) api.use(instanceServerMetricsRoutes(serverMetrics));
|
|
const pluginRegistry = pluginRegistryService(db);
|
|
const eventBus = createPluginEventBus();
|
|
setPluginEventBus(eventBus);
|
|
const jobStore = pluginJobStore(db);
|
|
const lifecycle = pluginLifecycleManager(db, { workerManager });
|
|
// Seed the plugin-provided connector lookups; readers refresh them again
|
|
// from the plugins table whenever they need a current answer.
|
|
setPluginConnectorWorkers(workerManager);
|
|
void loadPluginAppCatalog(db)
|
|
.catch((error) => logger.warn({ err: error }, "Could not load plugin connector catalog"));
|
|
const scheduler = createPluginJobScheduler({
|
|
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
|
db,
|
|
jobStore,
|
|
workerManager,
|
|
});
|
|
const toolDispatcher = createPluginToolDispatcher({
|
|
workerManager,
|
|
lifecycleManager: lifecycle,
|
|
db,
|
|
});
|
|
const gatewayOAuthAccess = toolAccessService(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
trustedLocalStdioRuntimeHost,
|
|
});
|
|
const toolActionDeliveries = toolActionDeliveryService(db, heartbeatService(db, { pluginWorkerManager: workerManager }));
|
|
const toolGateway = createToolGatewayService(db, {
|
|
budgetHooks: { cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
|
|
onToolActionSettled: (id) => toolActionDeliveries.deliver(id),
|
|
pluginToolDispatcher: toolDispatcher,
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
trustedLocalStdioRuntimeHost,
|
|
oauthGrantRefresher: (input) =>
|
|
gatewayOAuthAccess.refreshOAuthGrantCredentials(input),
|
|
});
|
|
// Issue routes are intentionally mounted after the gateway is constructed because
|
|
// issue approval endpoints delegate to it. The intervening routers use distinct
|
|
// route prefixes, so this dependency does not change issue-route precedence.
|
|
api.use(issueRoutes(db, opts.storageService, {
|
|
chatRunRetries: chatChannels,
|
|
pluginWorkerManager: workerManager,
|
|
approveToolActionRequest: (input) => toolGateway.approveActionRequest(input),
|
|
declineToolActionRequest: (input) => toolGateway.declineActionRequest(input),
|
|
}));
|
|
api.use(slackToolRoutes(db, opts.authPublicBaseUrl));
|
|
app.locals.toolGateway = toolGateway;
|
|
registerAssignedMcpGateway(db, toolGateway);
|
|
app.locals.toolActionDeliveries = toolActionDeliveries;
|
|
app.use(mcpGatewayProtocolRoutes(toolGateway));
|
|
api.use(decisionModelRoutes(db));
|
|
api.use(agentObservabilityRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure }));
|
|
api.use(fastResponseRoutes(db));
|
|
api.use(linkPreviewRoutes(db));
|
|
api.use(promptInjectionPreventionRoutes(db));
|
|
api.use(aiConnectionRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure, trustedLocalStdioRuntimeHost, pluginWorkerManager: workerManager }));
|
|
api.use(
|
|
toolAccessRoutes(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
authPublicBaseUrl: opts.authPublicBaseUrl,
|
|
trustedLocalStdioRuntimeHost,
|
|
toolGateway,
|
|
connectionIntentHeartbeat,
|
|
}),
|
|
);
|
|
api.use(connectionIntentBoardRoutes(db, connectionIntentHeartbeat));
|
|
api.use(
|
|
smokeLabRoutes(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
}),
|
|
);
|
|
const jobCoordinator = createPluginJobCoordinator({
|
|
db,
|
|
lifecycle,
|
|
scheduler,
|
|
jobStore,
|
|
});
|
|
const hostServiceCleanup = createPluginHostServiceCleanup(
|
|
lifecycle,
|
|
hostServicesDisposers,
|
|
);
|
|
let viteHtmlRenderer: ReturnType<typeof createCachedViteHtmlRenderer> | null =
|
|
null;
|
|
let viteDevServer: { close(): Promise<void> } | null = null;
|
|
let viteHmrServer: HttpServer | null = null;
|
|
const loader = pluginLoader(
|
|
db,
|
|
{
|
|
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
|
|
migrationDb: opts.pluginMigrationDb,
|
|
assertPackageActivation: bundledPackageResolver?.assertActivation ?? distributionPluginActivationGuard(bundledCatalogRoot, distributionPlugins, managedAutoInstallKeys),
|
|
resolvePackageRoot: bundledPackageResolver?.resolve,
|
|
prebuiltOnly: managedAutoInstallKeys !== null,
|
|
enableLocalFilesystem: managedAutoInstallKeys === null,
|
|
enableNpmDiscovery: managedAutoInstallKeys === null,
|
|
},
|
|
{
|
|
workerManager,
|
|
eventBus,
|
|
jobScheduler: scheduler,
|
|
jobStore,
|
|
toolDispatcher,
|
|
lifecycleManager: lifecycle,
|
|
instanceInfo: {
|
|
instanceId: opts.instanceId ?? "default",
|
|
hostVersion: opts.hostVersion ?? serverVersion,
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
},
|
|
buildHostHandlers: (pluginId, manifest) => {
|
|
const notifyWorker = (method: string, params: unknown) => {
|
|
const handle = workerManager.getWorker(pluginId);
|
|
if (handle) handle.notify(method, params);
|
|
};
|
|
const services = buildHostServices(
|
|
db,
|
|
pluginId,
|
|
manifest.id,
|
|
eventBus,
|
|
notifyWorker,
|
|
{
|
|
pluginWorkerManager: workerManager,
|
|
manifest,
|
|
},
|
|
);
|
|
hostServicesDisposers.set(pluginId, () => services.dispose());
|
|
return createHostClientHandlers({
|
|
pluginId,
|
|
capabilities: manifest.capabilities,
|
|
services,
|
|
});
|
|
},
|
|
},
|
|
);
|
|
runtimePluginLoader = loader;
|
|
workerManager.setActivationTimeoutRecovery?.(plugin => loader.requestActivationTimeoutRecovery(plugin));
|
|
const browserUse = browserUseService(db, undefined,
|
|
{ cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
|
|
(session, run) => toolGateway.browserUseSessionAuthorized({ ...session, runId: run.heartbeatRunId, invocationId: run.invocationId }));
|
|
api.use(browserUseRoutes(db, browserUse));
|
|
api.use(toolGatewayRoutes(db, toolGateway));
|
|
api.use(mcpAppsRoutes(db, toolGateway));
|
|
api.use(mcpSkillsRoutes(db, toolGateway));
|
|
api.use(
|
|
pluginRoutes(
|
|
db,
|
|
loader,
|
|
{ scheduler, jobStore },
|
|
{ workerManager },
|
|
{ toolDispatcher },
|
|
{ workerManager },
|
|
{ toolGateway },
|
|
),
|
|
);
|
|
api.use(
|
|
adapterRoutes({
|
|
db,
|
|
resolveRunnerTarget: (companyId, environmentId) => resolveAgentRunnerTargetForCompany(db, companyId, environmentId),
|
|
getNativeRunnerEnabled: async () =>
|
|
(await instanceSettingsService(db).getExperimental())
|
|
.enableNativeRunner === true,
|
|
getOpenAiDotEnabled: async () =>
|
|
(await instanceSettingsService(db).getExperimental())
|
|
.enableOpenAiDot === true,
|
|
}),
|
|
);
|
|
api.use(
|
|
accessRoutes(db, {
|
|
deploymentMode: opts.deploymentMode,
|
|
deploymentExposure: opts.deploymentExposure,
|
|
bindHost: opts.bindHost,
|
|
allowedHostnames: opts.allowedHostnames,
|
|
authPublicBaseUrl: opts.authPublicBaseUrl,
|
|
}),
|
|
);
|
|
let publicMcpEvents: PublicMcpEvents | null = null;
|
|
const grokDispatcher = createGrokRoutineDispatcher(db, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
let dotMcpEvents: PublicMcpEvents | null = null;
|
|
if (publicMcpOAuth) {
|
|
const dispatch = createMcpApiDispatch(api, app.locals);
|
|
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
|
|
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
|
});
|
|
const transfers = createPublicMcpTransfers(db, publicMcpOAuth, dispatch, opts.storageService);
|
|
publicMcpIngress.use(transfers.router);
|
|
publicMcpIngress.use(publicMcpIngressRoutes(publicMcpOAuth, createPublicMcpExecutor(db, publicMcpOAuth, dispatch, transfers), publicMcpEvents));
|
|
const dotOAuth = createPublicMcpOAuth(db, { ...publicMcpOAuth.config, resource: publicMcpOAuth.config.origin + "/mcp/runner" });
|
|
dotPrincipalOAuth = dotOAuth;
|
|
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive() });
|
|
const grokTransfers = createPublicMcpTransfers(db, dotOAuth, dispatch, opts.storageService, { pathPrefix: "/mcp/runner/files",
|
|
agentSession: { authorize: (principal, sessionId, taskId, write, database) => dotRunnerBroker(database ?? db).authorizeFileSession(principal, sessionId, taskId, write) } });
|
|
publicMcpIngress.use(grokTransfers.router);
|
|
publicMcpIngress.use(publicMcpIngressRoutes(dotOAuth, createPublicMcpExecutor(db, dotOAuth, dispatch), dotMcpEvents, createDotRunnerMcpTools(db, connectionIntentHeartbeat, grokTransfers)));
|
|
api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth));
|
|
api.use(dotRunnerRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
|
|
api.use(grokBotRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
|
|
}
|
|
|
|
api.use(museBoardRoutes(db, opts.authPublicBaseUrl));
|
|
api.use(museQualificationRoutes(db));
|
|
if (serverMetrics) app.use("/api", serverMetricsAgentPreHandler(serverMetrics));
|
|
app.use("/api", api);
|
|
app.use("/api", (_req, res) => {
|
|
res.status(404).json({ error: "API route not found" });
|
|
});
|
|
app.use(
|
|
pluginUiStaticRoutes(db, {
|
|
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
|
|
resolvePackageRoot: bundledPackageResolver?.resolve,
|
|
}),
|
|
);
|
|
|
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
|
if (opts.uiMode === "static") {
|
|
// Try published location first (server/ui-dist/), then monorepo dev location (../../ui/dist)
|
|
const candidates = [
|
|
path.resolve(__dirname, "../ui-dist"),
|
|
path.resolve(__dirname, "../../ui/dist"),
|
|
];
|
|
const uiDist = candidates.find((p) =>
|
|
fs.existsSync(path.join(p, "index.html")),
|
|
);
|
|
if (uiDist) {
|
|
staticUi.use(staticUiCompression());
|
|
// Hashed asset files (Vite emits them under /assets/<name>.<hash>.<ext>)
|
|
// never change once built, so they can be cached aggressively.
|
|
staticUi.use(
|
|
"/assets",
|
|
staticUiFiles(path.join(uiDist, "assets"), {
|
|
maxAge: "1y",
|
|
immutable: true,
|
|
}),
|
|
);
|
|
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
|
|
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
|
|
// Publish while the routes below are built; the outcome is awaited
|
|
// before the first background work starts. The promise settles to the
|
|
// failure instead of rejecting, so it is never an unhandled rejection.
|
|
app.locals.retainedUiAssetsPublication = trackIdleWork(publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
|
|
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"))
|
|
.then(() => null, (error: unknown) => ({ error })));
|
|
staticUi.use("/assets", retained.serve);
|
|
}
|
|
// Serve root/index through the same runtime HTML transform as SPA routes.
|
|
staticUi.get(["/", "/index.html"], (_req, res) => {
|
|
res.type("html").set("Cache-Control", "no-cache").send(readBrandedStaticIndexHtml(uiDist));
|
|
});
|
|
// Non-hashed static files (favicon.ico, manifest, robots.txt, etc.):
|
|
// short cache so operators who swap them out see the new version
|
|
// reasonably fast, with must-revalidate overrides for index.html and
|
|
// sw.js (see staticUiCacheControl for why those two).
|
|
staticUi.use(
|
|
staticUiFiles(uiDist, {
|
|
maxAge: "1h",
|
|
setHeaders(res, filePath) {
|
|
const override = staticUiCacheControl(filePath);
|
|
if (override) {
|
|
res.set("Cache-Control", override);
|
|
}
|
|
},
|
|
}),
|
|
);
|
|
// SPA fallback. Only for non-asset routes — if the browser asks for
|
|
// /assets/something.js that doesn't exist, we must NOT serve the HTML
|
|
// shell: the browser would try to load it as a JavaScript module, fail
|
|
// with a MIME-type error, and cache that broken response. Return 404
|
|
// instead. The index.html response itself is no-cache so a subsequent
|
|
// deploy's updated asset hashes are picked up on next load.
|
|
staticUi.get(/.*/, (req, res) => {
|
|
if (req.path.startsWith("/assets/")) {
|
|
res.status(404).end();
|
|
return;
|
|
}
|
|
res
|
|
.status(200)
|
|
.set("Content-Type", "text/html")
|
|
.set("Cache-Control", "no-cache")
|
|
.end(readBrandedStaticIndexHtml(uiDist));
|
|
});
|
|
} else {
|
|
console.warn("[paperclip] UI dist not found; running in API-only mode");
|
|
}
|
|
if (process.env.PAPERCLIP_MANAGED_RUNTIME_EXPOSURE === "tailscale_https") {
|
|
// The managed-runtime supervisor waits for the app port AND its derived
|
|
// Vite HMR companion port to bind before publishing the service. Static
|
|
// mode has no Vite, so bind the same placeholder listener dev mode uses
|
|
// or the supervisor kills a healthy server at the readiness deadline
|
|
// (PAP-18043).
|
|
const hmrServer = createHttpServer((_req, res) => {
|
|
res.writeHead(426, { "Content-Type": "text/plain" });
|
|
res.end("Upgrade Required");
|
|
});
|
|
await listenViteHmrServer(
|
|
hmrServer,
|
|
resolveViteHmrPort(opts.serverPort),
|
|
opts.bindHost,
|
|
);
|
|
viteHmrServer = hmrServer;
|
|
}
|
|
}
|
|
|
|
if (opts.uiMode === "vite-dev") {
|
|
const uiRoot = path.resolve(__dirname, "../../ui");
|
|
const publicUiRoot = path.resolve(uiRoot, "public");
|
|
const hmrPort = resolveViteHmrPort(opts.serverPort);
|
|
const hmrHost = resolveViteHmrHost(opts.bindHost);
|
|
const hmrProtocol = resolveViteHmrProtocol(
|
|
process.env.PAPERCLIP_VITE_HMR_PROTOCOL,
|
|
);
|
|
const hmrServer = createHttpServer((_req, res) => {
|
|
res.writeHead(426, { "Content-Type": "text/plain" });
|
|
res.end("Upgrade Required");
|
|
});
|
|
const { createServer: createViteServer } = await import("vite");
|
|
const configuredViteCacheDir = process.env.PAPERCLIP_VITE_CACHE_DIR?.trim();
|
|
const vite = await createViteServer({
|
|
root: uiRoot,
|
|
...(configuredViteCacheDir
|
|
? { cacheDir: path.resolve(configuredViteCacheDir) }
|
|
: {}),
|
|
appType: "custom",
|
|
// Vite otherwise discovers every HTML entry below the UI root. Generated
|
|
// Storybook output can reference dependencies that are intentionally not
|
|
// part of the application install, poisoning a clean embedded dev-server
|
|
// cache before the browser opens. The embedded UI has one real entry.
|
|
optimizeDeps: { entries: [path.resolve(uiRoot, "index.html")] },
|
|
server: {
|
|
// Listener binding and browser HMR hostname are deliberately separate:
|
|
// exposed branch runtimes stay loopback-only while the browser uses the
|
|
// current MagicDNS hostname through the broker's HTTPS listener.
|
|
host: opts.bindHost,
|
|
middlewareMode: true,
|
|
hmr: {
|
|
server: hmrServer,
|
|
...(hmrHost ? { host: hmrHost } : {}),
|
|
...(hmrProtocol ? { protocol: hmrProtocol } : {}),
|
|
port: hmrPort,
|
|
clientPort: hmrPort,
|
|
},
|
|
allowedHosts: privateHostnameGateEnabled
|
|
? Array.from(privateHostnameAllowSet)
|
|
: undefined,
|
|
},
|
|
});
|
|
try {
|
|
await listenViteHmrServer(hmrServer, hmrPort, opts.bindHost);
|
|
} catch (error) {
|
|
await vite.close();
|
|
throw error;
|
|
}
|
|
viteDevServer = vite;
|
|
viteHmrServer = hmrServer;
|
|
viteHtmlRenderer = createCachedViteHtmlRenderer({
|
|
vite,
|
|
uiRoot,
|
|
brandHtml: applyUiBranding,
|
|
});
|
|
const renderViteHtml = viteHtmlRenderer;
|
|
|
|
if (fs.existsSync(publicUiRoot)) {
|
|
staticUi.use(staticUiFiles(publicUiRoot, { index: false }));
|
|
}
|
|
staticUi.get(/.*/, async (req, res, next) => {
|
|
if (!shouldServeViteDevHtml(req)) {
|
|
next();
|
|
return;
|
|
}
|
|
try {
|
|
const html = await renderViteHtml.render(req.originalUrl);
|
|
res.status(200).set({ "Content-Type": "text/html" }).end(html);
|
|
} catch (err) {
|
|
next(err);
|
|
}
|
|
});
|
|
staticUi.use(vite.middlewares);
|
|
}
|
|
|
|
app.use(staticUi);
|
|
app.use(errorHandler);
|
|
|
|
// A build whose asset names collide with different published bytes refuses
|
|
// to start, so the outcome is known before any job, queue, or run starts.
|
|
const retainedUiAssetsPublication = await (app.locals.retainedUiAssetsPublication as Promise<{ error: unknown } | null> | undefined);
|
|
if (retainedUiAssetsPublication) throw retainedUiAssetsPublication.error;
|
|
|
|
jobCoordinator.start();
|
|
const deliveryWork = createDeliveryWorkCoordinator({
|
|
owner: db,
|
|
canRun: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
|
canReconcile: () => !isWarmStandby(),
|
|
onError: (err, queue) => logger.error({ err, queue }, "Delivery reconciliation failed"),
|
|
});
|
|
app.locals.deliveryWork = deliveryWork;
|
|
await scheduler.start(deliveryWork);
|
|
const agentLifecycleDelivery = deliveryWork.register(DELIVERY_QUEUES.agentLifecycle, {
|
|
retryMs: 1_000,
|
|
run: () => agentLifecycle.sweep(),
|
|
hasPending: () => agentLifecycle.hasPending(),
|
|
nextRunAt: () => agentLifecycle.nextRunAt(),
|
|
});
|
|
agentLifecycle.setWake(agentLifecycleDelivery.wake);
|
|
const clientMirror = startMcpClientMirrorSync(db, deliveryWork, mcpConfig?.origin,{authorizationOrigin:mcpConfig?.authorizationOrigin});
|
|
const unregisterMcpOwners = registerIdleMcpOwners(db, { config: mcpConfig, personal: publicMcpEvents, dot: dotMcpEvents, clientMirrorInstanceId:clientMirror.instanceId });
|
|
startMcpRevocationIndexSync(db, deliveryWork);
|
|
startMcpIngressHandoffExpiry(db, deliveryWork);
|
|
publicMcpEvents?.start(deliveryWork);
|
|
dotMcpEvents?.start(deliveryWork);
|
|
if (publicMcpOAuth) grokDispatcher.start(deliveryWork);
|
|
const stopMuseMaintenance = startMuseMaintenance(db, deliveryWork, async () => {
|
|
const state = await connectionIntentHeartbeat.resolveSchedulingSuppression(true);
|
|
return !state.suppressed || state.reason === "task_drain";
|
|
});
|
|
const resourceLifecycleDelivery = registerPluginLifecycleDelivery(deliveryWork, db, workerManager, lifecycle);
|
|
await resourceLifecycleDelivery.ready;
|
|
if (opts.feedbackExportService) {
|
|
deliveryWork.register(DELIVERY_QUEUES.feedback, {
|
|
retryMs: FEEDBACK_EXPORT_FLUSH_INTERVAL_MS,
|
|
run: (signal) => opts.feedbackExportService!.flushPendingFeedbackTraces({ signal }),
|
|
hasPending: () => opts.feedbackExportService!.hasPendingFeedbackTraces(),
|
|
});
|
|
}
|
|
registerChatDeliveryWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
|
|
registerChatActionWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
|
|
chatChannels.voice.registerReportWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.voice.registerRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.voice.registerReplyWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.voice.inbound.registerCleanupWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.voice.inbound.registerRetentionWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
registerGitHubReviewProjectionWork(db, deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerGitHubReviewCheckWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerGitHubTerminalResponseWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerGitHubAcknowledgementWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatFailedRetryNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
chatRunMilestoneWork(db, opts.authPublicBaseUrl).register(
|
|
deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
|
);
|
|
emailChannels.start();
|
|
chatChannels.registerGitHubWebhookRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
const importSpoolCleanup = registerImportSpoolCleanupWork(db, deliveryWork,
|
|
resolveDefaultImportTransferSpoolRoot(), () => !isWarmStandby() && !isIdleTaskDrainActive());
|
|
registerBrowserUseCleanup(deliveryWork, browserUse, () => !isIdleTaskDrainActive());
|
|
|
|
|
|
void companyTransferRunService
|
|
.recoverStrandedApplyingRuns(db)
|
|
.then((recovered) => {
|
|
if (recovered.length > 0) {
|
|
logger.warn(
|
|
{ count: recovered.length, runIds: recovered },
|
|
"failed company transfer runs stranded in applying by a restart",
|
|
);
|
|
}
|
|
})
|
|
.catch((err) => {
|
|
logger.error({ err }, "stranded company transfer apply recovery failed");
|
|
});
|
|
void toolDispatcher.initialize().catch((err) => {
|
|
logger.error({ err }, "Failed to initialize plugin tool dispatcher");
|
|
});
|
|
const devWatcher = managedAutoInstallKeys !== null ? null : createPluginDevWatcher(
|
|
lifecycle,
|
|
async (pluginId) =>
|
|
(await pluginRegistry.getById(pluginId))?.packagePath ?? null,
|
|
undefined,
|
|
{ unwatchedRoots: [bundledCatalogRoot] },
|
|
);
|
|
// Auto-provision bundled plugins so their providers are registered for
|
|
// agent runs. Bundles are excluded from the pnpm
|
|
// workspace and built standalone into the image (see Dockerfile), then
|
|
// installed here from their local paths. This runs BEFORE loadAll() so
|
|
// loadAll() can activate them in the same startup pass.
|
|
//
|
|
// Workers are started exactly once, by loadAll(): the `lifecycle` manager
|
|
// above is constructed without a runtime-capable loader
|
|
// (pluginLifecycleManager(db, { workerManager }) — no `loader` option), so
|
|
// the lifecycle.load() that ensureBundledPlugins performs per newly
|
|
// installed bundle only records the `ready` status and does not spawn a
|
|
// worker (see activateReadyPlugin in services/plugin-lifecycle.ts).
|
|
//
|
|
// Managed instances (`plugins.autoInstall` from PAPERCLIP_MANAGED_CONFIG)
|
|
// drive the key list from the control plane; self-hosted instances keep
|
|
// the pre-existing behavior of ensuring only the kubernetes bundle.
|
|
//
|
|
// Resolution is deliberately synchronous and NOT fail-safe: an
|
|
// unknown key or a path escaping the bundled catalog root throws out of
|
|
// createApp so a managed instance refuses to start (positive allowlist,
|
|
// fail closed).
|
|
// SAFETY: installation is fully fail-safe. Any failure
|
|
// (missing bundle, install error, load error) is caught, logged, and
|
|
// swallowed per plugin so the server ALWAYS finishes booting. A degraded
|
|
// boot (a provider unavailable, some agents cannot run) is strictly
|
|
// preferable to a crash loop.
|
|
//
|
|
// The chain is not awaited here (createApp stays fast), but the settled
|
|
// promise is exposed via `app.locals.bundledPluginsStartup` so boot steps
|
|
// that must not outrun plugin availability — managed sandbox environments
|
|
// (`applyManagedEnvironments`) run before the heartbeat resumes queued
|
|
// runs — can sequence on it. It never rejects.
|
|
const bundledPluginsStartup = ensureBundledPlugins(
|
|
bundledPluginInstalls,
|
|
{ registry: pluginRegistry, loader, lifecycle, logger },
|
|
// Managed mode reinstalls soft-uninstalled bundles (the control plane
|
|
// owns provisioning); self-hosted leaves an operator's uninstall alone.
|
|
// Disabled plugins never start automatically. Added distribution permissions
|
|
// still enter upgrade_pending so enabling them requires an operator decision.
|
|
{ reinstallUninstalled: managedAutoInstallKeys !== null, preservePackagePath: managedAutoInstallKeys !== null },
|
|
)
|
|
.then(() => loader.loadAll())
|
|
.then((result) => {
|
|
if (!result) return;
|
|
for (const loaded of result.results) {
|
|
if (devWatcher && loaded.success && loaded.plugin.packagePath) {
|
|
devWatcher.watch(loaded.plugin.id, loaded.plugin.packagePath);
|
|
}
|
|
}
|
|
})
|
|
.catch((err) => {
|
|
logger.error({ err }, "Failed to load ready plugins on startup");
|
|
});
|
|
app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
|
|
// Manifest ids the managed auto-install list resolves to; the managed
|
|
// environments bootstrap checks plugin entries against them.
|
|
app.locals.managedBundledPluginKeys = managedBundledPluginKeys;
|
|
void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying."));
|
|
// Carry over legacy plugin tool access once plugin workers have started,
|
|
// since a plugin connection is healthy only while its worker runs.
|
|
void bundledPluginsStartup.then(() => completeLegacyPluginToolAccess(db))
|
|
.catch((error) => logger.warn({ err: error }, "Could not keep legacy plugin tool access"));
|
|
// The shutdown hook runs at most once. It caches the in-flight promise, so a
|
|
// second caller (for example the `exit` handler) awaits the same completion
|
|
// instead of starting a second teardown.
|
|
let appServicesShutdown: Promise<void> | null = null;
|
|
const shutdownAppServices = (): Promise<void> => {
|
|
if (appServicesShutdown) return appServicesShutdown;
|
|
appServicesShutdown = finishServiceShutdown(() => scheduler.stop(), async () => {
|
|
await loader.stopActivationTimeoutRecovery();
|
|
await agentLifecycle.stop();
|
|
unregisterMcpOwners();
|
|
await publicMcpEvents?.stop();
|
|
await dotMcpEvents?.stop();
|
|
jobCoordinator.stop();
|
|
await finishServiceShutdown(() => resourceLifecycleDelivery.dispose(), async () => {
|
|
importSpoolCleanup.unsubscribe();
|
|
await deliveryWork.stop();
|
|
devWatcher?.close();
|
|
viteHtmlRenderer?.dispose();
|
|
void viteDevServer?.close().catch(() => undefined);
|
|
viteHmrServer?.close();
|
|
hostServiceCleanup.disposeAll();
|
|
hostServiceCleanup.teardown();
|
|
await stopMuseMaintenance();
|
|
await emailChannels.shutdown();
|
|
await chatChannels.shutdown();
|
|
// End the avatar worker pool, if a request ever started one, so no
|
|
// render outlives the HTTP teardown.
|
|
await agentAvatars.close();
|
|
// Cancel every live setup-token login session and AWAIT the cancellation,
|
|
// so each direct child stops and the server releases each lease before the
|
|
// caller stops the database and the provider. A lease release that
|
|
// fails stays a durable record for the startup reaper.
|
|
await setupTokenLoginService?.shutdown();
|
|
if (serverMetrics) stopServerMetrics(serverMetrics);
|
|
});
|
|
});
|
|
return appServicesShutdown;
|
|
};
|
|
app.locals.paperclipShutdown = shutdownAppServices;
|
|
|
|
// The `exit` event is synchronous. It cannot await the teardown, so it runs
|
|
// the best-effort cleanup and drops the returned promise. The orderly signal
|
|
// path awaits `shutdownAppServices` in full before the process exits.
|
|
process.once("exit", () => {
|
|
void shutdownAppServices();
|
|
});
|
|
process.once("beforeExit", () => {
|
|
void flushPluginLogBuffer();
|
|
});
|
|
|
|
trackIdleRequestHandlers(app);
|
|
return app;
|
|
}
|