Files
PaperClipAI/server/src/app.ts
T
DottaandClaude Opus 5.5 f4cb69205e Let plugin workers serve connector tools (plugin transport) (#22019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types to the Connectors catalog
(#22004), but only for transports the host already runs (remote MCP,
local stdio)
> - A plugin cannot yet serve a connector's tools itself, and legacy
plugin tools (top-level manifest `tools`) still reach every agent
without a connection or install
> - A plugin should be able to ship a full connector: settings,
credentials, tools, grants, policies and audit, like a built-in one
> - This pull request adds the `plugin` transport, the SDK API to serve
it, and moves legacy plugin tools under the same activation rules
without changing today's access
> - The benefit is that any connector can live in a plugin, and agents
get plugin tools only when an operator enables them

## Linked Issues or Issue Description

Refs #22004

**Subsystem affected**
Plugins (SDK, worker protocol, manifest), Connectors (tool access,
catalog discovery), tool gateway, database.

**Problem or motivation**
A plugin connector can only describe an existing transport. There is no
way for a plugin worker to serve a connector's tools with per-connection
settings and credentials. Legacy plugin tools are visible to every agent
as soon as the plugin is enabled, which is the opposite of the connector
activation rules.

**Proposed solution**
Add a `plugin` transport. The gateway keeps every decision (grant,
policy, approval, rate limit, audit) and calls the plugin worker with
the connection, its settings and a per-call credential. Legacy top-level
tools become one implicit connector per plugin. Existing installations
keep exactly their current access through a one-time migration.

**Alternatives considered**
Move legacy tools to connection tool names (`app.<connection>:<tool>`).
Rejected: names change, so existing tool policies and agent instructions
that name `plugin:tool` would stop matching (a possible widening or
loss). Legacy tools keep their names and handlers; only their visibility
is now gated by an install of the implicit connector.

## What Changed

- `packages/db`: migration `0491` widens
`tool_connections_transport_check` with `plugin` (`NOT VALID`, annotated
`online_compatible`), and adds `plugin_connector_legacy_access`, seeded
with one row per installed legacy-tools plugin and company.
- `packages/shared`: method schema gains `settingsSchema` and `tools`
(`{ kind: "static", list: [{ name, displayName?, description?, risk,
parametersSchema }] } | { kind: "dynamic" }`), only for `transport:
"plugin"`; tool names must match `[a-z0-9_]`; plugin-served methods use
`none` or `api_key`. The connect API accepts `settings`.
- `packages/plugins/sdk`: `ToolRunContext.connection` (`id`, `appSlug`,
`methodKey`, `grantId`, `settings`, `credential?`);
`ctx.connectors.register(appSlug, toolName, fn)` and
`ctx.connectors.listTools(appSlug, fn)`; new `listConnectorTools` RPC;
`executeTool` routes connector calls by `appSlug`. A worker refuses to
start when a declared static tool (or dynamic lister) has no handler.
The test harness gains `executeConnectorTool` and `listConnectorTools`.
- `server/src/services/plugin-connector-runtime.ts`: routes one call to
the plugin's worker; a stopped worker is a
`PluginWorkerUnavailableError`.
- Connect flow (`tool-access.ts`): a `plugin` method creates a
`paperclip_plugin` application and a `plugin` connection with `config =
{ sourceTemplateKey, connectionMethodKey, settings }`, validated against
`settingsSchema`. Credentials, grants, credential policy, installs and
profiles use the existing code.
- Catalog: static tools come from the definition, dynamic tools from
`listConnectorTools`, with the existing version hashing and retirement.
Tool risk comes from the declaration.
- Gateway: plugin connections join connected-tool discovery and both
execution paths (direct and approved actions). For each call the gateway
resolves the grant and, for `api_key`, the credential for that call
only. The run context comes only from the session. A stopped worker
returns `503 plugin_connector_unavailable` with `retryable: true`. The
token broker refuses to export plugin connector credentials.
- Legacy tools: a plugin with top-level `tools` and no `apps` gets an
implicit connector (slug `default`, named after the plugin, no
authentication). Its tools keep their names, handlers and today's access
decision, and an agent sees them only when a connection of that
connector is installed for it or for the company. The connector's saved
action rules (Off, Ask first) can only restrict them.
`completeLegacyPluginToolAccess` turns each migration row into such a
connection with a company install, at boot and when a disabled plugin is
enabled again; it is single-flight across servers, resumes a draft left
by an interrupted attempt, and never overrides an operator's later
changes.
- Catalog: tools a dynamic connector stops offering are retired from
discovery; `listConnectorTools` carries the connection's `companyId`,
and uses the acting person's grant for personal connections.
- Board: the setup flow renders `settingsSchema` with the shared JSON
Schema form, sends `settings`, and restores saved settings on resume and
reconnect.
- `create-paperclip-plugin --template connector` now scaffolds a plugin
connector (`apps` + `ctx.connectors.register` + a harness test). SDK
README and `PLUGIN_SPEC.md` describe connectors; top-level `tools` stay
documented as legacy.
- `inferToolRisk` moves to `tool-risk-inference.ts` so the catalog can
reuse it.

Design notes where the spec left room:
- OAuth for `plugin`-transport methods is not supported in this PR
(`none` and `api_key` only); OAuth plugin connectors can use
`mcp_remote` today. Noted as follow-up.
- Board "Test" calls on a plugin connection pass an empty
`agentId`/`runId`, because there is no agent run.
- Companies created after the upgrade get no automatic legacy
connection; they follow the normal activation rules.

## Verification

- `server`: `npx vitest run
src/__tests__/plugin-connectors-plugin-transport.test.ts` (new):
- rules 1-3: a ready plugin and configured accounts give no tools; an
install for agent A gives only A the tools; agent B is denied on
invocation;
- the worker receives `runContext.connection` with the right id, method,
grant, settings and the call's credential; spoofed arguments do not
change it; each account calls with its own key; results do not contain
the key;
- invalid settings are rejected (422); a stopped worker is `503
plugin_connector_unavailable` (retryable); a disabled plugin hides the
tools;
- dynamic tools are listed through the worker and a tool it stops
offering is retired;
- rule 5: a new installation of a legacy-tools plugin gives no tools; an
installation recorded by the migration keeps the same tool names for
every agent, through one company-wide connection (two concurrent
carry-overs make one; an interrupted draft is finished; a retry keeps an
operator's restriction), and is not also exposed as connection tools;
Ask first and Off on the implicit connector restrict the legacy tools;
  - the token broker refuses to export a plugin connection's credential.
- Existing gateway plugin-tool tests (`tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`) now set up the
legacy grant an existing installation has, and pass unchanged otherwise.
- Regression: `plugin-connectors-circleback-parity`, `tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`,
`tool-access-service` (846 passed), plus `connector-tool-policy`,
`plugin-lifecycle-restart`, `remote-mcp-connectors`,
`ai-connection-router` in an earlier run.
- UI: `AppsConnect.test.tsx` covers sending plugin settings from the
settings schema.
- SDK: `tests/worker-rpc-host.test.ts` (start refusal without handlers,
connector routing, tool listing). Scaffold: `entrypoints.test.ts`.
- `packages/db`: `pnpm typecheck` including `check:migrations`; server
layered typecheck (sources and tests); `tsc --noEmit` for shared, SDK,
scaffold and UI.

## Risks

- Behavior change for new installations: legacy plugin tools no longer
reach agents until an operator connects the plugin's connector.
Installations that exist at upgrade time keep their tools through the
migration rows.
- The migration widens a check constraint (`NOT VALID`) and copies
plugin × company rows into a new table. Both are fast; the previous
release never writes `plugin`.
- Calls depend on the plugin worker; when it is down the agent gets a
retryable error.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:09:42 +00:00

1509 lines
71 KiB
TypeScript

import { authorizationReadScope } from "./services/authorization-read-scope.js";
import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js";
import { startMcpClientMirrorSync } from "./services/public-mcp/client-mirror-sync.js";
import { registerIdleMcpOwners } from "./services/public-mcp/idle-sleep-proof.js";
import { startMcpRevocationIndexSync } from "./services/public-mcp/revocation-index-sync.js";
import { startMcpIngressHandoffExpiry } from "./services/public-mcp/ingress-handoff-expiry.js";
import { runnerAttachmentDownloadRoutes } from "./services/native-runtime/runner-attachment-download.js";
import { nativeRunMcpRoutes } from "./routes/native-run-mcp.js";
import { connectionEventWakeRoutes } from "./routes/connection-event-wake.js";
import { registerGitHubReviewProjectionWork } from "./services/chat-channels/github/review-projection-work.js";
import { finishServiceShutdown } from "./shutdown.js";
import { promptInjectionPreventionRoutes } from "./routes/prompt-injection-prevention.js";
import { voiceSessionRoutes, voiceWebhookRoutes } from "./routes/voice-sessions.js";
import { eq } from "drizzle-orm";
import { chatEndpoints, dedicatedDbConnectionStats } from "@paperclipai/db";
import { fastResponseRoutes } from "./routes/fast-responses.js";
import { fastResponseService } from "./services/fast-responses.js";
import { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
import { createLifecycleDriver } from "./services/agent-lifecycle-driver.js";
import { startAgentLifecycle } from "./services/agent-lifecycle.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
import { staticUiFiles } from "./middleware/static-ui.js";
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
import { loadStorageConfig } from "./storage/config.js";
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
import { customerSuccessRoutes } from "./routes/customer-success.js";
import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js";
import type { CloudWarmStandby } from "./services/cloud-warm-standby.js";
import { browserUseRoutes } from "./routes/browser-use.js";
import { mcpAppsRoutes } from "./routes/mcp-apps.js";
import { mcpSkillsRoutes } from "./routes/mcp-skills.js";
import { registerChatActionWork } from "./services/chat-action-work.js";
import { registerChatDeliveryWork } from "./services/chat-delivery-work.js";
import { registerBrowserUseCleanup } from "./services/browser-use-work.js";
import { browserUseService } from "./services/browser-use.js";
import { slackToolRoutes } from "./routes/slack-tools.js";
import { createPublicMcpOAuth, publicMcpConfig } from "./services/public-mcp/oauth.js";
import { createDotPrincipalHttp } from "./modules/external-agents/index.js";
import { createDotRunnerMcpTools, dotRunnerBroker } from "./services/dot-runner-broker.js";
import { agentProfileAvatarRoutes } from "./routes/agent-profile-avatar.js";
import { museTransportRoutes, museBoardRoutes } from "./routes/muse.js";
import { museQualificationRoutes } from "./routes/muse-qualification.js";
import { startMuseMaintenance } from "./services/muse-maintenance.js";
import { grokBotRoutes } from "./routes/grok-bot.js";
import { createGrokRoutineDispatcher } from "./services/grok-bot/dispatcher.js";
import { dotRunnerRoutes } from "./routes/dot-runner.js";
import { createPublicMcpTransfers } from "./services/public-mcp/file-transfers.js";
import { createMcpApiDispatch, createPublicMcpExecutor } from "./services/public-mcp/capabilities.js";
import { createPublicMcpEvents, type PublicMcpEvents } from "./services/public-mcp/events.js";
import { publicMcpIngressRoutes, publicMcpManagementRoutes } from "./routes/public-mcp.js";
import { agentAvatarRoutes } from "./routes/agent-avatars.js";
import { decisionModelRoutes } from "./routes/decision-models.js";
import { agentObservabilityRoutes } from "./routes/agent-observability.js";
import { aiConnectionRoutes } from "./routes/ai-connections.js";
import { taskWorkspaceRoutes } from "./routes/task-workspaces.js";
import { projectToolRoutes } from "./routes/project-tools.js";
import { emailChannelService } from "./services/email-channels.js";
import { emailRoutes, emailWebhookRoutes } from "./routes/email.js";
import { toolActionDeliveryService } from "./services/tool-action-delivery.js";
import { registerAssignedMcpGateway } from "./services/native-runtime/assigned-mcp-tools.js";
import express, { Router, type Request as ExpressRequest } from "express";
import {
createServer as createHttpServer,
type Server as HttpServer,
} from "node:http";
import path from "node:path";
import fs from "node:fs";
import { fileURLToPath } from "node:url";
import type { Db } from "@paperclipai/db";
import {
derivePaperclipViteHmrPort,
type DeploymentExposure,
type DeploymentMode,
} from "@paperclipai/shared";
import type { InspectDatabaseBackupHealthOptions } from "./services/database-backup-health.js";
import type { StorageService } from "./storage/types.js";
import { httpLogger, errorHandler } from "./middleware/index.js";
import { skillAccessRequestContext } from "./services/skill-access.js";
import { actorMiddleware } from "./middleware/auth.js";
import { boardMutationGuard } from "./middleware/board-mutation-guard.js";
import {
privateHostnameGuard,
resolvePrivateHostnameAllowSet,
} from "./middleware/private-hostname-guard.js";
import {
applyTrustProxy,
parseTrustProxyEnv,
} from "./middleware/trust-proxy.js";
import {
resolveDefaultImportTransferSpoolRoot,
} from "./services/company-import-transfers.js";
import { registerImportSpoolCleanupWork } from "./services/import-spool-cleanup-work.js";
import { companyTransferRunService } from "./services/company-transfer-runs.js";
import { healthRoutes } from "./routes/health.js";
import { readinessRoutes } from "./routes/ready.js";
import { livenessRoutes } from "./routes/liveness.js";
import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js";
import { cloudControlMiddleware } from "./middleware/cloud-control.js";
import { runnerCallOutcomeMiddleware } from "./middleware/runner-call-outcome.js";
import { cloudRoutes } from "./routes/cloud.js";
import { companyRoutes } from "./routes/companies.js";
import { companySkillRoutes } from "./routes/company-skills.js";
import { companySkillPolicyRoutes } from "./routes/company-skill-policy.js";
import { inboxAgentPolicyRoutes } from "./routes/inbox-agent-policy.js";
import { builtInAgentRoutes } from "./routes/built-in-agents.js";
import { folderRoutes } from "./routes/folders.js";
import { summarySlotRoutes } from "./routes/summary-slots.js";
import { statusCardRoutes } from "./routes/status-cards.js";
import { teamsCatalogRoutes } from "./routes/teams-catalog.js";
import { agentRoutes } from "./routes/agents.js";
import { agentInteractionPermissionRoutes } from "./routes/agent-interaction-permissions.js";
import { agentPermissionGrantRoutes } from "./routes/agent-permission-grants.js";
import { resolveAgentRunnerTargetForCompany } from "./services/agent-runner-selection.js";
import type { SetupTokenSessionService } from "./services/setup-token-session.js";
import {
buildSetupTokenLoginTransport,
createProductionSetupTokenSandboxProvider,
createProductionSetupTokenCleanupStore,
createSetupTokenSecretWriter,
createWorkerBoundLoginPtyOpener,
} from "./services/setup-token-transport-binding.js";
import { environmentService } from "./services/environments.js";
import { environmentRuntimeService } from "./services/environment-runtime.js";
import { projectRoutes } from "./routes/projects.js";
import { issueRoutes } from "./routes/issues.js";
import { linkPreviewRoutes } from "./routes/link-previews.js";
import { issueTreeControlRoutes } from "./routes/issue-tree-control.js";
import { caseRoutes } from "./routes/cases.js";
import { caseHandoffRoutes } from "./routes/case-handoffs.js";
import { workflowRoutes } from "./routes/workflows.js";
import { fileResourceRoutes } from "./routes/file-resources.js";
import { createRemoteWorkspaceFiles } from "./services/remote-workspace-files.js";
import { createSavedWorkspaceLoader } from "./services/remote-workspaces.js";
import { createSavedWorkspaceArchiveOpener } from "./services/saved-workspace-archive.js";
import { routineRoutes } from "./routes/routines.js";
import { environmentRoutes } from "./routes/environments.js";
import { tailscaleConnectionRoutes } from "./routes/tailscale-connections.js";
import { executionWorkspaceRoutes } from "./routes/execution-workspaces.js";
import { goalRoutes } from "./routes/goals.js";
import { onboardingSeedRoutes } from "./routes/onboarding-seed.js";
import { boardChatRoutes } from "./routes/board-chat.js";
import { approvalRoutes } from "./routes/approvals.js";
import { secretRoutes } from "./routes/secrets.js";
import { toolAccessRoutes } from "./routes/tool-access.js";
import {
chatChannelRoutes,
chatWebhookRoutes,
} from "./routes/chat-channels.js";
import { smokeLabRoutes } from "./routes/smoke-lab.js";
import { costRoutes } from "./routes/costs.js";
import { agentCommentaryRoutes } from "./routes/agent-commentary.js";
import { agentChatMessageRoutes } from "./routes/agent-chat-messages.js";
import { activityRoutes } from "./routes/activity.js";
import { dashboardRoutes } from "./routes/dashboard.js";
import { attentionRoutes } from "./routes/attention.js";
import { decisionTrainingRoutes } from "./routes/decision-training.js";
import { decisionRoutes } from "./routes/decisions.js";
import { decisionQueueRoutes } from "./routes/decision-queues.js";
import type { DecisionServiceOptions } from "./services/decisions.js";
import { userProfileRoutes } from "./routes/user-profiles.js";
import { sidebarBadgeRoutes } from "./routes/sidebar-badges.js";
import { sidebarPreferenceRoutes } from "./routes/sidebar-preferences.js";
import { announcementRoutes } from "./routes/announcements.js";
import { serverVersion } from "./version.js";
import { workspacePreferenceRoutes } from "./routes/workspace-preferences.js";
import { primaryAgentRoutes } from "./routes/primary-agent.js";
import { resourceMembershipRoutes } from "./routes/resource-memberships.js";
import { inboxDismissalRoutes } from "./routes/inbox-dismissals.js";
import { instanceSettingsRoutes } from "./routes/instance-settings.js";
import { ephemeralEnrollmentRoutes } from "./routes/ephemeral-enrollment.js";
import { instanceSettingsService } from "./services/instance-settings.js";
import { openApiRoutes } from "./routes/openapi.js";
import {
instanceDatabaseBackupRoutes,
type InstanceDatabaseBackupService,
} from "./routes/instance-database-backups.js";
import { llmRoutes } from "./routes/llms.js";
import { authRoutes } from "./routes/auth.js";
import { resolvePostHogProxyMount } from "./routes/posthog-proxy.js";
import { assetRoutes } from "./routes/assets.js";
import { accessRoutes } from "./routes/access.js";
import { pluginRoutes } from "./routes/plugins.js";
import {
mcpGatewayProtocolRoutes,
toolGatewayRoutes,
} from "./routes/tool-gateway.js";
import {
connectionIntentBoardRoutes,
runtimeConnectionIntentRoutes,
} from "./routes/connection-intents.js";
import { adapterRoutes } from "./routes/adapters.js";
import { managedAgentProfileRoutes } from "./routes/managed-agent-profiles.js";
import { remoteAgentProfileRoutes } from "./routes/remote-agent-profiles.js";
import { pluginUiStaticRoutes } from "./routes/plugin-ui-static.js";
import { readBrandedStaticIndexHtml } from "./static-index-html.js";
import { staticUiCacheControl } from "./static-ui-cache.js";
import { staticUiCompression } from "./middleware/static-ui-compression.js";
import { applyUiBranding } from "./ui-branding.js";
import { logger } from "./middleware/logger.js";
import {
DEFAULT_LOCAL_PLUGIN_DIR,
pluginLoader,
type PluginLoader,
} from "./services/plugin-loader.js";
import {
SELF_HOSTED_AUTO_INSTALL_KEYS,
BUNDLED_PLUGIN_CATALOG,
ensureBundledPlugins,
resolveBundledCatalogRoot,
resolveBundledPluginInstalls,
resolveManagedBundledPluginKeys,
} from "./services/bundled-plugins.js";
import { createBundledPluginPackageResolver } from "./services/bundled-plugin-package.js";
import { readDistributionPluginCatalog, distributionPluginActivationGuard } from "./services/distribution-plugin-catalog.js";
import {
createPluginWorkerManager,
type PluginWorkerManager,
} from "./services/plugin-worker-manager.js";
import { createPluginJobScheduler } from "./services/plugin-job-scheduler.js";
import { pluginJobStore } from "./services/plugin-job-store.js";
import { createPluginToolDispatcher } from "./services/plugin-tool-dispatcher.js";
import { createToolGatewayService } from "./services/tool-gateway.js";
import { toolAccessService } from "./services/tool-access.js";
import { chatChannelService } from "./services/chat-channels.js";
import type { BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js";
import { nativeChatProgressWork } from "./services/chat-native-progress-work.js";
import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-work.js";
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
import { heartbeatService } from "./services/heartbeat.js";
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
import { loadPluginAppCatalog } from "./services/plugin-app-catalog.js";
import { setPluginConnectorWorkers } from "./services/plugin-connector-runtime.js";
import { completeLegacyPluginToolAccess } from "./services/plugin-connector-legacy-access.js";
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
import {
buildHostServices,
flushPluginLogBuffer,
} from "./services/plugin-host-services.js";
import { createPluginEventBus } from "./services/plugin-event-bus.js";
import { setPluginEventBus } from "./services/activity-log.js";
import { createPluginDevWatcher } from "./services/plugin-dev-watcher.js";
import { createPluginHostServiceCleanup } from "./services/plugin-host-service-cleanup.js";
import { pluginRegistryService } from "./services/plugin-registry.js";
import { createHostClientHandlers } from "@paperclipai/plugin-sdk";
import type { BetterAuthSessionResult } from "./auth/better-auth.js";
import { createCachedViteHtmlRenderer } from "./vite-html-renderer.js";
import {
DEFAULT_JSON_BODY_LIMIT,
PORTABLE_JSON_BODY_LIMIT,
} from "./http/body-limits.js";
import { COMPANY_IMPORT_API_PATH } from "./routes/company-import-paths.js";
import { apiCompression } from "./middleware/api-compression.js";
import { apiServerTiming } from "./middleware/api-server-timing.js";
import { instanceServerMetricsRoutes } from "./routes/instance-server-metrics.js";
import {
serverMetricsAgentPreHandler,
serverMetricsArrival,
serverMetricsEnabledFromEnv,
startServerMetrics,
stopServerMetrics,
} from "./services/server-metrics.js";
import { chatWebhookBodyParser } from "./middleware/chat-webhook-body.js";
import { postHogRequestContextMiddlewareForEnv } from "./services/posthog-direct/request-context.js";
import { createChatWebhookDiagnostics } from "./services/chat-webhook-diagnostics.js";
type UiMode = "none" | "static" | "vite-dev";
const FEEDBACK_EXPORT_FLUSH_INTERVAL_MS = 5_000;
const VITE_DEV_ASSET_PREFIXES = [
"/@fs/",
"/@id/",
"/@react-refresh",
"/@vite/",
"/assets/",
"/node_modules/",
"/src/",
];
const VITE_DEV_STATIC_PATHS = new Set([
"/apple-touch-icon.png",
"/favicon-16x16.png",
"/favicon-32x32.png",
"/favicon.ico",
"/favicon.svg",
"/site.webmanifest",
"/sw.js",
]);
export function isDatabaseConnectionUnavailableError(err: unknown): boolean {
const error = err as { code?: unknown; message?: unknown; cause?: unknown };
if (error?.code === "ECONNREFUSED") return true;
return Boolean(
error?.cause && isDatabaseConnectionUnavailableError(error.cause),
);
}
export function resolveViteHmrPort(serverPort: number): number {
return derivePaperclipViteHmrPort(serverPort);
}
export function resolveViteHmrHost(bindHost: string): string | undefined {
const normalized = bindHost.trim().toLowerCase();
if (
normalized === "0.0.0.0" ||
normalized === "::" ||
normalized === "127.0.0.1" ||
normalized === "::1" ||
normalized === "localhost"
)
return undefined;
return bindHost;
}
export function resolveViteHmrProtocol(
value: string | undefined,
): "ws" | "wss" | undefined {
if (!value) return undefined;
if (value === "ws" || value === "wss") return value;
throw new Error("PAPERCLIP_VITE_HMR_PROTOCOL must be ws or wss");
}
export function listenViteHmrServer(
server: HttpServer,
port: number,
bindHost: string,
): Promise<void> {
return new Promise((resolve, reject) => {
const onError = (error: Error) => {
server.off("listening", onListening);
reject(error);
};
const onListening = () => {
server.off("error", onError);
resolve();
};
server.once("error", onError);
server.once("listening", onListening);
server.listen(port, bindHost);
});
}
export function shouldServeViteDevHtml(req: ExpressRequest): boolean {
const pathname = req.path;
if (VITE_DEV_STATIC_PATHS.has(pathname)) return false;
if (VITE_DEV_ASSET_PREFIXES.some((prefix) => pathname.startsWith(prefix)))
return false;
return req.accepts(["html"]) === "html";
}
export function shouldEnablePrivateHostnameGuard(opts: {
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
}): boolean {
return (
opts.deploymentExposure === "private" &&
(opts.deploymentMode === "local_trusted" ||
opts.deploymentMode === "authenticated")
);
}
export function createManagedBundledPluginWorkerRecovery(input: {
managedBundledPluginKeys: readonly string[];
workerManager: Pick<
PluginWorkerManager,
"getWorker" | "isRunning" | "stopWorker"
>;
getLoader: () => Pick<PluginLoader, "loadSingle"> | null;
}): (plugin: { id: string; pluginKey: string }) => Promise<boolean> {
const recoverablePluginKeys = new Set(input.managedBundledPluginKeys);
const inFlightStarts = new Map<string, Promise<boolean>>();
// A failed attempt can leave behind the dead handle it registered (e.g. the
// worker process died during initialize, which kills the process without
// scheduling a restart). No pre-existing handle survives to a recovery
// attempt — recovery only starts when getWorker() was empty — so discarding
// the dead handle lets a later capability request retry instead of being
// blocked by the handle-presence gate until the process restarts. Handles
// in starting/running/backoff states belong to the worker manager's own
// lifecycle and are left alone.
const discardDeadRecoveryHandle = async (plugin: {
id: string;
pluginKey: string;
}) => {
const handle = input.workerManager.getWorker(plugin.id);
if (!handle || (handle.status !== "crashed" && handle.status !== "stopped"))
return;
try {
await input.workerManager.stopWorker(plugin.id);
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"failed to discard dead plugin worker handle after recovery failure",
);
}
};
return async (plugin) => {
if (!recoverablePluginKeys.has(plugin.pluginKey)) return false;
const inFlight = inFlightStarts.get(plugin.id);
if (inFlight) return inFlight;
const startPromise = (async () => {
if (input.workerManager.getWorker(plugin.id)) {
return input.workerManager.isRunning(plugin.id);
}
const loader = input.getLoader();
if (!loader) return false;
try {
const result = await loader.loadSingle(plugin.id, {
markErrorOnFailure: false,
});
if (
result.success === true ||
input.workerManager.isRunning(plugin.id)
) {
return true;
}
await discardDeadRecoveryHandle(plugin);
return false;
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"managed bundled plugin lazy worker recovery failed",
);
await discardDeadRecoveryHandle(plugin);
throw err;
}
})();
inFlightStarts.set(plugin.id, startPromise);
try {
return await startPromise;
} finally {
if (inFlightStarts.get(plugin.id) === startPromise) {
inFlightStarts.delete(plugin.id);
}
}
};
}
export async function createApp(
db: Db,
opts: {
cloudWarmStandby?: CloudWarmStandby;
uiMode: UiMode;
serverPort: number;
storageService: StorageService;
feedbackExportService?: {
hasPendingFeedbackTraces(): Promise<boolean>;
flushPendingFeedbackTraces(input?: {
companyId?: string;
traceId?: string;
limit?: number;
now?: Date;
signal?: AbortSignal;
}): Promise<unknown>;
};
databaseBackupService?: InstanceDatabaseBackupService;
prepareIdleDatabaseBackup?: () => Promise<boolean>;
idleDeadlineOwners?: IdleDeadlineOwners;
/** Filled with the chat service's Retry callback for background budget-resume delivery. */
budgetResumeDelivery?: BudgetResumeDelivery;
databaseBackupHealth?: InspectDatabaseBackupHealthOptions;
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
allowedHostnames: string[];
bindHost: string;
authPublicBaseUrl?: string;
chatWebhookPublicBaseUrl?: string;
authReady: boolean;
companyDeletionEnabled: boolean;
announcements?: { enabled: boolean; feedUrl: string };
instanceId?: string;
hostVersion?: string;
localPluginDir?: string;
pluginMigrationDb?: Db;
pluginWorkerManager?: PluginWorkerManager;
decisionServiceOptions: DecisionServiceOptions;
betterAuthHandler?: express.RequestHandler;
resolveSession?: (
req: ExpressRequest,
) => Promise<BetterAuthSessionResult | null>;
/**
* `plugins.autoInstall` from the managed config (PAPERCLIP_MANAGED_CONFIG).
* `null`/absent ⇒ self-hosted: only the built-in kubernetes bundle is
* ensured, exactly as before. A managed list is resolved against the
* bundled catalog fail-to-start (see services/bundled-plugins.ts).
*/
managedPluginAutoInstall?: readonly string[] | null;
/** Test override for the bundled plugin catalog root. */
bundledPluginCatalogRoot?: string;
serverMetricsEnabled?: boolean;
},
) {
const app = express();
const serverMetrics = (opts.serverMetricsEnabled ?? serverMetricsEnabledFromEnv(process.env)) ? startServerMetrics(() => dedicatedDbConnectionStats(db)) : null;
if (serverMetrics) app.use("/api", serverMetricsArrival());
// First, so the reported server time includes admission and database waits.
app.use("/api", apiServerTiming());
app.use(databaseOperationMiddleware);
app.use(idleAdmissionMiddleware);
const staticUi = express.Router();
app.locals.paperclipDb = db;
const isWarmStandby = opts.cloudWarmStandby ?? (() => false);
const health = healthRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authReady: opts.authReady,
companyDeletionEnabled: opts.companyDeletionEnabled,
databaseBackupHealth: opts.databaseBackupHealth,
isWarmStandby,
});
const captureRawBody = (
req: express.Request,
_res: express.Response,
buf: Buffer,
) => {
(req as unknown as { rawBody: Buffer }).rawBody = buf;
};
// Respect the operator's `TRUST_PROXY` env var (see middleware/trust-proxy.ts).
// Default is unset → Express trusts nothing, which is the only safe choice
// when the server may be reachable without a known reverse proxy in front.
applyTrustProxy(app, parseTrustProxyEnv(process.env.TRUST_PROXY));
// Browser analytics on Paperclip Cloud only. Mounted before every body
// parser so PostHog payloads stream through unread; see routes/posthog-proxy.ts.
const posthogProxy = resolvePostHogProxyMount(process.env);
if (posthogProxy) app.use(posthogProxy.path, posthogProxy.router);
app.use(
COMPANY_IMPORT_API_PATH,
express.json({
limit: PORTABLE_JSON_BODY_LIMIT,
verify: captureRawBody,
}),
);
// Chat providers sign the exact request bytes. Capture every webhook media
// type before the global JSON parser so JSON events and form-encoded action
// callbacks are verified against the provider's original body.
app.use(
"/api/chat-webhooks",
createChatWebhookDiagnostics(),
chatWebhookBodyParser,
);
let dotPrincipalOAuth: ReturnType<typeof createPublicMcpOAuth> | null = null;
const dotPrincipalHttp = createDotPrincipalHttp(() => dotPrincipalOAuth);
app.use(dotPrincipalHttp.bodyParser);
const jsonBodyParser = express.json({ limit: DEFAULT_JSON_BODY_LIMIT, verify: captureRawBody });
// File tickets authenticate before parsing bytes; JSON attachments must remain bytes too.
app.use((req, res, next) => (dotPrincipalHttp.matchesPost(req) || ["/mcp/files/upload", "/mcp/runner/files/upload"].includes(req.path)) ? next() : jsonBodyParser(req, res, next));
app.use("/api", apiCompression());
app.use(httpLogger);
// Paperclip Cloud only: link server analytics events to the browser session
// that caused them. Self-hosted servers never mount this, so posthog-js
// tracing headers are ignored there.
const postHogRequestContext = postHogRequestContextMiddlewareForEnv();
if (postHogRequestContext) app.use(postHogRequestContext);
const privateHostnameGateEnabled = shouldEnablePrivateHostnameGuard({
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
});
const privateHostnameAllowSet = resolvePrivateHostnameAllowSet({
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
});
app.use(
privateHostnameGuard({
enabled: privateHostnameGateEnabled,
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
}),
);
let mcpConfig: ReturnType<typeof publicMcpConfig> = null;
try { mcpConfig = publicMcpConfig(process.env, opts.authPublicBaseUrl); }
catch { logger.warn("Assistant connections require an HTTPS public URL (HTTP loopback is allowed for development)."); }
const publicMcpOAuth = mcpConfig ? createPublicMcpOAuth(db, mcpConfig) : null;
const publicMcpIngress = Router();
publicMcpIngress.use(dotPrincipalHttp.router);
app.use(cloudRuntimeIdentityMiddleware(db));
// Routine probes stop before session/actor resolution, even after claim.
// Keep hostname validation and the signed-bootstrap endpoint guard above.
app.use("/api/liveness", livenessRoutes());
app.use("/api/ready", readinessRoutes());
// A signed claim above commits identity before any normal request can seed
// company data. Unclaimed probes bypass session resolution as well as SQL.
app.use(cloudWarmStandbyMiddleware(isWarmStandby, health, staticUi));
const workerManager = opts.pluginWorkerManager ?? createPluginWorkerManager();
app.use("/api/customer-success/v1", customerSuccessRoutes(db, opts.storageService, {
pluginWorkerManager: workerManager,
}));
app.use(museTransportRoutes(db, opts.authPublicBaseUrl));
app.use(publicMcpIngress);
// Connection-intent tools carry their own short-lived, run-bound bearer and
// must be reachable by remote adapters that intentionally do not receive an
// agent API key. Every request revalidates the active heartbeat row.
app.use(runtimeConnectionIntentRoutes(db));
app.use(nativeRunMcpRoutes());
app.use(runnerAttachmentDownloadRoutes(db, opts.storageService));
const hostServicesDisposers = new Map<string, () => void>();
let lifecyclePluginsReady = false;
const agentLifecycle = startAgentLifecycle(db, createLifecycleDriver(db, workerManager), () => lifecyclePluginsReady && !isWarmStandby() && !isIdleTaskDrainActive());
const connectionIntentHeartbeat = heartbeatService(db, {
pluginWorkerManager: workerManager,
});
const chatChannels = chatChannelService(db, {
allowLocalVoiceBoard: opts.deploymentMode === "local_trusted",
deferWebhookProcessing: true,
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
githubWizardOrigin: opts.deploymentMode === "local_trusted"
&& ["127.0.0.1", "localhost", "::1"].includes(opts.bindHost ?? "")
&& Number.isInteger(opts.serverPort) && opts.serverPort! > 0
? `http://${opts.bindHost === "::1" ? "[::1]" : opts.bindHost}:${opts.serverPort}` : null,
webhookPublicBaseUrl: opts.chatWebhookPublicBaseUrl,
resolveNativeQuestion: (interaction) =>
deliverNativeQuestionResponse(db, interaction),
storage: opts.storageService,
});
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
// A restart's automatic Retry of a connected conversation needs the same callback.
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
// Voice capabilities are verified by the signed callback route, not by
// board/agent bearer authentication. Mount before actorMiddleware so its
// per-session Bearer cannot be mistaken for an agent API key.
app.use(voiceWebhookRoutes(db, chatChannels.voice));
app.use(
actorMiddleware(db, {
deploymentMode: opts.deploymentMode,
resolveSession: opts.resolveSession,
}),
);
// After the actor middleware on purpose: a valid Cloud control assertion
// REPLACES whatever actor the request otherwise resolved to, and only on
// the one endpoint it authorizes (see the middleware for the contract).
app.use(cloudControlMiddleware());
app.use(authorizationReadScope);
app.use(runnerCallOutcomeMiddleware(db));
app.use(connectionEventWakeRoutes(db));
app.use(skillAccessRequestContext);
app.use("/api/auth", authRoutes(db));
if (opts.betterAuthHandler) {
app.all("/api/auth/{*authPath}", opts.betterAuthHandler);
}
app.use(llmRoutes(db));
// Provider-authenticated ingress is intentionally outside the board
// mutation guard. The Chat SDK adapter verifies the provider signature
// before Paperclip persists or acts on any event.
const emailChannels = emailChannelService(db, {
isReconciliationEnabled: () => !isWarmStandby(),
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
heartbeat: connectionIntentHeartbeat,
storage: opts.storageService,
publicBaseUrl: opts.chatWebhookPublicBaseUrl ?? opts.authPublicBaseUrl,
});
app.use(emailWebhookRoutes(emailChannels));
app.locals.fastResponses = fastResponseService(db, {
authorizeExternal: async (tx, request) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
return endpoint?.provider === "agentmail" ? emailChannels.authorizeFastResponse(tx, request) : chatChannels.authorizeFastResponse(tx, request);
},
publishExternal: async (tx, request, commentId, text) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
if (endpoint?.provider !== "agentmail") return false;
await emailChannels.publishFastResponse(tx, request, commentId, text);
return true;
},
budgetHooks: { cancelWorkForScope: connectionIntentHeartbeat.cancelBudgetScopeWork },
});
app.use(chatWebhookRoutes(chatChannels));
// The instance validates single-use registration state and its trusted
// current origin. These exact callback routes are the public setup returns.
app.get("/api/chat-github/manifest/callback", async (req, res) => {
res.set("Cache-Control", "no-store");
res.set("Referrer-Policy", "no-referrer");
const redirect = await chatChannels.githubWizard.directCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor);
res.redirect(303, redirect);
});
app.get("/api/chat-github/cloud/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.cloudCallback(String(req.query.state ?? ""), String(req.query.registration ?? ""), typeof req.query.claim === "string" ? req.query.claim : undefined, req.actor));
});
app.get("/api/chat-github/identity/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.identityCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor));
});
const managedAutoInstallKeys = resolveManagedBundledPluginKeys(opts.managedPluginAutoInstall);
const bundledCatalogRoot =
opts.bundledPluginCatalogRoot ?? resolveBundledCatalogRoot(process.env);
const distributionPlugins = readDistributionPluginCatalog(bundledCatalogRoot, BUNDLED_PLUGIN_CATALOG);
const bundledPluginInstalls = resolveBundledPluginInstalls(
managedAutoInstallKeys ?? SELF_HOSTED_AUTO_INSTALL_KEYS,
{
catalogRoot: bundledCatalogRoot,
env: process.env,
enforceCatalogRoot: managedAutoInstallKeys !== null,
distributionPlugins,
},
);
const bundledPackageResolver = managedAutoInstallKeys === null ? undefined : createBundledPluginPackageResolver(bundledPluginInstalls);
const managedBundledPluginKeys =
managedAutoInstallKeys !== null
? bundledPluginInstalls.map((install) => install.pluginKey)
: [];
let runtimePluginLoader: Pick<PluginLoader, "loadSingle"> | null = null;
// A sibling process can install a managed bundled plugin while this process
// skips the mid-install row, then finish the row after this process's
// loadAll() pass. The capabilities route may recover only those managed
// bundles by starting their ready-but-unstarted worker lazily.
const recoverManagedBundledPluginWorker =
managedAutoInstallKeys !== null
? createManagedBundledPluginWorkerRecovery({
managedBundledPluginKeys,
workerManager,
getLoader: () => runtimePluginLoader,
})
: undefined;
// Mount API routes
const api = Router();
const agentAvatars = agentAvatarRoutes();
api.use(agentAvatars.router);
api.use(boardMutationGuard());
api.use("/health", health);
api.use(openApiRoutes());
api.use("/cloud", cloudRoutes());
api.use("/companies", companyRoutes(db, opts.storageService));
api.use(llmRoutes(db));
api.use(folderRoutes(db));
api.use(companySkillRoutes(db, { pluginWorkerManager: workerManager }));
api.use(companySkillPolicyRoutes(db));
api.use(inboxAgentPolicyRoutes(db));
api.use(agentCommentaryRoutes(db));
api.use(agentChatMessageRoutes(db));
api.use(builtInAgentRoutes(db));
api.use(summarySlotRoutes(db));
api.use(statusCardRoutes(db));
api.use(teamsCatalogRoutes(db));
// The setup-token login session service. The router builds it and hands it
// back through the callback below, so the shutdown hook can cancel every live
// session (SR-4).
let setupTokenLoginService: SetupTokenSessionService | null = null;
// The dedicated proxy IP or CIDR allowlist for the confidential setup-token
// login responses (SR-7). The global `TRUST_PROXY` setting does not satisfy
// the guard; an operator sets this allowlist to the real TLS-terminating
// proxy addresses. An empty value keeps the confidential responses on direct
// TLS (or a `local_trusted` loopback peer) only.
const setupTokenLoginProxyAllowlist = (
process.env.CLAUDE_LOGIN_TRUSTED_PROXIES ?? ""
)
.split(",")
.map((entry) => entry.trim())
.filter((entry) => entry.length > 0);
// The explicit operator declaration that a platform edge terminates TLS for
// every client request (SR-7). This complements the allowlist for managed
// platforms (Railway, Render, Fly, and the like) where the app socket is
// always plain HTTP and the edge-proxy peer addresses are not stable or
// documented, so `CLAUDE_LOGIN_TRUSTED_PROXIES` cannot express them. It is a
// dedicated, single-purpose setting; the guard still never reads the global
// `TRUST_PROXY` value.
const setupTokenLoginEdgeTlsTerminated = /^(1|true|yes|on)$/i.test(
(process.env.CLAUDE_LOGIN_EDGE_TLS_TERMINATED ?? "").trim(),
);
// Bind the production setup-token login transport. It carries the live lease
// manager, the login-process factory over the sandbox pseudo-terminal, and the
// durable cleanup store. The factory passes only the fixed command
// `CLAUDE_SETUP_TOKEN_COMMAND`; it never reads a command from a
// route, a request body, or an adapter configuration. The durable store and the
// startup reaper are live now, so a restart reaps a leftover lease.
//
// The live sandbox pseudo-terminal opener binds inside the sandbox provider
// worker, so the server process does not hold the raw sandbox process. The
// opener drives the worker through the plugin worker manager route gate.
// The manager mints a host-owned route identifier, permits one
// active credential pseudo-terminal per worker, binds the worker session
// identifier one time for output only, and terminalizes the route on every open
// failure path. With the opener supplied, the provider acquires a lease and the
// start route drives a live login instead of the fixed 503.
const setupTokenLoginTransport = buildSetupTokenLoginTransport({
sandbox: createProductionSetupTokenSandboxProvider({
db,
environments: environmentService(db),
environmentRuntime: environmentRuntimeService(db, {
pluginWorkerManager: workerManager,
}),
openLivePtySession: createWorkerBoundLoginPtyOpener({
workerManager,
environments: environmentService(db),
log: (line) => logger.info(line),
}),
log: (line) => logger.info(line),
}),
store: createProductionSetupTokenCleanupStore(db),
// Bind the atomic credential-claim writer, so a completed login transitions
// the durable row to `stored` and stores the minted token in one control-plane
// transaction. The writer reads the company and the owner only from the
// immutable session scope. The confirm-replacement flow owns rotation. Without
// this writer the router falls back to the deferred, fail-closed 503 and never
// stores the token.
completeCredential: createSetupTokenSecretWriter({ db }),
// Forward the login runner diagnostic lines to the server logger. The
// runner is the sole producer, and every line is a fixed, non-secret
// literal. Without this sink the diagnostics fall back to a no-op in
// production, so a failed login leaves no log trail.
log: (line) => logger.info(line),
});
api.use(
agentRoutes(db, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
deploymentMode: opts.deploymentMode,
confidentialProxyAllowlist: setupTokenLoginProxyAllowlist,
confidentialEdgeTlsTerminated: setupTokenLoginEdgeTlsTerminated,
setupTokenLogin: setupTokenLoginTransport,
onSetupTokenLoginService: (service) => {
// Capture the service, so the graceful-shutdown hook cancels every live
// session and releases each lease. The standalone scheduled reaper owns
// the startup and interval lease cleanup now (SR-4).
setupTokenLoginService = service;
},
}),
);
api.use(agentPermissionGrantRoutes(db));
api.use(agentInteractionPermissionRoutes(db));
api.use(assetRoutes(db, opts.storageService));
api.use(agentProfileAvatarRoutes(db, opts.storageService));
api.use(projectToolRoutes(db));
api.use(taskWorkspaceRoutes(db));
api.use(projectRoutes(db));
api.use(caseRoutes(db, opts.storageService));
api.use(caseHandoffRoutes(db));
api.use(workflowRoutes(db));
api.use(issueTreeControlRoutes(db, { pluginWorkerManager: workerManager }));
api.use(fileResourceRoutes(db, { remoteFiles: createRemoteWorkspaceFiles(db, {
environmentRuntime: environmentRuntimeService(db, { pluginWorkerManager: workerManager }),
loadSavedWorkspace: createSavedWorkspaceLoader(db, opts.storageService),
openSavedWorkspaceArchive: createSavedWorkspaceArchiveOpener(db, opts.storageService),
}) }));
api.use(routineRoutes(db, { pluginWorkerManager: workerManager }));
api.use(tailscaleConnectionRoutes(db));
api.use(
environmentRoutes(db, {
pluginWorkerManager: workerManager,
retryPendingEnvironmentCleanup: input => connectionIntentHeartbeat.sweepPendingCleanupLeases({ environmentRetry: input }),
recoverMissingPluginWorker: recoverManagedBundledPluginWorker
? {
pluginKeys: managedBundledPluginKeys,
startWorker: recoverManagedBundledPluginWorker,
}
: undefined,
}),
);
api.use(executionWorkspaceRoutes(db, { pluginWorkerManager: workerManager }));
api.use(emailRoutes(db, emailChannels));
api.use(goalRoutes(db));
api.use(onboardingSeedRoutes(db));
api.use(boardChatRoutes(db, { deploymentMode: opts.deploymentMode }));
api.use(voiceSessionRoutes(db, chatChannels.voice));
api.use(approvalRoutes(db, { pluginWorkerManager: workerManager }));
api.use(secretRoutes(db));
api.use(managedAgentProfileRoutes(db));
api.use(remoteAgentProfileRoutes(db));
api.use(
chatChannelRoutes(db, {
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
storage: opts.storageService,
service: chatChannels,
}),
);
const trustedLocalStdioRuntimeHost =
process.env.PAPERCLIP_TRUSTED_MCP_RUNTIME_HOST ??
process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ??
null;
api.use(costRoutes(db, { pluginWorkerManager: workerManager }));
api.use(activityRoutes(db));
api.use(dashboardRoutes(db));
api.use(attentionRoutes(db, {
pluginWorkerManager: workerManager,
decisionServiceOptions: opts.decisionServiceOptions,
}));
api.use(decisionTrainingRoutes(db));
api.use(decisionRoutes(db, opts.decisionServiceOptions));
api.use(decisionQueueRoutes(db));
api.use(userProfileRoutes(db));
api.use(sidebarBadgeRoutes(db));
api.use(sidebarPreferenceRoutes(db));
api.use(announcementRoutes(db, { ...opts.announcements, version: opts.hostVersion ?? serverVersion }));
api.use(resourceMembershipRoutes(db));
api.use(primaryAgentRoutes(db));
api.use(workspacePreferenceRoutes(db));
api.use(inboxDismissalRoutes(db));
api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup, opts.idleDeadlineOwners,
input => resourceLifecycleDelivery.prepareForIdleSleep(input)));
api.use(ephemeralEnrollmentRoutes(db));
if (opts.databaseBackupService) {
api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService));
}
if (serverMetrics) api.use(instanceServerMetricsRoutes(serverMetrics));
const pluginRegistry = pluginRegistryService(db);
const eventBus = createPluginEventBus();
setPluginEventBus(eventBus);
const jobStore = pluginJobStore(db);
const lifecycle = pluginLifecycleManager(db, { workerManager });
// Seed the plugin-provided connector lookups; readers refresh them again
// from the plugins table whenever they need a current answer.
setPluginConnectorWorkers(workerManager);
void loadPluginAppCatalog(db)
.catch((error) => logger.warn({ err: error }, "Could not load plugin connector catalog"));
const scheduler = createPluginJobScheduler({
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
db,
jobStore,
workerManager,
});
const toolDispatcher = createPluginToolDispatcher({
workerManager,
lifecycleManager: lifecycle,
db,
});
const gatewayOAuthAccess = toolAccessService(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
});
const toolActionDeliveries = toolActionDeliveryService(db, heartbeatService(db, { pluginWorkerManager: workerManager }));
const toolGateway = createToolGatewayService(db, {
budgetHooks: { cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
onToolActionSettled: (id) => toolActionDeliveries.deliver(id),
pluginToolDispatcher: toolDispatcher,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
oauthGrantRefresher: (input) =>
gatewayOAuthAccess.refreshOAuthGrantCredentials(input),
});
// Issue routes are intentionally mounted after the gateway is constructed because
// issue approval endpoints delegate to it. The intervening routers use distinct
// route prefixes, so this dependency does not change issue-route precedence.
api.use(issueRoutes(db, opts.storageService, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
approveToolActionRequest: (input) => toolGateway.approveActionRequest(input),
declineToolActionRequest: (input) => toolGateway.declineActionRequest(input),
}));
api.use(slackToolRoutes(db, opts.authPublicBaseUrl));
app.locals.toolGateway = toolGateway;
registerAssignedMcpGateway(db, toolGateway);
app.locals.toolActionDeliveries = toolActionDeliveries;
app.use(mcpGatewayProtocolRoutes(toolGateway));
api.use(decisionModelRoutes(db));
api.use(agentObservabilityRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure }));
api.use(fastResponseRoutes(db));
api.use(linkPreviewRoutes(db));
api.use(promptInjectionPreventionRoutes(db));
api.use(aiConnectionRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure, trustedLocalStdioRuntimeHost, pluginWorkerManager: workerManager }));
api.use(
toolAccessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authPublicBaseUrl: opts.authPublicBaseUrl,
trustedLocalStdioRuntimeHost,
toolGateway,
connectionIntentHeartbeat,
}),
);
api.use(connectionIntentBoardRoutes(db, connectionIntentHeartbeat));
api.use(
smokeLabRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
}),
);
const jobCoordinator = createPluginJobCoordinator({
db,
lifecycle,
scheduler,
jobStore,
});
const hostServiceCleanup = createPluginHostServiceCleanup(
lifecycle,
hostServicesDisposers,
);
let viteHtmlRenderer: ReturnType<typeof createCachedViteHtmlRenderer> | null =
null;
let viteDevServer: { close(): Promise<void> } | null = null;
let viteHmrServer: HttpServer | null = null;
const loader = pluginLoader(
db,
{
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
migrationDb: opts.pluginMigrationDb,
assertPackageActivation: bundledPackageResolver?.assertActivation ?? distributionPluginActivationGuard(bundledCatalogRoot, distributionPlugins, managedAutoInstallKeys),
resolvePackageRoot: bundledPackageResolver?.resolve,
prebuiltOnly: managedAutoInstallKeys !== null,
enableLocalFilesystem: managedAutoInstallKeys === null,
enableNpmDiscovery: managedAutoInstallKeys === null,
},
{
workerManager,
eventBus,
jobScheduler: scheduler,
jobStore,
toolDispatcher,
lifecycleManager: lifecycle,
instanceInfo: {
instanceId: opts.instanceId ?? "default",
hostVersion: opts.hostVersion ?? serverVersion,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
},
buildHostHandlers: (pluginId, manifest) => {
const notifyWorker = (method: string, params: unknown) => {
const handle = workerManager.getWorker(pluginId);
if (handle) handle.notify(method, params);
};
const services = buildHostServices(
db,
pluginId,
manifest.id,
eventBus,
notifyWorker,
{
pluginWorkerManager: workerManager,
manifest,
},
);
hostServicesDisposers.set(pluginId, () => services.dispose());
return createHostClientHandlers({
pluginId,
capabilities: manifest.capabilities,
services,
});
},
},
);
runtimePluginLoader = loader;
workerManager.setActivationTimeoutRecovery?.(plugin => loader.requestActivationTimeoutRecovery(plugin));
const browserUse = browserUseService(db, undefined,
{ cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
(session, run) => toolGateway.browserUseSessionAuthorized({ ...session, runId: run.heartbeatRunId, invocationId: run.invocationId }));
api.use(browserUseRoutes(db, browserUse));
api.use(toolGatewayRoutes(db, toolGateway));
api.use(mcpAppsRoutes(db, toolGateway));
api.use(mcpSkillsRoutes(db, toolGateway));
api.use(
pluginRoutes(
db,
loader,
{ scheduler, jobStore },
{ workerManager },
{ toolDispatcher },
{ workerManager },
{ toolGateway },
),
);
api.use(
adapterRoutes({
db,
resolveRunnerTarget: (companyId, environmentId) => resolveAgentRunnerTargetForCompany(db, companyId, environmentId),
getNativeRunnerEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableNativeRunner === true,
getOpenAiDotEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableOpenAiDot === true,
}),
);
api.use(
accessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
bindHost: opts.bindHost,
allowedHostnames: opts.allowedHostnames,
authPublicBaseUrl: opts.authPublicBaseUrl,
}),
);
let publicMcpEvents: PublicMcpEvents | null = null;
const grokDispatcher = createGrokRoutineDispatcher(db, () => !isWarmStandby() && !isIdleTaskDrainActive());
let dotMcpEvents: PublicMcpEvents | null = null;
if (publicMcpOAuth) {
const dispatch = createMcpApiDispatch(api, app.locals);
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
});
const transfers = createPublicMcpTransfers(db, publicMcpOAuth, dispatch, opts.storageService);
publicMcpIngress.use(transfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(publicMcpOAuth, createPublicMcpExecutor(db, publicMcpOAuth, dispatch, transfers), publicMcpEvents));
const dotOAuth = createPublicMcpOAuth(db, { ...publicMcpOAuth.config, resource: publicMcpOAuth.config.origin + "/mcp/runner" });
dotPrincipalOAuth = dotOAuth;
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive() });
const grokTransfers = createPublicMcpTransfers(db, dotOAuth, dispatch, opts.storageService, { pathPrefix: "/mcp/runner/files",
agentSession: { authorize: (principal, sessionId, taskId, write, database) => dotRunnerBroker(database ?? db).authorizeFileSession(principal, sessionId, taskId, write) } });
publicMcpIngress.use(grokTransfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(dotOAuth, createPublicMcpExecutor(db, dotOAuth, dispatch), dotMcpEvents, createDotRunnerMcpTools(db, connectionIntentHeartbeat, grokTransfers)));
api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth));
api.use(dotRunnerRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
api.use(grokBotRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
}
api.use(museBoardRoutes(db, opts.authPublicBaseUrl));
api.use(museQualificationRoutes(db));
if (serverMetrics) app.use("/api", serverMetricsAgentPreHandler(serverMetrics));
app.use("/api", api);
app.use("/api", (_req, res) => {
res.status(404).json({ error: "API route not found" });
});
app.use(
pluginUiStaticRoutes(db, {
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
resolvePackageRoot: bundledPackageResolver?.resolve,
}),
);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
if (opts.uiMode === "static") {
// Try published location first (server/ui-dist/), then monorepo dev location (../../ui/dist)
const candidates = [
path.resolve(__dirname, "../ui-dist"),
path.resolve(__dirname, "../../ui/dist"),
];
const uiDist = candidates.find((p) =>
fs.existsSync(path.join(p, "index.html")),
);
if (uiDist) {
staticUi.use(staticUiCompression());
// Hashed asset files (Vite emits them under /assets/<name>.<hash>.<ext>)
// never change once built, so they can be cached aggressively.
staticUi.use(
"/assets",
staticUiFiles(path.join(uiDist, "assets"), {
maxAge: "1y",
immutable: true,
}),
);
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
// Publish while the routes below are built; the outcome is awaited
// before the first background work starts. The promise settles to the
// failure instead of rejecting, so it is never an unhandled rejection.
app.locals.retainedUiAssetsPublication = trackIdleWork(publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"))
.then(() => null, (error: unknown) => ({ error })));
staticUi.use("/assets", retained.serve);
}
// Serve root/index through the same runtime HTML transform as SPA routes.
staticUi.get(["/", "/index.html"], (_req, res) => {
res.type("html").set("Cache-Control", "no-cache").send(readBrandedStaticIndexHtml(uiDist));
});
// Non-hashed static files (favicon.ico, manifest, robots.txt, etc.):
// short cache so operators who swap them out see the new version
// reasonably fast, with must-revalidate overrides for index.html and
// sw.js (see staticUiCacheControl for why those two).
staticUi.use(
staticUiFiles(uiDist, {
maxAge: "1h",
setHeaders(res, filePath) {
const override = staticUiCacheControl(filePath);
if (override) {
res.set("Cache-Control", override);
}
},
}),
);
// SPA fallback. Only for non-asset routes — if the browser asks for
// /assets/something.js that doesn't exist, we must NOT serve the HTML
// shell: the browser would try to load it as a JavaScript module, fail
// with a MIME-type error, and cache that broken response. Return 404
// instead. The index.html response itself is no-cache so a subsequent
// deploy's updated asset hashes are picked up on next load.
staticUi.get(/.*/, (req, res) => {
if (req.path.startsWith("/assets/")) {
res.status(404).end();
return;
}
res
.status(200)
.set("Content-Type", "text/html")
.set("Cache-Control", "no-cache")
.end(readBrandedStaticIndexHtml(uiDist));
});
} else {
console.warn("[paperclip] UI dist not found; running in API-only mode");
}
if (process.env.PAPERCLIP_MANAGED_RUNTIME_EXPOSURE === "tailscale_https") {
// The managed-runtime supervisor waits for the app port AND its derived
// Vite HMR companion port to bind before publishing the service. Static
// mode has no Vite, so bind the same placeholder listener dev mode uses
// or the supervisor kills a healthy server at the readiness deadline
// (PAP-18043).
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
await listenViteHmrServer(
hmrServer,
resolveViteHmrPort(opts.serverPort),
opts.bindHost,
);
viteHmrServer = hmrServer;
}
}
if (opts.uiMode === "vite-dev") {
const uiRoot = path.resolve(__dirname, "../../ui");
const publicUiRoot = path.resolve(uiRoot, "public");
const hmrPort = resolveViteHmrPort(opts.serverPort);
const hmrHost = resolveViteHmrHost(opts.bindHost);
const hmrProtocol = resolveViteHmrProtocol(
process.env.PAPERCLIP_VITE_HMR_PROTOCOL,
);
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
const { createServer: createViteServer } = await import("vite");
const configuredViteCacheDir = process.env.PAPERCLIP_VITE_CACHE_DIR?.trim();
const vite = await createViteServer({
root: uiRoot,
...(configuredViteCacheDir
? { cacheDir: path.resolve(configuredViteCacheDir) }
: {}),
appType: "custom",
// Vite otherwise discovers every HTML entry below the UI root. Generated
// Storybook output can reference dependencies that are intentionally not
// part of the application install, poisoning a clean embedded dev-server
// cache before the browser opens. The embedded UI has one real entry.
optimizeDeps: { entries: [path.resolve(uiRoot, "index.html")] },
server: {
// Listener binding and browser HMR hostname are deliberately separate:
// exposed branch runtimes stay loopback-only while the browser uses the
// current MagicDNS hostname through the broker's HTTPS listener.
host: opts.bindHost,
middlewareMode: true,
hmr: {
server: hmrServer,
...(hmrHost ? { host: hmrHost } : {}),
...(hmrProtocol ? { protocol: hmrProtocol } : {}),
port: hmrPort,
clientPort: hmrPort,
},
allowedHosts: privateHostnameGateEnabled
? Array.from(privateHostnameAllowSet)
: undefined,
},
});
try {
await listenViteHmrServer(hmrServer, hmrPort, opts.bindHost);
} catch (error) {
await vite.close();
throw error;
}
viteDevServer = vite;
viteHmrServer = hmrServer;
viteHtmlRenderer = createCachedViteHtmlRenderer({
vite,
uiRoot,
brandHtml: applyUiBranding,
});
const renderViteHtml = viteHtmlRenderer;
if (fs.existsSync(publicUiRoot)) {
staticUi.use(staticUiFiles(publicUiRoot, { index: false }));
}
staticUi.get(/.*/, async (req, res, next) => {
if (!shouldServeViteDevHtml(req)) {
next();
return;
}
try {
const html = await renderViteHtml.render(req.originalUrl);
res.status(200).set({ "Content-Type": "text/html" }).end(html);
} catch (err) {
next(err);
}
});
staticUi.use(vite.middlewares);
}
app.use(staticUi);
app.use(errorHandler);
// A build whose asset names collide with different published bytes refuses
// to start, so the outcome is known before any job, queue, or run starts.
const retainedUiAssetsPublication = await (app.locals.retainedUiAssetsPublication as Promise<{ error: unknown } | null> | undefined);
if (retainedUiAssetsPublication) throw retainedUiAssetsPublication.error;
jobCoordinator.start();
const deliveryWork = createDeliveryWorkCoordinator({
owner: db,
canRun: () => !isWarmStandby() && !isIdleTaskDrainActive(),
canReconcile: () => !isWarmStandby(),
onError: (err, queue) => logger.error({ err, queue }, "Delivery reconciliation failed"),
});
app.locals.deliveryWork = deliveryWork;
await scheduler.start(deliveryWork);
const agentLifecycleDelivery = deliveryWork.register(DELIVERY_QUEUES.agentLifecycle, {
retryMs: 1_000,
run: () => agentLifecycle.sweep(),
hasPending: () => agentLifecycle.hasPending(),
nextRunAt: () => agentLifecycle.nextRunAt(),
});
agentLifecycle.setWake(agentLifecycleDelivery.wake);
const clientMirror = startMcpClientMirrorSync(db, deliveryWork, mcpConfig?.origin,{authorizationOrigin:mcpConfig?.authorizationOrigin});
const unregisterMcpOwners = registerIdleMcpOwners(db, { config: mcpConfig, personal: publicMcpEvents, dot: dotMcpEvents, clientMirrorInstanceId:clientMirror.instanceId });
startMcpRevocationIndexSync(db, deliveryWork);
startMcpIngressHandoffExpiry(db, deliveryWork);
publicMcpEvents?.start(deliveryWork);
dotMcpEvents?.start(deliveryWork);
if (publicMcpOAuth) grokDispatcher.start(deliveryWork);
const stopMuseMaintenance = startMuseMaintenance(db, deliveryWork, async () => {
const state = await connectionIntentHeartbeat.resolveSchedulingSuppression(true);
return !state.suppressed || state.reason === "task_drain";
});
const resourceLifecycleDelivery = registerPluginLifecycleDelivery(deliveryWork, db, workerManager, lifecycle);
await resourceLifecycleDelivery.ready;
if (opts.feedbackExportService) {
deliveryWork.register(DELIVERY_QUEUES.feedback, {
retryMs: FEEDBACK_EXPORT_FLUSH_INTERVAL_MS,
run: (signal) => opts.feedbackExportService!.flushPendingFeedbackTraces({ signal }),
hasPending: () => opts.feedbackExportService!.hasPendingFeedbackTraces(),
});
}
registerChatDeliveryWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
registerChatActionWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
chatChannels.voice.registerReportWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerReplyWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerCleanupWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerRetentionWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
registerGitHubReviewProjectionWork(db, deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubReviewCheckWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubTerminalResponseWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubAcknowledgementWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatFailedRetryNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatRunMilestoneWork(db, opts.authPublicBaseUrl).register(
deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive(),
);
emailChannels.start();
chatChannels.registerGitHubWebhookRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
const importSpoolCleanup = registerImportSpoolCleanupWork(db, deliveryWork,
resolveDefaultImportTransferSpoolRoot(), () => !isWarmStandby() && !isIdleTaskDrainActive());
registerBrowserUseCleanup(deliveryWork, browserUse, () => !isIdleTaskDrainActive());
void companyTransferRunService
.recoverStrandedApplyingRuns(db)
.then((recovered) => {
if (recovered.length > 0) {
logger.warn(
{ count: recovered.length, runIds: recovered },
"failed company transfer runs stranded in applying by a restart",
);
}
})
.catch((err) => {
logger.error({ err }, "stranded company transfer apply recovery failed");
});
void toolDispatcher.initialize().catch((err) => {
logger.error({ err }, "Failed to initialize plugin tool dispatcher");
});
const devWatcher = managedAutoInstallKeys !== null ? null : createPluginDevWatcher(
lifecycle,
async (pluginId) =>
(await pluginRegistry.getById(pluginId))?.packagePath ?? null,
undefined,
{ unwatchedRoots: [bundledCatalogRoot] },
);
// Auto-provision bundled plugins so their providers are registered for
// agent runs. Bundles are excluded from the pnpm
// workspace and built standalone into the image (see Dockerfile), then
// installed here from their local paths. This runs BEFORE loadAll() so
// loadAll() can activate them in the same startup pass.
//
// Workers are started exactly once, by loadAll(): the `lifecycle` manager
// above is constructed without a runtime-capable loader
// (pluginLifecycleManager(db, { workerManager }) — no `loader` option), so
// the lifecycle.load() that ensureBundledPlugins performs per newly
// installed bundle only records the `ready` status and does not spawn a
// worker (see activateReadyPlugin in services/plugin-lifecycle.ts).
//
// Managed instances (`plugins.autoInstall` from PAPERCLIP_MANAGED_CONFIG)
// drive the key list from the control plane; self-hosted instances keep
// the pre-existing behavior of ensuring only the kubernetes bundle.
//
// Resolution is deliberately synchronous and NOT fail-safe: an
// unknown key or a path escaping the bundled catalog root throws out of
// createApp so a managed instance refuses to start (positive allowlist,
// fail closed).
// SAFETY: installation is fully fail-safe. Any failure
// (missing bundle, install error, load error) is caught, logged, and
// swallowed per plugin so the server ALWAYS finishes booting. A degraded
// boot (a provider unavailable, some agents cannot run) is strictly
// preferable to a crash loop.
//
// The chain is not awaited here (createApp stays fast), but the settled
// promise is exposed via `app.locals.bundledPluginsStartup` so boot steps
// that must not outrun plugin availability — managed sandbox environments
// (`applyManagedEnvironments`) run before the heartbeat resumes queued
// runs — can sequence on it. It never rejects.
const bundledPluginsStartup = ensureBundledPlugins(
bundledPluginInstalls,
{ registry: pluginRegistry, loader, lifecycle, logger },
// Managed mode reinstalls soft-uninstalled bundles (the control plane
// owns provisioning); self-hosted leaves an operator's uninstall alone.
// Disabled plugins never start automatically. Added distribution permissions
// still enter upgrade_pending so enabling them requires an operator decision.
{ reinstallUninstalled: managedAutoInstallKeys !== null, preservePackagePath: managedAutoInstallKeys !== null },
)
.then(() => loader.loadAll())
.then((result) => {
if (!result) return;
for (const loaded of result.results) {
if (devWatcher && loaded.success && loaded.plugin.packagePath) {
devWatcher.watch(loaded.plugin.id, loaded.plugin.packagePath);
}
}
})
.catch((err) => {
logger.error({ err }, "Failed to load ready plugins on startup");
});
app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
// Manifest ids the managed auto-install list resolves to; the managed
// environments bootstrap checks plugin entries against them.
app.locals.managedBundledPluginKeys = managedBundledPluginKeys;
void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying."));
// Carry over legacy plugin tool access once plugin workers have started,
// since a plugin connection is healthy only while its worker runs.
void bundledPluginsStartup.then(() => completeLegacyPluginToolAccess(db))
.catch((error) => logger.warn({ err: error }, "Could not keep legacy plugin tool access"));
// The shutdown hook runs at most once. It caches the in-flight promise, so a
// second caller (for example the `exit` handler) awaits the same completion
// instead of starting a second teardown.
let appServicesShutdown: Promise<void> | null = null;
const shutdownAppServices = (): Promise<void> => {
if (appServicesShutdown) return appServicesShutdown;
appServicesShutdown = finishServiceShutdown(() => scheduler.stop(), async () => {
await loader.stopActivationTimeoutRecovery();
await agentLifecycle.stop();
unregisterMcpOwners();
await publicMcpEvents?.stop();
await dotMcpEvents?.stop();
jobCoordinator.stop();
await finishServiceShutdown(() => resourceLifecycleDelivery.dispose(), async () => {
importSpoolCleanup.unsubscribe();
await deliveryWork.stop();
devWatcher?.close();
viteHtmlRenderer?.dispose();
void viteDevServer?.close().catch(() => undefined);
viteHmrServer?.close();
hostServiceCleanup.disposeAll();
hostServiceCleanup.teardown();
await stopMuseMaintenance();
await emailChannels.shutdown();
await chatChannels.shutdown();
// End the avatar worker pool, if a request ever started one, so no
// render outlives the HTTP teardown.
await agentAvatars.close();
// Cancel every live setup-token login session and AWAIT the cancellation,
// so each direct child stops and the server releases each lease before the
// caller stops the database and the provider. A lease release that
// fails stays a durable record for the startup reaper.
await setupTokenLoginService?.shutdown();
if (serverMetrics) stopServerMetrics(serverMetrics);
});
});
return appServicesShutdown;
};
app.locals.paperclipShutdown = shutdownAppServices;
// The `exit` event is synchronous. It cannot await the teardown, so it runs
// the best-effort cleanup and drops the returned promise. The orderly signal
// path awaits `shutdownAppServices` in full before the process exits.
process.once("exit", () => {
void shutdownAppServices();
});
process.once("beforeExit", () => {
void flushPluginLogBuffer();
});
trackIdleRequestHandlers(app);
return app;
}