Files
PaperClipAI/server/src/app.ts
T
DottaandClaude Opus 5.5 b5ba548112 feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line
that a real-agent ramp harness checks (#21695)
> - Four finish-line criteria need numbers from inside each server
process: main-thread CPU, run event lateness, agent API time before the
handler, and missed lease renewals
> - The server had no way to give them, so every harness run was at best
"incomplete"
> - This pull request adds one instance-admin endpoint that returns
those numbers per process over a short sliding window, off by default,
and connects the harness to it
> - The benefit is that a ramp run can measure every server criterion it
can see, with near-zero cost when on and no cost when off

## Linked Issues or Issue Description

Refs #21695

**Subsystem affected**

Server (instance routes, run event append, native lease renewal),
database client (statement counter), and the scale-ramp harness in
`scripts/scale-ramp/`.

**Problem or motivation**

The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the
server has no endpoint for it. The criteria `main_thread_cpu`,
`run_event_lateness_p99`, `agent_api_pre_handler_p99` and
`lease_renewals_missed_zero` are therefore always "not measured", and no
run can pass.

**Proposed solution**

Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of
one server process over the last 50 to 60 s. Only an instance admin can
read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start,
default off) turns on collection and the route. Off, the path answers
404 and no collector starts.

**Alternatives considered**

- The postgres.js `debug` option for the statement counter: rejected,
because it captures a stack (`new Error()`) for every query.
- Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected,
because it includes GC and libuv threads. It is returned as `processCpu`
instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux
only, null elsewhere).
- Default on: the measured cost is negligible (numbers below), but the
event-loop monitor wakes the process every 20 ms when idle, and this is
a new instance-wide surface. It stays off; the runbook says how to turn
it on for a ramp run.

## What Changed

- `server/src/services/server-metrics.ts`: a per-process collector. It
uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20
ms), `process.cpuUsage()`, the main thread's
`/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native
`perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing
on the hot path awaits, allocates per call or does I/O.
- `server/src/routes/instance-server-metrics.ts`: `GET
/api/instance/server-metrics`, guarded by the existing
`assertInstanceAdmin` (the check that `/instance/database-backups` and
other instance routes use). It returns numbers, a random per-process
`processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no
SQL, names, IDs or hostnames.
- `server/src/app.ts`: when the flag is on, two middlewares time agent
API requests from arrival to the API router, the route is mounted, and
shutdown stops the collector. When the flag is off, nothing is mounted.
- The snapshot also carries `dedicatedConnections` from
`dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at
request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit`
since start. The harness ignores it today; it shows dedicated-connection
queueing in a restart burst.
- `packages/db/src/statement-count.ts` and
`lazy-transaction-queries.ts`: a Drizzle logger that increments one
counter per statement replaces Drizzle's no-op logger. No edit to
`client.ts`.
- `paperclip-control-plane-port.ts`: one call records runner `emittedAt`
to committed row for each committed (not duplicate) runner event.
- `native-execution-lease.ts`: one call counts a lease whose renewal
deadline passed.
- Harness: missed lease renewals are summed per `processId` (several
processes, or a restart, no longer cancel out). A metrics URL on the
target origin gets the board token in the `Authorization` header;
`PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin
gets no token.
- Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the
serializer's own output. A server test asserts the serializer produces
exactly this file, and a harness test parses it and checks every field.
- `rules/company-scope-exceptions.mdc`: adds the snapshot as an
instance-wide exception with its reason. The runbook in the plan doc
documents the path, the flag, the credential and each field.

| Field | Provided | Source |
| --- | --- | --- |
| `mainThreadCpu` | yes (Linux; null elsewhere) |
`/proc/self/task/<pid>/stat` utime+stime |
| `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` |
| `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s
p99 in the window |
| `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT
or raw driver calls) |
| `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue
time |
| `runEventLatenessP99Ms` | yes (null with no events) | runner event
append |
| `agentApiPreHandlerP99Ms` | yes (null with no agent requests) |
arrival and router-entry middlewares |
| `leaseRenewalsMissed` | yes, counter since process start | native
lease deadline timer |

## Verification

- `cd server && npx vitest run
src/__tests__/instance-server-metrics-routes.test.ts
src/__tests__/server-metrics.test.ts`: through `createApp` on the
embedded test database. 200 with the exact field set for an instance
admin; 403 for a company owner who is not an instance admin, for an
agent key, and unauthenticated; 404 and no collector when the flag is
off; values move after agent requests, statements, a late run event and
a missed renewal; the body has no IDs, keys or SQL.
- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks
(red/green checked for the lease hook by removing the call).
- `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts
src/client-options.test.ts`: the counter counts statements inside
transactions and savepoints.
- `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p
scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture
contract and the board-token header.
- `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in
`packages/db`: pass.

Overhead, measured on a shared, loaded laptop (load average 8 to 20),
median with min and max:

| Measure | Off | On |
| --- | --- | --- |
| Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) |
3.36 ns (3.32 to 6.16) |
| CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us
(20.3 to 42.4) | 32.3 us (20.7 to 45.5) |
| CPU per HTTP request, client and server in one process (2,000 x 6) |
67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) |
| Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08%
(0.05 to 0.09) |
| Record one run event (Date.parse + histogram) | - | 148 ns (135 to
188) |
| One snapshot | - | 55 us |

The statement and request differences are inside the run-to-run spread.
The direct cost is about 3 ns per statement, against about 30 us of CPU
per statement.

## Risks

- Low risk. Off by default; off, only the statement counter runs, and it
costs one increment more than Drizzle's no-op logger.
- `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other
platforms. The harness then uses `eventLoopUtilization`.
- Run event lateness compares the runner's clock with the server's
clock, so clock skew adds to it. Negative values count as zero.
- With several processes, each request reaches one process. The harness
keeps the highest gauge and sums counter growth per process, but a
process it never samples is not seen.
- The statement counter does not count BEGIN/COMMIT or raw postgres.js
calls.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:13:45 -06:00

1484 lines
69 KiB
TypeScript

import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js";
import { startMcpClientMirrorSync } from "./services/public-mcp/client-mirror-sync.js";
import { registerIdleMcpOwners } from "./services/public-mcp/idle-sleep-proof.js";
import { startMcpRevocationIndexSync } from "./services/public-mcp/revocation-index-sync.js";
import { startMcpIngressHandoffExpiry } from "./services/public-mcp/ingress-handoff-expiry.js";
import { runnerAttachmentDownloadRoutes } from "./services/native-runtime/runner-attachment-download.js";
import { nativeRunMcpRoutes } from "./routes/native-run-mcp.js";
import { connectionEventWakeRoutes } from "./routes/connection-event-wake.js";
import { registerGitHubReviewProjectionWork } from "./services/chat-channels/github/review-projection-work.js";
import { finishServiceShutdown } from "./shutdown.js";
import { promptInjectionPreventionRoutes } from "./routes/prompt-injection-prevention.js";
import { voiceSessionRoutes, voiceWebhookRoutes } from "./routes/voice-sessions.js";
import { eq } from "drizzle-orm";
import { chatEndpoints, dedicatedDbConnectionStats } from "@paperclipai/db";
import { fastResponseRoutes } from "./routes/fast-responses.js";
import { fastResponseService } from "./services/fast-responses.js";
import { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
import { createLifecycleDriver } from "./services/agent-lifecycle-driver.js";
import { startAgentLifecycle } from "./services/agent-lifecycle.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
import { staticUiFiles } from "./middleware/static-ui.js";
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
import { loadStorageConfig } from "./storage/config.js";
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
import { customerSuccessRoutes } from "./routes/customer-success.js";
import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js";
import type { CloudWarmStandby } from "./services/cloud-warm-standby.js";
import { browserUseRoutes } from "./routes/browser-use.js";
import { mcpAppsRoutes } from "./routes/mcp-apps.js";
import { mcpSkillsRoutes } from "./routes/mcp-skills.js";
import { registerChatActionWork } from "./services/chat-action-work.js";
import { registerChatDeliveryWork } from "./services/chat-delivery-work.js";
import { registerBrowserUseCleanup } from "./services/browser-use-work.js";
import { browserUseService } from "./services/browser-use.js";
import { slackToolRoutes } from "./routes/slack-tools.js";
import { createPublicMcpOAuth, publicMcpConfig } from "./services/public-mcp/oauth.js";
import { createDotPrincipalHttp } from "./modules/external-agents/index.js";
import { createDotRunnerMcpTools, dotRunnerBroker } from "./services/dot-runner-broker.js";
import { agentProfileAvatarRoutes } from "./routes/agent-profile-avatar.js";
import { museTransportRoutes, museBoardRoutes } from "./routes/muse.js";
import { museQualificationRoutes } from "./routes/muse-qualification.js";
import { startMuseMaintenance } from "./services/muse-maintenance.js";
import { grokBotRoutes } from "./routes/grok-bot.js";
import { createGrokRoutineDispatcher } from "./services/grok-bot/dispatcher.js";
import { dotRunnerRoutes } from "./routes/dot-runner.js";
import { createPublicMcpTransfers } from "./services/public-mcp/file-transfers.js";
import { createMcpApiDispatch, createPublicMcpExecutor } from "./services/public-mcp/capabilities.js";
import { createPublicMcpEvents, type PublicMcpEvents } from "./services/public-mcp/events.js";
import { publicMcpIngressRoutes, publicMcpManagementRoutes } from "./routes/public-mcp.js";
import { agentAvatarRoutes } from "./routes/agent-avatars.js";
import { decisionModelRoutes } from "./routes/decision-models.js";
import { agentObservabilityRoutes } from "./routes/agent-observability.js";
import { aiConnectionRoutes } from "./routes/ai-connections.js";
import { taskWorkspaceRoutes } from "./routes/task-workspaces.js";
import { projectToolRoutes } from "./routes/project-tools.js";
import { emailChannelService } from "./services/email-channels.js";
import { emailRoutes, emailWebhookRoutes } from "./routes/email.js";
import { toolActionDeliveryService } from "./services/tool-action-delivery.js";
import { registerAssignedMcpGateway } from "./services/native-runtime/assigned-mcp-tools.js";
import express, { Router, type Request as ExpressRequest } from "express";
import {
createServer as createHttpServer,
type Server as HttpServer,
} from "node:http";
import path from "node:path";
import fs from "node:fs";
import { fileURLToPath } from "node:url";
import type { Db } from "@paperclipai/db";
import {
derivePaperclipViteHmrPort,
type DeploymentExposure,
type DeploymentMode,
} from "@paperclipai/shared";
import type { InspectDatabaseBackupHealthOptions } from "./services/database-backup-health.js";
import type { StorageService } from "./storage/types.js";
import { httpLogger, errorHandler } from "./middleware/index.js";
import { skillAccessRequestContext } from "./services/skill-access.js";
import { actorMiddleware } from "./middleware/auth.js";
import { boardMutationGuard } from "./middleware/board-mutation-guard.js";
import {
privateHostnameGuard,
resolvePrivateHostnameAllowSet,
} from "./middleware/private-hostname-guard.js";
import {
applyTrustProxy,
parseTrustProxyEnv,
} from "./middleware/trust-proxy.js";
import {
resolveDefaultImportTransferSpoolRoot,
} from "./services/company-import-transfers.js";
import { registerImportSpoolCleanupWork } from "./services/import-spool-cleanup-work.js";
import { companyTransferRunService } from "./services/company-transfer-runs.js";
import { healthRoutes } from "./routes/health.js";
import { readinessRoutes } from "./routes/ready.js";
import { livenessRoutes } from "./routes/liveness.js";
import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js";
import { cloudControlMiddleware } from "./middleware/cloud-control.js";
import { runnerCallOutcomeMiddleware } from "./middleware/runner-call-outcome.js";
import { cloudRoutes } from "./routes/cloud.js";
import { companyRoutes } from "./routes/companies.js";
import { companySkillRoutes } from "./routes/company-skills.js";
import { companySkillPolicyRoutes } from "./routes/company-skill-policy.js";
import { inboxAgentPolicyRoutes } from "./routes/inbox-agent-policy.js";
import { builtInAgentRoutes } from "./routes/built-in-agents.js";
import { folderRoutes } from "./routes/folders.js";
import { summarySlotRoutes } from "./routes/summary-slots.js";
import { statusCardRoutes } from "./routes/status-cards.js";
import { teamsCatalogRoutes } from "./routes/teams-catalog.js";
import { agentRoutes } from "./routes/agents.js";
import { agentInteractionPermissionRoutes } from "./routes/agent-interaction-permissions.js";
import { agentPermissionGrantRoutes } from "./routes/agent-permission-grants.js";
import { resolveAgentRunnerTargetForCompany } from "./services/agent-runner-selection.js";
import type { SetupTokenSessionService } from "./services/setup-token-session.js";
import {
buildSetupTokenLoginTransport,
createProductionSetupTokenSandboxProvider,
createProductionSetupTokenCleanupStore,
createSetupTokenSecretWriter,
createWorkerBoundLoginPtyOpener,
} from "./services/setup-token-transport-binding.js";
import { environmentService } from "./services/environments.js";
import { environmentRuntimeService } from "./services/environment-runtime.js";
import { projectRoutes } from "./routes/projects.js";
import { issueRoutes } from "./routes/issues.js";
import { linkPreviewRoutes } from "./routes/link-previews.js";
import { issueTreeControlRoutes } from "./routes/issue-tree-control.js";
import { caseRoutes } from "./routes/cases.js";
import { caseHandoffRoutes } from "./routes/case-handoffs.js";
import { workflowRoutes } from "./routes/workflows.js";
import { fileResourceRoutes } from "./routes/file-resources.js";
import { createRemoteWorkspaceFiles } from "./services/remote-workspace-files.js";
import { createSavedWorkspaceLoader } from "./services/remote-workspaces.js";
import { createSavedWorkspaceArchiveOpener } from "./services/saved-workspace-archive.js";
import { routineRoutes } from "./routes/routines.js";
import { environmentRoutes } from "./routes/environments.js";
import { tailscaleConnectionRoutes } from "./routes/tailscale-connections.js";
import { executionWorkspaceRoutes } from "./routes/execution-workspaces.js";
import { goalRoutes } from "./routes/goals.js";
import { onboardingSeedRoutes } from "./routes/onboarding-seed.js";
import { boardChatRoutes } from "./routes/board-chat.js";
import { approvalRoutes } from "./routes/approvals.js";
import { secretRoutes } from "./routes/secrets.js";
import { toolAccessRoutes } from "./routes/tool-access.js";
import {
chatChannelRoutes,
chatWebhookRoutes,
} from "./routes/chat-channels.js";
import { smokeLabRoutes } from "./routes/smoke-lab.js";
import { costRoutes } from "./routes/costs.js";
import { agentCommentaryRoutes } from "./routes/agent-commentary.js";
import { agentChatMessageRoutes } from "./routes/agent-chat-messages.js";
import { activityRoutes } from "./routes/activity.js";
import { dashboardRoutes } from "./routes/dashboard.js";
import { attentionRoutes } from "./routes/attention.js";
import { decisionTrainingRoutes } from "./routes/decision-training.js";
import { decisionRoutes } from "./routes/decisions.js";
import { decisionQueueRoutes } from "./routes/decision-queues.js";
import type { DecisionServiceOptions } from "./services/decisions.js";
import { userProfileRoutes } from "./routes/user-profiles.js";
import { sidebarBadgeRoutes } from "./routes/sidebar-badges.js";
import { sidebarPreferenceRoutes } from "./routes/sidebar-preferences.js";
import { announcementRoutes } from "./routes/announcements.js";
import { serverVersion } from "./version.js";
import { workspacePreferenceRoutes } from "./routes/workspace-preferences.js";
import { primaryAgentRoutes } from "./routes/primary-agent.js";
import { resourceMembershipRoutes } from "./routes/resource-memberships.js";
import { inboxDismissalRoutes } from "./routes/inbox-dismissals.js";
import { instanceSettingsRoutes } from "./routes/instance-settings.js";
import { ephemeralEnrollmentRoutes } from "./routes/ephemeral-enrollment.js";
import { instanceSettingsService } from "./services/instance-settings.js";
import { openApiRoutes } from "./routes/openapi.js";
import {
instanceDatabaseBackupRoutes,
type InstanceDatabaseBackupService,
} from "./routes/instance-database-backups.js";
import { llmRoutes } from "./routes/llms.js";
import { authRoutes } from "./routes/auth.js";
import { assetRoutes } from "./routes/assets.js";
import { accessRoutes } from "./routes/access.js";
import { pluginRoutes } from "./routes/plugins.js";
import {
mcpGatewayProtocolRoutes,
toolGatewayRoutes,
} from "./routes/tool-gateway.js";
import {
connectionIntentBoardRoutes,
runtimeConnectionIntentRoutes,
} from "./routes/connection-intents.js";
import { adapterRoutes } from "./routes/adapters.js";
import { managedAgentProfileRoutes } from "./routes/managed-agent-profiles.js";
import { remoteAgentProfileRoutes } from "./routes/remote-agent-profiles.js";
import { pluginUiStaticRoutes } from "./routes/plugin-ui-static.js";
import { readBrandedStaticIndexHtml } from "./static-index-html.js";
import { staticUiCacheControl } from "./static-ui-cache.js";
import { staticUiCompression } from "./middleware/static-ui-compression.js";
import { applyUiBranding } from "./ui-branding.js";
import { logger } from "./middleware/logger.js";
import {
DEFAULT_LOCAL_PLUGIN_DIR,
pluginLoader,
type PluginLoader,
} from "./services/plugin-loader.js";
import {
SELF_HOSTED_AUTO_INSTALL_KEYS,
BUNDLED_PLUGIN_CATALOG,
ensureBundledPlugins,
resolveBundledCatalogRoot,
resolveBundledPluginInstalls,
resolveManagedBundledPluginKeys,
} from "./services/bundled-plugins.js";
import { createBundledPluginPackageResolver } from "./services/bundled-plugin-package.js";
import { readDistributionPluginCatalog, distributionPluginActivationGuard } from "./services/distribution-plugin-catalog.js";
import {
createPluginWorkerManager,
type PluginWorkerManager,
} from "./services/plugin-worker-manager.js";
import { createPluginJobScheduler } from "./services/plugin-job-scheduler.js";
import { pluginJobStore } from "./services/plugin-job-store.js";
import { createPluginToolDispatcher } from "./services/plugin-tool-dispatcher.js";
import { createToolGatewayService } from "./services/tool-gateway.js";
import { toolAccessService } from "./services/tool-access.js";
import { chatChannelService } from "./services/chat-channels.js";
import type { BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js";
import { nativeChatProgressWork } from "./services/chat-native-progress-work.js";
import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-work.js";
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
import { heartbeatService } from "./services/heartbeat.js";
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
import {
buildHostServices,
flushPluginLogBuffer,
} from "./services/plugin-host-services.js";
import { createPluginEventBus } from "./services/plugin-event-bus.js";
import { setPluginEventBus } from "./services/activity-log.js";
import { createPluginDevWatcher } from "./services/plugin-dev-watcher.js";
import { createPluginHostServiceCleanup } from "./services/plugin-host-service-cleanup.js";
import { pluginRegistryService } from "./services/plugin-registry.js";
import { createHostClientHandlers } from "@paperclipai/plugin-sdk";
import type { BetterAuthSessionResult } from "./auth/better-auth.js";
import { createCachedViteHtmlRenderer } from "./vite-html-renderer.js";
import {
DEFAULT_JSON_BODY_LIMIT,
PORTABLE_JSON_BODY_LIMIT,
} from "./http/body-limits.js";
import { COMPANY_IMPORT_API_PATH } from "./routes/company-import-paths.js";
import { apiCompression } from "./middleware/api-compression.js";
import { apiServerTiming } from "./middleware/api-server-timing.js";
import { instanceServerMetricsRoutes } from "./routes/instance-server-metrics.js";
import {
serverMetricsAgentPreHandler,
serverMetricsArrival,
serverMetricsEnabledFromEnv,
startServerMetrics,
stopServerMetrics,
} from "./services/server-metrics.js";
import { chatWebhookBodyParser } from "./middleware/chat-webhook-body.js";
import { createChatWebhookDiagnostics } from "./services/chat-webhook-diagnostics.js";
type UiMode = "none" | "static" | "vite-dev";
const FEEDBACK_EXPORT_FLUSH_INTERVAL_MS = 5_000;
const VITE_DEV_ASSET_PREFIXES = [
"/@fs/",
"/@id/",
"/@react-refresh",
"/@vite/",
"/assets/",
"/node_modules/",
"/src/",
];
const VITE_DEV_STATIC_PATHS = new Set([
"/apple-touch-icon.png",
"/favicon-16x16.png",
"/favicon-32x32.png",
"/favicon.ico",
"/favicon.svg",
"/site.webmanifest",
"/sw.js",
]);
export function isDatabaseConnectionUnavailableError(err: unknown): boolean {
const error = err as { code?: unknown; message?: unknown; cause?: unknown };
if (error?.code === "ECONNREFUSED") return true;
return Boolean(
error?.cause && isDatabaseConnectionUnavailableError(error.cause),
);
}
export function resolveViteHmrPort(serverPort: number): number {
return derivePaperclipViteHmrPort(serverPort);
}
export function resolveViteHmrHost(bindHost: string): string | undefined {
const normalized = bindHost.trim().toLowerCase();
if (
normalized === "0.0.0.0" ||
normalized === "::" ||
normalized === "127.0.0.1" ||
normalized === "::1" ||
normalized === "localhost"
)
return undefined;
return bindHost;
}
export function resolveViteHmrProtocol(
value: string | undefined,
): "ws" | "wss" | undefined {
if (!value) return undefined;
if (value === "ws" || value === "wss") return value;
throw new Error("PAPERCLIP_VITE_HMR_PROTOCOL must be ws or wss");
}
export function listenViteHmrServer(
server: HttpServer,
port: number,
bindHost: string,
): Promise<void> {
return new Promise((resolve, reject) => {
const onError = (error: Error) => {
server.off("listening", onListening);
reject(error);
};
const onListening = () => {
server.off("error", onError);
resolve();
};
server.once("error", onError);
server.once("listening", onListening);
server.listen(port, bindHost);
});
}
export function shouldServeViteDevHtml(req: ExpressRequest): boolean {
const pathname = req.path;
if (VITE_DEV_STATIC_PATHS.has(pathname)) return false;
if (VITE_DEV_ASSET_PREFIXES.some((prefix) => pathname.startsWith(prefix)))
return false;
return req.accepts(["html"]) === "html";
}
export function shouldEnablePrivateHostnameGuard(opts: {
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
}): boolean {
return (
opts.deploymentExposure === "private" &&
(opts.deploymentMode === "local_trusted" ||
opts.deploymentMode === "authenticated")
);
}
export function createManagedBundledPluginWorkerRecovery(input: {
managedBundledPluginKeys: readonly string[];
workerManager: Pick<
PluginWorkerManager,
"getWorker" | "isRunning" | "stopWorker"
>;
getLoader: () => Pick<PluginLoader, "loadSingle"> | null;
}): (plugin: { id: string; pluginKey: string }) => Promise<boolean> {
const recoverablePluginKeys = new Set(input.managedBundledPluginKeys);
const inFlightStarts = new Map<string, Promise<boolean>>();
// A failed attempt can leave behind the dead handle it registered (e.g. the
// worker process died during initialize, which kills the process without
// scheduling a restart). No pre-existing handle survives to a recovery
// attempt — recovery only starts when getWorker() was empty — so discarding
// the dead handle lets a later capability request retry instead of being
// blocked by the handle-presence gate until the process restarts. Handles
// in starting/running/backoff states belong to the worker manager's own
// lifecycle and are left alone.
const discardDeadRecoveryHandle = async (plugin: {
id: string;
pluginKey: string;
}) => {
const handle = input.workerManager.getWorker(plugin.id);
if (!handle || (handle.status !== "crashed" && handle.status !== "stopped"))
return;
try {
await input.workerManager.stopWorker(plugin.id);
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"failed to discard dead plugin worker handle after recovery failure",
);
}
};
return async (plugin) => {
if (!recoverablePluginKeys.has(plugin.pluginKey)) return false;
const inFlight = inFlightStarts.get(plugin.id);
if (inFlight) return inFlight;
const startPromise = (async () => {
if (input.workerManager.getWorker(plugin.id)) {
return input.workerManager.isRunning(plugin.id);
}
const loader = input.getLoader();
if (!loader) return false;
try {
const result = await loader.loadSingle(plugin.id, {
markErrorOnFailure: false,
});
if (
result.success === true ||
input.workerManager.isRunning(plugin.id)
) {
return true;
}
await discardDeadRecoveryHandle(plugin);
return false;
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"managed bundled plugin lazy worker recovery failed",
);
await discardDeadRecoveryHandle(plugin);
throw err;
}
})();
inFlightStarts.set(plugin.id, startPromise);
try {
return await startPromise;
} finally {
if (inFlightStarts.get(plugin.id) === startPromise) {
inFlightStarts.delete(plugin.id);
}
}
};
}
export async function createApp(
db: Db,
opts: {
cloudWarmStandby?: CloudWarmStandby;
uiMode: UiMode;
serverPort: number;
storageService: StorageService;
feedbackExportService?: {
hasPendingFeedbackTraces(): Promise<boolean>;
flushPendingFeedbackTraces(input?: {
companyId?: string;
traceId?: string;
limit?: number;
now?: Date;
signal?: AbortSignal;
}): Promise<unknown>;
};
databaseBackupService?: InstanceDatabaseBackupService;
prepareIdleDatabaseBackup?: () => Promise<boolean>;
idleDeadlineOwners?: IdleDeadlineOwners;
/** Filled with the chat service's Retry callback for background budget-resume delivery. */
budgetResumeDelivery?: BudgetResumeDelivery;
databaseBackupHealth?: InspectDatabaseBackupHealthOptions;
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
allowedHostnames: string[];
bindHost: string;
authPublicBaseUrl?: string;
chatWebhookPublicBaseUrl?: string;
authReady: boolean;
companyDeletionEnabled: boolean;
announcements?: { enabled: boolean; feedUrl: string };
instanceId?: string;
hostVersion?: string;
localPluginDir?: string;
pluginMigrationDb?: Db;
pluginWorkerManager?: PluginWorkerManager;
decisionServiceOptions: DecisionServiceOptions;
betterAuthHandler?: express.RequestHandler;
resolveSession?: (
req: ExpressRequest,
) => Promise<BetterAuthSessionResult | null>;
/**
* `plugins.autoInstall` from the managed config (PAPERCLIP_MANAGED_CONFIG).
* `null`/absent ⇒ self-hosted: only the built-in kubernetes bundle is
* ensured, exactly as before. A managed list is resolved against the
* bundled catalog fail-to-start (see services/bundled-plugins.ts).
*/
managedPluginAutoInstall?: readonly string[] | null;
/** Test override for the bundled plugin catalog root. */
bundledPluginCatalogRoot?: string;
serverMetricsEnabled?: boolean;
},
) {
const app = express();
const serverMetrics = (opts.serverMetricsEnabled ?? serverMetricsEnabledFromEnv(process.env)) ? startServerMetrics(() => dedicatedDbConnectionStats(db)) : null;
if (serverMetrics) app.use("/api", serverMetricsArrival());
// First, so the reported server time includes admission and database waits.
app.use("/api", apiServerTiming());
app.use(databaseOperationMiddleware);
app.use(idleAdmissionMiddleware);
const staticUi = express.Router();
app.locals.paperclipDb = db;
const isWarmStandby = opts.cloudWarmStandby ?? (() => false);
const health = healthRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authReady: opts.authReady,
companyDeletionEnabled: opts.companyDeletionEnabled,
databaseBackupHealth: opts.databaseBackupHealth,
isWarmStandby,
});
const captureRawBody = (
req: express.Request,
_res: express.Response,
buf: Buffer,
) => {
(req as unknown as { rawBody: Buffer }).rawBody = buf;
};
// Respect the operator's `TRUST_PROXY` env var (see middleware/trust-proxy.ts).
// Default is unset → Express trusts nothing, which is the only safe choice
// when the server may be reachable without a known reverse proxy in front.
applyTrustProxy(app, parseTrustProxyEnv(process.env.TRUST_PROXY));
app.use(
COMPANY_IMPORT_API_PATH,
express.json({
limit: PORTABLE_JSON_BODY_LIMIT,
verify: captureRawBody,
}),
);
// Chat providers sign the exact request bytes. Capture every webhook media
// type before the global JSON parser so JSON events and form-encoded action
// callbacks are verified against the provider's original body.
app.use(
"/api/chat-webhooks",
createChatWebhookDiagnostics(),
chatWebhookBodyParser,
);
let dotPrincipalOAuth: ReturnType<typeof createPublicMcpOAuth> | null = null;
const dotPrincipalHttp = createDotPrincipalHttp(() => dotPrincipalOAuth);
app.use(dotPrincipalHttp.bodyParser);
const jsonBodyParser = express.json({ limit: DEFAULT_JSON_BODY_LIMIT, verify: captureRawBody });
// File tickets authenticate before parsing bytes; JSON attachments must remain bytes too.
app.use((req, res, next) => (dotPrincipalHttp.matchesPost(req) || ["/mcp/files/upload", "/mcp/runner/files/upload"].includes(req.path)) ? next() : jsonBodyParser(req, res, next));
app.use("/api", apiCompression());
app.use(httpLogger);
const privateHostnameGateEnabled = shouldEnablePrivateHostnameGuard({
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
});
const privateHostnameAllowSet = resolvePrivateHostnameAllowSet({
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
});
app.use(
privateHostnameGuard({
enabled: privateHostnameGateEnabled,
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
}),
);
let mcpConfig: ReturnType<typeof publicMcpConfig> = null;
try { mcpConfig = publicMcpConfig(process.env, opts.authPublicBaseUrl); }
catch { logger.warn("Assistant connections require an HTTPS public URL (HTTP loopback is allowed for development)."); }
const publicMcpOAuth = mcpConfig ? createPublicMcpOAuth(db, mcpConfig) : null;
const publicMcpIngress = Router();
publicMcpIngress.use(dotPrincipalHttp.router);
app.use(cloudRuntimeIdentityMiddleware(db));
// Routine probes stop before session/actor resolution, even after claim.
// Keep hostname validation and the signed-bootstrap endpoint guard above.
app.use("/api/liveness", livenessRoutes());
app.use("/api/ready", readinessRoutes());
// A signed claim above commits identity before any normal request can seed
// company data. Unclaimed probes bypass session resolution as well as SQL.
app.use(cloudWarmStandbyMiddleware(isWarmStandby, health, staticUi));
const workerManager = opts.pluginWorkerManager ?? createPluginWorkerManager();
app.use("/api/customer-success/v1", customerSuccessRoutes(db, opts.storageService, {
pluginWorkerManager: workerManager,
}));
app.use(museTransportRoutes(db, opts.authPublicBaseUrl));
app.use(publicMcpIngress);
// Connection-intent tools carry their own short-lived, run-bound bearer and
// must be reachable by remote adapters that intentionally do not receive an
// agent API key. Every request revalidates the active heartbeat row.
app.use(runtimeConnectionIntentRoutes(db));
app.use(nativeRunMcpRoutes());
app.use(runnerAttachmentDownloadRoutes(db, opts.storageService));
const hostServicesDisposers = new Map<string, () => void>();
let lifecyclePluginsReady = false;
const agentLifecycle = startAgentLifecycle(db, createLifecycleDriver(db, workerManager), () => lifecyclePluginsReady && !isWarmStandby() && !isIdleTaskDrainActive());
const connectionIntentHeartbeat = heartbeatService(db, {
pluginWorkerManager: workerManager,
});
const chatChannels = chatChannelService(db, {
allowLocalVoiceBoard: opts.deploymentMode === "local_trusted",
deferWebhookProcessing: true,
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
githubWizardOrigin: opts.deploymentMode === "local_trusted"
&& ["127.0.0.1", "localhost", "::1"].includes(opts.bindHost ?? "")
&& Number.isInteger(opts.serverPort) && opts.serverPort! > 0
? `http://${opts.bindHost === "::1" ? "[::1]" : opts.bindHost}:${opts.serverPort}` : null,
webhookPublicBaseUrl: opts.chatWebhookPublicBaseUrl,
resolveNativeQuestion: (interaction) =>
deliverNativeQuestionResponse(db, interaction),
storage: opts.storageService,
});
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
// A restart's automatic Retry of a connected conversation needs the same callback.
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
// Voice capabilities are verified by the signed callback route, not by
// board/agent bearer authentication. Mount before actorMiddleware so its
// per-session Bearer cannot be mistaken for an agent API key.
app.use(voiceWebhookRoutes(db, chatChannels.voice));
app.use(
actorMiddleware(db, {
deploymentMode: opts.deploymentMode,
resolveSession: opts.resolveSession,
}),
);
// After the actor middleware on purpose: a valid Cloud control assertion
// REPLACES whatever actor the request otherwise resolved to, and only on
// the one endpoint it authorizes (see the middleware for the contract).
app.use(cloudControlMiddleware());
app.use(runnerCallOutcomeMiddleware(db));
app.use(connectionEventWakeRoutes(db));
app.use(skillAccessRequestContext);
app.use("/api/auth", authRoutes(db));
if (opts.betterAuthHandler) {
app.all("/api/auth/{*authPath}", opts.betterAuthHandler);
}
app.use(llmRoutes(db));
// Provider-authenticated ingress is intentionally outside the board
// mutation guard. The Chat SDK adapter verifies the provider signature
// before Paperclip persists or acts on any event.
const emailChannels = emailChannelService(db, {
isReconciliationEnabled: () => !isWarmStandby(),
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
heartbeat: connectionIntentHeartbeat,
storage: opts.storageService,
publicBaseUrl: opts.chatWebhookPublicBaseUrl ?? opts.authPublicBaseUrl,
});
app.use(emailWebhookRoutes(emailChannels));
app.locals.fastResponses = fastResponseService(db, {
authorizeExternal: async (tx, request) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
return endpoint?.provider === "agentmail" ? emailChannels.authorizeFastResponse(tx, request) : chatChannels.authorizeFastResponse(tx, request);
},
publishExternal: async (tx, request, commentId, text) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
if (endpoint?.provider !== "agentmail") return false;
await emailChannels.publishFastResponse(tx, request, commentId, text);
return true;
},
budgetHooks: { cancelWorkForScope: connectionIntentHeartbeat.cancelBudgetScopeWork },
});
app.use(chatWebhookRoutes(chatChannels));
// The instance validates single-use registration state and its trusted
// current origin. These exact callback routes are the public setup returns.
app.get("/api/chat-github/manifest/callback", async (req, res) => {
res.set("Cache-Control", "no-store");
res.set("Referrer-Policy", "no-referrer");
const redirect = await chatChannels.githubWizard.directCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor);
res.redirect(303, redirect);
});
app.get("/api/chat-github/cloud/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.cloudCallback(String(req.query.state ?? ""), String(req.query.registration ?? ""), typeof req.query.claim === "string" ? req.query.claim : undefined, req.actor));
});
app.get("/api/chat-github/identity/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.identityCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor));
});
const managedAutoInstallKeys = resolveManagedBundledPluginKeys(opts.managedPluginAutoInstall);
const bundledCatalogRoot =
opts.bundledPluginCatalogRoot ?? resolveBundledCatalogRoot(process.env);
const distributionPlugins = readDistributionPluginCatalog(bundledCatalogRoot, BUNDLED_PLUGIN_CATALOG);
const bundledPluginInstalls = resolveBundledPluginInstalls(
managedAutoInstallKeys ?? SELF_HOSTED_AUTO_INSTALL_KEYS,
{
catalogRoot: bundledCatalogRoot,
env: process.env,
enforceCatalogRoot: managedAutoInstallKeys !== null,
distributionPlugins,
},
);
const bundledPackageResolver = managedAutoInstallKeys === null ? undefined : createBundledPluginPackageResolver(bundledPluginInstalls);
const managedBundledPluginKeys =
managedAutoInstallKeys !== null
? bundledPluginInstalls.map((install) => install.pluginKey)
: [];
let runtimePluginLoader: Pick<PluginLoader, "loadSingle"> | null = null;
// A sibling process can install a managed bundled plugin while this process
// skips the mid-install row, then finish the row after this process's
// loadAll() pass. The capabilities route may recover only those managed
// bundles by starting their ready-but-unstarted worker lazily.
const recoverManagedBundledPluginWorker =
managedAutoInstallKeys !== null
? createManagedBundledPluginWorkerRecovery({
managedBundledPluginKeys,
workerManager,
getLoader: () => runtimePluginLoader,
})
: undefined;
// Mount API routes
const api = Router();
const agentAvatars = agentAvatarRoutes();
api.use(agentAvatars.router);
api.use(boardMutationGuard());
api.use("/health", health);
api.use(openApiRoutes());
api.use("/cloud", cloudRoutes());
api.use("/companies", companyRoutes(db, opts.storageService));
api.use(llmRoutes(db));
api.use(folderRoutes(db));
api.use(companySkillRoutes(db, { pluginWorkerManager: workerManager }));
api.use(companySkillPolicyRoutes(db));
api.use(inboxAgentPolicyRoutes(db));
api.use(agentCommentaryRoutes(db));
api.use(agentChatMessageRoutes(db));
api.use(builtInAgentRoutes(db));
api.use(summarySlotRoutes(db));
api.use(statusCardRoutes(db));
api.use(teamsCatalogRoutes(db));
// The setup-token login session service. The router builds it and hands it
// back through the callback below, so the shutdown hook can cancel every live
// session (SR-4).
let setupTokenLoginService: SetupTokenSessionService | null = null;
// The dedicated proxy IP or CIDR allowlist for the confidential setup-token
// login responses (SR-7). The global `TRUST_PROXY` setting does not satisfy
// the guard; an operator sets this allowlist to the real TLS-terminating
// proxy addresses. An empty value keeps the confidential responses on direct
// TLS (or a `local_trusted` loopback peer) only.
const setupTokenLoginProxyAllowlist = (
process.env.CLAUDE_LOGIN_TRUSTED_PROXIES ?? ""
)
.split(",")
.map((entry) => entry.trim())
.filter((entry) => entry.length > 0);
// The explicit operator declaration that a platform edge terminates TLS for
// every client request (SR-7). This complements the allowlist for managed
// platforms (Railway, Render, Fly, and the like) where the app socket is
// always plain HTTP and the edge-proxy peer addresses are not stable or
// documented, so `CLAUDE_LOGIN_TRUSTED_PROXIES` cannot express them. It is a
// dedicated, single-purpose setting; the guard still never reads the global
// `TRUST_PROXY` value.
const setupTokenLoginEdgeTlsTerminated = /^(1|true|yes|on)$/i.test(
(process.env.CLAUDE_LOGIN_EDGE_TLS_TERMINATED ?? "").trim(),
);
// Bind the production setup-token login transport. It carries the live lease
// manager, the login-process factory over the sandbox pseudo-terminal, and the
// durable cleanup store. The factory passes only the fixed command
// `CLAUDE_SETUP_TOKEN_COMMAND`; it never reads a command from a
// route, a request body, or an adapter configuration. The durable store and the
// startup reaper are live now, so a restart reaps a leftover lease.
//
// The live sandbox pseudo-terminal opener binds inside the sandbox provider
// worker, so the server process does not hold the raw sandbox process. The
// opener drives the worker through the plugin worker manager route gate.
// The manager mints a host-owned route identifier, permits one
// active credential pseudo-terminal per worker, binds the worker session
// identifier one time for output only, and terminalizes the route on every open
// failure path. With the opener supplied, the provider acquires a lease and the
// start route drives a live login instead of the fixed 503.
const setupTokenLoginTransport = buildSetupTokenLoginTransport({
sandbox: createProductionSetupTokenSandboxProvider({
db,
environments: environmentService(db),
environmentRuntime: environmentRuntimeService(db, {
pluginWorkerManager: workerManager,
}),
openLivePtySession: createWorkerBoundLoginPtyOpener({
workerManager,
environments: environmentService(db),
log: (line) => logger.info(line),
}),
log: (line) => logger.info(line),
}),
store: createProductionSetupTokenCleanupStore(db),
// Bind the atomic credential-claim writer, so a completed login transitions
// the durable row to `stored` and stores the minted token in one control-plane
// transaction. The writer reads the company and the owner only from the
// immutable session scope. The confirm-replacement flow owns rotation. Without
// this writer the router falls back to the deferred, fail-closed 503 and never
// stores the token.
completeCredential: createSetupTokenSecretWriter({ db }),
// Forward the login runner diagnostic lines to the server logger. The
// runner is the sole producer, and every line is a fixed, non-secret
// literal. Without this sink the diagnostics fall back to a no-op in
// production, so a failed login leaves no log trail.
log: (line) => logger.info(line),
});
api.use(
agentRoutes(db, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
deploymentMode: opts.deploymentMode,
confidentialProxyAllowlist: setupTokenLoginProxyAllowlist,
confidentialEdgeTlsTerminated: setupTokenLoginEdgeTlsTerminated,
setupTokenLogin: setupTokenLoginTransport,
onSetupTokenLoginService: (service) => {
// Capture the service, so the graceful-shutdown hook cancels every live
// session and releases each lease. The standalone scheduled reaper owns
// the startup and interval lease cleanup now (SR-4).
setupTokenLoginService = service;
},
}),
);
api.use(agentPermissionGrantRoutes(db));
api.use(agentInteractionPermissionRoutes(db));
api.use(assetRoutes(db, opts.storageService));
api.use(agentProfileAvatarRoutes(db, opts.storageService));
api.use(projectToolRoutes(db));
api.use(taskWorkspaceRoutes(db));
api.use(projectRoutes(db));
api.use(caseRoutes(db, opts.storageService));
api.use(caseHandoffRoutes(db));
api.use(workflowRoutes(db));
api.use(issueTreeControlRoutes(db, { pluginWorkerManager: workerManager }));
api.use(fileResourceRoutes(db, { remoteFiles: createRemoteWorkspaceFiles(db, {
environmentRuntime: environmentRuntimeService(db, { pluginWorkerManager: workerManager }),
loadSavedWorkspace: createSavedWorkspaceLoader(db, opts.storageService),
openSavedWorkspaceArchive: createSavedWorkspaceArchiveOpener(db, opts.storageService),
}) }));
api.use(routineRoutes(db, { pluginWorkerManager: workerManager }));
api.use(tailscaleConnectionRoutes(db));
api.use(
environmentRoutes(db, {
pluginWorkerManager: workerManager,
retryPendingEnvironmentCleanup: input => connectionIntentHeartbeat.sweepPendingCleanupLeases({ environmentRetry: input }),
recoverMissingPluginWorker: recoverManagedBundledPluginWorker
? {
pluginKeys: managedBundledPluginKeys,
startWorker: recoverManagedBundledPluginWorker,
}
: undefined,
}),
);
api.use(executionWorkspaceRoutes(db, { pluginWorkerManager: workerManager }));
api.use(emailRoutes(db, emailChannels));
api.use(goalRoutes(db));
api.use(onboardingSeedRoutes(db));
api.use(boardChatRoutes(db, { deploymentMode: opts.deploymentMode }));
api.use(voiceSessionRoutes(db, chatChannels.voice));
api.use(approvalRoutes(db, { pluginWorkerManager: workerManager }));
api.use(secretRoutes(db));
api.use(managedAgentProfileRoutes(db));
api.use(remoteAgentProfileRoutes(db));
api.use(
chatChannelRoutes(db, {
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
storage: opts.storageService,
service: chatChannels,
}),
);
const trustedLocalStdioRuntimeHost =
process.env.PAPERCLIP_TRUSTED_MCP_RUNTIME_HOST ??
process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ??
null;
api.use(costRoutes(db, { pluginWorkerManager: workerManager }));
api.use(activityRoutes(db));
api.use(dashboardRoutes(db));
api.use(attentionRoutes(db, {
pluginWorkerManager: workerManager,
decisionServiceOptions: opts.decisionServiceOptions,
}));
api.use(decisionTrainingRoutes(db));
api.use(decisionRoutes(db, opts.decisionServiceOptions));
api.use(decisionQueueRoutes(db));
api.use(userProfileRoutes(db));
api.use(sidebarBadgeRoutes(db));
api.use(sidebarPreferenceRoutes(db));
api.use(announcementRoutes(db, { ...opts.announcements, version: opts.hostVersion ?? serverVersion }));
api.use(resourceMembershipRoutes(db));
api.use(primaryAgentRoutes(db));
api.use(workspacePreferenceRoutes(db));
api.use(inboxDismissalRoutes(db));
api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup, opts.idleDeadlineOwners,
input => resourceLifecycleDelivery.prepareForIdleSleep(input)));
api.use(ephemeralEnrollmentRoutes(db));
if (opts.databaseBackupService) {
api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService));
}
if (serverMetrics) api.use(instanceServerMetricsRoutes(serverMetrics));
const pluginRegistry = pluginRegistryService(db);
const eventBus = createPluginEventBus();
setPluginEventBus(eventBus);
const jobStore = pluginJobStore(db);
const lifecycle = pluginLifecycleManager(db, { workerManager });
const scheduler = createPluginJobScheduler({
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
db,
jobStore,
workerManager,
});
const toolDispatcher = createPluginToolDispatcher({
workerManager,
lifecycleManager: lifecycle,
db,
});
const gatewayOAuthAccess = toolAccessService(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
});
const toolActionDeliveries = toolActionDeliveryService(db, heartbeatService(db, { pluginWorkerManager: workerManager }));
const toolGateway = createToolGatewayService(db, {
budgetHooks: { cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
onToolActionSettled: (id) => toolActionDeliveries.deliver(id),
pluginToolDispatcher: toolDispatcher,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
oauthGrantRefresher: (input) =>
gatewayOAuthAccess.refreshOAuthGrantCredentials(input),
});
// Issue routes are intentionally mounted after the gateway is constructed because
// issue approval endpoints delegate to it. The intervening routers use distinct
// route prefixes, so this dependency does not change issue-route precedence.
api.use(issueRoutes(db, opts.storageService, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
approveToolActionRequest: (input) => toolGateway.approveActionRequest(input),
declineToolActionRequest: (input) => toolGateway.declineActionRequest(input),
}));
api.use(slackToolRoutes(db, opts.authPublicBaseUrl));
app.locals.toolGateway = toolGateway;
registerAssignedMcpGateway(db, toolGateway);
app.locals.toolActionDeliveries = toolActionDeliveries;
app.use(mcpGatewayProtocolRoutes(toolGateway));
api.use(decisionModelRoutes(db));
api.use(agentObservabilityRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure }));
api.use(fastResponseRoutes(db));
api.use(linkPreviewRoutes(db));
api.use(promptInjectionPreventionRoutes(db));
api.use(aiConnectionRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure, trustedLocalStdioRuntimeHost, pluginWorkerManager: workerManager }));
api.use(
toolAccessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authPublicBaseUrl: opts.authPublicBaseUrl,
trustedLocalStdioRuntimeHost,
toolGateway,
connectionIntentHeartbeat,
}),
);
api.use(connectionIntentBoardRoutes(db, connectionIntentHeartbeat));
api.use(
smokeLabRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
}),
);
const jobCoordinator = createPluginJobCoordinator({
db,
lifecycle,
scheduler,
jobStore,
});
const hostServiceCleanup = createPluginHostServiceCleanup(
lifecycle,
hostServicesDisposers,
);
let viteHtmlRenderer: ReturnType<typeof createCachedViteHtmlRenderer> | null =
null;
let viteDevServer: { close(): Promise<void> } | null = null;
let viteHmrServer: HttpServer | null = null;
const loader = pluginLoader(
db,
{
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
migrationDb: opts.pluginMigrationDb,
assertPackageActivation: bundledPackageResolver?.assertActivation ?? distributionPluginActivationGuard(bundledCatalogRoot, distributionPlugins, managedAutoInstallKeys),
resolvePackageRoot: bundledPackageResolver?.resolve,
prebuiltOnly: managedAutoInstallKeys !== null,
enableLocalFilesystem: managedAutoInstallKeys === null,
enableNpmDiscovery: managedAutoInstallKeys === null,
},
{
workerManager,
eventBus,
jobScheduler: scheduler,
jobStore,
toolDispatcher,
lifecycleManager: lifecycle,
instanceInfo: {
instanceId: opts.instanceId ?? "default",
hostVersion: opts.hostVersion ?? serverVersion,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
},
buildHostHandlers: (pluginId, manifest) => {
const notifyWorker = (method: string, params: unknown) => {
const handle = workerManager.getWorker(pluginId);
if (handle) handle.notify(method, params);
};
const services = buildHostServices(
db,
pluginId,
manifest.id,
eventBus,
notifyWorker,
{
pluginWorkerManager: workerManager,
manifest,
},
);
hostServicesDisposers.set(pluginId, () => services.dispose());
return createHostClientHandlers({
pluginId,
capabilities: manifest.capabilities,
services,
});
},
},
);
runtimePluginLoader = loader;
workerManager.setActivationTimeoutRecovery?.(plugin => loader.requestActivationTimeoutRecovery(plugin));
const browserUse = browserUseService(db, undefined,
{ cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
(session, run) => toolGateway.browserUseSessionAuthorized({ ...session, runId: run.heartbeatRunId, invocationId: run.invocationId }));
api.use(browserUseRoutes(db, browserUse));
api.use(toolGatewayRoutes(db, toolGateway));
api.use(mcpAppsRoutes(db, toolGateway));
api.use(mcpSkillsRoutes(db, toolGateway));
api.use(
pluginRoutes(
db,
loader,
{ scheduler, jobStore },
{ workerManager },
{ toolDispatcher },
{ workerManager },
{ toolGateway },
),
);
api.use(
adapterRoutes({
db,
resolveRunnerTarget: (companyId, environmentId) => resolveAgentRunnerTargetForCompany(db, companyId, environmentId),
getNativeRunnerEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableNativeRunner === true,
getOpenAiDotEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableOpenAiDot === true,
}),
);
api.use(
accessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
bindHost: opts.bindHost,
allowedHostnames: opts.allowedHostnames,
authPublicBaseUrl: opts.authPublicBaseUrl,
}),
);
let publicMcpEvents: PublicMcpEvents | null = null;
const grokDispatcher = createGrokRoutineDispatcher(db, () => !isWarmStandby() && !isIdleTaskDrainActive());
let dotMcpEvents: PublicMcpEvents | null = null;
if (publicMcpOAuth) {
const dispatch = createMcpApiDispatch(api, app.locals);
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
});
const transfers = createPublicMcpTransfers(db, publicMcpOAuth, dispatch, opts.storageService);
publicMcpIngress.use(transfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(publicMcpOAuth, createPublicMcpExecutor(db, publicMcpOAuth, dispatch, transfers), publicMcpEvents));
const dotOAuth = createPublicMcpOAuth(db, { ...publicMcpOAuth.config, resource: publicMcpOAuth.config.origin + "/mcp/runner" });
dotPrincipalOAuth = dotOAuth;
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive() });
const grokTransfers = createPublicMcpTransfers(db, dotOAuth, dispatch, opts.storageService, { pathPrefix: "/mcp/runner/files",
agentSession: { authorize: (principal, sessionId, taskId, write, database) => dotRunnerBroker(database ?? db).authorizeFileSession(principal, sessionId, taskId, write) } });
publicMcpIngress.use(grokTransfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(dotOAuth, createPublicMcpExecutor(db, dotOAuth, dispatch), dotMcpEvents, createDotRunnerMcpTools(db, connectionIntentHeartbeat, grokTransfers)));
api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth));
api.use(dotRunnerRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
api.use(grokBotRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
}
api.use(museBoardRoutes(db, opts.authPublicBaseUrl));
api.use(museQualificationRoutes(db));
if (serverMetrics) app.use("/api", serverMetricsAgentPreHandler(serverMetrics));
app.use("/api", api);
app.use("/api", (_req, res) => {
res.status(404).json({ error: "API route not found" });
});
app.use(
pluginUiStaticRoutes(db, {
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
resolvePackageRoot: bundledPackageResolver?.resolve,
}),
);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
if (opts.uiMode === "static") {
// Try published location first (server/ui-dist/), then monorepo dev location (../../ui/dist)
const candidates = [
path.resolve(__dirname, "../ui-dist"),
path.resolve(__dirname, "../../ui/dist"),
];
const uiDist = candidates.find((p) =>
fs.existsSync(path.join(p, "index.html")),
);
if (uiDist) {
staticUi.use(staticUiCompression());
// Hashed asset files (Vite emits them under /assets/<name>.<hash>.<ext>)
// never change once built, so they can be cached aggressively.
staticUi.use(
"/assets",
staticUiFiles(path.join(uiDist, "assets"), {
maxAge: "1y",
immutable: true,
}),
);
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
// Publish while the routes below are built; the outcome is awaited
// before the first background work starts. The promise settles to the
// failure instead of rejecting, so it is never an unhandled rejection.
app.locals.retainedUiAssetsPublication = trackIdleWork(publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"))
.then(() => null, (error: unknown) => ({ error })));
staticUi.use("/assets", retained.serve);
}
// Serve root/index through the same runtime HTML transform as SPA routes.
staticUi.get(["/", "/index.html"], (_req, res) => {
res.type("html").set("Cache-Control", "no-cache").send(readBrandedStaticIndexHtml(uiDist));
});
// Non-hashed static files (favicon.ico, manifest, robots.txt, etc.):
// short cache so operators who swap them out see the new version
// reasonably fast, with must-revalidate overrides for index.html and
// sw.js (see staticUiCacheControl for why those two).
staticUi.use(
staticUiFiles(uiDist, {
maxAge: "1h",
setHeaders(res, filePath) {
const override = staticUiCacheControl(filePath);
if (override) {
res.set("Cache-Control", override);
}
},
}),
);
// SPA fallback. Only for non-asset routes — if the browser asks for
// /assets/something.js that doesn't exist, we must NOT serve the HTML
// shell: the browser would try to load it as a JavaScript module, fail
// with a MIME-type error, and cache that broken response. Return 404
// instead. The index.html response itself is no-cache so a subsequent
// deploy's updated asset hashes are picked up on next load.
staticUi.get(/.*/, (req, res) => {
if (req.path.startsWith("/assets/")) {
res.status(404).end();
return;
}
res
.status(200)
.set("Content-Type", "text/html")
.set("Cache-Control", "no-cache")
.end(readBrandedStaticIndexHtml(uiDist));
});
} else {
console.warn("[paperclip] UI dist not found; running in API-only mode");
}
if (process.env.PAPERCLIP_MANAGED_RUNTIME_EXPOSURE === "tailscale_https") {
// The managed-runtime supervisor waits for the app port AND its derived
// Vite HMR companion port to bind before publishing the service. Static
// mode has no Vite, so bind the same placeholder listener dev mode uses
// or the supervisor kills a healthy server at the readiness deadline
// (PAP-18043).
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
await listenViteHmrServer(
hmrServer,
resolveViteHmrPort(opts.serverPort),
opts.bindHost,
);
viteHmrServer = hmrServer;
}
}
if (opts.uiMode === "vite-dev") {
const uiRoot = path.resolve(__dirname, "../../ui");
const publicUiRoot = path.resolve(uiRoot, "public");
const hmrPort = resolveViteHmrPort(opts.serverPort);
const hmrHost = resolveViteHmrHost(opts.bindHost);
const hmrProtocol = resolveViteHmrProtocol(
process.env.PAPERCLIP_VITE_HMR_PROTOCOL,
);
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
const { createServer: createViteServer } = await import("vite");
const configuredViteCacheDir = process.env.PAPERCLIP_VITE_CACHE_DIR?.trim();
const vite = await createViteServer({
root: uiRoot,
...(configuredViteCacheDir
? { cacheDir: path.resolve(configuredViteCacheDir) }
: {}),
appType: "custom",
// Vite otherwise discovers every HTML entry below the UI root. Generated
// Storybook output can reference dependencies that are intentionally not
// part of the application install, poisoning a clean embedded dev-server
// cache before the browser opens. The embedded UI has one real entry.
optimizeDeps: { entries: [path.resolve(uiRoot, "index.html")] },
server: {
// Listener binding and browser HMR hostname are deliberately separate:
// exposed branch runtimes stay loopback-only while the browser uses the
// current MagicDNS hostname through the broker's HTTPS listener.
host: opts.bindHost,
middlewareMode: true,
hmr: {
server: hmrServer,
...(hmrHost ? { host: hmrHost } : {}),
...(hmrProtocol ? { protocol: hmrProtocol } : {}),
port: hmrPort,
clientPort: hmrPort,
},
allowedHosts: privateHostnameGateEnabled
? Array.from(privateHostnameAllowSet)
: undefined,
},
});
try {
await listenViteHmrServer(hmrServer, hmrPort, opts.bindHost);
} catch (error) {
await vite.close();
throw error;
}
viteDevServer = vite;
viteHmrServer = hmrServer;
viteHtmlRenderer = createCachedViteHtmlRenderer({
vite,
uiRoot,
brandHtml: applyUiBranding,
});
const renderViteHtml = viteHtmlRenderer;
if (fs.existsSync(publicUiRoot)) {
staticUi.use(staticUiFiles(publicUiRoot, { index: false }));
}
staticUi.get(/.*/, async (req, res, next) => {
if (!shouldServeViteDevHtml(req)) {
next();
return;
}
try {
const html = await renderViteHtml.render(req.originalUrl);
res.status(200).set({ "Content-Type": "text/html" }).end(html);
} catch (err) {
next(err);
}
});
staticUi.use(vite.middlewares);
}
app.use(staticUi);
app.use(errorHandler);
// A build whose asset names collide with different published bytes refuses
// to start, so the outcome is known before any job, queue, or run starts.
const retainedUiAssetsPublication = await (app.locals.retainedUiAssetsPublication as Promise<{ error: unknown } | null> | undefined);
if (retainedUiAssetsPublication) throw retainedUiAssetsPublication.error;
jobCoordinator.start();
const deliveryWork = createDeliveryWorkCoordinator({
owner: db,
canRun: () => !isWarmStandby() && !isIdleTaskDrainActive(),
canReconcile: () => !isWarmStandby(),
onError: (err, queue) => logger.error({ err, queue }, "Delivery reconciliation failed"),
});
app.locals.deliveryWork = deliveryWork;
await scheduler.start(deliveryWork);
const agentLifecycleDelivery = deliveryWork.register(DELIVERY_QUEUES.agentLifecycle, {
retryMs: 1_000,
run: () => agentLifecycle.sweep(),
hasPending: () => agentLifecycle.hasPending(),
nextRunAt: () => agentLifecycle.nextRunAt(),
});
agentLifecycle.setWake(agentLifecycleDelivery.wake);
const clientMirror = startMcpClientMirrorSync(db, deliveryWork, mcpConfig?.origin,{authorizationOrigin:mcpConfig?.authorizationOrigin});
const unregisterMcpOwners = registerIdleMcpOwners(db, { config: mcpConfig, personal: publicMcpEvents, dot: dotMcpEvents, clientMirrorInstanceId:clientMirror.instanceId });
startMcpRevocationIndexSync(db, deliveryWork);
startMcpIngressHandoffExpiry(db, deliveryWork);
publicMcpEvents?.start(deliveryWork);
dotMcpEvents?.start(deliveryWork);
if (publicMcpOAuth) grokDispatcher.start(deliveryWork);
const stopMuseMaintenance = startMuseMaintenance(db, deliveryWork, async () => {
const state = await connectionIntentHeartbeat.resolveSchedulingSuppression(true);
return !state.suppressed || state.reason === "task_drain";
});
const resourceLifecycleDelivery = registerPluginLifecycleDelivery(deliveryWork, db, workerManager, lifecycle);
await resourceLifecycleDelivery.ready;
if (opts.feedbackExportService) {
deliveryWork.register(DELIVERY_QUEUES.feedback, {
retryMs: FEEDBACK_EXPORT_FLUSH_INTERVAL_MS,
run: (signal) => opts.feedbackExportService!.flushPendingFeedbackTraces({ signal }),
hasPending: () => opts.feedbackExportService!.hasPendingFeedbackTraces(),
});
}
registerChatDeliveryWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
registerChatActionWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
chatChannels.voice.registerReportWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerReplyWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerCleanupWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerRetentionWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
registerGitHubReviewProjectionWork(db, deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubReviewCheckWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubTerminalResponseWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubAcknowledgementWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatFailedRetryNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatRunMilestoneWork(db, opts.authPublicBaseUrl).register(
deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive(),
);
emailChannels.start();
chatChannels.registerGitHubWebhookRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
const importSpoolCleanup = registerImportSpoolCleanupWork(db, deliveryWork,
resolveDefaultImportTransferSpoolRoot(), () => !isWarmStandby() && !isIdleTaskDrainActive());
registerBrowserUseCleanup(deliveryWork, browserUse, () => !isIdleTaskDrainActive());
void companyTransferRunService
.recoverStrandedApplyingRuns(db)
.then((recovered) => {
if (recovered.length > 0) {
logger.warn(
{ count: recovered.length, runIds: recovered },
"failed company transfer runs stranded in applying by a restart",
);
}
})
.catch((err) => {
logger.error({ err }, "stranded company transfer apply recovery failed");
});
void toolDispatcher.initialize().catch((err) => {
logger.error({ err }, "Failed to initialize plugin tool dispatcher");
});
const devWatcher = managedAutoInstallKeys !== null ? null : createPluginDevWatcher(
lifecycle,
async (pluginId) =>
(await pluginRegistry.getById(pluginId))?.packagePath ?? null,
undefined,
{ unwatchedRoots: [bundledCatalogRoot] },
);
// Auto-provision bundled plugins so their providers are registered for
// agent runs. Bundles are excluded from the pnpm
// workspace and built standalone into the image (see Dockerfile), then
// installed here from their local paths. This runs BEFORE loadAll() so
// loadAll() can activate them in the same startup pass.
//
// Workers are started exactly once, by loadAll(): the `lifecycle` manager
// above is constructed without a runtime-capable loader
// (pluginLifecycleManager(db, { workerManager }) — no `loader` option), so
// the lifecycle.load() that ensureBundledPlugins performs per newly
// installed bundle only records the `ready` status and does not spawn a
// worker (see activateReadyPlugin in services/plugin-lifecycle.ts).
//
// Managed instances (`plugins.autoInstall` from PAPERCLIP_MANAGED_CONFIG)
// drive the key list from the control plane; self-hosted instances keep
// the pre-existing behavior of ensuring only the kubernetes bundle.
//
// Resolution is deliberately synchronous and NOT fail-safe: an
// unknown key or a path escaping the bundled catalog root throws out of
// createApp so a managed instance refuses to start (positive allowlist,
// fail closed).
// SAFETY: installation is fully fail-safe. Any failure
// (missing bundle, install error, load error) is caught, logged, and
// swallowed per plugin so the server ALWAYS finishes booting. A degraded
// boot (a provider unavailable, some agents cannot run) is strictly
// preferable to a crash loop.
//
// The chain is not awaited here (createApp stays fast), but the settled
// promise is exposed via `app.locals.bundledPluginsStartup` so boot steps
// that must not outrun plugin availability — managed sandbox environments
// (`applyManagedEnvironments`) run before the heartbeat resumes queued
// runs — can sequence on it. It never rejects.
const bundledPluginsStartup = ensureBundledPlugins(
bundledPluginInstalls,
{ registry: pluginRegistry, loader, lifecycle, logger },
// Managed mode reinstalls soft-uninstalled bundles (the control plane
// owns provisioning); self-hosted leaves an operator's uninstall alone.
// Disabled plugins never start automatically. Added distribution permissions
// still enter upgrade_pending so enabling them requires an operator decision.
{ reinstallUninstalled: managedAutoInstallKeys !== null, preservePackagePath: managedAutoInstallKeys !== null },
)
.then(() => loader.loadAll())
.then((result) => {
if (!result) return;
for (const loaded of result.results) {
if (devWatcher && loaded.success && loaded.plugin.packagePath) {
devWatcher.watch(loaded.plugin.id, loaded.plugin.packagePath);
}
}
})
.catch((err) => {
logger.error({ err }, "Failed to load ready plugins on startup");
});
app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
// Manifest ids the managed auto-install list resolves to; the managed
// environments bootstrap checks plugin entries against them.
app.locals.managedBundledPluginKeys = managedBundledPluginKeys;
void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying."));
// The shutdown hook runs at most once. It caches the in-flight promise, so a
// second caller (for example the `exit` handler) awaits the same completion
// instead of starting a second teardown.
let appServicesShutdown: Promise<void> | null = null;
const shutdownAppServices = (): Promise<void> => {
if (appServicesShutdown) return appServicesShutdown;
appServicesShutdown = finishServiceShutdown(() => scheduler.stop(), async () => {
await loader.stopActivationTimeoutRecovery();
await agentLifecycle.stop();
unregisterMcpOwners();
await publicMcpEvents?.stop();
await dotMcpEvents?.stop();
jobCoordinator.stop();
await finishServiceShutdown(() => resourceLifecycleDelivery.dispose(), async () => {
importSpoolCleanup.unsubscribe();
await deliveryWork.stop();
devWatcher?.close();
viteHtmlRenderer?.dispose();
void viteDevServer?.close().catch(() => undefined);
viteHmrServer?.close();
hostServiceCleanup.disposeAll();
hostServiceCleanup.teardown();
await stopMuseMaintenance();
await emailChannels.shutdown();
await chatChannels.shutdown();
// End the avatar worker pool, if a request ever started one, so no
// render outlives the HTTP teardown.
await agentAvatars.close();
// Cancel every live setup-token login session and AWAIT the cancellation,
// so each direct child stops and the server releases each lease before the
// caller stops the database and the provider. A lease release that
// fails stays a durable record for the startup reaper.
await setupTokenLoginService?.shutdown();
if (serverMetrics) stopServerMetrics(serverMetrics);
});
});
return appServicesShutdown;
};
app.locals.paperclipShutdown = shutdownAppServices;
// The `exit` event is synchronous. It cannot await the teardown, so it runs
// the best-effort cleanup and drops the returned promise. The orderly signal
// path awaits `shutdownAppServices` in full before the process exits.
process.once("exit", () => {
void shutdownAppServices();
});
process.once("beforeExit", () => {
void flushPluginLogBuffer();
});
trackIdleRequestHandlers(app);
return app;
}