Files
PaperClipAI/server/src/app.ts
T
DottaandClaude Opus 5.5 b853cc28ae perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server process starts, recovers saved work, and then reports
ready on `/api/ready`. The deploy health gate waits for that signal.
> - On a busy staging workspace, one boot took 85 s to bind the listener
and 4.4 minutes to report ready. Small, idle stacks take 20–90 s.
> - Two parts of that time do not need to block readiness. The server
published several hundred UI assets one at a time before it bound the
listener. After the listener bound, it waited for the first pass of each
periodic recovery queue, and those passes grow with the backlog.
> - The volume-free data gate, which can refuse a start, ran last. By
then the server had already dispatched queued runs that a refusal would
abandon.
> - This pull request moves the refusal gate to the start, publishes the
assets 8 at a time with replayable bodies, and runs the backlog-sized
first passes after readiness, in their original order.
> - The benefit is a faster time from container start to ready, and no
work admitted before a gate that can refuse to start.

## Linked Issues or Issue Description

No public issue exists. This is the problem description (enhancement
template).

**Is your enhancement related to a problem? Please describe.**
Server boot time grows with the saved backlog. These timestamps come
from the log of one staging boot (release 87fe4d9793). The container
started at 01:16:56. The listener bound at 01:18:21, 85 s after start.
The fresh queued-run dispatch completed at 01:18:25. The full queued-run
dispatch pass completed at 01:20:14. The volume-free data gate ran at
01:21:20. Readiness came about 4.4 minutes after start. During startup
the log showed hundreds of `An error was encountered in a non-retryable
streaming request.` lines.

**Describe the solution you'd like**
Readiness waits only for the correctness gates and for the adoption of
old runs. Work that the periodic queues already retry runs after
readiness. A gate that can refuse to start runs before the server admits
any work.

**Describe alternatives you've considered**
- Move all post-listen recovery to the background. We rejected this.
Native recovery can fail closed and must still fail the start. It also
adopts old runs.
- Skip asset re-publication when the image is unchanged. We rejected
this. The collision check (never overwrite different bytes) needs a read
per asset.

**Additional context**
Startup map (line numbers are from `origin/master` 82e0393598):
- Blocks the listener: migrations and the embedded database
(`server/src/index.ts:369-741`); config and identity (`:753-774`);
backfills (`:821-833`, where
`sweepSupersededPendingRequestConfirmations` scans every pending
confirmation); `qualifyCloudConfigurationForCurrentImage` (`:956`,
synchronous package hashing); `createApp` (`:964`), which awaited serial
UI asset publication (`server/src/app.ts:1110`,
`server/src/retained-ui-assets.ts:69-105`).
- Blocks readiness after the listener: first passes of tool reviews,
gateway sessions and delivery workers (`index.ts:1529-1558`); the
terminal workspace reaper (`:1652`); native recovery and hot-restart
adoption (`:1707-1779`); the first passes of 17 recovery queues
(`:1782-1890`), including the full queued-run dispatch (`:1805-1811`);
backup registration (`:1979-1999`); the data gate and package
qualification (`:2020-2026`).
- Grows with the backlog: the queue first passes (queued runs, pending
deliveries, watchdogs, native sessions and results), the terminal
workspace sweep, the native recovery scan, and the confirmation sweep.
UI asset publication grows with the build (hundreds of assets), not the
backlog.

## What Changed

- **Volume-free data gate runs first.** `assertEphemeralServerData` now
runs right after `claimVolumeFreeControllerState` and the runtime config
load. That is before `createApp`, before the listener binds, and before
any queued-run dispatch. It used to run at the end of startup. The
package qualification gate stays at the end, because bundled plugin
installs can change its inputs.
- **UI asset publication is faster but still a gate before work.**
`createApp` starts the publication while it builds its routes. It awaits
the result before its first job, queue or run starts, which is the same
point as on master. A colliding build still refuses to start before it
admits any work (Greptile review).
- **Publication sends Buffer bodies, 8 at a time.** Each asset is read
once into a Buffer. The S3 client can replay a Buffer body; it cannot
replay a stream. The new code sends no stream bodies, so it no longer
reaches the SDK path that logs "non-retryable streaming request". After
the first failure, no new assets start. A collision found on one asset
outranks a storage outage on another (Greptile review).
- **Backlog-sized first passes run after readiness.** The 17 queue
registrations after native recovery (residual recovery through
stopped-instruction collection) now run in a tracked background
sequence, in the same order. Each pass is isolated. A pass that fails is
logged with its name, and the later passes, the housekeeping and the
scheduler still run. A queue whose first pass failed stays registered,
and the coordinator retries it. Native recovery and hot-restart adoption
still complete before ready. Idle sleep stays blocked while the sequence
runs, and shutdown waits for it like other scheduler work.
- **Queued turns keep the right status.** A new flag,
`startupQueuedRunPassOutstanding()`, stays true until the first full
queued-run pass ends. Until then the execution projection still shows
"Waiting for server restart". The flag clears even when that pass fails,
because the dispatch queue then owns those turns.
- **Startup timeline log.** At readiness the server logs the duration of
each startup step (`startup timeline at readiness`). It logs again when
the background passes finish. No OpenTelemetry collector is needed.
- **Benchmark.** A new script,
`scripts/benchmark-retained-ui-assets.ts`, measures publication against
storage with a fixed round-trip latency. It can fail when time exceeds a
budget (`RETAINED_UI_BENCH_MAX_MS`).

Why each moved step is safe:
- **Data gate (earlier).** Its inputs are the migrated database and the
owner marker. `claimVolumeFreeControllerState` still runs first.
Interaction with #20952 and #20990, which are both in this branch: those
changes made the gate tolerate
`native_harness_backups_require_migration` and
`local_instruction_copies_require_settlement` when the filesystem
carries the volume-free owner marker. That covered the case where this
boot's own dispatch and recovery wrote such run state before the late
gate ran. With the gate first, no work of this boot can write run state
before the gate: the app, its queues, run admission, native recovery and
dispatch all start after it. A test checks this order. So the gate sees
only state that a previous process left, and the owner-marker tolerance
still decides that state. This is strictly safer. Nothing relied on the
gate running after recovery: the only work that could clear a blocker
was background work that the old position never awaited. Effect: a
refused start no longer leaves dispatched runs behind.
- **UI asset publication.** The await point is unchanged, so the
gate-before-work order matches master. Only the time it takes changes.
- **Recovery queue first passes (after ready).** Each queue owns its
work and retries it on deadlines, commit wakes, and intervals. A first
pass that ends after ready behaves like a later pass. Interrupted runs
are still classified and adopted by native recovery before ready, and
they resume without a Retry. No wait is added before a turn starts:
turns can start sooner. The HTTP API already served requests during
recovery, because readiness never gated requests. Observable changes:
- The deploy gate passes earlier. A queue that throws while it registers
is now logged instead of failing the start. Each queue's `ready` already
absorbed run errors.
- `startup queued-run dispatch pass complete` can now appear after
`Server startup recovery complete`.

Left alone (safety unclear or not measured):
- The first passes of tool review recovery, gateway sessions, the
delivery workers, the GitHub workers, and the terminal workspace reaper.
These run before native recovery, and I did not prove the ordering safe.
- `qualifyCloudConfigurationForCurrentImage` before `createApp`. It
hashes plugin packages synchronously; its result is only a warning
there.
- The confirmation sweep and principal access backfill before the
listener. They scan whole tables.

## Verification

- Red/green: with `server/src/index.ts`, `app.ts` and
`retained-ui-assets.ts` reverted to master, these new tests fail. With
the change they pass:
  - `server-startup-feedback-export.test.ts`:
- refuses ineligible volume-free data before the app, the listener or
any queued-run dispatch
- reports ready while the first queued-run dispatch pass still runs,
then finishes it and the passes after it
- runs every deferred recovery pass after readiness when one throws,
logs it, and keeps the failed queue registered (fails on the first
revision of this PR, passes now)
  - `retained-ui-assets.test.ts`:
    - re-publishes with bounded parallel, replayable writes
- refuses a colliding build even when another asset's storage outage
fails first (fails with `failure ??=`, passes now)
- Regression guards that also pass: "checks volume-free data once,
before any of this boot's own startup work can write run state" and
"stops starting new assets after one collides".
- Focused suites pass after merging `origin/master`:
`server-startup-feedback-export` (131 tests), `retained-ui-assets`,
`execution-projection`, `liveness`, `heartbeat-startup-fresh-dispatch`,
`ephemeral-server-startup-run-state`, `ephemeral-server-data`,
`ephemeral-migration`, `ephemeral-server-contract` and
`static-index-html`. `tsc --noEmit -p server/tsconfig.json` is clean.
- Benchmark (measured locally, 400 synthetic assets, 25 ms simulated
round trip): `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-retained-ui-assets.ts`

  | | serial (before) | 8 in flight (after) |
  | --- | --- | --- |
  | first publish | 10.5 s | 1.3 s |
  | restart re-publish | 21.3 s | 2.7 s |

This time is still before the listener, as on master, but it is about 8
times shorter and overlaps route construction.
- Local boot (measured, empty instance, embedded PostgreSQL, 3 boots
each, a shared and noisy machine):
  - listener to ready: before 385–427 ms; after 212–303 ms.
- Spawn to listener was 5.5–12.8 s in both cases. Module load and the
embedded database dominate it, and it does not differ.
- Not measured: the hosted saving. From the staging log, the inferred
saving is the queued-run pass and the passes after it (about 175 s:
01:18:25 to 01:21:20), plus most of the serial asset publication before
the listener. Confirm with the new `startup timeline at readiness` log
line after deploy.

## Risks

- **Medium: readiness semantics change.** The deploy gate now passes
before the first recovery passes finish. The old server can leave
earlier while the new server still dispatches its backlog. Queued turns
show "Waiting for server restart" until the dispatch pass ends, and the
queues retry on their own.
- **Low: earlier data gate.** It can refuse a start that the old order
would have admitted only if background work cleared a blocker first.
That was a race before this change.
- **Low: parallel publication against throttling storage.** Eight
requests at a time, with the existing per-asset backoff (0.5–8 s). A
persistent outage still lets the server start on local files.
- No migrations. No API contract changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, subagents for code search). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21010 (retry a throttled asset upload).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:47 -06:00

1462 lines
68 KiB
TypeScript

import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js";
import { startMcpClientMirrorSync } from "./services/public-mcp/client-mirror-sync.js";
import { registerIdleMcpOwners } from "./services/public-mcp/idle-sleep-proof.js";
import { startMcpRevocationIndexSync } from "./services/public-mcp/revocation-index-sync.js";
import { startMcpIngressHandoffExpiry } from "./services/public-mcp/ingress-handoff-expiry.js";
import { runnerAttachmentDownloadRoutes } from "./services/native-runtime/runner-attachment-download.js";
import { nativeRunMcpRoutes } from "./routes/native-run-mcp.js";
import { connectionEventWakeRoutes } from "./routes/connection-event-wake.js";
import { registerGitHubReviewProjectionWork } from "./services/chat-channels/github/review-projection-work.js";
import { finishServiceShutdown } from "./shutdown.js";
import { promptInjectionPreventionRoutes } from "./routes/prompt-injection-prevention.js";
import { voiceSessionRoutes, voiceWebhookRoutes } from "./routes/voice-sessions.js";
import { eq } from "drizzle-orm";
import { chatEndpoints } from "@paperclipai/db";
import { fastResponseRoutes } from "./routes/fast-responses.js";
import { fastResponseService } from "./services/fast-responses.js";
import { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
import { createLifecycleDriver } from "./services/agent-lifecycle-driver.js";
import { startAgentLifecycle } from "./services/agent-lifecycle.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
import { staticUiFiles } from "./middleware/static-ui.js";
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
import { loadStorageConfig } from "./storage/config.js";
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
import { customerSuccessRoutes } from "./routes/customer-success.js";
import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js";
import type { CloudWarmStandby } from "./services/cloud-warm-standby.js";
import { browserUseRoutes } from "./routes/browser-use.js";
import { registerChatActionWork } from "./services/chat-action-work.js";
import { registerChatDeliveryWork } from "./services/chat-delivery-work.js";
import { registerBrowserUseCleanup } from "./services/browser-use-work.js";
import { browserUseService } from "./services/browser-use.js";
import { slackToolRoutes } from "./routes/slack-tools.js";
import { createPublicMcpOAuth, publicMcpConfig } from "./services/public-mcp/oauth.js";
import { createDotPrincipalHttp } from "./modules/external-agents/index.js";
import { createDotRunnerMcpTools, dotRunnerBroker } from "./services/dot-runner-broker.js";
import { agentProfileAvatarRoutes } from "./routes/agent-profile-avatar.js";
import { museTransportRoutes, museBoardRoutes } from "./routes/muse.js";
import { museQualificationRoutes } from "./routes/muse-qualification.js";
import { startMuseMaintenance } from "./services/muse-maintenance.js";
import { grokBotRoutes } from "./routes/grok-bot.js";
import { createGrokRoutineDispatcher } from "./services/grok-bot/dispatcher.js";
import { dotRunnerRoutes } from "./routes/dot-runner.js";
import { createPublicMcpTransfers } from "./services/public-mcp/file-transfers.js";
import { createMcpApiDispatch, createPublicMcpExecutor } from "./services/public-mcp/capabilities.js";
import { createPublicMcpEvents, type PublicMcpEvents } from "./services/public-mcp/events.js";
import { publicMcpIngressRoutes, publicMcpManagementRoutes } from "./routes/public-mcp.js";
import { agentAvatarRoutes } from "./routes/agent-avatars.js";
import { decisionModelRoutes } from "./routes/decision-models.js";
import { agentObservabilityRoutes } from "./routes/agent-observability.js";
import { aiConnectionRoutes } from "./routes/ai-connections.js";
import { taskWorkspaceRoutes } from "./routes/task-workspaces.js";
import { projectToolRoutes } from "./routes/project-tools.js";
import { emailChannelService } from "./services/email-channels.js";
import { emailRoutes, emailWebhookRoutes } from "./routes/email.js";
import { toolActionDeliveryService } from "./services/tool-action-delivery.js";
import { registerAssignedMcpGateway } from "./services/native-runtime/assigned-mcp-tools.js";
import express, { Router, type Request as ExpressRequest } from "express";
import {
createServer as createHttpServer,
type Server as HttpServer,
} from "node:http";
import path from "node:path";
import fs from "node:fs";
import { fileURLToPath } from "node:url";
import type { Db } from "@paperclipai/db";
import {
derivePaperclipViteHmrPort,
type DeploymentExposure,
type DeploymentMode,
} from "@paperclipai/shared";
import type { InspectDatabaseBackupHealthOptions } from "./services/database-backup-health.js";
import type { StorageService } from "./storage/types.js";
import { httpLogger, errorHandler } from "./middleware/index.js";
import { skillAccessRequestContext } from "./services/skill-access.js";
import { actorMiddleware } from "./middleware/auth.js";
import { boardMutationGuard } from "./middleware/board-mutation-guard.js";
import {
privateHostnameGuard,
resolvePrivateHostnameAllowSet,
} from "./middleware/private-hostname-guard.js";
import {
applyTrustProxy,
parseTrustProxyEnv,
} from "./middleware/trust-proxy.js";
import {
resolveDefaultImportTransferSpoolRoot,
} from "./services/company-import-transfers.js";
import { registerImportSpoolCleanupWork } from "./services/import-spool-cleanup-work.js";
import { companyTransferRunService } from "./services/company-transfer-runs.js";
import { healthRoutes } from "./routes/health.js";
import { readinessRoutes } from "./routes/ready.js";
import { livenessRoutes } from "./routes/liveness.js";
import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js";
import { cloudControlMiddleware } from "./middleware/cloud-control.js";
import { cloudRoutes } from "./routes/cloud.js";
import { companyRoutes } from "./routes/companies.js";
import { companySkillRoutes } from "./routes/company-skills.js";
import { companySkillPolicyRoutes } from "./routes/company-skill-policy.js";
import { inboxAgentPolicyRoutes } from "./routes/inbox-agent-policy.js";
import { builtInAgentRoutes } from "./routes/built-in-agents.js";
import { folderRoutes } from "./routes/folders.js";
import { summarySlotRoutes } from "./routes/summary-slots.js";
import { statusCardRoutes } from "./routes/status-cards.js";
import { teamsCatalogRoutes } from "./routes/teams-catalog.js";
import { agentRoutes } from "./routes/agents.js";
import { agentInteractionPermissionRoutes } from "./routes/agent-interaction-permissions.js";
import { agentPermissionGrantRoutes } from "./routes/agent-permission-grants.js";
import { resolveAgentRunnerTargetForCompany } from "./services/agent-runner-selection.js";
import type { SetupTokenSessionService } from "./services/setup-token-session.js";
import {
buildSetupTokenLoginTransport,
createProductionSetupTokenSandboxProvider,
createProductionSetupTokenCleanupStore,
createSetupTokenSecretWriter,
createWorkerBoundLoginPtyOpener,
} from "./services/setup-token-transport-binding.js";
import { environmentService } from "./services/environments.js";
import { environmentRuntimeService } from "./services/environment-runtime.js";
import { projectRoutes } from "./routes/projects.js";
import { issueRoutes } from "./routes/issues.js";
import { linkPreviewRoutes } from "./routes/link-previews.js";
import { issueTreeControlRoutes } from "./routes/issue-tree-control.js";
import { caseRoutes } from "./routes/cases.js";
import { caseHandoffRoutes } from "./routes/case-handoffs.js";
import { workflowRoutes } from "./routes/workflows.js";
import { fileResourceRoutes } from "./routes/file-resources.js";
import { createRemoteWorkspaceFiles } from "./services/remote-workspace-files.js";
import { createSavedWorkspaceLoader } from "./services/remote-workspaces.js";
import { createSavedWorkspaceArchiveOpener } from "./services/saved-workspace-archive.js";
import { routineRoutes } from "./routes/routines.js";
import { environmentRoutes } from "./routes/environments.js";
import { executionWorkspaceRoutes } from "./routes/execution-workspaces.js";
import { goalRoutes } from "./routes/goals.js";
import { onboardingSeedRoutes } from "./routes/onboarding-seed.js";
import { boardChatRoutes } from "./routes/board-chat.js";
import { approvalRoutes } from "./routes/approvals.js";
import { secretRoutes } from "./routes/secrets.js";
import { toolAccessRoutes } from "./routes/tool-access.js";
import {
chatChannelRoutes,
chatWebhookRoutes,
} from "./routes/chat-channels.js";
import { smokeLabRoutes } from "./routes/smoke-lab.js";
import { costRoutes } from "./routes/costs.js";
import { agentCommentaryRoutes } from "./routes/agent-commentary.js";
import { agentChatMessageRoutes } from "./routes/agent-chat-messages.js";
import { activityRoutes } from "./routes/activity.js";
import { dashboardRoutes } from "./routes/dashboard.js";
import { attentionRoutes } from "./routes/attention.js";
import { decisionTrainingRoutes } from "./routes/decision-training.js";
import { decisionRoutes } from "./routes/decisions.js";
import { decisionQueueRoutes } from "./routes/decision-queues.js";
import type { DecisionServiceOptions } from "./services/decisions.js";
import { userProfileRoutes } from "./routes/user-profiles.js";
import { sidebarBadgeRoutes } from "./routes/sidebar-badges.js";
import { sidebarPreferenceRoutes } from "./routes/sidebar-preferences.js";
import { announcementRoutes } from "./routes/announcements.js";
import { serverVersion } from "./version.js";
import { workspacePreferenceRoutes } from "./routes/workspace-preferences.js";
import { primaryAgentRoutes } from "./routes/primary-agent.js";
import { resourceMembershipRoutes } from "./routes/resource-memberships.js";
import { inboxDismissalRoutes } from "./routes/inbox-dismissals.js";
import { instanceSettingsRoutes } from "./routes/instance-settings.js";
import { ephemeralEnrollmentRoutes } from "./routes/ephemeral-enrollment.js";
import { instanceSettingsService } from "./services/instance-settings.js";
import { openApiRoutes } from "./routes/openapi.js";
import {
instanceDatabaseBackupRoutes,
type InstanceDatabaseBackupService,
} from "./routes/instance-database-backups.js";
import { llmRoutes } from "./routes/llms.js";
import { authRoutes } from "./routes/auth.js";
import { assetRoutes } from "./routes/assets.js";
import { accessRoutes } from "./routes/access.js";
import { pluginRoutes } from "./routes/plugins.js";
import {
mcpGatewayProtocolRoutes,
toolGatewayRoutes,
} from "./routes/tool-gateway.js";
import {
connectionIntentBoardRoutes,
runtimeConnectionIntentRoutes,
} from "./routes/connection-intents.js";
import { adapterRoutes } from "./routes/adapters.js";
import { managedAgentProfileRoutes } from "./routes/managed-agent-profiles.js";
import { remoteAgentProfileRoutes } from "./routes/remote-agent-profiles.js";
import { pluginUiStaticRoutes } from "./routes/plugin-ui-static.js";
import { readBrandedStaticIndexHtml } from "./static-index-html.js";
import { staticUiCacheControl } from "./static-ui-cache.js";
import { staticUiCompression } from "./middleware/static-ui-compression.js";
import { applyUiBranding } from "./ui-branding.js";
import { logger } from "./middleware/logger.js";
import {
DEFAULT_LOCAL_PLUGIN_DIR,
pluginLoader,
type PluginLoader,
} from "./services/plugin-loader.js";
import {
SELF_HOSTED_AUTO_INSTALL_KEYS,
BUNDLED_PLUGIN_CATALOG,
ensureBundledPlugins,
resolveBundledCatalogRoot,
resolveBundledPluginInstalls,
resolveManagedBundledPluginKeys,
} from "./services/bundled-plugins.js";
import { createBundledPluginPackageResolver } from "./services/bundled-plugin-package.js";
import { readDistributionPluginCatalog, distributionPluginActivationGuard } from "./services/distribution-plugin-catalog.js";
import {
createPluginWorkerManager,
type PluginWorkerManager,
} from "./services/plugin-worker-manager.js";
import { createPluginJobScheduler } from "./services/plugin-job-scheduler.js";
import { pluginJobStore } from "./services/plugin-job-store.js";
import { createPluginToolDispatcher } from "./services/plugin-tool-dispatcher.js";
import { createToolGatewayService } from "./services/tool-gateway.js";
import { toolAccessService } from "./services/tool-access.js";
import { chatChannelService } from "./services/chat-channels.js";
import type { BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js";
import { nativeChatProgressWork } from "./services/chat-native-progress-work.js";
import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-work.js";
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
import { heartbeatService } from "./services/heartbeat.js";
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
import {
buildHostServices,
flushPluginLogBuffer,
} from "./services/plugin-host-services.js";
import { createPluginEventBus } from "./services/plugin-event-bus.js";
import { setPluginEventBus } from "./services/activity-log.js";
import { createPluginDevWatcher } from "./services/plugin-dev-watcher.js";
import { createPluginHostServiceCleanup } from "./services/plugin-host-service-cleanup.js";
import { pluginRegistryService } from "./services/plugin-registry.js";
import { createHostClientHandlers } from "@paperclipai/plugin-sdk";
import type { BetterAuthSessionResult } from "./auth/better-auth.js";
import { createCachedViteHtmlRenderer } from "./vite-html-renderer.js";
import {
DEFAULT_JSON_BODY_LIMIT,
PORTABLE_JSON_BODY_LIMIT,
} from "./http/body-limits.js";
import { COMPANY_IMPORT_API_PATH } from "./routes/company-import-paths.js";
import { apiCompression } from "./middleware/api-compression.js";
import { apiServerTiming } from "./middleware/api-server-timing.js";
import { chatWebhookBodyParser } from "./middleware/chat-webhook-body.js";
import { createChatWebhookDiagnostics } from "./services/chat-webhook-diagnostics.js";
type UiMode = "none" | "static" | "vite-dev";
const FEEDBACK_EXPORT_FLUSH_INTERVAL_MS = 5_000;
const VITE_DEV_ASSET_PREFIXES = [
"/@fs/",
"/@id/",
"/@react-refresh",
"/@vite/",
"/assets/",
"/node_modules/",
"/src/",
];
const VITE_DEV_STATIC_PATHS = new Set([
"/apple-touch-icon.png",
"/favicon-16x16.png",
"/favicon-32x32.png",
"/favicon.ico",
"/favicon.svg",
"/site.webmanifest",
"/sw.js",
]);
export function isDatabaseConnectionUnavailableError(err: unknown): boolean {
const error = err as { code?: unknown; message?: unknown; cause?: unknown };
if (error?.code === "ECONNREFUSED") return true;
return Boolean(
error?.cause && isDatabaseConnectionUnavailableError(error.cause),
);
}
export function resolveViteHmrPort(serverPort: number): number {
return derivePaperclipViteHmrPort(serverPort);
}
export function resolveViteHmrHost(bindHost: string): string | undefined {
const normalized = bindHost.trim().toLowerCase();
if (
normalized === "0.0.0.0" ||
normalized === "::" ||
normalized === "127.0.0.1" ||
normalized === "::1" ||
normalized === "localhost"
)
return undefined;
return bindHost;
}
export function resolveViteHmrProtocol(
value: string | undefined,
): "ws" | "wss" | undefined {
if (!value) return undefined;
if (value === "ws" || value === "wss") return value;
throw new Error("PAPERCLIP_VITE_HMR_PROTOCOL must be ws or wss");
}
export function listenViteHmrServer(
server: HttpServer,
port: number,
bindHost: string,
): Promise<void> {
return new Promise((resolve, reject) => {
const onError = (error: Error) => {
server.off("listening", onListening);
reject(error);
};
const onListening = () => {
server.off("error", onError);
resolve();
};
server.once("error", onError);
server.once("listening", onListening);
server.listen(port, bindHost);
});
}
export function shouldServeViteDevHtml(req: ExpressRequest): boolean {
const pathname = req.path;
if (VITE_DEV_STATIC_PATHS.has(pathname)) return false;
if (VITE_DEV_ASSET_PREFIXES.some((prefix) => pathname.startsWith(prefix)))
return false;
return req.accepts(["html"]) === "html";
}
export function shouldEnablePrivateHostnameGuard(opts: {
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
}): boolean {
return (
opts.deploymentExposure === "private" &&
(opts.deploymentMode === "local_trusted" ||
opts.deploymentMode === "authenticated")
);
}
export function createManagedBundledPluginWorkerRecovery(input: {
managedBundledPluginKeys: readonly string[];
workerManager: Pick<
PluginWorkerManager,
"getWorker" | "isRunning" | "stopWorker"
>;
getLoader: () => Pick<PluginLoader, "loadSingle"> | null;
}): (plugin: { id: string; pluginKey: string }) => Promise<boolean> {
const recoverablePluginKeys = new Set(input.managedBundledPluginKeys);
const inFlightStarts = new Map<string, Promise<boolean>>();
// A failed attempt can leave behind the dead handle it registered (e.g. the
// worker process died during initialize, which kills the process without
// scheduling a restart). No pre-existing handle survives to a recovery
// attempt — recovery only starts when getWorker() was empty — so discarding
// the dead handle lets a later capability request retry instead of being
// blocked by the handle-presence gate until the process restarts. Handles
// in starting/running/backoff states belong to the worker manager's own
// lifecycle and are left alone.
const discardDeadRecoveryHandle = async (plugin: {
id: string;
pluginKey: string;
}) => {
const handle = input.workerManager.getWorker(plugin.id);
if (!handle || (handle.status !== "crashed" && handle.status !== "stopped"))
return;
try {
await input.workerManager.stopWorker(plugin.id);
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"failed to discard dead plugin worker handle after recovery failure",
);
}
};
return async (plugin) => {
if (!recoverablePluginKeys.has(plugin.pluginKey)) return false;
const inFlight = inFlightStarts.get(plugin.id);
if (inFlight) return inFlight;
const startPromise = (async () => {
if (input.workerManager.getWorker(plugin.id)) {
return input.workerManager.isRunning(plugin.id);
}
const loader = input.getLoader();
if (!loader) return false;
try {
const result = await loader.loadSingle(plugin.id, {
markErrorOnFailure: false,
});
if (
result.success === true ||
input.workerManager.isRunning(plugin.id)
) {
return true;
}
await discardDeadRecoveryHandle(plugin);
return false;
} catch (err) {
logger.warn(
{
pluginId: plugin.id,
pluginKey: plugin.pluginKey,
err: err instanceof Error ? err.message : String(err),
},
"managed bundled plugin lazy worker recovery failed",
);
await discardDeadRecoveryHandle(plugin);
throw err;
}
})();
inFlightStarts.set(plugin.id, startPromise);
try {
return await startPromise;
} finally {
if (inFlightStarts.get(plugin.id) === startPromise) {
inFlightStarts.delete(plugin.id);
}
}
};
}
export async function createApp(
db: Db,
opts: {
cloudWarmStandby?: CloudWarmStandby;
uiMode: UiMode;
serverPort: number;
storageService: StorageService;
feedbackExportService?: {
hasPendingFeedbackTraces(): Promise<boolean>;
flushPendingFeedbackTraces(input?: {
companyId?: string;
traceId?: string;
limit?: number;
now?: Date;
signal?: AbortSignal;
}): Promise<unknown>;
};
databaseBackupService?: InstanceDatabaseBackupService;
prepareIdleDatabaseBackup?: () => Promise<boolean>;
idleDeadlineOwners?: IdleDeadlineOwners;
/** Filled with the chat service's Retry callback for background budget-resume delivery. */
budgetResumeDelivery?: BudgetResumeDelivery;
databaseBackupHealth?: InspectDatabaseBackupHealthOptions;
deploymentMode: DeploymentMode;
deploymentExposure: DeploymentExposure;
allowedHostnames: string[];
bindHost: string;
authPublicBaseUrl?: string;
chatWebhookPublicBaseUrl?: string;
authReady: boolean;
companyDeletionEnabled: boolean;
announcements?: { enabled: boolean; feedUrl: string };
instanceId?: string;
hostVersion?: string;
localPluginDir?: string;
pluginMigrationDb?: Db;
pluginWorkerManager?: PluginWorkerManager;
decisionServiceOptions: DecisionServiceOptions;
betterAuthHandler?: express.RequestHandler;
resolveSession?: (
req: ExpressRequest,
) => Promise<BetterAuthSessionResult | null>;
/**
* `plugins.autoInstall` from the managed config (PAPERCLIP_MANAGED_CONFIG).
* `null`/absent ⇒ self-hosted: only the built-in kubernetes bundle is
* ensured, exactly as before. A managed list is resolved against the
* bundled catalog fail-to-start (see services/bundled-plugins.ts).
*/
managedPluginAutoInstall?: readonly string[] | null;
/** Test override for the bundled plugin catalog root. */
bundledPluginCatalogRoot?: string;
},
) {
const app = express();
// First, so the reported server time includes admission and database waits.
app.use("/api", apiServerTiming());
app.use(databaseOperationMiddleware);
app.use(idleAdmissionMiddleware);
const staticUi = express.Router();
app.locals.paperclipDb = db;
const isWarmStandby = opts.cloudWarmStandby ?? (() => false);
const health = healthRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authReady: opts.authReady,
companyDeletionEnabled: opts.companyDeletionEnabled,
databaseBackupHealth: opts.databaseBackupHealth,
isWarmStandby,
});
const captureRawBody = (
req: express.Request,
_res: express.Response,
buf: Buffer,
) => {
(req as unknown as { rawBody: Buffer }).rawBody = buf;
};
// Respect the operator's `TRUST_PROXY` env var (see middleware/trust-proxy.ts).
// Default is unset → Express trusts nothing, which is the only safe choice
// when the server may be reachable without a known reverse proxy in front.
applyTrustProxy(app, parseTrustProxyEnv(process.env.TRUST_PROXY));
app.use(
COMPANY_IMPORT_API_PATH,
express.json({
limit: PORTABLE_JSON_BODY_LIMIT,
verify: captureRawBody,
}),
);
// Chat providers sign the exact request bytes. Capture every webhook media
// type before the global JSON parser so JSON events and form-encoded action
// callbacks are verified against the provider's original body.
app.use(
"/api/chat-webhooks",
createChatWebhookDiagnostics(),
chatWebhookBodyParser,
);
let dotPrincipalOAuth: ReturnType<typeof createPublicMcpOAuth> | null = null;
const dotPrincipalHttp = createDotPrincipalHttp(() => dotPrincipalOAuth);
app.use(dotPrincipalHttp.bodyParser);
const jsonBodyParser = express.json({ limit: DEFAULT_JSON_BODY_LIMIT, verify: captureRawBody });
// File tickets authenticate before parsing bytes; JSON attachments must remain bytes too.
app.use((req, res, next) => (dotPrincipalHttp.matchesPost(req) || ["/mcp/files/upload", "/mcp/runner/files/upload"].includes(req.path)) ? next() : jsonBodyParser(req, res, next));
app.use("/api", apiCompression());
app.use(httpLogger);
const privateHostnameGateEnabled = shouldEnablePrivateHostnameGuard({
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
});
const privateHostnameAllowSet = resolvePrivateHostnameAllowSet({
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
});
app.use(
privateHostnameGuard({
enabled: privateHostnameGateEnabled,
allowedHostnames: opts.allowedHostnames,
bindHost: opts.bindHost,
}),
);
let mcpConfig: ReturnType<typeof publicMcpConfig> = null;
try { mcpConfig = publicMcpConfig(process.env, opts.authPublicBaseUrl); }
catch { logger.warn("Assistant connections require an HTTPS public URL (HTTP loopback is allowed for development)."); }
const publicMcpOAuth = mcpConfig ? createPublicMcpOAuth(db, mcpConfig) : null;
const publicMcpIngress = Router();
publicMcpIngress.use(dotPrincipalHttp.router);
app.use(cloudRuntimeIdentityMiddleware(db));
// Routine probes stop before session/actor resolution, even after claim.
// Keep hostname validation and the signed-bootstrap endpoint guard above.
app.use("/api/liveness", livenessRoutes());
app.use("/api/ready", readinessRoutes());
// A signed claim above commits identity before any normal request can seed
// company data. Unclaimed probes bypass session resolution as well as SQL.
app.use(cloudWarmStandbyMiddleware(isWarmStandby, health, staticUi));
const workerManager = opts.pluginWorkerManager ?? createPluginWorkerManager();
app.use("/api/customer-success/v1", customerSuccessRoutes(db, opts.storageService, {
pluginWorkerManager: workerManager,
}));
app.use(museTransportRoutes(db, opts.authPublicBaseUrl));
app.use(publicMcpIngress);
// Connection-intent tools carry their own short-lived, run-bound bearer and
// must be reachable by remote adapters that intentionally do not receive an
// agent API key. Every request revalidates the active heartbeat row.
app.use(runtimeConnectionIntentRoutes(db));
app.use(nativeRunMcpRoutes());
app.use(runnerAttachmentDownloadRoutes(db, opts.storageService));
const hostServicesDisposers = new Map<string, () => void>();
let lifecyclePluginsReady = false;
const agentLifecycle = startAgentLifecycle(db, createLifecycleDriver(db, workerManager), () => lifecyclePluginsReady && !isWarmStandby() && !isIdleTaskDrainActive());
const connectionIntentHeartbeat = heartbeatService(db, {
pluginWorkerManager: workerManager,
});
const chatChannels = chatChannelService(db, {
allowLocalVoiceBoard: opts.deploymentMode === "local_trusted",
deferWebhookProcessing: true,
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
githubWizardOrigin: opts.deploymentMode === "local_trusted"
&& ["127.0.0.1", "localhost", "::1"].includes(opts.bindHost ?? "")
&& Number.isInteger(opts.serverPort) && opts.serverPort! > 0
? `http://${opts.bindHost === "::1" ? "[::1]" : opts.bindHost}:${opts.serverPort}` : null,
webhookPublicBaseUrl: opts.chatWebhookPublicBaseUrl,
resolveNativeQuestion: (interaction) =>
deliverNativeQuestionResponse(db, interaction),
storage: opts.storageService,
});
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
// A restart's automatic Retry of a connected conversation needs the same callback.
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
// Voice capabilities are verified by the signed callback route, not by
// board/agent bearer authentication. Mount before actorMiddleware so its
// per-session Bearer cannot be mistaken for an agent API key.
app.use(voiceWebhookRoutes(db, chatChannels.voice));
app.use(
actorMiddleware(db, {
deploymentMode: opts.deploymentMode,
resolveSession: opts.resolveSession,
}),
);
// After the actor middleware on purpose: a valid Cloud control assertion
// REPLACES whatever actor the request otherwise resolved to, and only on
// the one endpoint it authorizes (see the middleware for the contract).
app.use(cloudControlMiddleware());
app.use(connectionEventWakeRoutes(db));
app.use(skillAccessRequestContext);
app.use("/api/auth", authRoutes(db));
if (opts.betterAuthHandler) {
app.all("/api/auth/{*authPath}", opts.betterAuthHandler);
}
app.use(llmRoutes(db));
// Provider-authenticated ingress is intentionally outside the board
// mutation guard. The Chat SDK adapter verifies the provider signature
// before Paperclip persists or acts on any event.
const emailChannels = emailChannelService(db, {
isReconciliationEnabled: () => !isWarmStandby(),
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
heartbeat: connectionIntentHeartbeat,
storage: opts.storageService,
publicBaseUrl: opts.chatWebhookPublicBaseUrl ?? opts.authPublicBaseUrl,
});
app.use(emailWebhookRoutes(emailChannels));
app.locals.fastResponses = fastResponseService(db, {
authorizeExternal: async (tx, request) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
return endpoint?.provider === "agentmail" ? emailChannels.authorizeFastResponse(tx, request) : chatChannels.authorizeFastResponse(tx, request);
},
publishExternal: async (tx, request, commentId, text) => {
const [endpoint] = await tx.select({ provider: chatEndpoints.provider }).from(chatEndpoints).where(eq(chatEndpoints.id, request.endpointId!));
if (endpoint?.provider !== "agentmail") return false;
await emailChannels.publishFastResponse(tx, request, commentId, text);
return true;
},
budgetHooks: { cancelWorkForScope: connectionIntentHeartbeat.cancelBudgetScopeWork },
});
app.use(chatWebhookRoutes(chatChannels));
// The instance validates single-use registration state and its trusted
// current origin. These exact callback routes are the public setup returns.
app.get("/api/chat-github/manifest/callback", async (req, res) => {
res.set("Cache-Control", "no-store");
res.set("Referrer-Policy", "no-referrer");
const redirect = await chatChannels.githubWizard.directCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor);
res.redirect(303, redirect);
});
app.get("/api/chat-github/cloud/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.cloudCallback(String(req.query.state ?? ""), String(req.query.registration ?? ""), typeof req.query.claim === "string" ? req.query.claim : undefined, req.actor));
});
app.get("/api/chat-github/identity/callback", async (req, res) => {
res.set("Cache-Control", "no-store"); res.set("Referrer-Policy", "no-referrer");
res.redirect(303, await chatChannels.githubWizard.identityCallback(String(req.query.state ?? ""), String(req.query.code ?? ""), req.actor));
});
const managedAutoInstallKeys = resolveManagedBundledPluginKeys(opts.managedPluginAutoInstall);
const bundledCatalogRoot =
opts.bundledPluginCatalogRoot ?? resolveBundledCatalogRoot(process.env);
const distributionPlugins = readDistributionPluginCatalog(bundledCatalogRoot, BUNDLED_PLUGIN_CATALOG);
const bundledPluginInstalls = resolveBundledPluginInstalls(
managedAutoInstallKeys ?? SELF_HOSTED_AUTO_INSTALL_KEYS,
{
catalogRoot: bundledCatalogRoot,
env: process.env,
enforceCatalogRoot: managedAutoInstallKeys !== null,
distributionPlugins,
},
);
const bundledPackageResolver = managedAutoInstallKeys === null ? undefined : createBundledPluginPackageResolver(bundledPluginInstalls);
const managedBundledPluginKeys =
managedAutoInstallKeys !== null
? bundledPluginInstalls.map((install) => install.pluginKey)
: [];
let runtimePluginLoader: Pick<PluginLoader, "loadSingle"> | null = null;
// A sibling process can install a managed bundled plugin while this process
// skips the mid-install row, then finish the row after this process's
// loadAll() pass. The capabilities route may recover only those managed
// bundles by starting their ready-but-unstarted worker lazily.
const recoverManagedBundledPluginWorker =
managedAutoInstallKeys !== null
? createManagedBundledPluginWorkerRecovery({
managedBundledPluginKeys,
workerManager,
getLoader: () => runtimePluginLoader,
})
: undefined;
// Mount API routes
const api = Router();
const agentAvatars = agentAvatarRoutes();
api.use(agentAvatars.router);
api.use(boardMutationGuard());
api.use("/health", health);
api.use(openApiRoutes());
api.use("/cloud", cloudRoutes());
api.use("/companies", companyRoutes(db, opts.storageService));
api.use(llmRoutes(db));
api.use(folderRoutes(db));
api.use(companySkillRoutes(db, { pluginWorkerManager: workerManager }));
api.use(companySkillPolicyRoutes(db));
api.use(inboxAgentPolicyRoutes(db));
api.use(agentCommentaryRoutes(db));
api.use(agentChatMessageRoutes(db));
api.use(builtInAgentRoutes(db));
api.use(summarySlotRoutes(db));
api.use(statusCardRoutes(db));
api.use(teamsCatalogRoutes(db));
// The setup-token login session service. The router builds it and hands it
// back through the callback below, so the shutdown hook can cancel every live
// session (SR-4).
let setupTokenLoginService: SetupTokenSessionService | null = null;
// The dedicated proxy IP or CIDR allowlist for the confidential setup-token
// login responses (SR-7). The global `TRUST_PROXY` setting does not satisfy
// the guard; an operator sets this allowlist to the real TLS-terminating
// proxy addresses. An empty value keeps the confidential responses on direct
// TLS (or a `local_trusted` loopback peer) only.
const setupTokenLoginProxyAllowlist = (
process.env.CLAUDE_LOGIN_TRUSTED_PROXIES ?? ""
)
.split(",")
.map((entry) => entry.trim())
.filter((entry) => entry.length > 0);
// The explicit operator declaration that a platform edge terminates TLS for
// every client request (SR-7). This complements the allowlist for managed
// platforms (Railway, Render, Fly, and the like) where the app socket is
// always plain HTTP and the edge-proxy peer addresses are not stable or
// documented, so `CLAUDE_LOGIN_TRUSTED_PROXIES` cannot express them. It is a
// dedicated, single-purpose setting; the guard still never reads the global
// `TRUST_PROXY` value.
const setupTokenLoginEdgeTlsTerminated = /^(1|true|yes|on)$/i.test(
(process.env.CLAUDE_LOGIN_EDGE_TLS_TERMINATED ?? "").trim(),
);
// Bind the production setup-token login transport. It carries the live lease
// manager, the login-process factory over the sandbox pseudo-terminal, and the
// durable cleanup store. The factory passes only the fixed command
// `CLAUDE_SETUP_TOKEN_COMMAND`; it never reads a command from a
// route, a request body, or an adapter configuration. The durable store and the
// startup reaper are live now, so a restart reaps a leftover lease.
//
// The live sandbox pseudo-terminal opener binds inside the sandbox provider
// worker, so the server process does not hold the raw sandbox process. The
// opener drives the worker through the plugin worker manager route gate.
// The manager mints a host-owned route identifier, permits one
// active credential pseudo-terminal per worker, binds the worker session
// identifier one time for output only, and terminalizes the route on every open
// failure path. With the opener supplied, the provider acquires a lease and the
// start route drives a live login instead of the fixed 503.
const setupTokenLoginTransport = buildSetupTokenLoginTransport({
sandbox: createProductionSetupTokenSandboxProvider({
db,
environments: environmentService(db),
environmentRuntime: environmentRuntimeService(db, {
pluginWorkerManager: workerManager,
}),
openLivePtySession: createWorkerBoundLoginPtyOpener({
workerManager,
environments: environmentService(db),
log: (line) => logger.info(line),
}),
log: (line) => logger.info(line),
}),
store: createProductionSetupTokenCleanupStore(db),
// Bind the atomic credential-claim writer, so a completed login transitions
// the durable row to `stored` and stores the minted token in one control-plane
// transaction. The writer reads the company and the owner only from the
// immutable session scope. The confirm-replacement flow owns rotation. Without
// this writer the router falls back to the deferred, fail-closed 503 and never
// stores the token.
completeCredential: createSetupTokenSecretWriter({ db }),
// Forward the login runner diagnostic lines to the server logger. The
// runner is the sole producer, and every line is a fixed, non-secret
// literal. Without this sink the diagnostics fall back to a no-op in
// production, so a failed login leaves no log trail.
log: (line) => logger.info(line),
});
api.use(
agentRoutes(db, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
deploymentMode: opts.deploymentMode,
confidentialProxyAllowlist: setupTokenLoginProxyAllowlist,
confidentialEdgeTlsTerminated: setupTokenLoginEdgeTlsTerminated,
setupTokenLogin: setupTokenLoginTransport,
onSetupTokenLoginService: (service) => {
// Capture the service, so the graceful-shutdown hook cancels every live
// session and releases each lease. The standalone scheduled reaper owns
// the startup and interval lease cleanup now (SR-4).
setupTokenLoginService = service;
},
}),
);
api.use(agentPermissionGrantRoutes(db));
api.use(agentInteractionPermissionRoutes(db));
api.use(assetRoutes(db, opts.storageService));
api.use(agentProfileAvatarRoutes(db, opts.storageService));
api.use(projectToolRoutes(db));
api.use(taskWorkspaceRoutes(db));
api.use(projectRoutes(db));
api.use(caseRoutes(db, opts.storageService));
api.use(caseHandoffRoutes(db));
api.use(workflowRoutes(db));
api.use(issueTreeControlRoutes(db, { pluginWorkerManager: workerManager }));
api.use(fileResourceRoutes(db, { remoteFiles: createRemoteWorkspaceFiles(db, {
environmentRuntime: environmentRuntimeService(db, { pluginWorkerManager: workerManager }),
loadSavedWorkspace: createSavedWorkspaceLoader(db, opts.storageService),
openSavedWorkspaceArchive: createSavedWorkspaceArchiveOpener(db, opts.storageService),
}) }));
api.use(routineRoutes(db, { pluginWorkerManager: workerManager }));
api.use(
environmentRoutes(db, {
pluginWorkerManager: workerManager,
retryPendingEnvironmentCleanup: input => connectionIntentHeartbeat.sweepPendingCleanupLeases({ environmentRetry: input }),
recoverMissingPluginWorker: recoverManagedBundledPluginWorker
? {
pluginKeys: managedBundledPluginKeys,
startWorker: recoverManagedBundledPluginWorker,
}
: undefined,
}),
);
api.use(executionWorkspaceRoutes(db, { pluginWorkerManager: workerManager }));
api.use(emailRoutes(db, emailChannels));
api.use(goalRoutes(db));
api.use(onboardingSeedRoutes(db));
api.use(boardChatRoutes(db, { deploymentMode: opts.deploymentMode }));
api.use(voiceSessionRoutes(db, chatChannels.voice));
api.use(approvalRoutes(db, { pluginWorkerManager: workerManager }));
api.use(secretRoutes(db));
api.use(managedAgentProfileRoutes(db));
api.use(remoteAgentProfileRoutes(db));
api.use(
chatChannelRoutes(db, {
heartbeat: connectionIntentHeartbeat,
publicBaseUrl: opts.authPublicBaseUrl,
storage: opts.storageService,
service: chatChannels,
}),
);
const trustedLocalStdioRuntimeHost =
process.env.PAPERCLIP_TRUSTED_MCP_RUNTIME_HOST ??
process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ??
null;
api.use(costRoutes(db, { pluginWorkerManager: workerManager }));
api.use(activityRoutes(db));
api.use(dashboardRoutes(db));
api.use(attentionRoutes(db, {
pluginWorkerManager: workerManager,
decisionServiceOptions: opts.decisionServiceOptions,
}));
api.use(decisionTrainingRoutes(db));
api.use(decisionRoutes(db, opts.decisionServiceOptions));
api.use(decisionQueueRoutes(db));
api.use(userProfileRoutes(db));
api.use(sidebarBadgeRoutes(db));
api.use(sidebarPreferenceRoutes(db));
api.use(announcementRoutes(db, { ...opts.announcements, version: opts.hostVersion ?? serverVersion }));
api.use(resourceMembershipRoutes(db));
api.use(primaryAgentRoutes(db));
api.use(workspacePreferenceRoutes(db));
api.use(inboxDismissalRoutes(db));
api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup, opts.idleDeadlineOwners,
input => resourceLifecycleDelivery.prepareForIdleSleep(input)));
api.use(ephemeralEnrollmentRoutes(db));
if (opts.databaseBackupService) {
api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService));
}
const pluginRegistry = pluginRegistryService(db);
const eventBus = createPluginEventBus();
setPluginEventBus(eventBus);
const jobStore = pluginJobStore(db);
const lifecycle = pluginLifecycleManager(db, { workerManager });
const scheduler = createPluginJobScheduler({
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
db,
jobStore,
workerManager,
});
const toolDispatcher = createPluginToolDispatcher({
workerManager,
lifecycleManager: lifecycle,
db,
});
const gatewayOAuthAccess = toolAccessService(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
});
const toolActionDeliveries = toolActionDeliveryService(db, heartbeatService(db, { pluginWorkerManager: workerManager }));
const toolGateway = createToolGatewayService(db, {
budgetHooks: { cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
onToolActionSettled: (id) => toolActionDeliveries.deliver(id),
pluginToolDispatcher: toolDispatcher,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
trustedLocalStdioRuntimeHost,
oauthGrantRefresher: (input) =>
gatewayOAuthAccess.refreshOAuthGrantCredentials(input),
});
// Issue routes are intentionally mounted after the gateway is constructed because
// issue approval endpoints delegate to it. The intervening routers use distinct
// route prefixes, so this dependency does not change issue-route precedence.
api.use(issueRoutes(db, opts.storageService, {
chatRunRetries: chatChannels,
pluginWorkerManager: workerManager,
approveToolActionRequest: (input) => toolGateway.approveActionRequest(input),
declineToolActionRequest: (input) => toolGateway.declineActionRequest(input),
}));
api.use(slackToolRoutes(db, opts.authPublicBaseUrl));
app.locals.toolGateway = toolGateway;
registerAssignedMcpGateway(db, toolGateway);
app.locals.toolActionDeliveries = toolActionDeliveries;
app.use(mcpGatewayProtocolRoutes(toolGateway));
api.use(decisionModelRoutes(db));
api.use(agentObservabilityRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure }));
api.use(fastResponseRoutes(db));
api.use(linkPreviewRoutes(db));
api.use(promptInjectionPreventionRoutes(db));
api.use(aiConnectionRoutes(db, { deploymentMode: opts.deploymentMode, deploymentExposure: opts.deploymentExposure, trustedLocalStdioRuntimeHost, pluginWorkerManager: workerManager }));
api.use(
toolAccessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
authPublicBaseUrl: opts.authPublicBaseUrl,
trustedLocalStdioRuntimeHost,
toolGateway,
connectionIntentHeartbeat,
}),
);
api.use(connectionIntentBoardRoutes(db, connectionIntentHeartbeat));
api.use(
smokeLabRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
}),
);
const jobCoordinator = createPluginJobCoordinator({
db,
lifecycle,
scheduler,
jobStore,
});
const hostServiceCleanup = createPluginHostServiceCleanup(
lifecycle,
hostServicesDisposers,
);
let viteHtmlRenderer: ReturnType<typeof createCachedViteHtmlRenderer> | null =
null;
let viteDevServer: { close(): Promise<void> } | null = null;
let viteHmrServer: HttpServer | null = null;
const loader = pluginLoader(
db,
{
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
migrationDb: opts.pluginMigrationDb,
assertPackageActivation: bundledPackageResolver?.assertActivation ?? distributionPluginActivationGuard(bundledCatalogRoot, distributionPlugins, managedAutoInstallKeys),
resolvePackageRoot: bundledPackageResolver?.resolve,
prebuiltOnly: managedAutoInstallKeys !== null,
enableLocalFilesystem: managedAutoInstallKeys === null,
enableNpmDiscovery: managedAutoInstallKeys === null,
},
{
workerManager,
eventBus,
jobScheduler: scheduler,
jobStore,
toolDispatcher,
lifecycleManager: lifecycle,
instanceInfo: {
instanceId: opts.instanceId ?? "default",
hostVersion: opts.hostVersion ?? serverVersion,
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
},
buildHostHandlers: (pluginId, manifest) => {
const notifyWorker = (method: string, params: unknown) => {
const handle = workerManager.getWorker(pluginId);
if (handle) handle.notify(method, params);
};
const services = buildHostServices(
db,
pluginId,
manifest.id,
eventBus,
notifyWorker,
{
pluginWorkerManager: workerManager,
manifest,
},
);
hostServicesDisposers.set(pluginId, () => services.dispose());
return createHostClientHandlers({
pluginId,
capabilities: manifest.capabilities,
services,
});
},
},
);
runtimePluginLoader = loader;
workerManager.setActivationTimeoutRecovery?.(plugin => loader.requestActivationTimeoutRecovery(plugin));
const browserUse = browserUseService(db, undefined,
{ cancelWorkForScope: heartbeatService(db, { pluginWorkerManager: workerManager }).cancelBudgetScopeWork },
(session, run) => toolGateway.browserUseSessionAuthorized({ ...session, runId: run.heartbeatRunId, invocationId: run.invocationId }));
api.use(browserUseRoutes(db, browserUse));
api.use(toolGatewayRoutes(db, toolGateway));
api.use(
pluginRoutes(
db,
loader,
{ scheduler, jobStore },
{ workerManager },
{ toolDispatcher },
{ workerManager },
{ toolGateway },
),
);
api.use(
adapterRoutes({
db,
resolveRunnerTarget: (companyId, environmentId) => resolveAgentRunnerTargetForCompany(db, companyId, environmentId),
getNativeRunnerEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableNativeRunner === true,
getOpenAiDotEnabled: async () =>
(await instanceSettingsService(db).getExperimental())
.enableOpenAiDot === true,
}),
);
api.use(
accessRoutes(db, {
deploymentMode: opts.deploymentMode,
deploymentExposure: opts.deploymentExposure,
bindHost: opts.bindHost,
allowedHostnames: opts.allowedHostnames,
authPublicBaseUrl: opts.authPublicBaseUrl,
}),
);
let publicMcpEvents: PublicMcpEvents | null = null;
const grokDispatcher = createGrokRoutineDispatcher(db, () => !isWarmStandby() && !isIdleTaskDrainActive());
let dotMcpEvents: PublicMcpEvents | null = null;
if (publicMcpOAuth) {
const dispatch = createMcpApiDispatch(api, app.locals);
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
});
const transfers = createPublicMcpTransfers(db, publicMcpOAuth, dispatch, opts.storageService);
publicMcpIngress.use(transfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(publicMcpOAuth, createPublicMcpExecutor(db, publicMcpOAuth, dispatch, transfers), publicMcpEvents));
const dotOAuth = createPublicMcpOAuth(db, { ...publicMcpOAuth.config, resource: publicMcpOAuth.config.origin + "/mcp/runner" });
dotPrincipalOAuth = dotOAuth;
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive() });
const grokTransfers = createPublicMcpTransfers(db, dotOAuth, dispatch, opts.storageService, { pathPrefix: "/mcp/runner/files",
agentSession: { authorize: (principal, sessionId, taskId, write, database) => dotRunnerBroker(database ?? db).authorizeFileSession(principal, sessionId, taskId, write) } });
publicMcpIngress.use(grokTransfers.router);
publicMcpIngress.use(publicMcpIngressRoutes(dotOAuth, createPublicMcpExecutor(db, dotOAuth, dispatch), dotMcpEvents, createDotRunnerMcpTools(db, connectionIntentHeartbeat, grokTransfers)));
api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth));
api.use(dotRunnerRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
api.use(grokBotRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner"));
}
api.use(museBoardRoutes(db, opts.authPublicBaseUrl));
api.use(museQualificationRoutes(db));
app.use("/api", api);
app.use("/api", (_req, res) => {
res.status(404).json({ error: "API route not found" });
});
app.use(
pluginUiStaticRoutes(db, {
localPluginDir: opts.localPluginDir ?? DEFAULT_LOCAL_PLUGIN_DIR,
resolvePackageRoot: bundledPackageResolver?.resolve,
}),
);
const __dirname = path.dirname(fileURLToPath(import.meta.url));
if (opts.uiMode === "static") {
// Try published location first (server/ui-dist/), then monorepo dev location (../../ui/dist)
const candidates = [
path.resolve(__dirname, "../ui-dist"),
path.resolve(__dirname, "../../ui/dist"),
];
const uiDist = candidates.find((p) =>
fs.existsSync(path.join(p, "index.html")),
);
if (uiDist) {
staticUi.use(staticUiCompression());
// Hashed asset files (Vite emits them under /assets/<name>.<hash>.<ext>)
// never change once built, so they can be cached aggressively.
staticUi.use(
"/assets",
staticUiFiles(path.join(uiDist, "assets"), {
maxAge: "1y",
immutable: true,
}),
);
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
// Publish while the routes below are built; the outcome is awaited
// before the first background work starts. The promise settles to the
// failure instead of rejecting, so it is never an unhandled rejection.
app.locals.retainedUiAssetsPublication = trackIdleWork(publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"))
.then(() => null, (error: unknown) => ({ error })));
staticUi.use("/assets", retained.serve);
}
// Serve root/index through the same runtime HTML transform as SPA routes.
staticUi.get(["/", "/index.html"], (_req, res) => {
res.type("html").set("Cache-Control", "no-cache").send(readBrandedStaticIndexHtml(uiDist));
});
// Non-hashed static files (favicon.ico, manifest, robots.txt, etc.):
// short cache so operators who swap them out see the new version
// reasonably fast, with must-revalidate overrides for index.html and
// sw.js (see staticUiCacheControl for why those two).
staticUi.use(
staticUiFiles(uiDist, {
maxAge: "1h",
setHeaders(res, filePath) {
const override = staticUiCacheControl(filePath);
if (override) {
res.set("Cache-Control", override);
}
},
}),
);
// SPA fallback. Only for non-asset routes — if the browser asks for
// /assets/something.js that doesn't exist, we must NOT serve the HTML
// shell: the browser would try to load it as a JavaScript module, fail
// with a MIME-type error, and cache that broken response. Return 404
// instead. The index.html response itself is no-cache so a subsequent
// deploy's updated asset hashes are picked up on next load.
staticUi.get(/.*/, (req, res) => {
if (req.path.startsWith("/assets/")) {
res.status(404).end();
return;
}
res
.status(200)
.set("Content-Type", "text/html")
.set("Cache-Control", "no-cache")
.end(readBrandedStaticIndexHtml(uiDist));
});
} else {
console.warn("[paperclip] UI dist not found; running in API-only mode");
}
if (process.env.PAPERCLIP_MANAGED_RUNTIME_EXPOSURE === "tailscale_https") {
// The managed-runtime supervisor waits for the app port AND its derived
// Vite HMR companion port to bind before publishing the service. Static
// mode has no Vite, so bind the same placeholder listener dev mode uses
// or the supervisor kills a healthy server at the readiness deadline
// (PAP-18043).
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
await listenViteHmrServer(
hmrServer,
resolveViteHmrPort(opts.serverPort),
opts.bindHost,
);
viteHmrServer = hmrServer;
}
}
if (opts.uiMode === "vite-dev") {
const uiRoot = path.resolve(__dirname, "../../ui");
const publicUiRoot = path.resolve(uiRoot, "public");
const hmrPort = resolveViteHmrPort(opts.serverPort);
const hmrHost = resolveViteHmrHost(opts.bindHost);
const hmrProtocol = resolveViteHmrProtocol(
process.env.PAPERCLIP_VITE_HMR_PROTOCOL,
);
const hmrServer = createHttpServer((_req, res) => {
res.writeHead(426, { "Content-Type": "text/plain" });
res.end("Upgrade Required");
});
const { createServer: createViteServer } = await import("vite");
const configuredViteCacheDir = process.env.PAPERCLIP_VITE_CACHE_DIR?.trim();
const vite = await createViteServer({
root: uiRoot,
...(configuredViteCacheDir
? { cacheDir: path.resolve(configuredViteCacheDir) }
: {}),
appType: "custom",
// Vite otherwise discovers every HTML entry below the UI root. Generated
// Storybook output can reference dependencies that are intentionally not
// part of the application install, poisoning a clean embedded dev-server
// cache before the browser opens. The embedded UI has one real entry.
optimizeDeps: { entries: [path.resolve(uiRoot, "index.html")] },
server: {
// Listener binding and browser HMR hostname are deliberately separate:
// exposed branch runtimes stay loopback-only while the browser uses the
// current MagicDNS hostname through the broker's HTTPS listener.
host: opts.bindHost,
middlewareMode: true,
hmr: {
server: hmrServer,
...(hmrHost ? { host: hmrHost } : {}),
...(hmrProtocol ? { protocol: hmrProtocol } : {}),
port: hmrPort,
clientPort: hmrPort,
},
allowedHosts: privateHostnameGateEnabled
? Array.from(privateHostnameAllowSet)
: undefined,
},
});
try {
await listenViteHmrServer(hmrServer, hmrPort, opts.bindHost);
} catch (error) {
await vite.close();
throw error;
}
viteDevServer = vite;
viteHmrServer = hmrServer;
viteHtmlRenderer = createCachedViteHtmlRenderer({
vite,
uiRoot,
brandHtml: applyUiBranding,
});
const renderViteHtml = viteHtmlRenderer;
if (fs.existsSync(publicUiRoot)) {
staticUi.use(staticUiFiles(publicUiRoot, { index: false }));
}
staticUi.get(/.*/, async (req, res, next) => {
if (!shouldServeViteDevHtml(req)) {
next();
return;
}
try {
const html = await renderViteHtml.render(req.originalUrl);
res.status(200).set({ "Content-Type": "text/html" }).end(html);
} catch (err) {
next(err);
}
});
staticUi.use(vite.middlewares);
}
app.use(staticUi);
app.use(errorHandler);
// A build whose asset names collide with different published bytes refuses
// to start, so the outcome is known before any job, queue, or run starts.
const retainedUiAssetsPublication = await (app.locals.retainedUiAssetsPublication as Promise<{ error: unknown } | null> | undefined);
if (retainedUiAssetsPublication) throw retainedUiAssetsPublication.error;
jobCoordinator.start();
const deliveryWork = createDeliveryWorkCoordinator({
owner: db,
canRun: () => !isWarmStandby() && !isIdleTaskDrainActive(),
canReconcile: () => !isWarmStandby(),
onError: (err, queue) => logger.error({ err, queue }, "Delivery reconciliation failed"),
});
app.locals.deliveryWork = deliveryWork;
await scheduler.start(deliveryWork);
const agentLifecycleDelivery = deliveryWork.register(DELIVERY_QUEUES.agentLifecycle, {
retryMs: 1_000,
run: () => agentLifecycle.sweep(),
hasPending: () => agentLifecycle.hasPending(),
nextRunAt: () => agentLifecycle.nextRunAt(),
});
agentLifecycle.setWake(agentLifecycleDelivery.wake);
const clientMirror = startMcpClientMirrorSync(db, deliveryWork, mcpConfig?.origin,{authorizationOrigin:mcpConfig?.authorizationOrigin});
const unregisterMcpOwners = registerIdleMcpOwners(db, { config: mcpConfig, personal: publicMcpEvents, dot: dotMcpEvents, clientMirrorInstanceId:clientMirror.instanceId });
startMcpRevocationIndexSync(db, deliveryWork);
startMcpIngressHandoffExpiry(db, deliveryWork);
publicMcpEvents?.start(deliveryWork);
dotMcpEvents?.start(deliveryWork);
if (publicMcpOAuth) grokDispatcher.start(deliveryWork);
const stopMuseMaintenance = startMuseMaintenance(db, deliveryWork, async () => {
const state = await connectionIntentHeartbeat.resolveSchedulingSuppression(true);
return !state.suppressed || state.reason === "task_drain";
});
const resourceLifecycleDelivery = registerPluginLifecycleDelivery(deliveryWork, db, workerManager, lifecycle);
await resourceLifecycleDelivery.ready;
if (opts.feedbackExportService) {
deliveryWork.register(DELIVERY_QUEUES.feedback, {
retryMs: FEEDBACK_EXPORT_FLUSH_INTERVAL_MS,
run: (signal) => opts.feedbackExportService!.flushPendingFeedbackTraces({ signal }),
hasPending: () => opts.feedbackExportService!.hasPendingFeedbackTraces(),
});
}
registerChatDeliveryWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
registerChatActionWork(deliveryWork, chatChannels, () => !isIdleTaskDrainActive());
chatChannels.voice.registerReportWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.registerReplyWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerCleanupWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.voice.inbound.registerRetentionWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
registerGitHubReviewProjectionWork(db, deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubReviewCheckWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubTerminalResponseWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubAcknowledgementWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatFailedRetryNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatRunMilestoneWork(db, opts.authPublicBaseUrl).register(
deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive(),
);
emailChannels.start();
chatChannels.registerGitHubWebhookRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
const importSpoolCleanup = registerImportSpoolCleanupWork(db, deliveryWork,
resolveDefaultImportTransferSpoolRoot(), () => !isWarmStandby() && !isIdleTaskDrainActive());
registerBrowserUseCleanup(deliveryWork, browserUse, () => !isIdleTaskDrainActive());
void companyTransferRunService
.recoverStrandedApplyingRuns(db)
.then((recovered) => {
if (recovered.length > 0) {
logger.warn(
{ count: recovered.length, runIds: recovered },
"failed company transfer runs stranded in applying by a restart",
);
}
})
.catch((err) => {
logger.error({ err }, "stranded company transfer apply recovery failed");
});
void toolDispatcher.initialize().catch((err) => {
logger.error({ err }, "Failed to initialize plugin tool dispatcher");
});
const devWatcher = managedAutoInstallKeys !== null ? null : createPluginDevWatcher(
lifecycle,
async (pluginId) =>
(await pluginRegistry.getById(pluginId))?.packagePath ?? null,
undefined,
{ unwatchedRoots: [bundledCatalogRoot] },
);
// Auto-provision bundled plugins so their providers are registered for
// agent runs. Bundles are excluded from the pnpm
// workspace and built standalone into the image (see Dockerfile), then
// installed here from their local paths. This runs BEFORE loadAll() so
// loadAll() can activate them in the same startup pass.
//
// Workers are started exactly once, by loadAll(): the `lifecycle` manager
// above is constructed without a runtime-capable loader
// (pluginLifecycleManager(db, { workerManager }) — no `loader` option), so
// the lifecycle.load() that ensureBundledPlugins performs per newly
// installed bundle only records the `ready` status and does not spawn a
// worker (see activateReadyPlugin in services/plugin-lifecycle.ts).
//
// Managed instances (`plugins.autoInstall` from PAPERCLIP_MANAGED_CONFIG)
// drive the key list from the control plane; self-hosted instances keep
// the pre-existing behavior of ensuring only the kubernetes bundle.
//
// Resolution is deliberately synchronous and NOT fail-safe: an
// unknown key or a path escaping the bundled catalog root throws out of
// createApp so a managed instance refuses to start (positive allowlist,
// fail closed).
// SAFETY: installation is fully fail-safe. Any failure
// (missing bundle, install error, load error) is caught, logged, and
// swallowed per plugin so the server ALWAYS finishes booting. A degraded
// boot (a provider unavailable, some agents cannot run) is strictly
// preferable to a crash loop.
//
// The chain is not awaited here (createApp stays fast), but the settled
// promise is exposed via `app.locals.bundledPluginsStartup` so boot steps
// that must not outrun plugin availability — managed sandbox environments
// (`applyManagedEnvironments`) run before the heartbeat resumes queued
// runs — can sequence on it. It never rejects.
const bundledPluginsStartup = ensureBundledPlugins(
bundledPluginInstalls,
{ registry: pluginRegistry, loader, lifecycle, logger },
// Managed mode reinstalls soft-uninstalled bundles (the control plane
// owns provisioning); self-hosted leaves an operator's uninstall alone.
// Disabled plugins never start automatically. Added distribution permissions
// still enter upgrade_pending so enabling them requires an operator decision.
{ reinstallUninstalled: managedAutoInstallKeys !== null, preservePackagePath: managedAutoInstallKeys !== null },
)
.then(() => loader.loadAll())
.then((result) => {
if (!result) return;
for (const loaded of result.results) {
if (devWatcher && loaded.success && loaded.plugin.packagePath) {
devWatcher.watch(loaded.plugin.id, loaded.plugin.packagePath);
}
}
})
.catch((err) => {
logger.error({ err }, "Failed to load ready plugins on startup");
});
app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
// Manifest ids the managed auto-install list resolves to; the managed
// environments bootstrap checks plugin entries against them.
app.locals.managedBundledPluginKeys = managedBundledPluginKeys;
void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying."));
// The shutdown hook runs at most once. It caches the in-flight promise, so a
// second caller (for example the `exit` handler) awaits the same completion
// instead of starting a second teardown.
let appServicesShutdown: Promise<void> | null = null;
const shutdownAppServices = (): Promise<void> => {
if (appServicesShutdown) return appServicesShutdown;
appServicesShutdown = finishServiceShutdown(() => scheduler.stop(), async () => {
await loader.stopActivationTimeoutRecovery();
await agentLifecycle.stop();
unregisterMcpOwners();
await publicMcpEvents?.stop();
await dotMcpEvents?.stop();
jobCoordinator.stop();
await finishServiceShutdown(() => resourceLifecycleDelivery.dispose(), async () => {
importSpoolCleanup.unsubscribe();
await deliveryWork.stop();
devWatcher?.close();
viteHtmlRenderer?.dispose();
void viteDevServer?.close().catch(() => undefined);
viteHmrServer?.close();
hostServiceCleanup.disposeAll();
hostServiceCleanup.teardown();
await stopMuseMaintenance();
await emailChannels.shutdown();
await chatChannels.shutdown();
// End the avatar worker pool, if a request ever started one, so no
// render outlives the HTTP teardown.
await agentAvatars.close();
// Cancel every live setup-token login session and AWAIT the cancellation,
// so each direct child stops and the server releases each lease before the
// caller stops the database and the provider. A lease release that
// fails stays a durable record for the startup reaper.
await setupTokenLoginService?.shutdown();
});
});
return appServicesShutdown;
};
app.locals.paperclipShutdown = shutdownAppServices;
// The `exit` event is synchronous. It cannot await the teardown, so it runs
// the best-effort cleanup and drops the returned promise. The orderly signal
// path awaits `shutdownAppServices` in full before the process exits.
process.once("exit", () => {
void shutdownAppServices();
});
process.once("beforeExit", () => {
void flushPluginLogBuffer();
});
trackIdleRequestHandlers(app);
return app;
}