Files
PaperClipAI/server/src/routes/environments.ts
T
DottaandPaperclip d67e3507b2 feat(runner): qualify Codex platform packages and cache hosted builds (#16028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution needs a compatible installed daemon and provider
dependency.
> - The Codex-default work found missing platform packages and
clean-install startup failures.
> - Release builds also repeat dependency compilation and run
preparation serially.
> - This prerequisite packages four daemon targets and verifies
installed native startup.
> - It also caches hosted Cargo dependencies and overlaps JavaScript
preparation with compilation.
> - Agent creation defaults and experimental gates stay at the current
master behavior.

## Linked Issues or Issue Description

Refs #15683. Refs #15555. Refs #15422. Refs #15926. Refs #15932.

Extract runtime and installation support from #15683. This prerequisite
has 65 changed files. Creation, onboarding, imports, UI runner selection
and graduation remain in the dependent PR. Feature/access maps and the
Storybook source catalog stay synchronized. Existing agents keep their
execution choices.

## What Changed

- Correct the existing runner test fixtures. Reserve owned credential
quorum listeners through the existing bounded fixture helper. Keep
foreign listeners intact, run actual credential staging once, and
preserve its rejection and unchanged-auth assertions. Reuse the
durable-resume outer-limit correction. Wait for actual warm-completion
and held-authorization callback entry under the existing recovery
deadline and surface early request errors. Preserve every semantic
assertion.
- Retain bounded installation failure diagnostics through the existing
qualification receipt upload. Keep the original error, 10-minute
deadline and all success checks.
- Keep startup ownership until credential preparation and registration
drain. Reject duplicate and post-close startup/resume work. Observe
shutdown during normal and adopted authentication readiness. Preserve
the primary failure and retain private state when cleanup is
unconfirmed.
- Package Linux x64/ARM64 and macOS Apple Silicon/Intel daemons. Verify
target headers, source identities, manifests, and hashes before assembly
and installation.
- Reuse the existing release producer/consumer workflows. Cache Cargo
dependencies by target, OS, compiler, lockfile, and contract inputs.
Rebuild workspace binaries. Permit trusted cache writes only for
canonical exact-source master pushes.
- Overlap JavaScript preparation with native compilation. Join both
commands and failure/cancellation cleanup before artifact verification.
- Preserve shell positional arguments in the shared SSH runner and apply
environment assignments correctly. Reject environment option keys. Six
before/after controls and two real SSH subprocess paths pass with zero
provider calls; this fix lands before both Codex and Claude defaults.
- Admit verified suspended SSH backups during cold controller
continuation, using the same full-scope, integrity and lifecycle checks
as sandbox targets. Two positive SSH cases fail before; all 14
positive/negative matrix cases pass afterward. Active ownership and
corrupt/malformed/nonempty state remain rejected.
- Record measured sandbox platform facts through the existing explicit
environment probe. Bind facts to the company, effective provider
configuration, immutable OCI image and actual captured image binding.
Check the returned cleanup receipt. Reject client-authored proof and
stale or named boot identities. Creation and discovery allocate nothing.
Preserve connection-only behavior.
- Resolve the actual remote Codex target and preserve same-host
development overrides. Preserve runtime artifact verification. Use
actual protocol/model compatibility rather than a numerical CLI version
fence.
- Resolve the installed Codex login dependency on 64-bit Linux and Mac
independently of the daemon libc baseline. Preserve account homes and
safe errors.
- Fix actual clean-install TOML table comparison. Preserve strict
semantic checks, comments, private writes, and malformed-input
rejection. Preserve primary startup failures and bounded cleanup.
- Document GNU libc 2.39+ and stock Ubuntu sandbox prerequisites.
Qualify ARM startup with a scoped distribution profile and unchanged
global restrictions. Preserve Windows and Alpine deferral.
- Preserve selected and saved native Codex reasoning effort in the
existing UI builder, while excluding it from other providers. This small
shared correction prevents import edits higher in the stack from
dropping effort.
- Reuse existing installed-package checks and assertions. Paperclip
tarballs contain no Codex native payloads. Official npm dependencies
supply the consumer's Codex binaries. All Docker and native builds stay
remote.

## Verification

Candidate: `271d434c170e74bdedde65d0b13059a55419c943`. It includes
current master `e168512f7bdbb6af6ea01954c6d7ed80b91e903e` and preserves
prior candidates through ordinary commits.

Current [full
CI](https://github.com/paperclipai/paperclip/actions/runs/38346601686)
is successful on attempt 2. GitHub reports 54 successful checks and four
conditional skips, including required verify and browser aggregates,
workspace typecheck/tests/build, Linux DB controls, both runner shards
and [isolated native
compilation](https://github.com/paperclipai/paperclip/actions/runs/38346601609).
Fresh current-head Greptile is 5/5 with all four threads resolved. The
passing build's clean Grok/Codex report records CI merge source
`7a82a37438e6e448abc7cc5caa60dc2519a97be0`, 18 installed packages,
omitted Codex payloads, official host-only npm dependencies and zero
provider calls. Pi setup and its no-prompt lease verification also pass.
The first attempt's single 60-second Node archive download timeout
remains retained; its DNS/TLS/download cause remains unknown. One
unchanged-source retry was requested through the normal Build-job dialog
with debug logging disabled. GitHub's attempt 2 graph reports the other
CI jobs as well; no second retry or source/deadline relaxation was made.

The [final exact-source package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38347027535)
is successful on literal `271d434c170e74bdedde65d0b13059a55419c943`. It
passes all four native producers, public Linux package assembly and
clean installed consumers on Apple Silicon, Intel Mac and Linux ARM64.
The nine qualification-mode skips are intentional; full CI above covers
their ordinary gates. Native builds and Docker assembly ran remotely.
The predecessor [`dae808` package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38342104102)
remains literal historical evidence. Only tests and the dated record
differ from that source; runtime, package and workflow code is
unchanged. The reused credential fixture correction passes all 42
canonical Node 24 controls with synthetic data and independent
matching-source review, and the current hosted runner shard passes.
Production credential behavior is unchanged.

**Merge hold:** current-source CI, fresh 5/5 review and exact-source
package qualification pass. Actual code-owner approval and normal
protected merge remain pending. No slice has merged or deployed. GitHub
CLI authentication needs refresh; the normal signed-in embedded GitHub
UI provided this status without credential workarounds.

- The master merge preserves both Storybook source-review appendices and
the guarded test-home cleanup. All six existing cleanup controls pass
without skips. Master's isolated-home helper/test/setup are
byte-identical. The two existing whitespace lines in master's unrelated
privacy test remain outside this PR's diff.
- Canonical Node 24 passes all 24 existing public-install helper tests.
Three narrow added cases prove bounded/redacted failure metadata,
symlink exclusions, original exception identity, failed diagnostic
writes and no false failure on success. The Mac installer now uses
`--timing`; at most two 16 KiB tails and 32 structured entries per tail
enter the existing receipt upload. Raw argv, URLs, environment, error
text and stacks are omitted. A failed receipt explicitly remains
FAILED/unqualified/runtime-not-attempted. The 600,000 ms install
deadline and every real package/hash/payload/lock/lifecycle/runtime
oracle are unchanged.
- Canonical Node 24 reproduces the durable-resume outer timeout under
controlled fake latency. Only its outer test limit changes to 15
seconds; internal 500 ms/5 second deadlines and usage/effect assertions
remain. Controlled warm-recovery and before-bootstrap authorization
scheduling delays reproduce their callback polling failures, while the
explicit entry signals pass. Early recovery errors surface unchanged.
Independent review is clear. Historical mock binaries support the
fixture controls; fresh full hosted CI remains decisive. The original
hosted callback failure cause is inconclusive.
- The composed runner/surfaces/server/E2E TypeScript checks pass. Map
checks pass 35 areas/171 capability IDs. Token gates pass. The existing
Storybook catalog passes 223 files/2,210 stories; story IDs and
classifications remain intact. The current master had 25 inherited stale
fingerprints from three changed map recipes. Only those guarded
fingerprints and a source-only review note change; all 92 non-map
story/component/fixture inputs remain unchanged. Existing
classifications are retained. This proves source structure, not visual
or live acceptance.
- The 65-file contribution audit has no code file crossing from below
1,000 lines to above the limit. Earlier comment cleanup preserves
executable emissions, parsed workflows and action pins. Independent
cleanup/catalog and three-file reporting reviews are clear. No new
workflow, suite, grader or local native/Docker build was added.

**Prior `dae808` gate evidence:** fresh Greptile is 5/5 with four
resolved threads, isolated native compilation and the browser aggregate
pass. Its first runner shard passes the previous two corrections but
fails a different held-authorization cell at the implicit one-second
polling deadline. It records 1,672 passes/one failure/three skips. This
candidate applies the verified entry-signal correction without changing
production deadlines or any semantic assertions. Its original hosted
cause remains inconclusive until fresh CI.

**Previous-source evidence, kept literal:** the [exact `16a532` package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38338806076)
passes all four native producers, public Linux package assembly, Apple
Silicon, Intel Mac and Linux ARM64 installed consumers. Its fresh review
is 5/5 with all four threads resolved. Full CI records 52 successful
checks, four conditional skips and the two fixture failures corrected
here. This is not a green full-CI candidate or proof for the new source.

`64c074fb8da1b31f8d6042c4fef48dc4164e759b` passed [full
CI](https://github.com/paperclipai/paperclip/actions/runs/38332726938),
54 checks/four conditional skips, [isolated native
compilation](https://github.com/paperclipai/paperclip/actions/runs/38332726787/job/115085895328),
and fresh Greptile 5/5/all four resolved threads. The hosted transport
suite passed 246 tests and startup trust passed 22, closing that
source's local fake-binary limitation. These are historical after
reconciliation.

The [previous exact-source package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38332921463)
built all four platform daemons and passed public Linux package assembly
plus Apple Silicon and Linux ARM64 installed consumers. **Intel Mac
failed:** its initial scripts-disabled npm install timed out before
lifecycle/runtime assertions. The underlying download/resolution/host
cause is unknown because the old helper deleted the npm logs. This is
the concrete baseline for the diagnostic change. The failed campaign
remains failed; no all-platform signoff or automatic retry is claimed.

The earlier `a200495` [four-target
campaign](https://github.com/paperclipai/paperclip/actions/runs/38283695248)
and [scoped ARM
qualification](https://github.com/paperclipai/paperclip/actions/runs/38291535914)
remain component evidence. Cargo restores were cold misses; no
warm-cache speedup is claimed. Paperclip tarballs contain no Codex
payloads. Official npm dependencies supply Codex on the user's platform.

Current-source full workspace typecheck/tests/build, Linux DB controls,
static/Rust/native tests, all browser shards and exact-source package
consumers must pass. New-agent defaults and final managed onboarding
belong to #15683. Actual private Cloud composition, warm/cold signup,
task/follow-up and rollback compatibility remain later release gates.
Source audits found that ordinary managed onboarding does not run the
saved platform probe and Cloud distributes named snapshots without a
verified final boot-image binding. These dependent release blockers
remain open; prequalified E2E fixtures do not close them. Master adds
task-workspace migrations; the final image/migrator and previous-image
checks must include them. ARM paid task/follow-up and ordinary OS APT
installation are not proven.

## Risks

- Multi-target packages require source and header agreement. Missing or
mismatched artifacts fail explicitly. Existing execution choices remain
unchanged.
- Linux requires GNU libc 2.39+ and compatible sandbox prerequisites.
The scoped hosted ARM preparation does not qualify arbitrary containers
or kernels. Windows, 32-bit ARM, and Alpine remain deferred.
- Only dependency crates are cached. Preview jobs cannot write trusted
caches. Concurrent command cleanup must preserve the primary failure.
- Native-free Paperclip packaging retains official Codex npm
dependencies. No Codex payload bundling, schema migration, automatic
agent migration, or production deployment is included.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and parallel agents. The exact serving model snapshot and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-13 06:24:07 -05:00

1489 lines
59 KiB
TypeScript

import { Router, type Request } from "express";
import type { Db } from "@paperclipai/db";
import {
AGENT_ADAPTER_TYPES,
cancelEnvironmentCustomImageSetupSessionSchema,
createEnvironmentCustomImageTerminalSessionTokenSchema,
createEnvironmentSchema,
finishEnvironmentCustomImageSetupSessionSchema,
getEnvironmentCapabilities,
probeEnvironmentConfigSchema,
resolveDeclaredSandboxCapabilities,
redactEnvironmentCustomImageSetupSession,
redactEnvironmentCustomImageTemplate,
relinkEnvironmentCustomImageTemplateSchema,
startEnvironmentCustomImageSetupSessionSchema,
type EnvironmentDeleteBlastRadius,
updateEnvironmentSchema,
} from "@paperclipai/shared";
import { conflict, forbidden, unprocessable } from "../errors.js";
import { isCloudManagedInstance } from "../services/cloud-instance.js";
import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js";
import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js";
import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js";
import { validate } from "../middleware/validate.js";
import { logger } from "../middleware/logger.js";
import {
environmentCustomImageService,
issueService,
instanceSettingsService,
logActivity,
projectService,
} from "../services/index.js";
import {
environmentCustomImageTerminalConnectionRegistry,
environmentCustomImageTerminalSessionStore,
validateCustomImageSetupSshPayload,
type EnvironmentCustomImageTerminalPayloadValidationResult,
} from "../services/environment-custom-image-terminal-sessions.js";
import {
readCustomImageSetupSessionCompanyId,
requireFutureCustomImageSetupExpiry,
} from "../services/environment-custom-image-setup-session-utils.js";
import {
collectEnvironmentSecretRefs,
normalizeEnvironmentConfigForPersistence,
normalizeEnvironmentConfigForProbe,
readSshEnvironmentPrivateKeySecretId,
type ParsedEnvironmentConfig,
} from "../services/environment-config.js";
import { probeEnvironment } from "../services/environment-probe.js";
import { secretService } from "../services/secrets.js";
import {
listReadyPluginEnvironmentDrivers,
type ReadyPluginWorkerRecovery,
} from "../services/plugin-environment-driver.js";
import { getConfiguredSecretProvider } from "../secrets/configured-provider.js";
import { assertBoardOrgAccess, getActorInfo } from "./authz.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
import { environmentService } from "../services/environments.js";
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "../services/environment-custom-image-runtime.js";
import { environmentRuntimeService } from "../services/environment-runtime.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { closeWarmNativeSessionsForEnvironment } from "../services/native-runtime/native-session-executor.js";
function isPlainRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
/**
* Whether this environment row was provisioned by the platform: the
* managed-config environment provisioner marker, or the legacy managed
* Kubernetes wrapper marker (rows created by builds that predate the
* generalized provisioner and have not been adopted yet).
*/
export function isPlatformProvisionedEnvironment(environment: {
metadata: Record<string, unknown> | null;
}): boolean {
return (
environment.metadata?.managedByPaperclip === true ||
environment.metadata?.managedKubernetesSandbox === true
);
}
function redactSecretLikeConfigKeys(value: Record<string, unknown>): Record<string, unknown> {
const result: Record<string, unknown> = {};
for (const [key, child] of Object.entries(value)) {
if (SECRET_LIKE_CONFIG_KEY_PATTERN.test(key)) continue;
if (isPlainRecord(child)) {
result[key] = redactSecretLikeConfigKeys(child);
} else if (Array.isArray(child)) {
result[key] = child.map((element) =>
isPlainRecord(element) ? redactSecretLikeConfigKeys(element) : element,
);
} else {
result[key] = child;
}
}
return result;
}
/**
* Floor view of a platform-provisioned environment on a cloud-managed
* instance: credential-shaped config keys are never echoed — to ANY actor,
* including instance admins — while structural config (provider, image,
* template, region, ...) and the managed markers in `metadata` stay
* visible so admin surfaces can render the environment and show its
* platform-managed state.
*
* Env vars are the one tenant-owned field on a managed sandbox row: the
* platform never writes them (`ensureManagedSandboxEnvironment` reconciles
* name/config/metadata/status only), and tenants may edit them through the
* envVars-only PATCH exception below — so they echo back for round-trip
* editing. Everywhere else (the local slot, legacy kubernetes-marker rows
* that pre-generalization builds may have stamped platform values on) env
* vars stay blanked.
*/
export function applyPlatformProvisionedEnvironmentFloor<T extends {
driver?: string;
config: Record<string, unknown> | null;
envVars?: Record<string, unknown> | null;
metadata: Record<string, unknown> | null;
}>(environment: T): T {
const tenantEnvVarsVisible = isTenantEditableManagedSandbox(environment);
return {
...environment,
config: redactSecretLikeConfigKeys(isPlainRecord(environment.config) ? environment.config : {}),
...(Object.prototype.hasOwnProperty.call(environment, "envVars") && !tenantEnvVarsVisible
? { envVars: {} }
: {}),
};
}
/**
* Whether this platform-provisioned row participates in the tenant
* env-vars contract: the generalized managed sandbox slot only. The local
* slot and legacy kubernetes-marker rows do not — the tenant never runs
* on local under this regime, and legacy rows may carry platform-written
* values.
*/
function isTenantEditableManagedSandbox(environment: {
driver?: string;
metadata: Record<string, unknown> | null;
}): boolean {
return (
environment.driver === "sandbox" &&
environment.metadata?.managedByPaperclip === true &&
environment.metadata?.managedKubernetesSandbox !== true
);
}
const PLATFORM_PROVISIONED_MARKER_KEYS = [
"managedByPaperclip",
"managedKubernetesSandbox",
] as const;
/**
* Whether some bootstrap path on this instance currently owns the managed
* sandbox slot: the managed-config `environments` section
* (`applyManagedEnvironments`) or the forced execution-mode bootstrap
* (`PAPERCLIP_EXECUTION_MODE=kubernetes`). Both adopt and refresh the
* marked sandbox row on every boot. Fails closed: an unparseable document
* or env value counts as configured, keeping the slot protected (a
* malformed value refuses startup anyway, so a running server never hits
* the catch).
*/
function isManagedSandboxProvisioningConfigured(): boolean {
try {
if ((getManagedInstanceConfig()?.environments.length ?? 0) > 0) return true;
} catch {
return true;
}
try {
return parseExecutionPolicyBootstrapEnv(process.env) !== null;
} catch {
return true;
}
}
/**
* Whether this row occupies a provisioner-owned environment slot whose
* platform markers are LIVE state — i.e. some platform code path converges
* the markers on this exact row, so a marker here is never a stale
* leftover:
*
* - The single local row (`environments_local_driver_idx`): on a
* cloud-managed instance `ensureLocalEnvironment` adopts and stamps it
* from every caller (company creation, the heartbeat, run
* orchestration), so the local slot is platform-owned unconditionally.
* - The single marked sandbox row (`environments_managed_sandbox_idx`):
* owned only while a managed-sandbox bootstrap path is configured —
* `ensureManagedSandboxEnvironment` then adopts and refreshes whichever
* row holds the slot on every boot.
* - Any sandbox row bearing the legacy kubernetes marker while the
* persisted instance execution policy forces kubernetes:
* `findKubernetesEnvironment` selects marked rows (newest first) for
* every forced run, so under that policy each marked row is — or on the
* newest row's removal becomes — the live runtime row. This holds even
* with no bootstrap path configured: the per-run guard reads the
* persisted `executionMode`, which can outlive the env that seeded it.
*
* With no provisioning path configured and no forced-kubernetes policy the
* platform holds no claim on any sandbox row, so a platform marker there
* is stale by definition and stays recoverable via the marker-clear
* escape hatch below.
*/
async function isPlatformSlotEnvironment(
environment: { driver: string; metadata: Record<string, unknown> | null },
options: { isForcedKubernetesExecution: () => Promise<boolean> },
): Promise<boolean> {
if (environment.driver === "local") return true;
if (environment.driver !== "sandbox") return false;
if (
environment.metadata?.managedByPaperclip === true &&
isManagedSandboxProvisioningConfigured()
) {
return true;
}
return (
environment.metadata?.managedKubernetesSandbox === true &&
(await options.isForcedKubernetesExecution())
);
}
/**
* Returns true when the PATCH body is a metadata-only write whose only
* purpose is to clear platform markers (setting them to null/false). This is
* the escape hatch for rows that had these markers stamped through the old
* unrestricted API before the floor was introduced. It never applies to a
* row whose slot markers are live (see `isPlatformSlotEnvironment`) —
* clearing a live slot row's markers would reclassify it as
* tenant-managed and lift the write floor in two steps.
*/
function isPlatformMarkerClearOnlyPatch(body: unknown): boolean {
if (!isPlainRecord(body)) return false;
const bodyKeys = Object.keys(body);
if (bodyKeys.length !== 1 || bodyKeys[0] !== "metadata") return false;
const metadata = body.metadata;
if (!isPlainRecord(metadata)) return false;
const metaKeys = Object.keys(metadata);
if (metaKeys.length === 0) return false;
return metaKeys.every(
(key) =>
(PLATFORM_PROVISIONED_MARKER_KEYS as readonly string[]).includes(key) &&
(metadata[key] === null || metadata[key] === false),
);
}
/**
* Returns true when the PATCH body touches nothing but `envVars` (as a
* plain object). This is the one tenant edit the managed-environment write
* floor admits: agents need environment variables in their managed
* sandbox, and everything else on the row (name, driver, config, status,
* metadata) stays platform-owned.
*/
function isTenantEnvVarsOnlyPatch(body: unknown): boolean {
if (!isPlainRecord(body)) return false;
const bodyKeys = Object.keys(body);
return bodyKeys.length === 1 && bodyKeys[0] === "envVars" && isPlainRecord(body.envVars);
}
/**
* Floor: on cloud-managed instances, platform-provisioned rows are
* platform-owned runtime state — no actor, including instance admins, may
* update or delete them. Binds to the persisted row's markers, so a patch
* cannot strip the marker to lift the floor.
*
* Exceptions:
*
* - A metadata-only patch that only clears the marker keys is allowed so
* tenants can recover rows stamped with stale markers by the old
* unrestricted API — for every row except those whose markers are live
* platform state (see `isPlatformSlotEnvironment`): there, clearing the
* markers would let the very next write reclassify the row as
* tenant-managed and bypass this floor. Every marker outside a live slot
* is stale by construction, so no row is ever locked unrecoverably.
* - An envVars-only patch on the generalized managed sandbox row is
* allowed: env vars are the one tenant-owned field there (agents need
* their environment variables inside the managed sandbox), the platform
* never writes them, and the body shape guarantees nothing else — name,
* driver, config, status, metadata — rides along. DELETE still calls
* this with no options, so deletion stays blocked.
*/
async function assertPlatformProvisionedEnvironmentWritable(
environment: { driver: string; metadata: Record<string, unknown> | null },
options?: {
patchBody: unknown;
isForcedKubernetesExecution: () => Promise<boolean>;
},
): Promise<void> {
if (!isCloudManagedInstance() || !isPlatformProvisionedEnvironment(environment)) return;
if (
options !== undefined &&
isPlatformMarkerClearOnlyPatch(options.patchBody) &&
!(await isPlatformSlotEnvironment(environment, options))
) return;
if (
options !== undefined &&
isTenantEnvVarsOnlyPatch(options.patchBody) &&
isTenantEditableManagedSandbox(environment)
) return;
throw forbidden("Platform-provisioned environments are platform-managed on cloud-managed instances", {
code: "environment_platform_managed",
});
}
/**
* Floor: on cloud-managed instances the platform markers in `metadata` are
* reserved to the platform provisioner (which writes them at the service
* layer, not through these routes). A client payload that sets them to a
* truthy value is rejected — otherwise a tenant could stamp its own row as
* platform-provisioned and permanently lock it behind the write floor above.
* Clearing (null/false) is allowed so stale markers can be removed.
*/
function assertNoClientPlatformProvisionedMarkers(metadata: unknown): void {
if (isPlainRecord(metadata) && Object.prototype.hasOwnProperty.call(metadata, SANDBOX_PLATFORM_PROBE_METADATA_KEY)) {
throw unprocessable("Sandbox platform qualification is recorded only by an explicit environment probe.", { code: "environment_platform_probe_reserved" });
}
if (!isCloudManagedInstance() || !isPlainRecord(metadata)) return;
for (const key of PLATFORM_PROVISIONED_MARKER_KEYS) {
if (metadata[key] !== undefined && metadata[key] !== null && metadata[key] !== false) {
throw unprocessable(
`metadata.${key} is reserved to the platform on cloud-managed instances`,
{ code: "environment_platform_marker_reserved" },
);
}
}
}
export function environmentRoutes(
db: Db,
options: {
pluginWorkerManager?: PluginWorkerManager;
recoverMissingPluginWorker?: ReadyPluginWorkerRecovery;
} = {},
) {
const router = Router();
const svc = environmentService(db);
const environmentRuntime = environmentRuntimeService(db, {
pluginWorkerManager: options.pluginWorkerManager,
});
const customImages = environmentCustomImageService(db, {
pluginWorkerManager: options.pluginWorkerManager,
});
const executionWorkspaces = executionWorkspaceService(db);
const issues = issueService(db);
const instanceSettings = instanceSettingsService(db);
const projects = projectService(db);
const secrets = secretService(db);
const strictSecretsMode = process.env.PAPERCLIP_SECRETS_STRICT_MODE === "true";
function parseObject(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
function assertCanAccessInstanceEnvironments(req: Request) {
if (req.actor.type !== "board") {
throw forbidden("Instance environment management is restricted to board operators");
}
if (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin) return;
throw forbidden("Instance admin access required");
}
function assertCanReadInstanceEnvironments(req: Request) {
assertBoardOrgAccess(req);
}
function assertCustomImageCompanyAccess(req: Request, companyId: string) {
if (req.actor.type !== "board") {
throw forbidden("Board access required");
}
if (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin) return;
const allowedCompanies = req.actor.companyIds ?? [];
if (!allowedCompanies.includes(companyId)) {
throw forbidden("User does not have access to this company");
}
}
function canReadFullInstanceEnvironment(req: Request) {
return req.actor.type === "board"
&& (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin);
}
function redactEnvironmentForRestrictedView<T extends {
config: Record<string, unknown> | null;
envVars?: Record<string, unknown> | null;
metadata: Record<string, unknown> | null;
}>(environment: T): T {
return {
...environment,
config: {},
...(Object.prototype.hasOwnProperty.call(environment, "envVars") ? { envVars: {} } : {}),
metadata: null,
};
}
function presentEnvironmentForRead<T extends {
config: Record<string, unknown> | null;
envVars?: Record<string, unknown> | null;
metadata: Record<string, unknown> | null;
}>(req: Request, environment: T): T {
if (environment.metadata) {
const { [SANDBOX_PLATFORM_PROBE_METADATA_KEY]: _probe, ...metadata } = environment.metadata;
environment = { ...environment, metadata };
}
// Floor: on cloud-managed instances, platform-provisioned rows use the
// floored view — credential-shaped config keys are never echoed to any
// reader, structural config stays visible, and tenant env vars on the
// managed sandbox row round-trip for full readers only. Restricted
// readers keep the structural floor fields (the platform config carries
// no secrets by the managed-config contract) but never env vars — the
// same envVars posture the restricted view applies to every other
// environment, since tenant env vars can carry pasted credentials.
if (isCloudManagedInstance() && isPlatformProvisionedEnvironment(environment)) {
const floored = applyPlatformProvisionedEnvironmentFloor(environment);
if (canReadFullInstanceEnvironment(req)) {
return floored;
}
return {
...floored,
...(Object.prototype.hasOwnProperty.call(floored, "envVars") ? { envVars: {} } : {}),
};
}
return canReadFullInstanceEnvironment(req)
? environment
: redactEnvironmentForRestrictedView(environment);
}
async function assertCanReadSecretsForDraftProbe(req: Request, companyId: string) {
assertCanAccessInstanceEnvironments(req);
return companyId;
}
async function logInstanceEnvironmentActivity(input: {
actor: ReturnType<typeof getActorInfo>;
action: string;
entityId: string;
details: Record<string, unknown>;
}) {
const companyIds = await instanceSettings.listCompanyIds();
await Promise.all(
companyIds.map((companyId) =>
logActivity(db, {
companyId,
actorType: input.actor.actorType,
actorId: input.actor.actorId,
agentId: input.actor.agentId,
runId: input.actor.runId,
action: input.action,
entityType: "environment",
entityId: input.entityId,
details: input.details,
})
),
);
}
async function logEnvironmentCustomImageActivity(input: {
actor: ReturnType<typeof getActorInfo>;
companyId: string;
action: string;
entityId: string;
details: Record<string, unknown>;
}) {
await logActivity(db, {
companyId: input.companyId,
actorType: input.actor.actorType,
actorId: input.actor.actorId,
agentId: input.actor.agentId,
runId: input.actor.runId,
action: input.action,
entityType: "environment",
entityId: input.entityId,
details: input.details,
});
}
async function resolveCustomImageCompanyId(req: Request): Promise<string> {
const queryCompanyId =
typeof req.query.companyId === "string" && req.query.companyId.trim().length > 0
? req.query.companyId.trim()
: null;
if (queryCompanyId) {
assertCustomImageCompanyAccess(req, queryCompanyId);
return queryCompanyId;
}
if (req.actor.type === "board" && req.actor.companyIds?.length === 1) {
return req.actor.companyIds[0]!;
}
const companyIds = await instanceSettings.listCompanyIds();
if (companyIds.length === 1 && companyIds[0]) {
const companyId = companyIds[0];
assertCustomImageCompanyAccess(req, companyId);
return companyId;
}
throw unprocessable("companyId query parameter is required for environment customImage setup.");
}
async function resolveCustomImageSessionCompanyId(
req: Request,
session: { metadata?: Record<string, unknown> | null },
): Promise<string> {
const metadataCompanyId = readCustomImageSetupSessionCompanyId(session);
if (metadataCompanyId) {
assertCustomImageCompanyAccess(req, metadataCompanyId);
return metadataCompanyId;
}
return await resolveCustomImageCompanyId(req);
}
/**
* Pick the company context used to create new secrets from raw-pasted
* values, normalize env-var bindings, and resolve probe secrets. An
* explicit route param / query wins, then the single company the
* environment's bindings already live in, then the actor's own company,
* then the instance's only company (when exactly one exists).
* Bindings must never veto an explicit caller context: config-derived
* bindings live in the company that owns each referenced secret (see
* `replaceSecretRefsForInstanceTarget`), so an environment's bindings may
* legitimately sit in a different company — or several — than the board
* that is editing it.
*/
async function resolveEnvironmentSecretContextCompanyId(
req: Request,
environmentId: string,
options: { required: boolean },
): Promise<string | null> {
const routeCompanyId =
typeof req.params.companyId === "string" && req.params.companyId.trim().length > 0
? req.params.companyId.trim()
: typeof req.query.companyId === "string" && req.query.companyId.trim().length > 0
? req.query.companyId.trim()
: null;
if (routeCompanyId) return routeCompanyId;
const bindingCompanyIds = await secrets.listBindingCompanyIdsForTarget({
targetType: "environment",
targetId: environmentId,
});
if (bindingCompanyIds.length === 1) return bindingCompanyIds[0] ?? null;
if (req.actor.type === "agent" && req.actor.companyId) return req.actor.companyId;
if (req.actor.type === "board" && Array.isArray(req.actor.companyIds) && req.actor.companyIds.length === 1) {
return req.actor.companyIds[0] ?? null;
}
// Single-company instances have exactly one possible secret scope, so an
// actor whose memberships cannot pin a company (none, or several — e.g. an
// instance admin provisioned without a membership row) still resolves.
// Mirrors the fallback in `resolveCustomImageCompanyId`.
const instanceCompanyIds = await instanceSettings.listCompanyIds();
if (instanceCompanyIds.length === 1 && instanceCompanyIds[0]) {
return instanceCompanyIds[0];
}
if (!options.required) return null;
throw unprocessable(
"Environment secret management requires a companyId context during the instance-scoped transition.",
);
}
function summarizeEnvironmentUpdate(
patch: Record<string, unknown>,
environment: {
name: string;
driver: string;
status: string;
},
): Record<string, unknown> {
const details: Record<string, unknown> = {
changedFields: Object.keys(patch).sort(),
};
if (patch.name !== undefined) details.name = environment.name;
if (patch.driver !== undefined) details.driver = environment.driver;
if (patch.status !== undefined) details.status = environment.status;
if (patch.description !== undefined) details.descriptionChanged = true;
if (patch.config !== undefined) {
details.configChanged = true;
details.configTopLevelKeyCount =
patch.config && typeof patch.config === "object" && !Array.isArray(patch.config)
? Object.keys(patch.config as Record<string, unknown>).length
: 0;
}
if (patch.metadata !== undefined) {
details.metadataChanged = true;
details.metadataTopLevelKeyCount =
patch.metadata && typeof patch.metadata === "object" && !Array.isArray(patch.metadata)
? Object.keys(patch.metadata as Record<string, unknown>).length
: 0;
}
return details;
}
function environmentDeleteBlockMessage(impact: EnvironmentDeleteBlastRadius): string | null {
if (impact.staticReferences.isManagedLocal) {
return "Cannot delete the managed local environment.";
}
if (impact.staticReferences.isInstanceDefault) {
return "Cannot delete the current instance default environment. Set a new default environment before deleting this one.";
}
if (impact.pendingCleanupLeaseCount > 0) {
return "Cannot delete this environment while a sandbox cleanup is pending. Wait for the cleanup sweep to destroy the orphan sandbox, then retry.";
}
if (impact.reusableSandboxLeaseCount > 0) {
return "Cannot delete this environment while it has a reusable sandbox lease. Remove the associated execution workspace or issue so Paperclip can destroy the sandbox, then retry.";
}
return null;
}
function rejectEnvironmentDelete(input: {
actor: ReturnType<typeof getActorInfo>;
environment: { id: string; driver: string };
impact: EnvironmentDeleteBlastRadius;
// Sandboxes a consented delete destroyed before this rejection. Provider
// destruction is not transactional with the delete guard, so a rejection
// after a partial destroy must say what already happened.
destroyedReusableSandboxLeaseCount?: number;
}): never {
const message =
environmentDeleteBlockMessage(input.impact)
?? "Environment delete is currently blocked. Refresh the environment and retry.";
logger.warn(
{
environmentId: input.environment.id,
environmentDriver: input.environment.driver,
deleteBlockedReasons: input.impact.deleteBlockedReasons,
destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount ?? 0,
actorType: input.actor.actorType,
actorId: input.actor.actorId,
agentId: input.actor.agentId,
runId: input.actor.runId,
},
"environment delete rejected by guard",
);
throw conflict(message, {
deleteBlockedReasons: input.impact.deleteBlockedReasons,
...(input.destroyedReusableSandboxLeaseCount
? { destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount }
: {}),
});
}
function setupSessionActivityDetails(session: {
id: string;
environmentId: string;
provider: string;
status: string;
providerLeaseId: string | null;
baseTemplateRef: string | null;
connectionSummary?: Record<string, unknown> | null;
connectionSecretRef: string | null;
metadata?: Record<string, unknown> | null;
}) {
return redactEnvironmentCustomImageSetupSession({
sessionId: session.id,
environmentId: session.environmentId,
provider: session.provider,
status: session.status,
providerLeaseId: session.providerLeaseId,
baseTemplateRef: session.baseTemplateRef,
connectionSummary: session.connectionSummary,
connectionSecretRef: session.connectionSecretRef,
metadata: session.metadata,
});
}
function templateActivityDetails(template: {
id: string;
environmentId: string;
provider: string;
status: string;
templateKind: string;
templateRef: string | null;
sourceTemplateRef: string | null;
metadata?: Record<string, unknown> | null;
}) {
return redactEnvironmentCustomImageTemplate({
templateId: template.id,
environmentId: template.environmentId,
provider: template.provider,
status: template.status,
templateKind: template.templateKind,
templateRef: template.templateRef,
sourceTemplateRef: template.sourceTemplateRef,
metadata: template.metadata,
});
}
function throwTerminalPayloadValidationFailure(
failure: Extract<EnvironmentCustomImageTerminalPayloadValidationResult, { ok: false }>,
): never {
if (failure.status === 409) {
throw conflict(failure.message);
}
throw unprocessable(failure.message);
}
/**
* Managed-sandbox-only policy (`enableManagedSandboxOnly`): the local
* environment disappears from every read surface — the list and the
* by-id read — for every actor, including instance admins. The row
* itself stays in the database (company creation and the heartbeat
* depend on `ensureLocalEnvironment`); run selection independently
* refuses local under the same flag, so hiding here is presentation,
* not the enforcement boundary.
*/
async function isManagedSandboxOnlyInstance(): Promise<boolean> {
return (await instanceSettings.getExperimental()).enableManagedSandboxOnly === true;
}
router.get("/companies/:companyId/environments", async (req, res) => {
assertCanReadInstanceEnvironments(req);
const rows = await svc.list({
status: req.query.status as string | undefined,
driver: req.query.driver as string | undefined,
});
const visible = (await isManagedSandboxOnlyInstance())
? rows.filter((row) => row.driver !== "local")
: rows;
res.json(visible.map((row) => presentEnvironmentForRead(req, row)));
});
router.get("/environments/:id/delete-blast-radius", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const impact = await svc.getDeleteBlastRadius(req.params.id as string);
if (!impact) {
res.status(404).json({ error: "Environment not found" });
return;
}
res.json(impact);
});
router.get("/companies/:companyId/environments/capabilities", async (req, res) => {
assertCanReadInstanceEnvironments(req);
const pluginDrivers = await listReadyPluginEnvironmentDrivers({
db,
workerManager: options.pluginWorkerManager,
recoverMissingWorker: options.recoverMissingPluginWorker,
});
res.json(getEnvironmentCapabilities(
AGENT_ADAPTER_TYPES,
{
sandboxProviders: Object.fromEntries(pluginDrivers.map((driver) => [
driver.driverKey,
{
status: "supported" as const,
supportsSavedProbe: true,
supportsUnsavedProbe: true,
supportsRunExecution: true,
// Publish reusable-lease support only when the declaration allows it
// AND the live worker verified all reuse lifecycle methods, so the
// presentation matches the acquisition guard, which requires them.
// The declaration part uses the same resolver acquisition uses, so
// the nested `sandboxCapabilities` override wins over the legacy
// `supportsReusableLeases` flag: a manifest with legacy `true` and
// nested `false` presents as not reusable. Default an absent value
// to false with `=== true`. A ready worker that omits any reuse
// lifecycle method presents as not reusable, because acquisition
// would always fall back to an ephemeral lease.
supportsReusableLeases:
resolveDeclaredSandboxCapabilities(driver).reusableLeases === true
&& driver.reusableLeaseMethodsVerified,
supportsInteractiveSetup: driver.supportsInteractiveSetup,
interactiveSetupConnectionTypes: driver.interactiveSetupConnectionTypes,
supportsTemplateCapture: driver.supportsTemplateCapture,
templateRefKind: driver.templateRefKind,
templateConfigBinding: driver.templateConfigBinding,
supportsTemplateDelete: driver.supportsTemplateDelete,
supportsLoginPty: driver.supportsLoginPty ?? false,
displayName: driver.displayName,
description: driver.description,
source: "plugin" as const,
pluginKey: driver.pluginKey,
pluginId: driver.pluginId,
configSchema: driver.configSchema,
},
])),
},
));
});
router.get("/environments/:environmentId/custom-image-template", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
await resolveCustomImageCompanyId(req);
const overview = await customImages.getOverview({
environmentId: req.params.environmentId as string,
});
res.json(overview);
});
router.post(
"/environments/:environmentId/custom-image-setup-sessions",
validate(startEnvironmentCustomImageSetupSessionSchema),
async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const companyId = await resolveCustomImageCompanyId(req);
const actor = getActorInfo(req);
const result = await customImages.startSetupSession({
environmentId: req.params.environmentId as string,
templateId: req.body.templateId ?? null,
ttlSeconds: req.body.ttlSeconds ?? null,
actor: {
userId: actor.actorType === "user" ? actor.actorId : null,
agentId: actor.agentId,
},
secretContextCompanyId: companyId,
});
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_setup.started",
entityId: result.session.environmentId,
details: setupSessionActivityDetails(result.session),
});
res.status(201).json(result);
},
);
router.get("/environment-custom-image-setup-sessions/:sessionId", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const session = await customImages.getSessionById(req.params.sessionId as string);
if (!session) {
res.status(404).json({ error: "Environment customImage setup session not found" });
return;
}
await resolveCustomImageSessionCompanyId(req, session);
const result = await customImages.refreshSetupSession({
sessionId: session.id,
includeConnectionPayload: true,
});
res.json(result);
});
router.post(
"/environment-custom-image-setup-sessions/:sessionId/terminal-session-token",
validate(createEnvironmentCustomImageTerminalSessionTokenSchema),
async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const session = await customImages.getSessionById(req.params.sessionId as string);
if (!session) {
res.status(404).json({ error: "Environment customImage setup session not found" });
return;
}
const companyId = await resolveCustomImageSessionCompanyId(req, session);
const refreshed = await customImages.refreshSetupSession({
sessionId: session.id,
includeConnectionPayload: true,
});
const now = new Date();
if (refreshed.session.status !== "waiting_for_user") {
throw conflict(`Cannot create terminal session token from setup status "${refreshed.session.status}".`);
}
const setupExpiresAt = requireFutureCustomImageSetupExpiry(refreshed.session, now);
const payloadValidation = validateCustomImageSetupSshPayload(refreshed.connectionPayload, now);
if (!payloadValidation.ok) {
throwTerminalPayloadValidationFailure(payloadValidation);
}
const minted = environmentCustomImageTerminalSessionStore.create({
setupSessionId: refreshed.session.id,
companyId,
environmentId: refreshed.session.environmentId,
provider: refreshed.session.provider,
ssh: payloadValidation.ssh,
setupExpiresAt,
connectionExpiresAt: payloadValidation.connectionExpiresAt,
now,
});
const actor = getActorInfo(req);
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_terminal_session_token.created",
entityId: refreshed.session.environmentId,
details: {
session: setupSessionActivityDetails(refreshed.session),
terminalSession: {
connectionType: "ssh",
connectExpiresAt: minted.session.connectExpiresAt.toISOString(),
sessionExpiresAt: minted.session.sessionExpiresAt.toISOString(),
},
},
});
res.status(201).json({
id: minted.session.id,
token: minted.token,
expiresAt: minted.session.connectExpiresAt.toISOString(),
setupSessionId: minted.session.setupSessionId,
environmentId: minted.session.environmentId,
connectionType: "ssh",
websocketPath:
`/api/environment-custom-image-setup-sessions/${encodeURIComponent(minted.session.setupSessionId)}/terminal/ws`
+ `?terminalSessionId=${encodeURIComponent(minted.session.id)}`,
});
},
);
router.post(
"/environment-custom-image-setup-sessions/:sessionId/finish",
validate(finishEnvironmentCustomImageSetupSessionSchema),
async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const session = await customImages.getSessionById(req.params.sessionId as string);
if (!session) {
res.status(404).json({ error: "Environment customImage setup session not found" });
return;
}
const companyId = await resolveCustomImageSessionCompanyId(req, session);
const actor = getActorInfo(req);
const result = await customImages.finishSetupSession({
sessionId: session.id,
metadata: req.body.metadata,
});
environmentCustomImageTerminalSessionStore.deleteBySetupSessionId(session.id);
environmentCustomImageTerminalConnectionRegistry.closeBySetupSessionId(session.id, "setup_finished");
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_setup.finished",
entityId: result.session.environmentId,
details: {
session: setupSessionActivityDetails(result.session),
template: templateActivityDetails(result.template),
},
});
res.json(result);
},
);
router.post(
"/environment-custom-image-setup-sessions/:sessionId/cancel",
validate(cancelEnvironmentCustomImageSetupSessionSchema),
async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const session = await customImages.getSessionById(req.params.sessionId as string);
if (!session) {
res.status(404).json({ error: "Environment customImage setup session not found" });
return;
}
const companyId = await resolveCustomImageSessionCompanyId(req, session);
const actor = getActorInfo(req);
const cancelled = await customImages.cancelSetupSession({
sessionId: session.id,
reason: req.body.reason ?? null,
});
environmentCustomImageTerminalSessionStore.deleteBySetupSessionId(session.id);
environmentCustomImageTerminalConnectionRegistry.closeBySetupSessionId(session.id, "setup_cancelled");
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_setup.cancelled",
entityId: cancelled.environmentId,
details: setupSessionActivityDetails(cancelled),
});
res.json(cancelled);
},
);
router.post("/environments/:environmentId/custom-image-template/rollback", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const companyId = await resolveCustomImageCompanyId(req);
const actor = getActorInfo(req);
const result = await customImages.rollbackTemplate({
environmentId: req.params.environmentId as string,
});
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_template.rolled_back",
entityId: req.params.environmentId as string,
details: {
activeTemplate: templateActivityDetails(result.activeTemplate),
supersededTemplate: templateActivityDetails(result.supersededTemplate),
},
});
res.json(result);
});
router.post(
"/environments/:environmentId/custom-image-template/relink",
validate(relinkEnvironmentCustomImageTemplateSchema),
async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const companyId = await resolveCustomImageCompanyId(req);
const actor = getActorInfo(req);
// The service classifies drift, re-stamps the fingerprint, and writes the
// activity row in one transaction. The route never classifies.
const result = await customImages.relinkActiveTemplate({
environmentId: req.params.environmentId as string,
confirmBootSourceDrift: req.body.confirmBootSourceDrift === true,
actor: {
actorType: actor.actorType,
actorId: actor.actorId,
agentId: actor.agentId,
runId: actor.runId,
agentApiKeyId: actor.agentApiKeyId,
},
companyId,
});
res.json(result);
},
);
router.delete("/environments/:environmentId/custom-image-template", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const companyId = await resolveCustomImageCompanyId(req);
const actor = getActorInfo(req);
const template = await customImages.disableTemplate({
environmentId: req.params.environmentId as string,
deleteProviderTemplate: req.query.deleteProviderTemplate === "true",
});
await logEnvironmentCustomImageActivity({
actor,
companyId,
action: "environment.custom_image_template.disabled",
entityId: req.params.environmentId as string,
details: templateActivityDetails(template),
});
res.json(template);
});
router.post("/companies/:companyId/environments", validate(createEnvironmentSchema), async (req, res) => {
const companyId = req.params.companyId as string;
assertCanAccessInstanceEnvironments(req);
assertNoClientPlatformProvisionedMarkers(req.body.metadata);
if (req.body.driver === "local") {
const existingLocal = await svc.list({ driver: "local" });
if (existingLocal.length > 0) {
throw conflict("A local environment already exists for this instance.");
}
}
const actor = getActorInfo(req);
const input = {
...req.body,
envVars: await secrets.normalizeEnvBindingsForPersistence(
companyId,
req.body.envVars,
{ strictMode: strictSecretsMode, fieldPath: "envVars" },
),
config: await normalizeEnvironmentConfigForPersistence({
db,
companyId,
environmentName: req.body.name,
driver: req.body.driver,
secretProvider: getConfiguredSecretProvider(),
config: req.body.config,
actor: {
agentId: actor.agentId,
userId: actor.actorType === "user" ? actor.actorId : null,
},
pluginWorkerManager: options.pluginWorkerManager,
}),
};
// Create the row and its binding rows atomically so an invalid secret
// ref cannot leave an environment persisted without its bindings.
const environment = await db.transaction(async (tx) => {
const created = await svc.create(input, undefined, { db: tx });
await secrets.replaceSecretRefsForInstanceTarget(
{ targetType: "environment", targetId: created.id },
await collectEnvironmentSecretRefs({ db, environment: created }),
{ db: tx },
);
await secrets.syncEnvBindingsForTarget(
companyId,
{ targetType: "environment", targetId: created.id },
created.envVars,
{ db: tx },
);
return created;
});
await logInstanceEnvironmentActivity({
actor,
action: "environment.created",
entityId: environment.id,
details: {
name: environment.name,
driver: environment.driver,
status: environment.status,
},
});
res.status(201).json(presentEnvironmentForRead(req, environment));
});
router.get("/environments/:id", async (req, res) => {
assertCanReadInstanceEnvironments(req);
const environment = await svc.getById(req.params.id as string);
if (!environment || (environment.driver === "local" && (await isManagedSandboxOnlyInstance()))) {
res.status(404).json({ error: "Environment not found" });
return;
}
res.json(presentEnvironmentForRead(req, environment));
});
router.get("/environments/:id/secret-refs", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const environment = await svc.getById(req.params.id as string);
if (!environment) {
res.status(404).json({ error: "Environment not found" });
return;
}
// Metadata only (name / status / owning company) — never secret values.
// Environments are instance-scoped while secrets are company-scoped, so
// the editor needs this to render refs whose secret a given company's
// picker cannot list. Gated by the same instance-level access check as
// environment editing.
const refs = await collectEnvironmentSecretRefs({ db, environment });
res.json({ refs: await secrets.describeSecretRefs(refs) });
});
router.get("/environments/:id/leases", async (req, res) => {
assertCanReadInstanceEnvironments(req);
const environment = await svc.getById(req.params.id as string);
if (!environment) {
res.status(404).json({ error: "Environment not found" });
return;
}
const leases = await svc.listLeases(environment.id, {
status: req.query.status as string | undefined,
});
res.json(leases);
});
router.get("/environment-leases/:leaseId", async (req, res) => {
assertCanReadInstanceEnvironments(req);
const lease = await svc.getLeaseById(req.params.leaseId as string);
if (!lease) {
res.status(404).json({ error: "Environment lease not found" });
return;
}
res.json(lease);
});
router.patch("/environments/:id", validate(updateEnvironmentSchema), async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const existing = await svc.getById(req.params.id as string);
if (!existing) {
res.status(404).json({ error: "Environment not found" });
return;
}
await assertPlatformProvisionedEnvironmentWritable(existing, {
patchBody: req.body,
isForcedKubernetesExecution: async () =>
isExecutionForcedToKubernetes({
executionMode: (await instanceSettings.getGeneral()).executionMode,
}),
});
assertNoClientPlatformProvisionedMarkers(req.body.metadata);
// The durable `pending_cleanup` lease row stores the provider, the provider
// lease id, and the immutable config metadata for an orphan sandbox. The
// teardown retry reads that row alone and never reads the current environment
// provider. So a provider change or an environment delete after the record
// lands cannot strand the teardown. That immutable record is the correctness
// invariant.
//
// This pre-transaction check is a best-effort fast-fail only. It rejects a
// provider change while a known `pending_cleanup` lease exists, so the
// operator resolves the cleanup first. An orphan record that lands after this
// check still carries its own immutable teardown context, so the
// time-of-check-to-time-of-use window here cannot strand a sandbox.
const changesProviderTarget =
(req.body.driver !== undefined && req.body.driver !== existing.driver) ||
req.body.config !== undefined;
if (changesProviderTarget && (await svc.hasUnresolvedPendingCleanupLeases(existing.id))) {
throw conflict(
"Cannot change the driver or provider config while a sandbox cleanup is pending. Wait for the cleanup sweep to destroy the orphan sandbox, then retry.",
{ code: "environment_pending_sandbox_cleanup" },
);
}
const actor = getActorInfo(req);
const nextDriver = req.body.driver ?? existing.driver;
const nextName = req.body.name ?? existing.name;
const companyIdForSecrets =
req.body.config !== undefined || req.body.driver !== undefined || req.body.envVars !== undefined
? await resolveEnvironmentSecretContextCompanyId(req, existing.id, { required: true })
: null;
const configSource =
req.body.config !== undefined
? req.body.driver !== undefined && req.body.driver !== existing.driver
? req.body.config
: {
...parseObject(existing.config),
...parseObject(req.body.config),
}
: req.body.driver !== undefined && req.body.driver !== existing.driver
? {}
: existing.config;
const patch = {
...req.body,
...(req.body.envVars !== undefined
? {
envVars: await secrets.normalizeEnvBindingsForPersistence(
companyIdForSecrets!,
req.body.envVars,
{ strictMode: strictSecretsMode, fieldPath: "envVars" },
),
}
: {}),
...(req.body.config !== undefined || req.body.driver !== undefined
? {
config: await normalizeEnvironmentConfigForPersistence({
db,
companyId: companyIdForSecrets!,
environmentName: nextName,
driver: nextDriver,
secretProvider: getConfiguredSecretProvider(),
config: configSource,
actor: {
agentId: actor.agentId,
userId: actor.actorType === "user" ? actor.actorId : null,
},
pluginWorkerManager: options.pluginWorkerManager,
}),
}
: {}),
};
// Persist the config change and its binding rows atomically: a binding
// ref that fails validation (e.g. a deleted secret) must roll the whole
// save back instead of leaving the config re-pointed with stale bindings.
const environment = await db.transaction(async (tx) => {
const updated = await svc.update(existing.id, patch, { db: tx });
if (!updated) return null;
if (patch.config !== undefined || patch.driver !== undefined) {
await secrets.replaceSecretRefsForInstanceTarget(
{ targetType: "environment", targetId: updated.id },
await collectEnvironmentSecretRefs({ db, environment: updated }),
{ db: tx },
);
}
if (patch.envVars !== undefined) {
await secrets.syncEnvBindingsForTarget(
companyIdForSecrets!,
{ targetType: "environment", targetId: updated.id },
updated.envVars,
{ db: tx },
);
}
return updated;
});
if (!environment) {
res.status(404).json({ error: "Environment not found" });
return;
}
let customImageReconciliation: Awaited<
ReturnType<typeof customImages.reconcileActiveTemplateForConfigChange>
> = { action: "none" };
if (patch.config !== undefined || patch.driver !== undefined) {
try {
customImageReconciliation = await customImages.reconcileActiveTemplateForConfigChange({
environmentId: environment.id,
previous: existing,
next: environment,
});
} catch {
// Reconciliation is best-effort; a failure must not fail the save.
}
}
await logInstanceEnvironmentActivity({
actor,
action: "environment.updated",
entityId: environment.id,
details: summarizeEnvironmentUpdate(patch as Record<string, unknown>, environment),
});
const presented = presentEnvironmentForRead(req, environment);
res.json(customImageReconciliation.action === "none"
? presented
: { ...presented, customImageReconciliation });
});
router.delete("/environments/:id", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const existing = await svc.getById(req.params.id as string);
if (!existing) {
res.status(404).json({ error: "Environment not found" });
return;
}
await assertPlatformProvisionedEnvironmentWritable(existing);
const actor = getActorInfo(req);
let impact = await svc.getDeleteBlastRadius(existing.id);
if (!impact) {
res.status(404).json({ error: "Environment not found" });
return;
}
// With explicit consent, destroy the environment's reusable sandbox leases
// so the delete can proceed — but only while those leases are the sole
// blocker. Destroying provider resources and then rejecting on another
// gate would be an irreversible action with nothing gained. The destroy
// must run before the delete: the driver needs the environment config to
// reach the provider. A lease whose teardown fails lands in
// `pending_cleanup`, and the re-fetched blast radius rejects below until
// the cleanup sweep resolves it — no sandbox is ever orphaned silently.
// A lease held by an in-flight run is skipped, keeps blocking, and the
// re-check rejects the delete without touching that run's sandbox.
//
// The gate above and the destroy are not one atomic step: provider calls
// cannot join a database transaction, so a blocker that lands in the
// window between them (a new instance default, a fresh pending cleanup)
// rejects the delete only after some sandboxes are already gone. Those
// sandboxes belonged to the environment the operator consented to
// destroy; the rejection reports the count so nothing is silent.
let destroyedReusableSandboxLeaseCount = 0;
if (
req.query.destroyReusableSandboxLeases === "true"
&& impact.reusableSandboxLeaseCount > 0
&& impact.deleteBlockedReasons.every((reason) => reason === "reusable_sandbox_lease")
) {
const warmSessions = await closeWarmNativeSessionsForEnvironment({
environmentId: existing.id,
reason: "environment deleted",
});
if (warmSessions.busy > 0 || warmSessions.failed > 0) {
throw conflict(
warmSessions.busy > 0
? "Cannot delete this environment while a native runner session is active. Wait for its run to finish, then retry."
: "Cannot delete this environment because its warm native runner did not shut down cleanly. Retry after the runner exits.",
{ nativeRunnerSessions: warmSessions },
);
}
const destroyResult =
await environmentRuntime.destroyReusableSandboxLeasesForEnvironment({
environmentId: existing.id,
failureReason: "environment_deleted",
});
destroyedReusableSandboxLeaseCount = destroyResult.destroyed;
impact = await svc.getDeleteBlastRadius(existing.id);
if (!impact) {
res.status(404).json({ error: "Environment not found" });
return;
}
}
if (!impact.canDelete) {
rejectEnvironmentDelete({
actor,
environment: existing,
impact,
destroyedReusableSandboxLeaseCount,
});
}
const removed = await svc.removeIfDeletable(existing.id);
if (!removed) {
const latestImpact = await svc.getDeleteBlastRadius(existing.id);
if (!latestImpact) {
res.status(404).json({ error: "Environment not found" });
return;
}
rejectEnvironmentDelete({ actor, environment: existing, impact: latestImpact });
}
const companyIds = await instanceSettings.listCompanyIds();
await Promise.all(
companyIds.flatMap((companyId) => [
executionWorkspaces.clearEnvironmentSelection(companyId, existing.id),
issues.clearExecutionWorkspaceEnvironmentSelection(companyId, existing.id),
projects.clearExecutionWorkspaceEnvironmentSelection(companyId, existing.id),
secrets.syncEnvBindingsForTarget(
companyId,
{ targetType: "environment", targetId: existing.id },
{},
),
secrets.syncSecretRefsForTarget(
companyId,
{ targetType: "environment", targetId: existing.id },
[],
{ replaceAll: true },
),
]),
);
const secretId = readSshEnvironmentPrivateKeySecretId(existing);
if (secretId) {
await secrets.remove(secretId);
}
await logInstanceEnvironmentActivity({
actor,
action: "environment.deleted",
entityId: removed.id,
details: {
name: removed.name,
driver: removed.driver,
status: removed.status,
...(destroyedReusableSandboxLeaseCount > 0
? { destroyedReusableSandboxLeaseCount }
: {}),
},
});
res.json({
...presentEnvironmentForRead(req, removed),
destroyedReusableSandboxLeaseCount,
});
});
router.post("/environments/:id/probe", async (req, res) => {
assertCanAccessInstanceEnvironments(req);
const environment = await svc.getById(req.params.id as string);
if (!environment) {
res.status(404).json({ error: "Environment not found" });
return;
}
const actor = getActorInfo(req);
const companyIdForSecrets = await resolveEnvironmentSecretContextCompanyId(req, environment.id, { required: false });
const companyIdForProbe = companyIdForSecrets
?? (environment.driver === "sandbox" ? await resolveCustomImageCompanyId(req) : null);
if (!companyIdForSecrets) {
const secretRefs = await collectEnvironmentSecretRefs({ db, environment });
if (secretRefs.length > 0) {
throw unprocessable(
"Environment probe requires an explicit companyId to resolve secret-backed config for this environment.",
);
}
}
const probe = await probeEnvironment(db, environment, {
companyId: companyIdForProbe,
pluginWorkerManager: options.pluginWorkerManager,
applyCustomImageTemplate: environment.driver === "sandbox",
acquireSandboxRuntimeLease: environment.driver === "sandbox",
});
await logInstanceEnvironmentActivity({
actor,
action: "environment.probed",
entityId: environment.id,
details: {
driver: environment.driver,
ok: probe.ok,
summary: probe.summary,
},
});
res.json(probe);
});
router.post(
"/companies/:companyId/environments/probe-config",
validate(probeEnvironmentConfigSchema),
async (req, res) => {
const companyId = req.params.companyId as string;
assertCanAccessInstanceEnvironments(req);
if (req.body.driver === "sandbox") {
await assertCanReadSecretsForDraftProbe(req, companyId);
}
const actor = getActorInfo(req);
const normalizedConfig = await normalizeEnvironmentConfigForProbe({
db,
companyId,
driver: req.body.driver,
config: req.body.config,
accessContext: {
actorType: actor.actorType,
actorId: actor.actorId,
actorSource: actor.actorSource,
heartbeatRunId: actor.runId,
},
pluginWorkerManager: options.pluginWorkerManager,
});
const environment = {
id: "unsaved",
companyId,
name: req.body.name?.trim() || "Unsaved environment",
description: req.body.description ?? null,
driver: req.body.driver,
status: "active" as const,
config: normalizedConfig,
envVars: {},
metadata: req.body.metadata ?? null,
createdAt: new Date(),
updatedAt: new Date(),
};
const probe = await probeEnvironment(db, environment, {
companyId,
pluginWorkerManager: options.pluginWorkerManager,
resolvedConfig: {
driver: req.body.driver,
config: normalizedConfig,
} as ParsedEnvironmentConfig,
});
await logInstanceEnvironmentActivity({
actor,
action: "environment.probed_unsaved",
entityId: "unsaved",
details: {
driver: environment.driver,
ok: probe.ok,
summary: probe.summary,
configTopLevelKeyCount: Object.keys(environment.config).length,
},
});
res.json(probe);
},
);
return router;
}