mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.
## Risks
- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1574 lines
64 KiB
TypeScript
1574 lines
64 KiB
TypeScript
import { Router, type Request } from "express";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
AGENT_ADAPTER_TYPES,
|
|
cancelEnvironmentCustomImageSetupSessionSchema,
|
|
createEnvironmentCustomImageTerminalSessionTokenSchema,
|
|
createEnvironmentSchema,
|
|
finishEnvironmentCustomImageSetupSessionSchema,
|
|
getEnvironmentCapabilities,
|
|
probeEnvironmentConfigSchema,
|
|
resolveDeclaredSandboxCapabilities,
|
|
redactEnvironmentCustomImageSetupSession,
|
|
redactEnvironmentCustomImageTemplate,
|
|
relinkEnvironmentCustomImageTemplateSchema,
|
|
startEnvironmentCustomImageSetupSessionSchema,
|
|
type EnvironmentDeleteBlastRadius,
|
|
updateEnvironmentSchema,
|
|
} from "@paperclipai/shared";
|
|
import { conflict, forbidden, unprocessable } from "../errors.js";
|
|
import { isCloudManagedInstance } from "../services/cloud-instance.js";
|
|
import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js";
|
|
import {
|
|
MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY,
|
|
managedPluginEnvironmentMarker,
|
|
} from "../services/managed-plugin-environments.js";
|
|
import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js";
|
|
import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js";
|
|
import { validate } from "../middleware/validate.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import {
|
|
environmentCustomImageService,
|
|
issueService,
|
|
instanceSettingsService,
|
|
logActivity,
|
|
projectService,
|
|
} from "../services/index.js";
|
|
import {
|
|
environmentCustomImageTerminalConnectionRegistry,
|
|
environmentCustomImageTerminalSessionStore,
|
|
validateCustomImageSetupSshPayload,
|
|
type EnvironmentCustomImageTerminalPayloadValidationResult,
|
|
} from "../services/environment-custom-image-terminal-sessions.js";
|
|
import {
|
|
readCustomImageSetupSessionCompanyId,
|
|
requireFutureCustomImageSetupExpiry,
|
|
} from "../services/environment-custom-image-setup-session-utils.js";
|
|
import {
|
|
collectEnvironmentSecretRefs,
|
|
normalizeEnvironmentConfigForPersistence,
|
|
normalizeEnvironmentConfigForProbe,
|
|
readSshEnvironmentPrivateKeySecretId,
|
|
type ParsedEnvironmentConfig,
|
|
} from "../services/environment-config.js";
|
|
import { probeEnvironment } from "../services/environment-probe.js";
|
|
import { secretService } from "../services/secrets.js";
|
|
import {
|
|
listReadyPluginEnvironmentDrivers,
|
|
type ReadyPluginWorkerRecovery,
|
|
assertAgentsReadyForEnvironmentChange,
|
|
} from "../services/plugin-environment-driver.js";
|
|
import { getConfiguredSecretProvider } from "../secrets/configured-provider.js";
|
|
import { assertBoard, assertBoardOrgAccess, boardAccessRequired, getActorInfo } from "./authz.js";
|
|
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
|
import { environmentService } from "../services/environments.js";
|
|
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "../services/environment-custom-image-runtime.js";
|
|
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
|
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
|
import { closeWarmNativeSessionsForEnvironment } from "../services/native-runtime/native-session-executor.js";
|
|
|
|
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
}
|
|
|
|
/**
|
|
* Whether this environment row was provisioned by the platform: the
|
|
* managed-config environment provisioner marker, the managed plugin
|
|
* environment marker (`"<pluginKey>:<driverKey>"`, stamped next to
|
|
* `managedByPaperclip` on a managed-config plugin entry's row), or the legacy
|
|
* managed Kubernetes wrapper marker (rows created by builds that predate the
|
|
* generalized provisioner and have not been adopted yet).
|
|
*/
|
|
export function isPlatformProvisionedEnvironment(environment: {
|
|
metadata: Record<string, unknown> | null;
|
|
}): boolean {
|
|
return (
|
|
environment.metadata?.managedByPaperclip === true ||
|
|
typeof environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY] === "string" ||
|
|
environment.metadata?.managedKubernetesSandbox === true
|
|
);
|
|
}
|
|
|
|
function redactSecretLikeConfigKeys(value: Record<string, unknown>): Record<string, unknown> {
|
|
const result: Record<string, unknown> = {};
|
|
for (const [key, child] of Object.entries(value)) {
|
|
if (SECRET_LIKE_CONFIG_KEY_PATTERN.test(key)) continue;
|
|
result[key] = redactSecretLikeConfigValue(child);
|
|
}
|
|
return result;
|
|
}
|
|
|
|
/** Objects and arrays at any depth, including arrays nested in arrays. */
|
|
function redactSecretLikeConfigValue(value: unknown): unknown {
|
|
if (isPlainRecord(value)) return redactSecretLikeConfigKeys(value);
|
|
if (Array.isArray(value)) return value.map(redactSecretLikeConfigValue);
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Floor view of a platform-provisioned environment on a cloud-managed
|
|
* instance: credential-shaped config keys are never echoed — to ANY actor,
|
|
* including instance admins — while structural config (provider, image,
|
|
* template, region, ...) and the managed markers in `metadata` stay
|
|
* visible so admin surfaces can render the environment and show its
|
|
* platform-managed state.
|
|
*
|
|
* Env vars are the one tenant-owned field on a managed sandbox row: the
|
|
* platform never writes them (`ensureManagedSandboxEnvironment` reconciles
|
|
* name/config/metadata/status only), and tenants may edit them through the
|
|
* envVars-only PATCH exception below — so they echo back for round-trip
|
|
* editing. Everywhere else (the local slot, legacy kubernetes-marker rows
|
|
* that pre-generalization builds may have stamped platform values on) env
|
|
* vars stay blanked.
|
|
*/
|
|
export function applyPlatformProvisionedEnvironmentFloor<T extends {
|
|
driver?: string;
|
|
config: Record<string, unknown> | null;
|
|
envVars?: Record<string, unknown> | null;
|
|
metadata: Record<string, unknown> | null;
|
|
}>(environment: T): T {
|
|
const tenantEnvVarsVisible = isTenantEditableManagedSandbox(environment);
|
|
return {
|
|
...environment,
|
|
config: redactSecretLikeConfigKeys(isPlainRecord(environment.config) ? environment.config : {}),
|
|
...(Object.prototype.hasOwnProperty.call(environment, "envVars") && !tenantEnvVarsVisible
|
|
? { envVars: {} }
|
|
: {}),
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Whether this platform-provisioned row participates in the tenant
|
|
* env-vars contract: the generalized managed sandbox slot only. The local
|
|
* slot and legacy kubernetes-marker rows do not — the tenant never runs
|
|
* on local under this regime, and legacy rows may carry platform-written
|
|
* values.
|
|
*/
|
|
function isTenantEditableManagedSandbox(environment: {
|
|
driver?: string;
|
|
metadata: Record<string, unknown> | null;
|
|
}): boolean {
|
|
return (
|
|
environment.driver === "sandbox" &&
|
|
environment.metadata?.managedByPaperclip === true &&
|
|
environment.metadata?.managedKubernetesSandbox !== true
|
|
);
|
|
}
|
|
|
|
const PLATFORM_PROVISIONED_MARKER_KEYS = [
|
|
"managedByPaperclip",
|
|
MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY,
|
|
"managedKubernetesSandbox",
|
|
] as const;
|
|
|
|
/**
|
|
* Whether some bootstrap path on this instance currently owns the managed
|
|
* sandbox slot: the managed-config `environments` section
|
|
* (`applyManagedEnvironments`) or the forced execution-mode bootstrap
|
|
* (`PAPERCLIP_EXECUTION_MODE=kubernetes`). Both adopt and refresh the
|
|
* marked sandbox row on every boot. Fails closed: an unparseable document
|
|
* or env value counts as configured, keeping the slot protected (a
|
|
* malformed value refuses startup anyway, so a running server never hits
|
|
* the catch).
|
|
*/
|
|
function isManagedSandboxProvisioningConfigured(): boolean {
|
|
try {
|
|
if ((getManagedInstanceConfig()?.environments.length ?? 0) > 0) return true;
|
|
} catch {
|
|
return true;
|
|
}
|
|
try {
|
|
return parseExecutionPolicyBootstrapEnv(process.env) !== null;
|
|
} catch {
|
|
return true;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Whether the managed-config document declares the plugin entry this row's
|
|
* marker names. Fails closed like `isManagedSandboxProvisioningConfigured`.
|
|
*/
|
|
function isManagedPluginEnvironmentDeclared(marker: string): boolean {
|
|
try {
|
|
return (getManagedInstanceConfig()?.pluginEnvironments ?? []).some(
|
|
(spec) => managedPluginEnvironmentMarker(spec) === marker,
|
|
);
|
|
} catch {
|
|
return true;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Whether this row occupies a provisioner-owned environment slot whose
|
|
* platform markers are LIVE state — i.e. some platform code path converges
|
|
* the markers on this exact row, so a marker here is never a stale
|
|
* leftover:
|
|
*
|
|
* - The single local row (`environments_local_driver_idx`): on a
|
|
* cloud-managed instance `ensureLocalEnvironment` adopts and stamps it
|
|
* from every caller (company creation, the heartbeat, run
|
|
* orchestration), so the local slot is platform-owned unconditionally.
|
|
* - The single marked sandbox row (`environments_managed_sandbox_idx`):
|
|
* owned only while a managed-sandbox bootstrap path is configured —
|
|
* `ensureManagedSandboxEnvironment` then adopts and refreshes whichever
|
|
* row holds the slot on every boot.
|
|
* - The row of a managed-config plugin entry (its
|
|
* `managedPluginEnvironment` marker) while the document declares that
|
|
* entry: `applyManagedEnvironments` converges the markers on it every boot.
|
|
* - Any sandbox row bearing the legacy kubernetes marker while the
|
|
* persisted instance execution policy forces kubernetes:
|
|
* `findKubernetesEnvironment` selects marked rows (newest first) for
|
|
* every forced run, so under that policy each marked row is — or on the
|
|
* newest row's removal becomes — the live runtime row. This holds even
|
|
* with no bootstrap path configured: the per-run guard reads the
|
|
* persisted `executionMode`, which can outlive the env that seeded it.
|
|
*
|
|
* With no provisioning path configured and no forced-kubernetes policy the
|
|
* platform holds no claim on any sandbox row, so a platform marker there
|
|
* is stale by definition and stays recoverable via the marker-clear
|
|
* escape hatch below.
|
|
*/
|
|
async function isPlatformSlotEnvironment(
|
|
environment: { driver: string; metadata: Record<string, unknown> | null },
|
|
options: { isForcedKubernetesExecution: () => Promise<boolean> },
|
|
): Promise<boolean> {
|
|
if (environment.driver === "local") return true;
|
|
if (environment.driver === "plugin") {
|
|
const pluginMarker = environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY];
|
|
return typeof pluginMarker === "string" && isManagedPluginEnvironmentDeclared(pluginMarker);
|
|
}
|
|
if (environment.driver !== "sandbox") return false;
|
|
if (
|
|
environment.metadata?.managedByPaperclip === true &&
|
|
isManagedSandboxProvisioningConfigured()
|
|
) {
|
|
return true;
|
|
}
|
|
return (
|
|
environment.metadata?.managedKubernetesSandbox === true &&
|
|
(await options.isForcedKubernetesExecution())
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Returns true when the PATCH body is a metadata-only write whose only
|
|
* purpose is to clear platform markers (setting them to null/false). This is
|
|
* the escape hatch for rows that had these markers stamped through the old
|
|
* unrestricted API before the floor was introduced. It never applies to a
|
|
* row whose slot markers are live (see `isPlatformSlotEnvironment`) —
|
|
* clearing a live slot row's markers would reclassify it as
|
|
* tenant-managed and lift the write floor in two steps.
|
|
*/
|
|
function isPlatformMarkerClearOnlyPatch(body: unknown): boolean {
|
|
if (!isPlainRecord(body)) return false;
|
|
const bodyKeys = Object.keys(body);
|
|
if (bodyKeys.length !== 1 || bodyKeys[0] !== "metadata") return false;
|
|
const metadata = body.metadata;
|
|
if (!isPlainRecord(metadata)) return false;
|
|
const metaKeys = Object.keys(metadata);
|
|
if (metaKeys.length === 0) return false;
|
|
return metaKeys.every(
|
|
(key) =>
|
|
(PLATFORM_PROVISIONED_MARKER_KEYS as readonly string[]).includes(key) &&
|
|
(metadata[key] === null || metadata[key] === false),
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Returns true when the PATCH body touches nothing but `envVars` (as a
|
|
* plain object). This is the one tenant edit the managed-environment write
|
|
* floor admits: agents need environment variables in their managed
|
|
* sandbox, and everything else on the row (name, driver, config, status,
|
|
* metadata) stays platform-owned.
|
|
*/
|
|
function isTenantEnvVarsOnlyPatch(body: unknown): boolean {
|
|
if (!isPlainRecord(body)) return false;
|
|
const bodyKeys = Object.keys(body);
|
|
return bodyKeys.length === 1 && bodyKeys[0] === "envVars" && isPlainRecord(body.envVars);
|
|
}
|
|
|
|
/**
|
|
* Floor: on cloud-managed instances, platform-provisioned rows are
|
|
* platform-owned runtime state — no actor, including instance admins, may
|
|
* update or delete them. Binds to the persisted row's markers, so a patch
|
|
* cannot strip the marker to lift the floor.
|
|
*
|
|
* Exceptions:
|
|
*
|
|
* - A metadata-only patch that only clears the marker keys is allowed so
|
|
* tenants can recover rows stamped with stale markers by the old
|
|
* unrestricted API — for every row except those whose markers are live
|
|
* platform state (see `isPlatformSlotEnvironment`): there, clearing the
|
|
* markers would let the very next write reclassify the row as
|
|
* tenant-managed and bypass this floor. Every marker outside a live slot
|
|
* is stale by construction, so no row is ever locked unrecoverably.
|
|
* - An envVars-only patch on the generalized managed sandbox row is
|
|
* allowed: env vars are the one tenant-owned field there (agents need
|
|
* their environment variables inside the managed sandbox), the platform
|
|
* never writes them, and the body shape guarantees nothing else — name,
|
|
* driver, config, status, metadata — rides along. DELETE still calls
|
|
* this with no options, so deletion stays blocked.
|
|
*/
|
|
async function assertPlatformProvisionedEnvironmentWritable(
|
|
environment: { driver: string; metadata: Record<string, unknown> | null },
|
|
options?: {
|
|
patchBody: unknown;
|
|
isForcedKubernetesExecution: () => Promise<boolean>;
|
|
},
|
|
): Promise<void> {
|
|
if (!isCloudManagedInstance() || !isPlatformProvisionedEnvironment(environment)) return;
|
|
if (
|
|
options !== undefined &&
|
|
isPlatformMarkerClearOnlyPatch(options.patchBody) &&
|
|
!(await isPlatformSlotEnvironment(environment, options))
|
|
) return;
|
|
if (
|
|
options !== undefined &&
|
|
isTenantEnvVarsOnlyPatch(options.patchBody) &&
|
|
isTenantEditableManagedSandbox(environment)
|
|
) return;
|
|
throw forbidden("Platform-provisioned environments are platform-managed on cloud-managed instances", {
|
|
code: "environment_platform_managed",
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Floor: on cloud-managed instances the platform markers in `metadata` are
|
|
* reserved to the platform provisioner (which writes them at the service
|
|
* layer, not through these routes). A client payload that sets them to a
|
|
* truthy value is rejected — otherwise a tenant could stamp its own row as
|
|
* platform-provisioned and permanently lock it behind the write floor above.
|
|
* Clearing (null/false) is allowed so stale markers can be removed.
|
|
*/
|
|
function assertNoClientPlatformProvisionedMarkers(metadata: unknown): void {
|
|
if (isPlainRecord(metadata) && Object.prototype.hasOwnProperty.call(metadata, SANDBOX_PLATFORM_PROBE_METADATA_KEY)) {
|
|
throw unprocessable("Sandbox platform qualification is recorded only by an explicit environment probe.", { code: "environment_platform_probe_reserved" });
|
|
}
|
|
if (!isCloudManagedInstance() || !isPlainRecord(metadata)) return;
|
|
for (const key of PLATFORM_PROVISIONED_MARKER_KEYS) {
|
|
if (metadata[key] !== undefined && metadata[key] !== null && metadata[key] !== false) {
|
|
throw unprocessable(
|
|
`metadata.${key} is reserved to the platform on cloud-managed instances`,
|
|
{ code: "environment_platform_marker_reserved" },
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
export function environmentRoutes(
|
|
db: Db,
|
|
options: {
|
|
pluginWorkerManager?: PluginWorkerManager;
|
|
recoverMissingPluginWorker?: ReadyPluginWorkerRecovery;
|
|
/** Runs one teardown attempt now for each pending cleanup of this
|
|
* environment, without the retry cooldown. Absent, a delete waits for the
|
|
* scheduled cleanup sweep. */
|
|
retryPendingEnvironmentCleanup?: (input: { environmentId: string; actorId: string }) => Promise<unknown>;
|
|
} = {},
|
|
) {
|
|
const router = Router();
|
|
const svc = environmentService(db);
|
|
const environmentRuntime = environmentRuntimeService(db, {
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
});
|
|
const customImages = environmentCustomImageService(db, {
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
});
|
|
const executionWorkspaces = executionWorkspaceService(db);
|
|
const issues = issueService(db);
|
|
const instanceSettings = instanceSettingsService(db);
|
|
const projects = projectService(db);
|
|
const secrets = secretService(db);
|
|
const strictSecretsMode = process.env.PAPERCLIP_SECRETS_STRICT_MODE === "true";
|
|
|
|
function parseObject(value: unknown): Record<string, unknown> {
|
|
return value && typeof value === "object" && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
function assertCanAccessInstanceEnvironments(req: Request) {
|
|
if (req.actor.type === "agent") throw boardAccessRequired(req);
|
|
if (req.actor.type !== "board") {
|
|
throw forbidden("Instance environment management is restricted to board operators");
|
|
}
|
|
if (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin) return;
|
|
throw forbidden("Instance admin access required");
|
|
}
|
|
|
|
function assertCanReadInstanceEnvironments(req: Request) {
|
|
assertBoardOrgAccess(req);
|
|
}
|
|
|
|
function assertCustomImageCompanyAccess(req: Request, companyId: string) {
|
|
assertBoard(req);
|
|
if (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin) return;
|
|
const allowedCompanies = req.actor.companyIds ?? [];
|
|
if (!allowedCompanies.includes(companyId)) {
|
|
throw forbidden("User does not have access to this company");
|
|
}
|
|
}
|
|
|
|
function canReadFullInstanceEnvironment(req: Request) {
|
|
return req.actor.type === "board"
|
|
&& (req.actor.source === "local_implicit" || req.actor.isInstanceAdmin);
|
|
}
|
|
|
|
function redactEnvironmentForRestrictedView<T extends {
|
|
config: Record<string, unknown> | null;
|
|
envVars?: Record<string, unknown> | null;
|
|
metadata: Record<string, unknown> | null;
|
|
}>(environment: T): T {
|
|
return {
|
|
...environment,
|
|
config: {},
|
|
...(Object.prototype.hasOwnProperty.call(environment, "envVars") ? { envVars: {} } : {}),
|
|
metadata: null,
|
|
};
|
|
}
|
|
|
|
function presentEnvironmentForRead<T extends {
|
|
driver: string;
|
|
config: Record<string, unknown> | null;
|
|
envVars?: Record<string, unknown> | null;
|
|
metadata: Record<string, unknown> | null;
|
|
}>(req: Request, environment: T): T {
|
|
if (environment.driver === "plugin") {
|
|
const { pluginKey, driverKey } = environment.config ?? {};
|
|
if (typeof pluginKey === "string" && typeof driverKey === "string") {
|
|
environment = { ...environment, pluginDriver: { pluginKey, driverKey } };
|
|
}
|
|
}
|
|
if (environment.metadata) {
|
|
const { [SANDBOX_PLATFORM_PROBE_METADATA_KEY]: _probe, ...metadata } = environment.metadata;
|
|
environment = { ...environment, metadata };
|
|
}
|
|
// Floor: on cloud-managed instances, platform-provisioned rows use the
|
|
// floored view — credential-shaped config keys are never echoed to any
|
|
// reader, structural config stays visible, and tenant env vars on the
|
|
// managed sandbox row round-trip for full readers only. Restricted
|
|
// readers keep the structural floor fields (the platform config carries
|
|
// no secrets by the managed-config contract) but never env vars — the
|
|
// same envVars posture the restricted view applies to every other
|
|
// environment, since tenant env vars can carry pasted credentials.
|
|
if (isCloudManagedInstance() && isPlatformProvisionedEnvironment(environment)) {
|
|
const floored = applyPlatformProvisionedEnvironmentFloor(environment);
|
|
if (canReadFullInstanceEnvironment(req)) {
|
|
return floored;
|
|
}
|
|
return {
|
|
...floored,
|
|
...(Object.prototype.hasOwnProperty.call(floored, "envVars") ? { envVars: {} } : {}),
|
|
};
|
|
}
|
|
return canReadFullInstanceEnvironment(req)
|
|
? environment
|
|
: redactEnvironmentForRestrictedView(environment);
|
|
}
|
|
|
|
async function assertCanReadSecretsForDraftProbe(req: Request, companyId: string) {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
return companyId;
|
|
}
|
|
|
|
async function logInstanceEnvironmentActivity(input: {
|
|
actor: ReturnType<typeof getActorInfo>;
|
|
action: string;
|
|
entityId: string;
|
|
details: Record<string, unknown>;
|
|
}) {
|
|
const companyIds = await instanceSettings.listCompanyIds();
|
|
await Promise.all(
|
|
companyIds.map((companyId) =>
|
|
logActivity(db, {
|
|
companyId,
|
|
actorType: input.actor.actorType,
|
|
actorId: input.actor.actorId,
|
|
agentId: input.actor.agentId,
|
|
runId: input.actor.runId,
|
|
action: input.action,
|
|
entityType: "environment",
|
|
entityId: input.entityId,
|
|
details: input.details,
|
|
})
|
|
),
|
|
);
|
|
}
|
|
|
|
async function logEnvironmentCustomImageActivity(input: {
|
|
actor: ReturnType<typeof getActorInfo>;
|
|
companyId: string;
|
|
action: string;
|
|
entityId: string;
|
|
details: Record<string, unknown>;
|
|
}) {
|
|
await logActivity(db, {
|
|
companyId: input.companyId,
|
|
actorType: input.actor.actorType,
|
|
actorId: input.actor.actorId,
|
|
agentId: input.actor.agentId,
|
|
runId: input.actor.runId,
|
|
action: input.action,
|
|
entityType: "environment",
|
|
entityId: input.entityId,
|
|
details: input.details,
|
|
});
|
|
}
|
|
|
|
async function resolveCustomImageCompanyId(req: Request): Promise<string> {
|
|
const queryCompanyId =
|
|
typeof req.query.companyId === "string" && req.query.companyId.trim().length > 0
|
|
? req.query.companyId.trim()
|
|
: null;
|
|
if (queryCompanyId) {
|
|
assertCustomImageCompanyAccess(req, queryCompanyId);
|
|
return queryCompanyId;
|
|
}
|
|
if (req.actor.type === "board" && req.actor.companyIds?.length === 1) {
|
|
return req.actor.companyIds[0]!;
|
|
}
|
|
const companyIds = await instanceSettings.listCompanyIds();
|
|
if (companyIds.length === 1 && companyIds[0]) {
|
|
const companyId = companyIds[0];
|
|
assertCustomImageCompanyAccess(req, companyId);
|
|
return companyId;
|
|
}
|
|
throw unprocessable("companyId query parameter is required for environment customImage setup.");
|
|
}
|
|
|
|
async function resolveCustomImageSessionCompanyId(
|
|
req: Request,
|
|
session: { metadata?: Record<string, unknown> | null },
|
|
): Promise<string> {
|
|
const metadataCompanyId = readCustomImageSetupSessionCompanyId(session);
|
|
if (metadataCompanyId) {
|
|
assertCustomImageCompanyAccess(req, metadataCompanyId);
|
|
return metadataCompanyId;
|
|
}
|
|
return await resolveCustomImageCompanyId(req);
|
|
}
|
|
|
|
/**
|
|
* Pick the company context used to create new secrets from raw-pasted
|
|
* values, normalize env-var bindings, and resolve probe secrets. An
|
|
* explicit route param / query wins, then the single company the
|
|
* environment's bindings already live in, then the actor's own company,
|
|
* then the instance's only company (when exactly one exists).
|
|
* Bindings must never veto an explicit caller context: config-derived
|
|
* bindings live in the company that owns each referenced secret (see
|
|
* `replaceSecretRefsForInstanceTarget`), so an environment's bindings may
|
|
* legitimately sit in a different company — or several — than the board
|
|
* that is editing it.
|
|
*/
|
|
async function resolveEnvironmentSecretContextCompanyId(
|
|
req: Request,
|
|
environmentId: string,
|
|
options: { required: boolean },
|
|
): Promise<string | null> {
|
|
const routeCompanyId =
|
|
typeof req.params.companyId === "string" && req.params.companyId.trim().length > 0
|
|
? req.params.companyId.trim()
|
|
: typeof req.query.companyId === "string" && req.query.companyId.trim().length > 0
|
|
? req.query.companyId.trim()
|
|
: null;
|
|
if (routeCompanyId) return routeCompanyId;
|
|
const bindingCompanyIds = await secrets.listBindingCompanyIdsForTarget({
|
|
targetType: "environment",
|
|
targetId: environmentId,
|
|
});
|
|
if (bindingCompanyIds.length === 1) return bindingCompanyIds[0] ?? null;
|
|
if (req.actor.type === "agent" && req.actor.companyId) return req.actor.companyId;
|
|
if (req.actor.type === "board" && Array.isArray(req.actor.companyIds) && req.actor.companyIds.length === 1) {
|
|
return req.actor.companyIds[0] ?? null;
|
|
}
|
|
// Single-company instances have exactly one possible secret scope, so an
|
|
// actor whose memberships cannot pin a company (none, or several — e.g. an
|
|
// instance admin provisioned without a membership row) still resolves.
|
|
// Mirrors the fallback in `resolveCustomImageCompanyId`.
|
|
const instanceCompanyIds = await instanceSettings.listCompanyIds();
|
|
if (instanceCompanyIds.length === 1 && instanceCompanyIds[0]) {
|
|
return instanceCompanyIds[0];
|
|
}
|
|
if (!options.required) return null;
|
|
throw unprocessable(
|
|
"Environment secret management requires a companyId context during the instance-scoped transition.",
|
|
);
|
|
}
|
|
|
|
function summarizeEnvironmentUpdate(
|
|
patch: Record<string, unknown>,
|
|
environment: {
|
|
name: string;
|
|
driver: string;
|
|
status: string;
|
|
},
|
|
): Record<string, unknown> {
|
|
const details: Record<string, unknown> = {
|
|
changedFields: Object.keys(patch).sort(),
|
|
};
|
|
|
|
if (patch.name !== undefined) details.name = environment.name;
|
|
if (patch.driver !== undefined) details.driver = environment.driver;
|
|
if (patch.status !== undefined) details.status = environment.status;
|
|
if (patch.description !== undefined) details.descriptionChanged = true;
|
|
if (patch.config !== undefined) {
|
|
details.configChanged = true;
|
|
details.configTopLevelKeyCount =
|
|
patch.config && typeof patch.config === "object" && !Array.isArray(patch.config)
|
|
? Object.keys(patch.config as Record<string, unknown>).length
|
|
: 0;
|
|
}
|
|
if (patch.metadata !== undefined) {
|
|
details.metadataChanged = true;
|
|
details.metadataTopLevelKeyCount =
|
|
patch.metadata && typeof patch.metadata === "object" && !Array.isArray(patch.metadata)
|
|
? Object.keys(patch.metadata as Record<string, unknown>).length
|
|
: 0;
|
|
}
|
|
|
|
return details;
|
|
}
|
|
|
|
function environmentDeleteBlockMessage(impact: EnvironmentDeleteBlastRadius): string | null {
|
|
if (impact.staticReferences.isManagedLocal) {
|
|
return "Cannot delete the managed local environment.";
|
|
}
|
|
if (impact.staticReferences.isInstanceDefault) {
|
|
return "Cannot delete the current instance default environment. Set a new default environment before deleting this one.";
|
|
}
|
|
if (impact.pendingCleanupLeaseCount > 0) {
|
|
return "Cannot delete this environment while a sandbox cleanup is pending. Wait for the cleanup sweep to destroy the orphan sandbox, then retry.";
|
|
}
|
|
if (impact.reusableSandboxLeaseCount > 0) {
|
|
if (impact.reusableSandboxLeaseHolders.some(holder => holder.leasePolicy === "retain_on_failure")) {
|
|
return "Cannot delete this environment while it keeps a failed run's workspace copy (a stopped sandbox kept for file recovery). Deleting the environment with consent destroys that sandbox and the files only it holds.";
|
|
}
|
|
return "Cannot delete this environment while it has a reusable sandbox lease. Remove the associated execution workspace or issue so Paperclip can destroy the sandbox, then retry.";
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function rejectEnvironmentDelete(input: {
|
|
actor: ReturnType<typeof getActorInfo>;
|
|
environment: { id: string; driver: string };
|
|
impact: EnvironmentDeleteBlastRadius;
|
|
// Sandboxes a consented delete destroyed before this rejection. Provider
|
|
// destruction is not transactional with the delete guard, so a rejection
|
|
// after a partial destroy must say what already happened.
|
|
destroyedReusableSandboxLeaseCount?: number;
|
|
}): never {
|
|
const message =
|
|
environmentDeleteBlockMessage(input.impact)
|
|
?? "Environment delete is currently blocked. Refresh the environment and retry.";
|
|
logger.warn(
|
|
{
|
|
environmentId: input.environment.id,
|
|
environmentDriver: input.environment.driver,
|
|
deleteBlockedReasons: input.impact.deleteBlockedReasons,
|
|
destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount ?? 0,
|
|
actorType: input.actor.actorType,
|
|
actorId: input.actor.actorId,
|
|
agentId: input.actor.agentId,
|
|
runId: input.actor.runId,
|
|
},
|
|
"environment delete rejected by guard",
|
|
);
|
|
throw conflict(message, {
|
|
deleteBlockedReasons: input.impact.deleteBlockedReasons,
|
|
...(input.destroyedReusableSandboxLeaseCount
|
|
? { destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount }
|
|
: {}),
|
|
});
|
|
}
|
|
|
|
function setupSessionActivityDetails(session: {
|
|
id: string;
|
|
environmentId: string;
|
|
provider: string;
|
|
status: string;
|
|
providerLeaseId: string | null;
|
|
baseTemplateRef: string | null;
|
|
connectionSummary?: Record<string, unknown> | null;
|
|
connectionSecretRef: string | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
}) {
|
|
return redactEnvironmentCustomImageSetupSession({
|
|
sessionId: session.id,
|
|
environmentId: session.environmentId,
|
|
provider: session.provider,
|
|
status: session.status,
|
|
providerLeaseId: session.providerLeaseId,
|
|
baseTemplateRef: session.baseTemplateRef,
|
|
connectionSummary: session.connectionSummary,
|
|
connectionSecretRef: session.connectionSecretRef,
|
|
metadata: session.metadata,
|
|
});
|
|
}
|
|
|
|
function templateActivityDetails(template: {
|
|
id: string;
|
|
environmentId: string;
|
|
provider: string;
|
|
status: string;
|
|
templateKind: string;
|
|
templateRef: string | null;
|
|
sourceTemplateRef: string | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
}) {
|
|
return redactEnvironmentCustomImageTemplate({
|
|
templateId: template.id,
|
|
environmentId: template.environmentId,
|
|
provider: template.provider,
|
|
status: template.status,
|
|
templateKind: template.templateKind,
|
|
templateRef: template.templateRef,
|
|
sourceTemplateRef: template.sourceTemplateRef,
|
|
metadata: template.metadata,
|
|
});
|
|
}
|
|
|
|
function throwTerminalPayloadValidationFailure(
|
|
failure: Extract<EnvironmentCustomImageTerminalPayloadValidationResult, { ok: false }>,
|
|
): never {
|
|
if (failure.status === 409) {
|
|
throw conflict(failure.message);
|
|
}
|
|
throw unprocessable(failure.message);
|
|
}
|
|
|
|
/**
|
|
* Managed-sandbox-only policy (`enableManagedSandboxOnly`): the local
|
|
* environment disappears from every read surface — the list and the
|
|
* by-id read — for every actor, including instance admins. The row
|
|
* itself stays in the database (company creation and the heartbeat
|
|
* depend on `ensureLocalEnvironment`); run selection independently
|
|
* refuses local under the same flag, so hiding here is presentation,
|
|
* not the enforcement boundary.
|
|
*/
|
|
async function isManagedSandboxOnlyInstance(): Promise<boolean> {
|
|
return (await instanceSettings.getExperimental()).enableManagedSandboxOnly === true;
|
|
}
|
|
|
|
router.get("/companies/:companyId/environments", async (req, res) => {
|
|
assertCanReadInstanceEnvironments(req);
|
|
const rows = await svc.list({
|
|
status: req.query.status as string | undefined,
|
|
driver: req.query.driver as string | undefined,
|
|
});
|
|
const visible = (await isManagedSandboxOnlyInstance())
|
|
? rows.filter((row) => row.driver !== "local")
|
|
: rows;
|
|
res.json(visible.map((row) => presentEnvironmentForRead(req, row)));
|
|
});
|
|
|
|
router.get("/environments/:id/delete-blast-radius", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const impact = await svc.getDeleteBlastRadius(req.params.id as string);
|
|
if (!impact) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
res.json(impact);
|
|
});
|
|
|
|
router.get("/companies/:companyId/environments/capabilities", async (req, res) => {
|
|
assertCanReadInstanceEnvironments(req);
|
|
const pluginDrivers = await listReadyPluginEnvironmentDrivers({
|
|
db,
|
|
workerManager: options.pluginWorkerManager,
|
|
recoverMissingWorker: options.recoverMissingPluginWorker,
|
|
});
|
|
const loginDrivers = await listReadyPluginEnvironmentDrivers({ db, workerManager: options.pluginWorkerManager, kind: "environment_driver" });
|
|
const capabilities = getEnvironmentCapabilities(
|
|
AGENT_ADAPTER_TYPES,
|
|
{
|
|
sandboxProviders: Object.fromEntries(pluginDrivers.map((driver) => [
|
|
driver.driverKey,
|
|
{
|
|
status: "supported" as const,
|
|
supportsSavedProbe: true,
|
|
supportsUnsavedProbe: true,
|
|
supportsRunExecution: true,
|
|
// Publish reusable-lease support only when the declaration allows it
|
|
// AND the live worker verified all reuse lifecycle methods, so the
|
|
// presentation matches the acquisition guard, which requires them.
|
|
// The declaration part uses the same resolver acquisition uses, so
|
|
// the nested `sandboxCapabilities` override wins over the legacy
|
|
// `supportsReusableLeases` flag: a manifest with legacy `true` and
|
|
// nested `false` presents as not reusable. Default an absent value
|
|
// to false with `=== true`. A ready worker that omits any reuse
|
|
// lifecycle method presents as not reusable, because acquisition
|
|
// would always fall back to an ephemeral lease.
|
|
supportsReusableLeases:
|
|
resolveDeclaredSandboxCapabilities(driver).reusableLeases === true
|
|
&& driver.reusableLeaseMethodsVerified,
|
|
supportsInteractiveSetup: driver.supportsInteractiveSetup,
|
|
interactiveSetupConnectionTypes: driver.interactiveSetupConnectionTypes,
|
|
supportsTemplateCapture: driver.supportsTemplateCapture,
|
|
templateRefKind: driver.templateRefKind,
|
|
templateConfigBinding: driver.templateConfigBinding,
|
|
supportsTemplateDelete: driver.supportsTemplateDelete,
|
|
supportsLoginPty: driver.supportsLoginPty ?? false,
|
|
displayName: driver.displayName,
|
|
description: driver.description,
|
|
source: "plugin" as const,
|
|
pluginKey: driver.pluginKey,
|
|
pluginId: driver.pluginId,
|
|
configSchema: driver.configSchema,
|
|
},
|
|
])),
|
|
},
|
|
);
|
|
res.json({ ...capabilities, loginPtyDrivers: Object.fromEntries(loginDrivers.map(driver => [
|
|
`${driver.pluginKey}:${driver.driverKey}`, { supportsLoginPty: driver.supportsLoginPty === true, supportsCodexLogin: driver.supportsCodexLogin === true },
|
|
])) });
|
|
});
|
|
|
|
router.get("/environments/:environmentId/custom-image-template", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
await resolveCustomImageCompanyId(req);
|
|
const overview = await customImages.getOverview({
|
|
environmentId: req.params.environmentId as string,
|
|
});
|
|
res.json(overview);
|
|
});
|
|
|
|
router.post(
|
|
"/environments/:environmentId/custom-image-setup-sessions",
|
|
validate(startEnvironmentCustomImageSetupSessionSchema),
|
|
async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const companyId = await resolveCustomImageCompanyId(req);
|
|
const actor = getActorInfo(req);
|
|
const result = await customImages.startSetupSession({
|
|
environmentId: req.params.environmentId as string,
|
|
templateId: req.body.templateId ?? null,
|
|
ttlSeconds: req.body.ttlSeconds ?? null,
|
|
actor: {
|
|
userId: actor.actorType === "user" ? actor.actorId : null,
|
|
agentId: actor.agentId,
|
|
},
|
|
secretContextCompanyId: companyId,
|
|
});
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_setup.started",
|
|
entityId: result.session.environmentId,
|
|
details: setupSessionActivityDetails(result.session),
|
|
});
|
|
res.status(201).json(result);
|
|
},
|
|
);
|
|
|
|
router.get("/environment-custom-image-setup-sessions/:sessionId", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const session = await customImages.getSessionById(req.params.sessionId as string);
|
|
if (!session) {
|
|
res.status(404).json({ error: "Environment customImage setup session not found" });
|
|
return;
|
|
}
|
|
await resolveCustomImageSessionCompanyId(req, session);
|
|
const result = await customImages.refreshSetupSession({
|
|
sessionId: session.id,
|
|
includeConnectionPayload: true,
|
|
});
|
|
res.json(result);
|
|
});
|
|
|
|
router.post(
|
|
"/environment-custom-image-setup-sessions/:sessionId/terminal-session-token",
|
|
validate(createEnvironmentCustomImageTerminalSessionTokenSchema),
|
|
async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const session = await customImages.getSessionById(req.params.sessionId as string);
|
|
if (!session) {
|
|
res.status(404).json({ error: "Environment customImage setup session not found" });
|
|
return;
|
|
}
|
|
const companyId = await resolveCustomImageSessionCompanyId(req, session);
|
|
|
|
const refreshed = await customImages.refreshSetupSession({
|
|
sessionId: session.id,
|
|
includeConnectionPayload: true,
|
|
});
|
|
const now = new Date();
|
|
if (refreshed.session.status !== "waiting_for_user") {
|
|
throw conflict(`Cannot create terminal session token from setup status "${refreshed.session.status}".`);
|
|
}
|
|
const setupExpiresAt = requireFutureCustomImageSetupExpiry(refreshed.session, now);
|
|
const payloadValidation = validateCustomImageSetupSshPayload(refreshed.connectionPayload, now);
|
|
if (!payloadValidation.ok) {
|
|
throwTerminalPayloadValidationFailure(payloadValidation);
|
|
}
|
|
|
|
const minted = environmentCustomImageTerminalSessionStore.create({
|
|
setupSessionId: refreshed.session.id,
|
|
companyId,
|
|
environmentId: refreshed.session.environmentId,
|
|
provider: refreshed.session.provider,
|
|
ssh: payloadValidation.ssh,
|
|
setupExpiresAt,
|
|
connectionExpiresAt: payloadValidation.connectionExpiresAt,
|
|
now,
|
|
});
|
|
const actor = getActorInfo(req);
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_terminal_session_token.created",
|
|
entityId: refreshed.session.environmentId,
|
|
details: {
|
|
session: setupSessionActivityDetails(refreshed.session),
|
|
terminalSession: {
|
|
connectionType: "ssh",
|
|
connectExpiresAt: minted.session.connectExpiresAt.toISOString(),
|
|
sessionExpiresAt: minted.session.sessionExpiresAt.toISOString(),
|
|
},
|
|
},
|
|
});
|
|
res.status(201).json({
|
|
id: minted.session.id,
|
|
token: minted.token,
|
|
expiresAt: minted.session.connectExpiresAt.toISOString(),
|
|
setupSessionId: minted.session.setupSessionId,
|
|
environmentId: minted.session.environmentId,
|
|
connectionType: "ssh",
|
|
websocketPath:
|
|
`/api/environment-custom-image-setup-sessions/${encodeURIComponent(minted.session.setupSessionId)}/terminal/ws`
|
|
+ `?terminalSessionId=${encodeURIComponent(minted.session.id)}`,
|
|
});
|
|
},
|
|
);
|
|
|
|
router.post(
|
|
"/environment-custom-image-setup-sessions/:sessionId/finish",
|
|
validate(finishEnvironmentCustomImageSetupSessionSchema),
|
|
async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const session = await customImages.getSessionById(req.params.sessionId as string);
|
|
if (!session) {
|
|
res.status(404).json({ error: "Environment customImage setup session not found" });
|
|
return;
|
|
}
|
|
const companyId = await resolveCustomImageSessionCompanyId(req, session);
|
|
const actor = getActorInfo(req);
|
|
const result = await customImages.finishSetupSession({
|
|
sessionId: session.id,
|
|
metadata: req.body.metadata,
|
|
});
|
|
environmentCustomImageTerminalSessionStore.deleteBySetupSessionId(session.id);
|
|
environmentCustomImageTerminalConnectionRegistry.closeBySetupSessionId(session.id, "setup_finished");
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_setup.finished",
|
|
entityId: result.session.environmentId,
|
|
details: {
|
|
session: setupSessionActivityDetails(result.session),
|
|
template: templateActivityDetails(result.template),
|
|
},
|
|
});
|
|
res.json(result);
|
|
},
|
|
);
|
|
|
|
router.post(
|
|
"/environment-custom-image-setup-sessions/:sessionId/cancel",
|
|
validate(cancelEnvironmentCustomImageSetupSessionSchema),
|
|
async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const session = await customImages.getSessionById(req.params.sessionId as string);
|
|
if (!session) {
|
|
res.status(404).json({ error: "Environment customImage setup session not found" });
|
|
return;
|
|
}
|
|
const companyId = await resolveCustomImageSessionCompanyId(req, session);
|
|
const actor = getActorInfo(req);
|
|
const cancelled = await customImages.cancelSetupSession({
|
|
sessionId: session.id,
|
|
reason: req.body.reason ?? null,
|
|
});
|
|
environmentCustomImageTerminalSessionStore.deleteBySetupSessionId(session.id);
|
|
environmentCustomImageTerminalConnectionRegistry.closeBySetupSessionId(session.id, "setup_cancelled");
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_setup.cancelled",
|
|
entityId: cancelled.environmentId,
|
|
details: setupSessionActivityDetails(cancelled),
|
|
});
|
|
res.json(cancelled);
|
|
},
|
|
);
|
|
|
|
router.post("/environments/:environmentId/custom-image-template/rollback", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const companyId = await resolveCustomImageCompanyId(req);
|
|
const actor = getActorInfo(req);
|
|
const result = await customImages.rollbackTemplate({
|
|
environmentId: req.params.environmentId as string,
|
|
});
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_template.rolled_back",
|
|
entityId: req.params.environmentId as string,
|
|
details: {
|
|
activeTemplate: templateActivityDetails(result.activeTemplate),
|
|
supersededTemplate: templateActivityDetails(result.supersededTemplate),
|
|
},
|
|
});
|
|
res.json(result);
|
|
});
|
|
|
|
router.post(
|
|
"/environments/:environmentId/custom-image-template/relink",
|
|
validate(relinkEnvironmentCustomImageTemplateSchema),
|
|
async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const companyId = await resolveCustomImageCompanyId(req);
|
|
const actor = getActorInfo(req);
|
|
// The service classifies drift, re-stamps the fingerprint, and writes the
|
|
// activity row in one transaction. The route never classifies.
|
|
const result = await customImages.relinkActiveTemplate({
|
|
environmentId: req.params.environmentId as string,
|
|
confirmBootSourceDrift: req.body.confirmBootSourceDrift === true,
|
|
actor: {
|
|
actorType: actor.actorType,
|
|
actorId: actor.actorId,
|
|
agentId: actor.agentId,
|
|
runId: actor.runId,
|
|
agentApiKeyId: actor.agentApiKeyId,
|
|
},
|
|
companyId,
|
|
});
|
|
res.json(result);
|
|
},
|
|
);
|
|
|
|
router.delete("/environments/:environmentId/custom-image-template", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const companyId = await resolveCustomImageCompanyId(req);
|
|
const actor = getActorInfo(req);
|
|
const template = await customImages.disableTemplate({
|
|
environmentId: req.params.environmentId as string,
|
|
deleteProviderTemplate: req.query.deleteProviderTemplate === "true",
|
|
});
|
|
await logEnvironmentCustomImageActivity({
|
|
actor,
|
|
companyId,
|
|
action: "environment.custom_image_template.disabled",
|
|
entityId: req.params.environmentId as string,
|
|
details: templateActivityDetails(template),
|
|
});
|
|
res.json(template);
|
|
});
|
|
|
|
router.post("/companies/:companyId/environments", validate(createEnvironmentSchema), async (req, res) => {
|
|
const companyId = req.params.companyId as string;
|
|
assertCanAccessInstanceEnvironments(req);
|
|
assertNoClientPlatformProvisionedMarkers(req.body.metadata);
|
|
if (req.body.driver === "local") {
|
|
const existingLocal = await svc.list({ driver: "local" });
|
|
if (existingLocal.length > 0) {
|
|
throw conflict("A local environment already exists for this instance.");
|
|
}
|
|
}
|
|
const actor = getActorInfo(req);
|
|
const input = {
|
|
...req.body,
|
|
envVars: await secrets.normalizeEnvBindingsForPersistence(
|
|
companyId,
|
|
req.body.envVars,
|
|
{ strictMode: strictSecretsMode, fieldPath: "envVars" },
|
|
),
|
|
config: await normalizeEnvironmentConfigForPersistence({
|
|
db,
|
|
companyId,
|
|
environmentName: req.body.name,
|
|
driver: req.body.driver,
|
|
secretProvider: getConfiguredSecretProvider(),
|
|
config: req.body.config,
|
|
actor: {
|
|
agentId: actor.agentId,
|
|
userId: actor.actorType === "user" ? actor.actorId : null,
|
|
},
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
}),
|
|
};
|
|
// Create the row and its binding rows atomically so an invalid secret
|
|
// ref cannot leave an environment persisted without its bindings.
|
|
const environment = await db.transaction(async (tx) => {
|
|
const created = await svc.create(input, undefined, { db: tx });
|
|
await secrets.replaceSecretRefsForInstanceTarget(
|
|
{ targetType: "environment", targetId: created.id },
|
|
await collectEnvironmentSecretRefs({ db, environment: created }),
|
|
{ db: tx },
|
|
);
|
|
await secrets.syncEnvBindingsForTarget(
|
|
companyId,
|
|
{ targetType: "environment", targetId: created.id },
|
|
created.envVars,
|
|
{ db: tx },
|
|
);
|
|
return created;
|
|
});
|
|
await logInstanceEnvironmentActivity({
|
|
actor,
|
|
action: "environment.created",
|
|
entityId: environment.id,
|
|
details: {
|
|
name: environment.name,
|
|
driver: environment.driver,
|
|
status: environment.status,
|
|
},
|
|
});
|
|
res.status(201).json(presentEnvironmentForRead(req, environment));
|
|
});
|
|
|
|
router.get("/environments/:id", async (req, res) => {
|
|
assertCanReadInstanceEnvironments(req);
|
|
const environment = await svc.getById(req.params.id as string);
|
|
if (!environment || (environment.driver === "local" && (await isManagedSandboxOnlyInstance()))) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
res.json(presentEnvironmentForRead(req, environment));
|
|
});
|
|
|
|
router.get("/environments/:id/secret-refs", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const environment = await svc.getById(req.params.id as string);
|
|
if (!environment) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
// Metadata only (name / status / owning company) — never secret values.
|
|
// Environments are instance-scoped while secrets are company-scoped, so
|
|
// the editor needs this to render refs whose secret a given company's
|
|
// picker cannot list. Gated by the same instance-level access check as
|
|
// environment editing.
|
|
const refs = await collectEnvironmentSecretRefs({ db, environment });
|
|
res.json({ refs: await secrets.describeSecretRefs(refs) });
|
|
});
|
|
|
|
router.get("/environments/:id/leases", async (req, res) => {
|
|
assertCanReadInstanceEnvironments(req);
|
|
const environment = await svc.getById(req.params.id as string);
|
|
if (!environment) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
const leases = await svc.listLeases(environment.id, {
|
|
status: req.query.status as string | undefined,
|
|
});
|
|
res.json(leases);
|
|
});
|
|
|
|
router.get("/environment-leases/:leaseId", async (req, res) => {
|
|
assertCanReadInstanceEnvironments(req);
|
|
const lease = await svc.getLeaseById(req.params.leaseId as string);
|
|
if (!lease) {
|
|
res.status(404).json({ error: "Environment lease not found" });
|
|
return;
|
|
}
|
|
res.json(lease);
|
|
});
|
|
|
|
router.patch("/environments/:id", validate(updateEnvironmentSchema), async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const existing = await svc.getById(req.params.id as string);
|
|
if (!existing) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
await assertPlatformProvisionedEnvironmentWritable(existing, {
|
|
patchBody: req.body,
|
|
isForcedKubernetesExecution: async () =>
|
|
isExecutionForcedToKubernetes({
|
|
executionMode: (await instanceSettings.getGeneral()).executionMode,
|
|
}),
|
|
});
|
|
assertNoClientPlatformProvisionedMarkers(req.body.metadata);
|
|
// The durable `pending_cleanup` lease row stores the provider, the provider
|
|
// lease id, and the immutable config metadata for an orphan sandbox. The
|
|
// teardown retry reads that row alone and never reads the current environment
|
|
// provider. So a provider change or an environment delete after the record
|
|
// lands cannot strand the teardown. That immutable record is the correctness
|
|
// invariant.
|
|
//
|
|
// This pre-transaction check is a best-effort fast-fail only. It rejects a
|
|
// provider change while a known `pending_cleanup` lease exists, so the
|
|
// operator resolves the cleanup first. An orphan record that lands after this
|
|
// check still carries its own immutable teardown context, so the
|
|
// time-of-check-to-time-of-use window here cannot strand a sandbox.
|
|
const changesProviderTarget =
|
|
(req.body.driver !== undefined && req.body.driver !== existing.driver) ||
|
|
req.body.config !== undefined;
|
|
if (changesProviderTarget && (await svc.hasUnresolvedPendingCleanupLeases(existing.id))) {
|
|
throw conflict(
|
|
"Cannot change the driver or provider config while a sandbox cleanup is pending. Wait for the cleanup sweep to destroy the orphan sandbox, then retry.",
|
|
{ code: "environment_pending_sandbox_cleanup" },
|
|
);
|
|
}
|
|
const actor = getActorInfo(req);
|
|
const nextDriver = req.body.driver ?? existing.driver;
|
|
const nextName = req.body.name ?? existing.name;
|
|
const companyIdForSecrets =
|
|
req.body.config !== undefined || req.body.driver !== undefined || req.body.envVars !== undefined
|
|
? await resolveEnvironmentSecretContextCompanyId(req, existing.id, { required: true })
|
|
: null;
|
|
const configSource =
|
|
req.body.config !== undefined
|
|
? req.body.driver !== undefined && req.body.driver !== existing.driver
|
|
? req.body.config
|
|
: {
|
|
...parseObject(existing.config),
|
|
...parseObject(req.body.config),
|
|
}
|
|
: req.body.driver !== undefined && req.body.driver !== existing.driver
|
|
? {}
|
|
: existing.config;
|
|
const patch = {
|
|
...req.body,
|
|
...(req.body.envVars !== undefined
|
|
? {
|
|
envVars: await secrets.normalizeEnvBindingsForPersistence(
|
|
companyIdForSecrets!,
|
|
req.body.envVars,
|
|
{ strictMode: strictSecretsMode, fieldPath: "envVars" },
|
|
),
|
|
}
|
|
: {}),
|
|
...(req.body.config !== undefined || req.body.driver !== undefined
|
|
? {
|
|
config: await normalizeEnvironmentConfigForPersistence({
|
|
db,
|
|
companyId: companyIdForSecrets!,
|
|
environmentName: nextName,
|
|
driver: nextDriver,
|
|
secretProvider: getConfiguredSecretProvider(),
|
|
config: configSource,
|
|
actor: {
|
|
agentId: actor.agentId,
|
|
userId: actor.actorType === "user" ? actor.actorId : null,
|
|
},
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
}),
|
|
}
|
|
: {}),
|
|
};
|
|
// Persist the config change and its binding rows atomically: a binding
|
|
// ref that fails validation (e.g. a deleted secret) must roll the whole
|
|
// save back instead of leaving the config re-pointed with stale bindings.
|
|
if (patch.config !== undefined || patch.driver !== undefined) {
|
|
const instanceDefault = (await instanceSettingsService(db).get()).defaultEnvironmentId;
|
|
await assertAgentsReadyForEnvironmentChange(db, existing, { driver: nextDriver, config: patch.config ?? existing.config },
|
|
{ environmentId: existing.id, inheritsDefault: instanceDefault === existing.id });
|
|
}
|
|
const environment = await db.transaction(async (tx) => {
|
|
const updated = await svc.update(existing.id, patch, { db: tx });
|
|
if (!updated) return null;
|
|
if (patch.config !== undefined || patch.driver !== undefined) {
|
|
await secrets.replaceSecretRefsForInstanceTarget(
|
|
{ targetType: "environment", targetId: updated.id },
|
|
await collectEnvironmentSecretRefs({ db, environment: updated }),
|
|
{ db: tx },
|
|
);
|
|
}
|
|
if (patch.envVars !== undefined) {
|
|
await secrets.syncEnvBindingsForTarget(
|
|
companyIdForSecrets!,
|
|
{ targetType: "environment", targetId: updated.id },
|
|
updated.envVars,
|
|
{ db: tx },
|
|
);
|
|
}
|
|
return updated;
|
|
});
|
|
if (!environment) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
let customImageReconciliation: Awaited<
|
|
ReturnType<typeof customImages.reconcileActiveTemplateForConfigChange>
|
|
> = { action: "none" };
|
|
if (patch.config !== undefined || patch.driver !== undefined) {
|
|
try {
|
|
customImageReconciliation = await customImages.reconcileActiveTemplateForConfigChange({
|
|
environmentId: environment.id,
|
|
previous: existing,
|
|
next: environment,
|
|
});
|
|
} catch {
|
|
// Reconciliation is best-effort; a failure must not fail the save.
|
|
}
|
|
}
|
|
await logInstanceEnvironmentActivity({
|
|
actor,
|
|
action: "environment.updated",
|
|
entityId: environment.id,
|
|
details: summarizeEnvironmentUpdate(patch as Record<string, unknown>, environment),
|
|
});
|
|
const presented = presentEnvironmentForRead(req, environment);
|
|
res.json(customImageReconciliation.action === "none"
|
|
? presented
|
|
: { ...presented, customImageReconciliation });
|
|
});
|
|
|
|
router.delete("/environments/:id", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const existing = await svc.getById(req.params.id as string);
|
|
if (!existing) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
await assertPlatformProvisionedEnvironmentWritable(existing);
|
|
const actor = getActorInfo(req);
|
|
let impact = await svc.getDeleteBlastRadius(existing.id);
|
|
if (!impact) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
// With explicit consent, destroy the environment's reusable sandbox leases
|
|
// so the delete can proceed — but only while those leases are the sole
|
|
// blocker. Destroying provider resources and then rejecting on another
|
|
// gate would be an irreversible action with nothing gained. The destroy
|
|
// must run before the delete: the driver needs the environment config to
|
|
// reach the provider. A lease whose teardown fails lands in
|
|
// `pending_cleanup`, and the re-fetched blast radius rejects below until
|
|
// the cleanup sweep resolves it — no sandbox is ever orphaned silently.
|
|
// A lease held by an in-flight run is skipped, keeps blocking, and the
|
|
// re-check rejects the delete without touching that run's sandbox.
|
|
//
|
|
// The gate above and the destroy are not one atomic step: provider calls
|
|
// cannot join a database transaction, so a blocker that lands in the
|
|
// window between them (a new instance default, a fresh pending cleanup)
|
|
// rejects the delete only after some sandboxes are already gone. Those
|
|
// sandboxes belonged to the environment the operator consented to
|
|
// destroy; the rejection reports the count so nothing is silent.
|
|
let destroyedReusableSandboxLeaseCount = 0;
|
|
// The same consent covers a pending sandbox cleanup: it is the environment's
|
|
// own sandbox, already queued for teardown. Run one attempt now instead of
|
|
// waiting out the retry cooldown. An attempt in flight elsewhere is not taken
|
|
// over, and a failed attempt keeps the lease, so the re-check still rejects.
|
|
// Consent covers the environment's restore-source sandboxes too: release
|
|
// their restore protection first, so neither the pending-cleanup retry nor
|
|
// the teardown below refuses them (including ones an earlier attempt left
|
|
// claimed for cleanup).
|
|
if (
|
|
req.query.destroyReusableSandboxLeases === "true"
|
|
&& impact.deleteBlockedReasons.every((reason) =>
|
|
reason === "pending_sandbox_cleanup" || reason === "reusable_sandbox_lease")
|
|
) {
|
|
await environmentRuntime.releaseRestoreSourcesForEnvironmentDelete({ environmentId: existing.id,
|
|
consentedBy: actor.actorId });
|
|
}
|
|
if (
|
|
req.query.destroyReusableSandboxLeases === "true"
|
|
&& impact.pendingCleanupLeaseCount > 0
|
|
&& options.retryPendingEnvironmentCleanup
|
|
&& impact.deleteBlockedReasons.every((reason) =>
|
|
reason === "pending_sandbox_cleanup" || reason === "reusable_sandbox_lease")
|
|
) {
|
|
await options.retryPendingEnvironmentCleanup({ environmentId: existing.id, actorId: actor.actorId });
|
|
impact = await svc.getDeleteBlastRadius(existing.id);
|
|
if (!impact) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
}
|
|
if (
|
|
req.query.destroyReusableSandboxLeases === "true"
|
|
&& impact.reusableSandboxLeaseCount > 0
|
|
&& impact.deleteBlockedReasons.every((reason) => reason === "reusable_sandbox_lease")
|
|
) {
|
|
const warmSessions = await closeWarmNativeSessionsForEnvironment({
|
|
environmentId: existing.id,
|
|
reason: "environment deleted",
|
|
forceStopUnsettled: true,
|
|
db,
|
|
});
|
|
if (warmSessions.busy > 0 || warmSessions.failed > 0) {
|
|
throw conflict(
|
|
warmSessions.busy > 0
|
|
? "Cannot delete this environment while a native runner session is active. Wait for its run to finish, then retry."
|
|
: `Cannot delete this environment because its warm native runner could not be stopped (${warmSessions.failures
|
|
.map((failure) => `run ${failure.runId}: ${failure.reason}`).join("; ")}). Retry after the runner exits.`,
|
|
{ nativeRunnerSessions: warmSessions },
|
|
);
|
|
}
|
|
const destroyResult =
|
|
await environmentRuntime.destroyReusableSandboxLeasesForEnvironment({
|
|
environmentId: existing.id,
|
|
failureReason: "environment_deleted",
|
|
consentedBy: actor.actorId,
|
|
});
|
|
destroyedReusableSandboxLeaseCount = destroyResult.destroyed;
|
|
impact = await svc.getDeleteBlastRadius(existing.id);
|
|
if (!impact) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
}
|
|
if (!impact.canDelete) {
|
|
rejectEnvironmentDelete({
|
|
actor,
|
|
environment: existing,
|
|
impact,
|
|
destroyedReusableSandboxLeaseCount,
|
|
});
|
|
}
|
|
|
|
const removed = await svc.removeIfDeletable(existing.id);
|
|
if (!removed) {
|
|
const latestImpact = await svc.getDeleteBlastRadius(existing.id);
|
|
if (!latestImpact) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
rejectEnvironmentDelete({ actor, environment: existing, impact: latestImpact });
|
|
}
|
|
const companyIds = await instanceSettings.listCompanyIds();
|
|
await Promise.all(
|
|
companyIds.flatMap((companyId) => [
|
|
executionWorkspaces.clearEnvironmentSelection(companyId, existing.id),
|
|
issues.clearExecutionWorkspaceEnvironmentSelection(companyId, existing.id),
|
|
projects.clearExecutionWorkspaceEnvironmentSelection(companyId, existing.id),
|
|
secrets.syncEnvBindingsForTarget(
|
|
companyId,
|
|
{ targetType: "environment", targetId: existing.id },
|
|
{},
|
|
),
|
|
secrets.syncSecretRefsForTarget(
|
|
companyId,
|
|
{ targetType: "environment", targetId: existing.id },
|
|
[],
|
|
{ replaceAll: true },
|
|
),
|
|
]),
|
|
);
|
|
const secretId = readSshEnvironmentPrivateKeySecretId(existing);
|
|
if (secretId) {
|
|
await secrets.remove(secretId);
|
|
}
|
|
await logInstanceEnvironmentActivity({
|
|
actor,
|
|
action: "environment.deleted",
|
|
entityId: removed.id,
|
|
details: {
|
|
name: removed.name,
|
|
driver: removed.driver,
|
|
status: removed.status,
|
|
...(destroyedReusableSandboxLeaseCount > 0
|
|
? { destroyedReusableSandboxLeaseCount }
|
|
: {}),
|
|
},
|
|
});
|
|
res.json({
|
|
...presentEnvironmentForRead(req, removed),
|
|
destroyedReusableSandboxLeaseCount,
|
|
});
|
|
});
|
|
|
|
router.post("/environments/:id/probe", async (req, res) => {
|
|
assertCanAccessInstanceEnvironments(req);
|
|
const environment = await svc.getById(req.params.id as string);
|
|
if (!environment) {
|
|
res.status(404).json({ error: "Environment not found" });
|
|
return;
|
|
}
|
|
const actor = getActorInfo(req);
|
|
const companyIdForSecrets = await resolveEnvironmentSecretContextCompanyId(req, environment.id, { required: false });
|
|
const companyIdForProbe = companyIdForSecrets
|
|
?? (environment.driver === "sandbox" ? await resolveCustomImageCompanyId(req) : null);
|
|
if (!companyIdForSecrets) {
|
|
const secretRefs = await collectEnvironmentSecretRefs({ db, environment });
|
|
if (secretRefs.length > 0) {
|
|
throw unprocessable(
|
|
"Environment probe requires an explicit companyId to resolve secret-backed config for this environment.",
|
|
);
|
|
}
|
|
}
|
|
const probe = await probeEnvironment(db, environment, {
|
|
companyId: companyIdForProbe,
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
applyCustomImageTemplate: environment.driver === "sandbox",
|
|
acquireSandboxRuntimeLease: environment.driver === "sandbox",
|
|
});
|
|
await logInstanceEnvironmentActivity({
|
|
actor,
|
|
action: "environment.probed",
|
|
entityId: environment.id,
|
|
details: {
|
|
driver: environment.driver,
|
|
ok: probe.ok,
|
|
summary: probe.summary,
|
|
},
|
|
});
|
|
res.json(probe);
|
|
});
|
|
|
|
router.post(
|
|
"/companies/:companyId/environments/probe-config",
|
|
validate(probeEnvironmentConfigSchema),
|
|
async (req, res) => {
|
|
const companyId = req.params.companyId as string;
|
|
assertCanAccessInstanceEnvironments(req);
|
|
if (req.body.driver === "sandbox") {
|
|
await assertCanReadSecretsForDraftProbe(req, companyId);
|
|
}
|
|
const actor = getActorInfo(req);
|
|
const normalizedConfig = await normalizeEnvironmentConfigForProbe({
|
|
db,
|
|
companyId,
|
|
driver: req.body.driver,
|
|
config: req.body.config,
|
|
accessContext: {
|
|
actorType: actor.actorType,
|
|
actorId: actor.actorId,
|
|
actorSource: actor.actorSource,
|
|
heartbeatRunId: actor.runId,
|
|
},
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
});
|
|
const environment = {
|
|
id: "unsaved",
|
|
companyId,
|
|
name: req.body.name?.trim() || "Unsaved environment",
|
|
description: req.body.description ?? null,
|
|
driver: req.body.driver,
|
|
status: "active" as const,
|
|
config: normalizedConfig,
|
|
envVars: {},
|
|
metadata: req.body.metadata ?? null,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
};
|
|
const probe = await probeEnvironment(db, environment, {
|
|
companyId,
|
|
pluginWorkerManager: options.pluginWorkerManager,
|
|
resolvedConfig: {
|
|
driver: req.body.driver,
|
|
config: normalizedConfig,
|
|
} as ParsedEnvironmentConfig,
|
|
});
|
|
await logInstanceEnvironmentActivity({
|
|
actor,
|
|
action: "environment.probed_unsaved",
|
|
entityId: "unsaved",
|
|
details: {
|
|
driver: environment.driver,
|
|
ok: probe.ok,
|
|
summary: probe.summary,
|
|
configTopLevelKeyCount: Object.keys(environment.config).length,
|
|
},
|
|
});
|
|
res.json(probe);
|
|
},
|
|
);
|
|
|
|
return router;
|
|
}
|