Files
PaperClipAI/server/src/services/environments.ts
T
DottaandPaperclip 57e977be72 feat: integrate Pi 1.0 into the experimental Runner (#14921)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The experimental Runner owns provider processes and durable
sessions.
> - Pi needs working task execution and human controls.
> - The five-PR stack must preserve changes already on master.
> - Each layer now carries the complete integrated source for a safe
sequential fallback.
> - This PR belongs to native GitHub stack #15602, ending at #14956.

## Linked Issues or Issue Description

Refs #14436, #14631, #14743 and #14956.

Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are
#14921, #14922, #14923, #14924 and #14956. The user authorized the
complete merge after checks pass. Existing `pi_local` execution is
unchanged. Accounting and wider provider/platform qualification remain
deferred.

## What Changed

- Recover missing final replies after workspace finalization changes
owners, using accepted-turn evidence without rerunning work or granting
external-chat publication.
- Preserve the admitted Pi instruction root across warm runs, while
retaining changed-root rejection.
- Give Pi a bounded 15-second default shutdown grace so stop, drain
acknowledgement and durable suspension can complete. Explicit deadlines
and other providers retain their existing behavior.
- Integrate the Pi 1.0 runtime and master contracts.
- Use Pi profile 22. Preserve explicit caller-selected models and exact
native thinking levels. Keep Pi's wrapper, helper, extension and
question/control behavior unchanged from the qualified profile-19
runtime.
- Preserve master's Dot lifecycle and consent fields, configured task
environment, status guards and current Codex/Claude dependency versions.
Cursor stays qualified. Copilot stays pending; profile 17 binds the
changed shared protocol validation sources.
- Exclude general AWS IAM credentials from Pi static/custom provider
bindings and selected task projections; preserve the provider-scoped
Bedrock bearer key. Profile 21 is retained as historical provenance.
Rust and cloud install probes use the current declaration.
- Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12
payload. Pin the patch and complete runtime closures. Include the patch
in normal installed setup tooling. Keep the upstream Pi shrinkwrap as
provenance and permit only this exact security correction.
- Include current attestation files in the Docker build context. Keep
the repository lockfile unchanged from master. CI and private image
builds resolve manifest changes before their frozen installation.

## Verification

- Full local `pnpm -r typecheck` passes, including Runner Rust, server
and UI. Focused integration checks pass: 194 Runner
admission/environment tests, 63 profile/credential tests with one
expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice
tests.
- Full local `pnpm build` passes on the final source.
- Fresh final-source checks pass: all 698 Rust workspace tests (32
binaries), 156 credential/profile/controller tests with one expected
skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests.
- The profile-21 Pi materializer passes on the native host with the
official pinned Node 24.21.0 and its npm. It verifies all 150 locked
packages, the patched dependency and the exact closure. Setup/package
bundle tests and UI token gates pass.
- The old hashes were reproduced for all three supported targets before
calculating the patched graph. New closure hashes are darwin-arm64
`282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`,
darwin-x64
`64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and
linux-x64
`713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`.
Independent native platform execution is separate from graph identity
reproduction.
- Historical cloud qualification remains unchanged: all seven core cases
pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`,
harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image
`sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`,
and accepted Sonnet 4.6/low fixture. All 215 canonical files and all
seven cleanup checks pass independent verification. These are profile-19
results and are not relabeled as fresh profile-22 runs.
- Current Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.
[The readiness
plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md)
preserves campaign and failed-attempt provenance.
- Merge only after every PR's current-head CI and fresh review pass.
Linux CI covers the full suites, build and browser tests. The local
embedded Postgres API-authority suite cannot start on this macOS/Node 26
host, so Linux CI must confirm that suite.

### Fresh profile-22 core qualification — 2026-10-08

All seven accepted core cases pass canonically on Pi profile 22, with
`openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low
thinking. This model is a fixture; production accepts the caller's
explicit Pi provider/model.

Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`.
Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`.
Immutable cloud image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`.
Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.

[Hosted Linux image and clean-install
verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023)
passes, including all 20 source-bound archives, normal CLI/Pi setup,
companion import and the production pack reader. This exact installation
source includes the latest master integration and the corrected Pi warm
instruction-root fence. Full local typecheck/build and current-head
hosted CI verify the final stack. All 13 focused real-root regressions
pass. The full local executor suite passed 662 tests; 15 database tests
could not start the Mac embedded PostgreSQL service. Hosted Linux CI
passes the full required verification and E2E checks. These fresh
results keep their own source identity; profile-19 results remain
historical.

| Core path | Canonical campaign | Retained archive SHA-256 |
| --- | --- | --- |
| File edit, validation, download and Done |
`pi-core22-replyfix-0-1791511228` | 23 files;
`a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` |
| Pending question and controller restart |
`pi-core22-replyfix-1-1791511376` | 33 files;
`6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` |
| Three-turn session/process/workspace continuity |
`pi-core22-replyfix-2-1791511587` | 23 files;
`7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` |
| Four typed questions and browser reconnects |
`pi-core22-replyfix-3-1791511881` | 42 files;
`9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` |
| Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22
files;
`a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` |
| Same-turn steering and permission denial |
`pi-core22-replyfix-5-1791512261` | 39 files;
`c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` |
| Stop during pending permission | `pi-core22-replyfix-6-1791512390` |
33 files;
`7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` |

All 215 canonical files (28714587 bytes) are independently
hash-verified. All seven cleanup grades pass, with no owned runtime
process or temporary root after each case. Automatic retries are zero.
The owned cloud host stopped normally after retention. The prior
profile-22 warm attempt remains failed and separately retained: archive
SHA-256
`1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its
original canonical classification is preserved. Diagnosis reproduced a
product bug comparing an agent-files root against an unset
checkpoint-only field. The fix stores the admitted physical root
separately from the adopted per-run collection capability. The real-root
regression fails before the fix and passes afterward, including
rejection of a changed physical root. Fixture, grader, model and all
seven accepted case IDs are unchanged; this fresh campaign tests
final-reply publication after file registration first. The intermediate
restart attempt also remains failed and retained: archive SHA-256
`5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its
original canonical classification is preserved. Pi resumed, wrote the
verified answer and completed its task; exact runner suspension was
proven, but idle stop consumed about 5.2s and left under 3s for the
drain acknowledgement. The Pi-only default shutdown grace is now 15s,
preserving a full 5s drain round trip and a finite suspension reserve.
Explicit caller deadlines, other provider defaults, literal drain
receipts and exact suspension identity checks remain unchanged. The
timing regression fails before this correction and passes afterward; all
18 focused settlement tests and Runner typecheck pass. The final-source
file attempt is also preserved as failed (`candidate_failure`), archive
SHA-256
`db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`.
Native edit, validation, exact downloadable artifact and Done/succeeded
all passed, and the exact final reply was durably recorded. A workspace
recovery owner completed before the live heartbeat reached presentation,
leaving that reply absent from task chat. Recovery now materializes only
a completed final reply from the accepted turn of an ordinary internal
Done task, preserving issue/run/contract binding, suppression,
external-chat authorization and same-run deduplication. The database
regression covers the generated file-preparation receipt, suppression,
unapproved external continuation and replay. Server typecheck and all 49
response-selection tests pass; hosted Linux verifies the database
regression because embedded PostgreSQL cannot start on this Mac.

The delayed-final-answer database regression passes on [the final
root-source Linux server
shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152),
alongside 1,108 passing tests. The first root Runner shard had one
unchanged durable-resume test exceed its 5-second timeout; the identical
top-source shard and the isolated exact test passed. One rerun of that
failed job and its required aggregate passed without source or test
changes. The original failed job log and the single-rerun receipt remain
retained.

### October 9 merge verification

Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All
current-head checks pass, including `ci / verify` and `ci / e2e`;
exact-head Greptile review is 5/5 with no unresolved threads. Current
master conflicts are resolved. The user authorized the maintainer
override of the code-owner review gate after these checks. The seven
retained live core cases remain bound to source
`3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image.

## Risks

- The security correction changes the dependency closure and profile
identity. Old sessions must reopen on the new profile. Exact identities
and credential bindings fail closed.
- The runner remains experimental and requires explicit selection.
Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E
model is a fixture.
- Accounting and the broad platform/provider matrix remain deferred.
This merge does not publish a release or deploy a service.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, editing and tool use. The exact serving ID and context
window are not exposed in this session. Final live qualification uses Pi
1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed
low thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 08:56:41 -05:00

1771 lines
71 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, desc, eq, inArray, isNull, ne, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agents,
builtInManagedResources,
companies,
companySecretBindings,
environmentCustomImageSetupSessions,
environmentLeases,
environments,
executionWorkspaces,
instanceSettings,
issues,
projects,
} from "@paperclipai/db";
import {
ENVIRONMENT_DRIVERS,
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
ENVIRONMENT_LEASE_POLICIES,
ENVIRONMENT_LEASE_STATUSES,
ENVIRONMENT_STATUSES,
type CreateEnvironment,
type Environment,
type EnvironmentDeleteBlastRadius,
type EnvironmentDeleteBlockedReason,
type EnvironmentLease,
type EnvironmentLeaseCleanupStatus,
type EnvironmentLeasePolicy,
type EnvironmentLeaseStatus,
type UpdateEnvironment,
} from "@paperclipai/shared";
import { conflict, forbidden } from "../errors.js";
import { logActivity } from "./activity-log.js";
import { isCloudManagedInstance } from "./cloud-instance.js";
import {
resourceStatus,
stockHash,
type ManagedResourceStockStatus,
} from "./managed-resource-drift.js";
type EnvironmentRow = typeof environments.$inferSelect;
type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
const DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION =
"Default execution environment for Paperclip runs on this machine.";
const DEFAULT_KUBERNETES_ENVIRONMENT_NAME = "Kubernetes Sandbox";
const DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION =
"Managed Kubernetes sandbox environment for hosted tenant execution.";
/** Provider key (== plugin driverKey) of the first-party Kubernetes sandbox provider. */
const KUBERNETES_PROVIDER_KEY = "kubernetes";
/** Metadata marker for the company's managed-by-config Kubernetes sandbox environment. */
const KUBERNETES_MANAGED_MARKER = "managedKubernetesSandbox";
const ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES = ["starting", "waiting_for_user", "capturing"] as const;
/**
* Configuration accepted by `ensureKubernetesEnvironment`. Mirrors the keys of
* the kubernetes sandbox-provider `configSchema` that an operator typically
* pins for a hosted cloud instance. Stored verbatim in `environment.config`
* (the plugin validates/defaults it via `kubernetesProviderConfigSchema` at
* lease time); `provider` is always forced to "kubernetes".
*/
export interface KubernetesEnvironmentConfigInput {
backend?: "sandbox-cr" | "job";
inCluster?: boolean;
runtimeClassName?: string;
egressMode?: "cilium" | "standard";
egressAllowFqdns?: string[];
egressAllowCidrs?: string[];
namespacePrefix?: string;
imageRegistry?: string;
adapterType?: string;
/**
* Sandbox lease RPC timeout in milliseconds. Read at lease time by
* `resolvePluginSandboxRpcTimeoutMs` to extend the worker-manager call
* timeout when acquiring a lease may take minutes (e.g. a cold node
* scale-up on an autoscale-to-zero pool). Stored verbatim in the
* environment config and validated by the sandbox config schema.
*/
timeoutMs?: number;
adapters?: import("@paperclipai/shared").AdapterRegistryEntry[];
[key: string]: unknown;
}
/**
* Input to `ensureManagedSandboxEnvironment`. Provider-agnostic: `provider`
* is the sandbox plugin's driver key and is forced into `config.provider`;
* the rest of `config` is stored verbatim for the plugin to validate at
* lease time.
*/
export interface ManagedSandboxEnvironmentInput {
/** Company whose managed-resource binding should be reconciled. Omit at instance boot to bind every company. */
companyId?: string;
name: string;
description?: string;
/** Sandbox provider key (the plugin's driverKey, e.g. "kubernetes", "daytona"). */
provider: string;
config?: Record<string, unknown>;
/**
* Extra metadata markers stamped on the managed row (e.g. the legacy
* kubernetes marker `managedKubernetesSandbox` that
* `findKubernetesEnvironment` keys on).
*/
extraMetadata?: Record<string, unknown>;
/** Version label recorded with the stock binding; hashes remain the drift authority. */
stockVersion?: string;
/**
* Asserts the caller's deployment gives no operator any path to hand-edit
* this row (currently only true for the PAPERCLIP_MANAGED_CONFIG applier,
* where `enableManagedSandboxOnly` removes the tenant's own environment
* choice entirely). When set, a plain content-hash mismatch against a real
* prior binding is treated as ordinary stock drift instead of an operator
* customization to protect — see the `operator_modified` handling below.
* Leave unset for any caller (self-hosted `kubernetes-execution-mode`
* bootstrap, tests, admin routes) where an operator could realistically
* have edited the row through the normal environments UI/API.
*/
platformFullyManaged?: boolean;
}
export type ManagedSandboxEnvironmentReconcileAction =
| "added"
| "updated"
| "unchanged"
| "skipped";
export interface ManagedSandboxEnvironmentReconcileResult {
environment: Environment;
action: ManagedSandboxEnvironmentReconcileAction;
/** Classification observed before this reconciliation wrote anything. */
stockStatus: ManagedResourceStockStatus;
/** True only when operator drift prevented the available stock update. */
updateAvailable: boolean;
stockHash: string;
}
function cloneRecord(value: unknown, fallback: Record<string, unknown> | null = null): Record<string, unknown> | null {
if (!value || typeof value !== "object" || Array.isArray(value)) return fallback;
return { ...(value as Record<string, unknown>) };
}
function readEnum<T extends string>(value: string | null, allowed: readonly T[], fieldName: string): T | null {
if (value === null) return null;
if ((allowed as readonly string[]).includes(value)) return value as T;
throw new Error(`Unexpected ${fieldName} value: ${value}`);
}
function hasConstraintName(error: unknown, constraintName: string): boolean {
if (typeof error !== "object" || error === null) return false;
const candidate = error as {
constraint?: unknown;
constraint_name?: unknown;
cause?: unknown;
};
return candidate.constraint === constraintName
|| candidate.constraint_name === constraintName
|| hasConstraintName(candidate.cause, constraintName);
}
function toEnvironment(row: EnvironmentRow): Environment {
return {
id: row.id,
name: row.name,
description: row.description ?? null,
driver: readEnum(row.driver, ENVIRONMENT_DRIVERS, "environment driver") ?? "local",
status: readEnum(row.status, ENVIRONMENT_STATUSES, "environment status") ?? "active",
config: cloneRecord(row.config, {}) ?? {},
envVars: cloneRecord(row.envVars, {}) ?? {},
metadata: cloneRecord(row.metadata),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
} as Environment;
}
type EnvironmentListFilters = {
status?: string;
driver?: string;
};
function resolveListFilters(
companyIdOrFilters?: string | EnvironmentListFilters,
maybeFilters?: EnvironmentListFilters,
): EnvironmentListFilters {
if (typeof companyIdOrFilters === "string") {
return maybeFilters ?? {};
}
return companyIdOrFilters ?? {};
}
function resolveCreateInput(
companyIdOrInput: string | CreateEnvironment,
maybeInput?: CreateEnvironment,
): CreateEnvironment {
if (typeof companyIdOrInput === "string") {
if (!maybeInput) throw new Error("Create environment input is required");
return maybeInput;
}
return companyIdOrInput;
}
function resolveKubernetesConfig(
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
maybeConfig?: KubernetesEnvironmentConfigInput,
): KubernetesEnvironmentConfigInput {
if (typeof companyIdOrConfig === "string") {
if (!maybeConfig) throw new Error("Kubernetes environment config is required");
return maybeConfig;
}
return companyIdOrConfig;
}
function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
return {
id: row.id,
companyId: row.companyId,
environmentId: row.environmentId,
executionWorkspaceId: row.executionWorkspaceId ?? null,
issueId: row.issueId ?? null,
heartbeatRunId: row.heartbeatRunId ?? null,
status: readEnum(row.status, ENVIRONMENT_LEASE_STATUSES, "environment lease status") ?? "active",
leasePolicy: readEnum(row.leasePolicy, ENVIRONMENT_LEASE_POLICIES, "environment lease policy") ?? "ephemeral",
provider: row.provider ?? null,
providerLeaseId: row.providerLeaseId ?? null,
acquiredAt: row.acquiredAt,
lastUsedAt: row.lastUsedAt,
expiresAt: row.expiresAt ?? null,
releasedAt: row.releasedAt ?? null,
failureReason: row.failureReason ?? null,
cleanupStatus: readEnum(
row.cleanupStatus,
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
"environment lease cleanup status",
),
metadata: cloneRecord(row.metadata),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
}
function countFromRows(rows: Array<{ count: number | string | null | undefined }>): number {
return Number(rows[0]?.count ?? 0);
}
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
type EnvironmentWriteDb = Pick<Db | DbTransaction, "select" | "insert" | "update" | "delete">;
const MANAGED_ENVIRONMENT_BUNDLE_KEY = "managed-sandbox-environment";
const MANAGED_ENVIRONMENT_RESOURCE_KIND = "environment";
const MANAGED_ENVIRONMENT_RESOURCE_KEY = "managed-sandbox";
const MANAGED_ENVIRONMENT_STOCK_VERSION = "managed-environment-v1";
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY = "_paperclipManagedArchiveToken";
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY = "_paperclipManagedArchiveToken";
function managedEnvironmentBaselineDefaults(
defaultsJson: Record<string, unknown>,
): Record<string, unknown> {
const baseline = { ...defaultsJson };
delete baseline[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
return baseline;
}
function withoutManagedEnvironmentArchiveToken(
metadata: Record<string, unknown> | null,
): Record<string, unknown> | null {
if (!metadata) return null;
const next = { ...metadata };
delete next[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
return next;
}
function managedMetadataKeys(
desiredMetadata: Record<string, unknown>,
bindings: Array<{ defaultsJson: Record<string, unknown> }>,
): string[] {
const keys = new Set([
"managedByPaperclip",
"managedSandboxProvider",
KUBERNETES_MANAGED_MARKER,
...Object.keys(desiredMetadata),
]);
for (const binding of bindings) {
const metadata = cloneRecord(binding.defaultsJson.metadata);
for (const key of Object.keys(metadata ?? {})) keys.add(key);
}
return [...keys].sort((left, right) => left.localeCompare(right));
}
function managedEnvironmentStock(input: {
name: string;
description: string | null;
config: Record<string, unknown>;
metadata: Record<string, unknown> | null;
status: string;
}, metadataKeys: readonly string[]): Record<string, unknown> {
const metadata = input.metadata ?? {};
return {
name: input.name,
description: input.description,
config: input.config,
metadata: Object.fromEntries(
metadataKeys.map((key) => [key, Object.prototype.hasOwnProperty.call(metadata, key) ? metadata[key] : null]),
),
status: input.status,
};
}
function mergeManagedEnvironmentMetadata(
current: Record<string, unknown> | null,
desired: Record<string, unknown>,
keys: readonly string[],
): Record<string, unknown> {
const merged = { ...(current ?? {}) };
for (const key of keys) {
const value = Object.prototype.hasOwnProperty.call(desired, key) ? desired[key] : null;
if (value === null || value === undefined) delete merged[key];
else merged[key] = value;
}
return merged;
}
export function environmentService(db: Db) {
/**
* Idempotently ensure THE Paperclip-managed sandbox environment for this
* instance, configured for an arbitrary sandbox provider plugin. Mirrors
* `ensureLocalEnvironment`; the partial unique index
* `environments_managed_sandbox_idx` enforces at most one managed sandbox
* row per instance, so this function owns that single slot regardless of
* provider:
*
* - A stock-controlled managed row advances to a new stock hash in the same
* transaction as its managed fields, including provider switches.
* - A stock-current row is returned without an environment write.
* - An operator-modified or previously unmanaged row is preserved and
* reported as skipped; its user-owned fields are never folded into stock.
*/
const ensureManagedSandboxEnvironment = async (
input: ManagedSandboxEnvironmentInput,
): Promise<ManagedSandboxEnvironmentReconcileResult> => {
const desiredConfig: Record<string, unknown> = {
...(input.config ?? {}),
provider: input.provider,
};
const desiredMetadata: Record<string, unknown> = {
managedByPaperclip: true,
managedSandboxProvider: input.provider,
...(input.extraMetadata ?? {}),
};
if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) {
desiredMetadata[KUBERNETES_MANAGED_MARKER] = null;
}
let activityCompanyIds: string[] = [];
let trackingInitialized = false;
let providerReactivated = false;
const reconciliation = await db.transaction(
async (tx): Promise<ManagedSandboxEnvironmentReconcileResult> => {
const companyIds = input.companyId
? [input.companyId]
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
activityCompanyIds = companyIds;
// Take the sandbox-row lock BEFORE reading the stock bindings. Two
// passes can reconcile the same slot concurrently (the boot ensure and
// the async provider-recovery reactivation, or two app builds during a
// rolling deploy). Concurrent passes serialize on this lock, and under
// READ COMMITTED each later statement sees a fresh snapshot — so a pass
// that blocks here then reads the bindings the winning pass committed,
// not a snapshot from before it waited. Reading bindings first left a
// window where a waiting pass compared the winner's fresh row against
// its own stale binding hash, misclassified the mismatch as
// `operator_modified`, and — once `platformFullyManaged` turns that
// into an update — rolled the row back to its own older stock.
const sandboxRows = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update");
const bindingConditions = and(
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
...(companyIds.length > 0 ? [inArray(builtInManagedResources.companyId, companyIds)] : []),
);
const bindings = companyIds.length > 0
? await tx.select().from(builtInManagedResources).where(bindingConditions)
: [];
trackingInitialized = bindings.length < companyIds.length;
const keys = managedMetadataKeys(desiredMetadata, bindings);
let row = sandboxRows.find(
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? sandboxRows.find((candidate) => candidate.name === input.name) ?? null;
const writeBindings = async (
environmentId: string,
stockVersion: string,
installedStockHash: string,
defaultsJson: Record<string, unknown>,
replace: boolean,
) => {
const targetCompanyIds = replace
? companyIds
: companyIds.filter(
(companyId) => !bindings.some((binding) => binding.companyId === companyId),
);
if (targetCompanyIds.length === 0) return;
const values = targetCompanyIds.map((companyId) => ({
companyId,
bundleKey: MANAGED_ENVIRONMENT_BUNDLE_KEY,
resourceKind: MANAGED_ENVIRONMENT_RESOURCE_KIND,
resourceKey: MANAGED_ENVIRONMENT_RESOURCE_KEY,
resourceId: environmentId,
stockVersion,
stockHash: installedStockHash,
defaultsJson,
}));
const insert = tx.insert(builtInManagedResources).values(values);
if (!replace) {
await insert.onConflictDoNothing({
target: [
builtInManagedResources.companyId,
builtInManagedResources.bundleKey,
builtInManagedResources.resourceKind,
builtInManagedResources.resourceKey,
],
});
return;
}
await insert.onConflictDoUpdate({
target: [
builtInManagedResources.companyId,
builtInManagedResources.bundleKey,
builtInManagedResources.resourceKind,
builtInManagedResources.resourceKey,
],
set: {
resourceId: environmentId,
stockVersion,
stockHash: installedStockHash,
defaultsJson,
updatedAt: new Date(),
},
});
};
if (!row) {
const nameOwner = await tx
.select()
.from(environments)
.where(eq(environments.name, input.name))
.then((rows) => rows[0] ?? null);
if (nameOwner && nameOwner.driver !== "sandbox") {
throw new Error(
`Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`,
);
}
const now = new Date();
const inserted = await tx
.insert(environments)
.values({
name: input.name,
description: input.description ?? null,
driver: "sandbox",
status: "active",
config: desiredConfig,
envVars: {},
metadata: mergeManagedEnvironmentMetadata(null, desiredMetadata, keys),
createdAt: now,
updatedAt: now,
})
// Either the managed-slot partial index or the global name index
// can select the concurrent winner. Treat both as convergence and
// reselect that winner under lock below.
.onConflictDoNothing()
.returning()
.then((rows) => rows[0] ?? null);
if (inserted) {
const stock = managedEnvironmentStock({
name: inserted.name,
description: inserted.description ?? null,
config: inserted.config,
metadata: inserted.metadata,
status: inserted.status,
}, keys);
const latestStockHash = stockHash(stock);
await writeBindings(
inserted.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
stock,
true,
);
return {
environment: toEnvironment(inserted),
action: "added",
stockStatus: "missing",
updateAvailable: false,
stockHash: latestStockHash,
};
}
row = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update")
.then(
(rows) => rows.find(
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? null,
);
if (!row) throw new Error("Failed to ensure managed sandbox environment");
}
const nameOwner = await tx
.select({ id: environments.id })
.from(environments)
.where(eq(environments.name, input.name))
.then((rows) => rows[0] ?? null);
const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name;
const desiredStock = managedEnvironmentStock({
name: desiredName,
description: input.description ?? null,
config: desiredConfig,
metadata: desiredMetadata,
status: "active",
}, keys);
const latestStockHash = stockHash(desiredStock);
const currentStock = managedEnvironmentStock({
name: row.name,
description: row.description ?? null,
config: row.config,
metadata: row.metadata,
status: row.status,
}, keys);
const currentHash = stockHash(currentStock);
const matchingBindings = bindings.filter((binding) => binding.resourceId === row!.id);
const rowArchiveToken = (row.metadata as Record<string, unknown> | null)
?.[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
let stockStatus = resourceStatus({
resourceId: row.id,
currentHash,
bindingStockHash: matchingBindings[0]?.stockHash ?? null,
latestStockHash,
});
if (stockStatus === "operator_modified") {
const stockControlledBinding = matchingBindings.find(
(binding) => resourceStatus({
resourceId: row!.id,
currentHash,
bindingStockHash: binding.stockHash,
latestStockHash,
}) === "stock_update_available",
);
if (stockControlledBinding) stockStatus = "stock_update_available";
}
const operatorReaffirmedArchive = row.status === "archived" && matchingBindings.some(
(binding) => {
const bindingToken = binding.defaultsJson[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
return binding.defaultsJson.status === "archived" &&
typeof bindingToken === "string" &&
bindingToken !== rowArchiveToken;
},
);
if (operatorReaffirmedArchive) stockStatus = "operator_modified";
// `platformFullyManaged` callers (currently: the PAPERCLIP_MANAGED_CONFIG
// applier) assert that nothing in their deployment can hand-edit this
// row — the product gives a cloud-harness tenant no path to it, unlike
// the general self-hosted contract this function otherwise protects
// (see "classifies operator drift" in environment-service.test.ts,
// which exercises a real operator edit and must keep winning). Under
// that assertion, a plain content-hash mismatch against a real prior
// binding can only be drift between two platform-driven reconciliation
// passes (e.g. a stock hash recorded by an older app build before a
// later stock field was added), never a customization to protect —
// apply it like any other stock-outdated row.
//
// Two things the bypass must never touch:
// - A row with NO matching binding. `row` can be a same-name sandbox
// row that was never Paperclip-managed (the fallback lookup above).
// It also reads as `operator_modified`, but there is no prior
// platform pass to have drifted from — adopting it would overwrite
// a tenant-created environment and stamp it managed. Require a
// binding for this exact row, so "prior binding" is enforced, not
// just documented.
// - Archive-reaffirmation. A `sandbox_image` update must never
// resurrect a row something else deliberately kept archived after
// Paperclip's own provider-unavailability archival, so that path
// still skips below regardless of this flag.
if (
input.platformFullyManaged &&
stockStatus === "operator_modified" &&
!operatorReaffirmedArchive &&
matchingBindings.length > 0
) {
stockStatus = "stock_update_available";
}
if (stockStatus === "operator_modified") {
const baseline = matchingBindings[0];
let baselineDefaults = baseline
? managedEnvironmentBaselineDefaults(baseline.defaultsJson)
: desiredStock;
let baselineHash = baseline?.stockHash ?? latestStockHash;
// Provider unavailability is an operational state transition, not
// an operator edit. If the binding records that Paperclip archived
// this row, restore only its availability status. Keep every other
// operator-modified field intact and leave the stock update pending.
// A manually archived row still has an active binding baseline, so
// it remains operator_modified and is not reactivated here.
const archivedByReconciler = row.status === "archived" &&
typeof rowArchiveToken === "string" &&
matchingBindings.some((binding) => {
const bindingDefaults = binding.defaultsJson;
return bindingDefaults.status === "archived" &&
bindingDefaults[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY] === rowArchiveToken;
});
if (archivedByReconciler) {
const reactivated = await tx
.update(environments)
.set({
status: "active",
metadata: withoutManagedEnvironmentArchiveToken(row.metadata),
updatedAt: new Date(),
})
.where(and(eq(environments.id, row.id), eq(environments.status, "archived")))
.returning()
.then((rows) => rows[0] ?? null);
if (!reactivated) {
throw new Error("Managed sandbox environment changed during reactivation");
}
row = reactivated;
providerReactivated = true;
for (const binding of matchingBindings) {
const reactivatedDefaults = {
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
status: "active",
};
const reactivatedHash = stockHash(reactivatedDefaults);
await tx
.update(builtInManagedResources)
.set({
stockHash: reactivatedHash,
defaultsJson: reactivatedDefaults,
updatedAt: new Date(),
})
.where(and(
eq(builtInManagedResources.id, binding.id),
eq(builtInManagedResources.resourceId, row.id),
));
if (binding.id === baseline?.id) {
baselineDefaults = reactivatedDefaults;
baselineHash = reactivatedHash;
}
}
}
await writeBindings(
row.id,
baseline?.stockVersion ?? input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
baselineHash,
baselineDefaults,
false,
);
return {
environment: toEnvironment(row),
action: "skipped",
stockStatus,
updateAvailable: true,
stockHash: latestStockHash,
};
}
if (stockStatus === "stock_current") {
await writeBindings(
row.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
desiredStock,
false,
);
return {
environment: toEnvironment(row),
action: "unchanged",
stockStatus,
updateAvailable: false,
stockHash: latestStockHash,
};
}
const updated = await tx
.update(environments)
.set({
name: desiredName,
description: input.description ?? null,
config: desiredConfig,
metadata: withoutManagedEnvironmentArchiveToken(
mergeManagedEnvironmentMetadata(row.metadata, desiredMetadata, keys),
),
status: "active",
updatedAt: new Date(),
})
.where(eq(environments.id, row.id))
.returning()
.then((rows) => rows[0] ?? null);
if (!updated) throw new Error("Managed sandbox environment changed during reconciliation");
await writeBindings(
updated.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
desiredStock,
true,
);
return {
environment: toEnvironment(updated),
action: "updated",
stockStatus,
updateAvailable: false,
stockHash: latestStockHash,
};
},
);
if (reconciliation.action !== "unchanged" || trackingInitialized) {
const action = reconciliation.action === "added"
? "environment.managed_stock_added"
: reconciliation.action === "updated"
? "environment.managed_stock_updated"
: reconciliation.action === "skipped"
? "environment.managed_stock_skipped"
: "environment.managed_stock_tracking_initialized";
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
companyId,
actorType: "system",
actorId: "managed-environment-reconciler",
action,
entityType: "environment",
entityId: reconciliation.environment.id,
details: {
provider: input.provider,
reconciliationAction: reconciliation.action,
stockStatus: reconciliation.stockStatus,
updateAvailable: reconciliation.updateAvailable,
stockHash: reconciliation.stockHash,
providerReactivated,
},
})));
}
return reconciliation;
};
/**
* Archive the Paperclip-managed sandbox row when its provider became
* unavailable (plugin missing, not ready, or its worker not running), so
* run scheduling stops selecting an environment whose lease acquisition
* cannot succeed (`resolveEnvironment` rejects non-active rows).
*
* Scoped to the row provisioned for the SAME provider: a row that a
* provider switch left on a different provider is not touched (the ensure
* path adopts it once the new provider is healthy). Reactivation is
* automatic — the next successful `ensureManagedSandboxEnvironment` stamps
* the row `active` again.
*
* Returns the archived environment, or null when there is no active
* managed row for this provider.
*/
const archiveManagedSandboxEnvironment = async (
input: { provider: string; companyId?: string },
): Promise<Environment | null> => {
let activityCompanyIds: string[] = [];
const archived = await db.transaction(async (tx) => {
const existing = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update")
.then(
(rows) => rows.find(
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? null,
);
if (!existing || existing.status !== "active") return null;
const rowProvider = (existing.metadata as Record<string, unknown> | null)
?.managedSandboxProvider;
if (rowProvider !== input.provider) return null;
const companyIds = input.companyId
? [input.companyId]
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
activityCompanyIds = companyIds;
const bindings = companyIds.length > 0
? await tx
.select()
.from(builtInManagedResources)
.where(and(
inArray(builtInManagedResources.companyId, companyIds),
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
eq(builtInManagedResources.resourceId, existing.id),
))
: [];
const archiveToken = randomUUID();
const archivedMetadata = {
...(existing.metadata ?? {}),
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]: archiveToken,
};
const archived = await tx
.update(environments)
.set({
status: "archived",
metadata: archivedMetadata,
updatedAt: new Date(),
})
.where(and(eq(environments.id, existing.id), eq(environments.status, "active")))
.returning()
.then((rows) => rows[0] ?? null);
if (!archived) return null;
// Archival is a Paperclip-owned availability transition. Record only
// that status change in each installed baseline. Deriving the new hash
// from defaultsJson keeps operator-modified row fields out of stock.
for (const binding of bindings) {
const archivedStock = {
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
status: "archived",
};
await tx
.update(builtInManagedResources)
.set({
stockHash: stockHash(archivedStock),
defaultsJson: {
...archivedStock,
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]: archiveToken,
},
updatedAt: new Date(),
})
.where(and(
eq(builtInManagedResources.id, binding.id),
eq(builtInManagedResources.resourceId, existing.id),
));
}
return toEnvironment(archived);
});
if (archived) {
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
companyId,
actorType: "system",
actorId: "managed-environment-reconciler",
action: "environment.managed_provider_unavailable_archived",
entityType: "environment",
entityId: archived.id,
details: { provider: input.provider },
})));
}
return archived;
};
return {
list: async (
companyIdOrFilters?: string | EnvironmentListFilters,
maybeFilters?: EnvironmentListFilters,
): Promise<Environment[]> => {
const filters = resolveListFilters(companyIdOrFilters, maybeFilters);
const conditions = [];
if (filters.status) conditions.push(eq(environments.status, filters.status));
if (filters.driver) conditions.push(eq(environments.driver, filters.driver));
const rows = await db
.select()
.from(environments)
.where(conditions.length > 0 ? and(...conditions) : undefined)
.orderBy(desc(environments.updatedAt), desc(environments.createdAt));
return rows.map(toEnvironment);
},
getById: async (id: string): Promise<Environment | null> => {
const row = await db.select().from(environments).where(eq(environments.id, id)).then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
/**
* List the companies that own an environment through a built-in managed
* resource binding. A managed sandbox row binds to each company that the
* instance provisions it for. An operator-created environment has no
* binding, so this returns an empty list. The caller treats an empty list
* as an instance-global environment with no company owner.
*/
listBoundCompanyIds: async (environmentId: string): Promise<string[]> => {
const rows = await db
.select({ companyId: builtInManagedResources.companyId })
.from(builtInManagedResources)
.where(
and(
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceId, environmentId),
),
);
return Array.from(new Set(rows.map((row) => row.companyId)));
},
getLeaseById: async (id: string): Promise<EnvironmentLease | null> => {
const row = await db
.select()
.from(environmentLeases)
.where(eq(environmentLeases.id, id))
.then((rows) => rows[0] ?? null);
return row ? toEnvironmentLease(row) : null;
},
/**
* Idempotently ensure THE local-driver environment row; the partial
* unique index `environments_local_driver_idx` enforces at most one per
* instance.
*
* On a cloud-managed instance an existing row is additionally ADOPTED —
* stamped `managedByPaperclip: true` (other metadata preserved) — so the
* single local slot is platform-owned there by construction, mirroring
* `ensureManagedSandboxEnvironment`'s adoption of the sandbox slot. This
* is what lets the environment-routes write floor treat a local row's
* platform markers as live state rather than a stale leftover: every
* caller (company creation, the heartbeat, run orchestration) converges
* the marker. Self-hosted instances keep the historical behavior:
* an existing row is returned untouched.
*/
ensureLocalEnvironment: async (_companyId?: string): Promise<Environment> => {
const now = new Date();
const insert = () =>
db
.insert(environments)
.values({
name: DEFAULT_LOCAL_ENVIRONMENT_NAME,
description: DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION,
driver: "local",
status: "active",
config: {},
envVars: {},
metadata: {
managedByPaperclip: true,
defaultForInstance: true,
},
createdAt: now,
updatedAt: now,
})
.onConflictDoNothing({
target: [environments.driver],
where: sql`${environments.driver} = 'local'`,
})
.returning()
.then((rows) => rows[0] ?? null);
const row = await insert().catch((error: unknown) => {
if (hasConstraintName(error, "environments_name_idx")) {
return null;
}
throw error;
});
if (row) return toEnvironment(row);
const existing = await db
.select()
.from(environments)
.where(eq(environments.driver, "local"))
.then((rows) => rows[0] ?? null);
if (!existing) {
throw new Error("Failed to ensure local environment");
}
const existingMetadata = (existing.metadata ?? {}) as Record<string, unknown>;
if (isCloudManagedInstance() && existingMetadata.managedByPaperclip !== true) {
const adopted = await db
.update(environments)
.set({
metadata: { ...existingMetadata, managedByPaperclip: true },
updatedAt: new Date(),
})
.where(eq(environments.id, existing.id))
.returning()
.then((rows) => rows[0] ?? existing);
return toEnvironment(adopted);
}
return toEnvironment(existing);
},
ensureManagedSandboxEnvironment,
archiveManagedSandboxEnvironment,
/**
* Idempotently ensure a managed Kubernetes sandbox environment exists for
* an instance, configured from instance/operator-supplied config. A thin
* wrapper over `ensureManagedSandboxEnvironment` that pins the provider to
* "kubernetes" and stamps the legacy marker `findKubernetesEnvironment`
* keys on. On subsequent calls stock-controlled config advances in place;
* operator modifications remain untouched for explicit review.
*/
ensureKubernetesEnvironment: async (
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
maybeConfig?: KubernetesEnvironmentConfigInput,
): Promise<Environment> => {
const config = resolveKubernetesConfig(companyIdOrConfig, maybeConfig);
return ensureManagedSandboxEnvironment({
companyId: typeof companyIdOrConfig === "string" ? companyIdOrConfig : undefined,
name: DEFAULT_KUBERNETES_ENVIRONMENT_NAME,
description: DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION,
provider: KUBERNETES_PROVIDER_KEY,
config,
extraMetadata: { [KUBERNETES_MANAGED_MARKER]: true },
}).then((result) => result.environment);
},
/**
* Find the active managed Kubernetes sandbox environment, if one
* exists. Read-only counterpart to `ensureKubernetesEnvironment` used by the
* per-run execution guard (which must not silently create config-less envs).
*/
findKubernetesEnvironment: async (_companyId?: string): Promise<Environment | null> => {
const rows = await db
.select()
.from(environments)
.where(
and(
eq(environments.driver, "sandbox"),
eq(environments.status, "active"),
),
)
.orderBy(desc(environments.updatedAt));
const match = rows.find(
(row) =>
(row.metadata as Record<string, unknown> | null)?.[KUBERNETES_MANAGED_MARKER] === true,
);
return match ? toEnvironment(match) : null;
},
/**
* Find the platform-managed sandbox environment (the single
* `managedByPaperclip`-marked slot row), if one exists. Read-only
* counterpart to `ensureManagedSandboxEnvironment`. The default
* (active-only) form serves the managed-sandbox-only run guard — which
* must fail closed rather than create a config-less environment when
* the slot is empty or archived (the provisioner archives it while its
* provider plugin is down). `includeArchived` serves reconciliation
* cleanup, which must find the row even after the provisioner archived
* it.
*/
findManagedSandboxEnvironment: async (
_companyId?: string,
options?: { includeArchived?: boolean },
): Promise<Environment | null> => {
const rows = await db
.select()
.from(environments)
.where(
options?.includeArchived === true
? eq(environments.driver, "sandbox")
: and(
eq(environments.driver, "sandbox"),
eq(environments.status, "active"),
),
)
.orderBy(desc(environments.updatedAt));
const match = rows.find(
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
);
return match ? toEnvironment(match) : null;
},
create: async (
companyIdOrInput: string | CreateEnvironment,
maybeInput?: CreateEnvironment,
options?: { db?: EnvironmentWriteDb },
): Promise<Environment> => {
const input = resolveCreateInput(companyIdOrInput, maybeInput);
const now = new Date();
const row = await (options?.db ?? db)
.insert(environments)
.values({
name: input.name,
description: input.description ?? null,
driver: input.driver,
status: input.status ?? "active",
config: input.config ?? {},
envVars: (input as CreateEnvironment & { envVars?: Record<string, unknown> }).envVars ?? {},
metadata: input.metadata ?? null,
createdAt: now,
updatedAt: now,
})
.returning()
.then((rows) => rows[0] ?? null)
.catch((error) => {
if (hasConstraintName(error, "environments_name_idx")) {
throw conflict(`An environment named "${input.name}" already exists for this instance.`);
}
if (hasConstraintName(error, "environments_local_driver_idx")) {
throw conflict("A local environment already exists for this instance.");
}
throw error;
});
if (!row) {
throw new Error("Failed to create environment");
}
return toEnvironment(row);
},
update: async (
id: string,
patch: UpdateEnvironment,
options?: { db?: EnvironmentWriteDb },
): Promise<Environment | null> => {
const writeDb = options?.db ?? db;
const values: Partial<typeof environments.$inferInsert> = {
updatedAt: new Date(),
};
if (patch.name !== undefined) values.name = patch.name;
if (patch.description !== undefined) values.description = patch.description ?? null;
if (patch.driver !== undefined) values.driver = patch.driver;
if (patch.status !== undefined) values.status = patch.status;
if (patch.config !== undefined) values.config = patch.config;
if ("envVars" in patch && patch.envVars !== undefined) {
values.envVars = (patch.envVars ?? {}) as Record<string, unknown>;
}
if (patch.metadata !== undefined) {
values.metadata = withoutManagedEnvironmentArchiveToken(patch.metadata ?? null);
} else if (patch.status !== undefined) {
const existingMetadata = await writeDb
.select({ metadata: environments.metadata })
.from(environments)
.where(eq(environments.id, id))
.then((rows) => rows[0]?.metadata ?? null);
values.metadata = withoutManagedEnvironmentArchiveToken(existingMetadata);
}
const row = await writeDb
.update(environments)
.set(values)
.where(eq(environments.id, id))
.returning()
.then((rows) => rows[0] ?? null)
.catch((error) => {
if (hasConstraintName(error, "environments_name_idx")) {
throw conflict(`An environment named "${patch.name}" already exists for this instance.`);
}
if (hasConstraintName(error, "environments_local_driver_idx")) {
throw conflict("A local environment already exists for this instance.");
}
throw error;
});
return row ? toEnvironment(row) : null;
},
remove: async (id: string): Promise<Environment | null> => {
const row = await db
.delete(environments)
.where(eq(environments.id, id))
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
removeIfDeletable: async (id: string): Promise<Environment | null> => {
const row = await db
.delete(environments)
.where(
and(
eq(environments.id, id),
ne(environments.driver, "local"),
sql`not exists (
select 1 from ${instanceSettings}
where ${instanceSettings.defaultEnvironmentId} = ${environments.id}
)`,
// A `pending_cleanup` lease is the durable teardown reference for an
// orphan sandbox. The environment foreign key uses
// `on delete set null`, so a delete keeps the lease row but drops its
// environment reference. This predicate refuses the delete while such
// a lease exists, so the operator resolves the cleanup first and the
// lease keeps its environment link. It runs in the same statement as
// the delete, so it also closes the check-to-delete race.
sql`not exists (
select 1 from ${environmentLeases}
where ${environmentLeases.environmentId} = ${environments.id}
and ${environmentLeases.status} = 'pending_cleanup'
)`,
// A reusable lease keeps a provider sandbox after a run releases
// it, including when the run failed but release succeeded.
// Deleting the environment would set its reference to
// null, and both the normal release path and scoped reusable cleanup
// require that environment context. Refuse the delete atomically
// until the owning issue/workspace destroys the reusable sandbox.
sql`not exists (
select 1 from ${environmentLeases}
where ${environmentLeases.environmentId} = ${environments.id}
and ${environmentLeases.leasePolicy} = 'reuse_by_environment'
and ${environmentLeases.status} in ('active', 'released', 'retained', 'failed')
)`,
),
)
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
/**
* Return true when the environment has one or more leases in the terminal
* `pending_cleanup` state. Each such lease is the only durable provider
* reference for an orphan sandbox that a teardown retry must destroy. The
* delete guard and the provider-change guard call this to protect that
* reference.
*/
hasUnresolvedPendingCleanupLeases: async (environmentId: string): Promise<boolean> => {
const rows = await db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, environmentId),
eq(environmentLeases.status, "pending_cleanup"),
),
);
return countFromRows(rows) > 0;
},
getDeleteBlastRadius: async (id: string): Promise<EnvironmentDeleteBlastRadius | null> => {
const environment = await db
.select({
id: environments.id,
driver: environments.driver,
})
.from(environments)
.where(eq(environments.id, id))
.then((rows) => rows[0] ?? null);
if (!environment) return null;
const [
instanceDefaultRows,
agentDefaultRows,
executionWorkspaceRows,
issueRows,
projectRows,
secretBindingRows,
activeLeaseRows,
pendingCleanupLeaseRows,
reusableSandboxLeaseRows,
activeSetupRows,
] = await Promise.all([
db
.select({ count: sql<number>`count(*)::int` })
.from(instanceSettings)
.where(eq(instanceSettings.defaultEnvironmentId, id)),
db
.select({ count: sql<number>`count(*)::int` })
.from(agents)
.where(eq(agents.defaultEnvironmentId, id)),
db
.select({ count: sql<number>`count(*)::int` })
.from(executionWorkspaces)
.where(sql`${executionWorkspaces.metadata} -> 'config' ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(issues)
.where(sql`${issues.executionWorkspaceSettings} ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(projects)
.where(sql`${projects.executionWorkspacePolicy} ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(companySecretBindings)
.where(
and(
eq(companySecretBindings.targetType, "environment"),
eq(companySecretBindings.targetId, id),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.status, "active"),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.status, "pending_cleanup"),
),
),
db
.select({
leaseId: environmentLeases.id,
executionWorkspaceId: environmentLeases.executionWorkspaceId,
executionWorkspaceName: executionWorkspaces.name,
issueId: environmentLeases.issueId,
issueIdentifier: issues.identifier,
issueTitle: issues.title,
})
.from(environmentLeases)
.leftJoin(executionWorkspaces, eq(environmentLeases.executionWorkspaceId, executionWorkspaces.id))
.leftJoin(issues, eq(environmentLeases.issueId, issues.id))
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentCustomImageSetupSessions)
.where(
and(
eq(environmentCustomImageSetupSessions.environmentId, id),
inArray(environmentCustomImageSetupSessions.status, [...ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES]),
),
),
]);
const isManagedLocal = environment.driver === "local";
const isInstanceDefault = countFromRows(instanceDefaultRows) > 0;
const pendingCleanupLeaseCount = countFromRows(pendingCleanupLeaseRows);
const reusableSandboxLeaseHolders = reusableSandboxLeaseRows.map((row) => ({
leaseId: row.leaseId,
executionWorkspaceId: row.executionWorkspaceId,
executionWorkspaceName: row.executionWorkspaceName,
issueId: row.issueId,
issueIdentifier: row.issueIdentifier,
issueTitle: row.issueTitle,
}));
const reusableSandboxLeaseCount = reusableSandboxLeaseHolders.length;
const deleteBlockedReasons: EnvironmentDeleteBlockedReason[] = [];
if (isManagedLocal) deleteBlockedReasons.push("managed_local");
if (isInstanceDefault) deleteBlockedReasons.push("instance_default");
// A `pending_cleanup` lease is the durable teardown reference for an orphan
// sandbox. The environment foreign key uses `on delete set null`, so a
// delete keeps the lease but drops its environment reference. Block the
// delete until the sweep resolves the lease, so the operator resolves the
// cleanup first and the lease keeps its environment link.
if (pendingCleanupLeaseCount > 0) deleteBlockedReasons.push("pending_sandbox_cleanup");
if (reusableSandboxLeaseCount > 0) deleteBlockedReasons.push("reusable_sandbox_lease");
const activeLeaseCount = countFromRows(activeLeaseRows);
const activeCustomImageSetupSessionCount = countFromRows(activeSetupRows);
return {
environmentId: id,
canDelete: deleteBlockedReasons.length === 0,
deleteBlockedReasons,
pendingCleanupLeaseCount,
reusableSandboxLeaseCount,
reusableSandboxLeaseHolders,
staticReferences: {
isManagedLocal,
isInstanceDefault,
agentDefaultCount: countFromRows(agentDefaultRows),
executionWorkspaceSelectionCount: countFromRows(executionWorkspaceRows),
issueSelectionCount: countFromRows(issueRows),
projectSelectionCount: countFromRows(projectRows),
secretBindingCount: countFromRows(secretBindingRows),
},
activeRuntimeUse: {
activeLeaseCount,
activeCustomImageSetupSessionCount,
hasActiveRuntimeUse: activeLeaseCount > 0 || activeCustomImageSetupSessionCount > 0,
},
};
},
listLeases: async (
environmentId: string,
filters: {
status?: string;
} = {},
): Promise<EnvironmentLease[]> => {
const conditions = [eq(environmentLeases.environmentId, environmentId)];
if (filters.status) conditions.push(eq(environmentLeases.status, filters.status));
const rows = await db
.select()
.from(environmentLeases)
.where(and(...conditions))
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
return rows.map(toEnvironmentLease);
},
acquireLease: async (input: {
companyId: string;
environmentId: string;
executionWorkspaceId?: string | null;
issueId?: string | null;
heartbeatRunId?: string | null;
leasePolicy?: EnvironmentLeasePolicy;
provider?: string | null;
providerLeaseId?: string | null;
expiresAt?: Date | null;
metadata?: Record<string, unknown> | null;
/**
* Atomically retire the previous database ownership record when this
* acquisition reuses the same provider resource for a new run. The
* provider resume happens before this write, so a failed transaction
* leaves the prior retained row recoverable instead of publishing two
* reusable owners for one sandbox.
*/
replacesReusableLeaseId?: string | null;
/**
* Reactivate the exact lease row already owned by this heartbeat run.
* Native same-run recovery re-enters environment startup after the
* provider process is interrupted; it must not manufacture a second
* database owner for the same run and provider resource.
*/
reusesReusableLeaseId?: string | null;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login routes set this to close the check-to-lease
* race: managed reconciliation can bind a sandbox to another company
* between the route guard and this acquire. When the environment is
* bound to a company other than `companyId`, the insert throws the 403
* `environment_company_mismatch` and no lease row is created (fail
* closed). An unbound (instance-global) environment stays open to every
* member. All other callers keep the plain, non-transactional insert.
*/
assertCompanyBinding?: boolean;
}): Promise<EnvironmentLease> => {
const now = new Date();
const values = {
companyId: input.companyId,
environmentId: input.environmentId,
executionWorkspaceId: input.executionWorkspaceId ?? null,
issueId: input.issueId ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
status: "active" as const,
leasePolicy: input.leasePolicy ?? "ephemeral",
provider: input.provider ?? null,
providerLeaseId: input.providerLeaseId ?? null,
acquiredAt: now,
lastUsedAt: now,
expiresAt: input.expiresAt ?? null,
releasedAt: null,
failureReason: null,
cleanupStatus: null,
metadata: input.metadata ?? null,
createdAt: now,
updatedAt: now,
};
if (
(input.replacesReusableLeaseId || input.reusesReusableLeaseId) &&
((!input.executionWorkspaceId && (!input.issueId || typeof input.metadata?.agentId !== "string")) || !input.providerLeaseId)
) {
throw new Error(
"A reusable lease handoff requires a workspace or agent-bound task, and a provider lease id.",
);
}
if (input.reusesReusableLeaseId && !input.heartbeatRunId) {
throw new Error(
"A same-run reusable lease reacquisition requires a heartbeat run id.",
);
}
if (input.replacesReusableLeaseId && input.reusesReusableLeaseId) {
throw new Error(
"A reusable lease cannot be replaced and reacquired in the same operation.",
);
}
const row =
input.assertCompanyBinding ||
input.replacesReusableLeaseId ||
input.reusesReusableLeaseId
? await db.transaction(async (tx) => {
if (input.assertCompanyBinding) {
// Lock the environment row first. Managed reconciliation locks the
// same sandbox environment rows with `for update` before it writes a
// company binding, so this lock serializes the two transactions on
// this row and closes the time-of-check to time-of-use window.
await tx
.select({ id: environments.id })
.from(environments)
.where(eq(environments.id, input.environmentId))
.for("update");
// Re-read the company binding inside the locked transaction. A
// binding a reconciliation committed after the route guard now
// appears here. Reject a foreign-company environment before the
// insert, so the login holds no lease.
const boundRows = await tx
.select({ companyId: builtInManagedResources.companyId })
.from(builtInManagedResources)
.where(
and(
eq(
builtInManagedResources.resourceKind,
MANAGED_ENVIRONMENT_RESOURCE_KIND,
),
eq(
builtInManagedResources.resourceId,
input.environmentId,
),
),
);
const boundCompanyIds = Array.from(
new Set(boundRows.map((boundRow) => boundRow.companyId)),
);
if (
boundCompanyIds.length > 0 &&
!boundCompanyIds.includes(input.companyId)
) {
throw forbidden(
"The selected environment belongs to another company.",
{
code: "environment_company_mismatch",
},
);
}
}
if (input.replacesReusableLeaseId) {
const retired = await tx
.update(environmentLeases)
.set({
status: "expired",
releasedAt: now,
lastUsedAt: now,
updatedAt: now,
cleanupStatus: "success",
})
.where(
and(
eq(environmentLeases.id, input.replacesReusableLeaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.environmentId, input.environmentId),
input.executionWorkspaceId
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
: and(
isNull(environmentLeases.executionWorkspaceId),
eq(environmentLeases.issueId, input.issueId!),
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
eq(
environmentLeases.providerLeaseId,
input.providerLeaseId!,
),
inArray(environmentLeases.status, [
"released",
"retained",
]),
),
)
.returning({ id: environmentLeases.id });
if (retired.length !== 1) {
throw conflict(
"Reusable sandbox lease ownership changed during acquisition.",
);
}
}
if (input.reusesReusableLeaseId) {
const reacquired = await tx
.update(environmentLeases)
.set({
status: "active",
releasedAt: null,
lastUsedAt: now,
expiresAt: input.expiresAt ?? null,
failureReason: null,
cleanupStatus: null,
metadata: input.metadata ?? null,
updatedAt: now,
})
.where(
and(
eq(environmentLeases.id, input.reusesReusableLeaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.environmentId, input.environmentId),
input.executionWorkspaceId
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
: and(
isNull(environmentLeases.executionWorkspaceId),
eq(environmentLeases.issueId, input.issueId!),
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
),
eq(
environmentLeases.heartbeatRunId,
input.heartbeatRunId!,
),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
eq(
environmentLeases.providerLeaseId,
input.providerLeaseId!,
),
inArray(environmentLeases.status, [
"active",
"released",
"retained",
]),
),
)
.returning()
.then((rows) => rows[0] ?? null);
if (!reacquired) {
throw conflict(
"Reusable sandbox lease ownership changed during reacquisition.",
);
}
return reacquired;
}
return tx
.insert(environmentLeases)
.values(values)
.returning()
.then((rows) => rows[0] ?? null);
})
: await db
.insert(environmentLeases)
.values(values)
.returning()
.then((rows) => rows[0] ?? null);
if (!row) {
throw new Error("Failed to acquire environment lease");
}
return toEnvironmentLease(row);
},
releaseLease: async (
id: string,
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed" | "retained" | "pending_cleanup"> = "released",
options?: {
expectedPendingCleanupAttemptId?: string;
failureReason?: string;
cleanupStatus?: EnvironmentLeaseCleanupStatus;
remoteExecutionTermination?: Record<string, unknown>;
},
) => {
const now = new Date();
const row = await db
.update(environmentLeases)
.set({
status,
releasedAt: status === "retained" ? null : now,
lastUsedAt: now,
updatedAt: now,
...(options?.failureReason !== undefined ? { failureReason: options.failureReason } : {}),
...(options?.cleanupStatus !== undefined ? { cleanupStatus: options.cleanupStatus } : {}),
// A later release without a receipt cannot reuse an earlier stop's
// authority (for example after a same-run lease resume).
metadata: options?.remoteExecutionTermination
? sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ remoteExecutionTermination: options.remoteExecutionTermination })}::jsonb`
: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'`,
})
.where(and(eq(environmentLeases.id, id), options?.expectedPendingCleanupAttemptId
? and(eq(environmentLeases.status, "pending_cleanup"),
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.expectedPendingCleanupAttemptId}`)
: undefined))
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironmentLease(row) : null;
},
/**
* Record a lease-less orphan sandbox directly in the terminal
* `pending_cleanup` state with one atomic insert. The sweep reads a row with
* status `pending_cleanup` and cleanup status `failed`, so this insert makes
* the orphan visible to recovery immediately. It never passes through the
* `active` state, so a process or database crash cannot strand the row in an
* intermediate state that no sweep finds. The insert skips the company
* binding assertion, so it records the orphan even for a foreign-bound
* environment.
*
* The insert runs in a transaction that first locks the environment row with
* `for update`. The delete path (`removeIfDeletable`) refuses a delete while a
* `pending_cleanup` lease exists, so this lock serializes the two writes:
*
* - The environment still exists: the insert records the orphan with the
* environment reference. A concurrent delete blocks on the lock, then its
* `not exists (pending_cleanup)` predicate fails, so the delete refuses.
* - The environment is already gone (a delete won the race before this
* insert): the insert records the orphan with a null environment
* reference. The row carries the immutable provider metadata the sweep
* needs, so the teardown still runs. This closes the window the finding
* describes, where the foreign-key insert fails after a delete.
*/
insertPendingCleanupLease: async (input: {
companyId: string;
environmentId: string;
executionWorkspaceId?: string | null;
issueId?: string | null;
heartbeatRunId?: string | null;
provider?: string | null;
providerLeaseId?: string | null;
metadata?: Record<string, unknown> | null;
failureReason: string;
}): Promise<EnvironmentLease> => {
const now = new Date();
const row = await db.transaction(async (tx) => {
// Lock the environment row so a concurrent delete cannot commit between
// this read and the insert. An absent row means a delete already removed
// the environment, so record the orphan with a null reference.
const environmentRows = await tx
.select({ id: environments.id })
.from(environments)
.where(eq(environments.id, input.environmentId))
.for("update");
const environmentIdForRow = environmentRows[0]?.id ?? null;
return tx
.insert(environmentLeases)
.values({
companyId: input.companyId,
environmentId: environmentIdForRow,
executionWorkspaceId: input.executionWorkspaceId ?? null,
issueId: input.issueId ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
status: "pending_cleanup" as const,
leasePolicy: "ephemeral",
provider: input.provider ?? null,
providerLeaseId: input.providerLeaseId ?? null,
acquiredAt: now,
lastUsedAt: now,
expiresAt: null,
releasedAt: now,
failureReason: input.failureReason,
cleanupStatus: "failed",
metadata: input.metadata ?? null,
createdAt: now,
updatedAt: now,
})
.returning()
.then((rows) => rows[0] ?? null);
});
if (!row) {
throw new Error("Failed to record pending sandbox cleanup lease");
}
return toEnvironmentLease(row);
},
updateLeaseMetadata: async (
id: string,
metadata: Record<string, unknown> | null,
): Promise<EnvironmentLease | null> => {
const row = await db
.update(environmentLeases)
.set({
metadata,
lastUsedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(environmentLeases.id, id))
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironmentLease(row) : null;
},
releaseLeasesForRun: async (
heartbeatRunId: string,
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
): Promise<EnvironmentLease[]> => {
const now = new Date();
const rows = await db
.update(environmentLeases)
.set({
status,
releasedAt: now,
lastUsedAt: now,
updatedAt: now,
})
.where(
and(
eq(environmentLeases.heartbeatRunId, heartbeatRunId),
eq(environmentLeases.status, "active"),
),
)
.returning();
return rows.map(toEnvironmentLease);
},
};
}