Files
PaperClipAI/server/src/services/environments.ts
T
DottaandPaperclip 45a3a92e6d fix(environments): persist sandbox ownership before creation (#18357)
## Thinking Path

> - Paperclip manages companies and the agents that do their work.
> - Remote execution creates provider resources that need a durable
owner.
> - The server could create a sandbox before it saved its lease.
> - A failed lease write could leave cleanup data only on the old web
server disk.
> - This change saves a company-scoped operation before creation and
transfers ownership atomically.
> - Recovery finds the original Daytona resource or records an explicit
manual reconciliation outcome.

## Linked Issues or Issue Description

**What happened?**
A sandbox could exist without a lease after a database failure or
company rebind. Failed cleanup recording could then create a local
fallback file. Replacing that web server could lose the cleanup
reference.

**Expected behavior**
A failed admission write makes no provider call. An interrupted creation
retains a committed operation. Recovery must not create a second sandbox
or destroy an attached lease.

**Steps to reproduce**
1. Run the disposable PostgreSQL cases in
`server/src/__tests__/sandbox-creation-operations.test.ts`.
2. Reject the lease insert after the controlled provider creates its
resource.
3. Start a new receiver process with a different empty web disk. It must
find the same resource and create one cleanup owner.

Related ownership report: Refs #14672. This change addresses new
allocations before lease persistence. It does not replace existing lease
recovery.

## What Changed

- Add an idempotent migration and a small creation-operation table.
Retain original company, environment and run identities across deletion.
- Commit creation intent before run, ad-hoc lease, and plugin task
replacement calls. Fence duplicate requests and ambiguous lease commits.
- Retain a known returned task resource ID in the existing operation
before attachment. Preserve manual reconciliation and never delete an
ambiguously attached resource.
- Keep company and run authority checks and atomic lease transfer with
the lease owner.
- Reuse Daytona names, labels, read-only inspection and claimed cleanup.
Keep provider payload handling in the provider.
- Retain nonsecret credential identity hashes. Refuse recovery lookup
under a changed credential value.
- Record manual reconciliation for missing/deleted or explicitly
inactive saved credentials, versions, and provider vaults. Keep
temporary resolution failures retryable. Never reactivate a disabled
credential.
- Use transaction signals and concrete deadlines. Failed records back
off independently. Manual reconciliation stays durable and quiet.
- Remove new orphan-file fallback writes from the covered paths. Keep
legacy file replay and its tests for cutover.
- Replace obsolete fallback-producer assertions with normal-entrypoint
durability tests. Keep cleanup, diagnostics and warm handoff coverage.

## Verification

- [Independent current-head
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320)
passed 190/190 combined tests, 40/40 incoming TestOps tests, full
workspace typecheck, and build. PR head
`958af986130f25e136065ad30d1fdd843c46255c` has tree
`e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined
commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL,
compiled Rust Runner and controlled object providers were used;
external-S3 integration remains outside this local proof.
- Current head `958af986130f25e136065ad30d1fdd843c46255c` normally
merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole
conflict was the maintenance test: a parameterized case now preserves
both legacy-local `local_storage` and valid shared-S3
`provider_mismatch` checks, including the parent local receipt, exact
version identity, missing-snapshot/no-seed refusal, and unchanged
persisted row. The three master TestOps files remain intact. No
production, schema, migration, or package changes were made after the
fully qualified prior head `0b4d05505e`. Independent review approved the
resolution; the full managed-skill-content file and Cloud guard controls
pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge
head, first
attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412):
54 successful checks and three intended skips. The additional Paperclip
review is neutral at 4/5; two external Superagent checks (Contributor
trust and Security Scan) remain in progress with zero annotations and no
assessment, so they are not passing scan evidence. Greptile reviewed
exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final
thread audit has zero unresolved threads. All completed evidence below
retains its recorded revision scope. The additional Paperclip Review on
`958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or
should-fix item. Its optional observation is that a generic create
rejection remains pending/manual even when an operator believes
allocation did not occur. This change preserves uncertainty because the
generic create-error contract does not prove provider absence; no new
error classification or provider-specific compensation was added.

- [Independent final-source
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247)
passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full
workspace typecheck, and build. The integration commit
`c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as
final PR head `0b4d05505e` (tree
`bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider
coverage passed; the separately guarded external-S3 suite was not run
because the bounded local MinIO pull was unavailable. This includes the
two final compatibility fixes.
- The first final-head CI attempt exposed one E2E timing failure in
`disconnect-write-recovery.spec.ts:181`: the title stayed unchanged
through the 15-second session-recovery poll. The same original case
passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all
UI/session/E2E source is identical across those heads. The failure log
and CI artifacts are retained; no cause is claimed. One failed-job-only
retry is authorized on this evidence, without changing a timeout,
assertion, or production behavior. [The single failed-job retry
completed
successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
all 35 E2E shard cases passed (6.6 minutes), including the original
session-recovery case in 14.0 seconds. The initial failure remains
disclosed and its cause remains unproven.
- Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two
independently reviewed integration fixes after the complete 7ec CI
matrix exposed two deterministic failures. Server 9/12 failed legacy
local-storage maintenance because the CLI reached the new Cloud server
startup guard; server 5/12 found the Daytona package version did not
match its bumped manifest. The failed run is retained at
https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is
not passing evidence and was not blindly retried.
- Maintenance now constructs the existing explicit storage dependency
after loading the required persisted Cloud snapshot, and passes it to
both maintenance owners. It can verify legacy local storage before
provider cutover while normal Cloud server startup remains guarded. The
original case reproduced red locally, then the complete
managed-skill-content and managed-cloud-prerequisites files passed 57/57
(32.05 seconds), including missing-snapshot refusal, unchanged stored
configuration, and existing server guard controls. Daytona package
version now matches manifest 0.1.10; the complete bundled package
identity file passed 14/14 (3.98 seconds), and explicit
manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42
seconds). An initial workspace-filter command matched no project and is
not counted as pass evidence. The final diff changes no sandbox runtime
or schema behavior. [Full CI is green on this exact final
head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
56 successful checks, three intended skips, one additional
incomplete/neutral Paperclip Review, and no pending or failed checks.
The neutral review is not a passing assessment. [Greptile reviewed the
exact final head at
5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969),
with no new actionable issues and all review threads resolved.
- Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626`
includes master managed-skills migration 0417 and shared-storage
prerequisites from `198218bf55` through normal merges. Sandbox creation
moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL
is byte-for-byte unchanged. The regenerated 0418 snapshot contains both
sandbox creation operations and the master skill mutation receipts, and
points to the master 0417 snapshot. Master SQL and journal history
remain intact. Exactly five snapshots remain, 0414 through 0418. The
only sandbox-specific changes since the approved credential policy are
migration metadata/number and the matching test/document references; the
sandbox runtime files are unchanged.
- [Independent final combined
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026)
passed on integration revision `e6f05b4cf0cd`, containing final sandbox
head `7ec06f77b7` and actual master `198218bf55`: migration/declaration
checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251
seconds), full workspace typecheck (81 seconds), and build (35 seconds).
- Final combined migration proof: fresh real PostgreSQL has both tables.
A separate original-0417 sandbox table with a retained pending row
survives two applications of the renamed 0418 SQL unchanged, with the
expected primary/index and foreign-key structure. Migration
numbering/safety checks pass. The normal runtime migration reapply case
passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first
temporary qualification harness stopped before any database assertion
because its external `.ts` file selected CommonJS; the `.mts` harness
corrected that setup and passed. Independent review approved the exact
metadata and reference diff. Earlier full suite and live-provider
results below retain their stated revision scope.

- Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd`
handles the remaining inactive-credential review warning. The existing
boundary accepts only concrete secret-owner 422 codes and exact
no-details disabled/coming-soon vault messages. Unknown 422,
authorization errors, provider wrappers, and temporary database/provider
failures still propagate. Independent source review approved this
three-file correction. No generic resolver, schema, provider protocol,
retry cap, or activation behavior changed.
- Creation/cleanup qualification at `c606d473b0`: all 39 cases pass
against real PostgreSQL (81.27 seconds). The new cases use actual
disabled/archived saved secrets, a valid disabled version, and provider
vaults created ready before normal API updates disable them or mark them
coming soon. Each remains manual through restart/repeated scans, with
zero provider lookup and no retry deadline. The independent plugin idle
blocker remains intact. The temporary 503 control retains retry
ownership and later recovers one cleanup owner. Before this correction,
all five inactive-state cases remained pending; that failed evidence is
retained. The prior 18-case known-ID task proof below belongs to
`237144fcf6`, whose helper and task path are unchanged by this
correction.

- Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also
addresses the independent task-replacement reliability warning. The
existing creation operation now saves only the known nonsecret returned
resource ID before expiry validation and attachment. The write is scoped
to operation/company/original environment and allows only a missing or
matching ID. It changes no ownership state and copies no resolved
configuration, credentials, or provider payload. Independent source
review approved this four-file fix.
- The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds).
Rejected attachments retain the ID through manual reconciliation and
repeated scans. A real committed attachment followed by an injected lost
response retains the same active lease and attached operation, with no
release or second create. Same-ID retries work; conflicting IDs and
foreign companies cannot change it. The initial three regression cases
failed because the ID was absent, before the fix.
- The shipped Daytona manifest does not declare task support, so it
cannot enter this generic replacement path. The dispatcher requires both
declared task support and an environmentTask worker method. Generic
drivers without creation lookup still use manual reconciliation, with a
known returned ID retained when the database write can commit. No
automatic lookup capability or unconditional compensation was added.

- Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0`
fixes the fresh review finding about missing credentials. Only known
secret-owner 404 results at the recovery secret-resolution boundary
select the existing manual outcome. Provider inspection and transient
database/provider errors remain retryable. Independent source review
approved this change.
- Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass
against isolated real PostgreSQL. The complete creation suite passes
16/16 (100.75 seconds); the complete cleanup coordinator suite passes
18/18 (99.83 seconds). The new cases use normal saved-secret creation
and deletion, confirm zero provider lookup, restart/repeated scans, no
automatic deadline, preserved independent plugin idle blocking, and no
additional idle blocker after the controlled worker supplies its quiet
proof. A temporary 503 resolution failure preserves pending ownership
and later recovers one cleanup owner. No additional live provider
resource was created.
- Failed attempts remain part of the evidence. The initial red run
reproduced the pending-operation bug and also exposed a numeric-deadline
test harness mistake, which was corrected. A later combined run reported
an existing replacement-receiver failure and a new idle-fixture
assertion failure; it was interrupted while the next test file remained
in imports. An isolated diagnostic confirmed that the manual state was
correct but the installed fixture plugin correctly blocked sleep without
its independent quiet proof. The corrected test retains that blocker
assertion before supplying the controlled worker witness. The unchanged
replacement-receiver case passes in the final complete 16-case run; its
earlier failure is not counted as a pass or assigned a proven cause. No
timeout or production idle guard was relaxed.
- Greptile reviewed previous combined head
`7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved
review threads. Its CI produced the two concrete integration failures
addressed above. The earlier CI and live evidence below remain
historical evidence, not a claim that the current full gates have
completed.

- Master integration update: merged `dbff3747b8` and then latest master
`76fde891cc` without rewriting branch history. Only `doc/DATABASE.md`
conflicted; the sandbox ownership, object-backed run evidence, and cloud
configuration sections all remain. The master merges changed no sandbox
implementation, migration, or provider code. Migration numbering/safety
checks pass and exactly five schema snapshots remain. At merge head
`f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused
creation/coordinator suites pass all 32 cases against isolated real
PostgreSQL (64.57 seconds). Those merge-head checks precede the
missing-credential fix below. The earlier live Daytona proof still
covers the unchanged normal creation/lookup/teardown path; the new
missing-credential branch is qualified with real PostgreSQL and
controlled provider hooks.

The optional Superagent Security Scan was inconclusive on prior head
`7ec06f77b7`: its pinned PR merge-base checkout failed or timed out,
with a neutral conclusion and zero annotations. An earlier scan could
not start within 120 seconds; a bounded rerun request returned HTTP 404.
These scanner failures are not passing security-scan evidence. On final
head `0b4d05505e`, the additional Paperclip Review check is neutral: its
assigned review execution did not complete and produced no validated
assessment. Fresh Greptile 5/5 and independent source review are
separate evidence. Previous-head required CI, independent review, and
the live Daytona qualification below passed.

- [Complete previous-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052)
passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful
checks and three intended skips, including typecheck, full tests, build
and E2E. Greptile is 5/5 on this head and both original findings are
resolved.

- Initial head: 200 PostgreSQL runtime, creation and coordinator tests
passed. The corrected head also passes all 32 creation and coordinator
tests, including deletion before/during lookup, a failed lookup after
deletion, restart, settled polling and actual idle-safety checks. Cases
cover failed admission, failed lease writes, concurrent recovery, lost
provider replies, ambiguous transfer retries, company rebind,
environment deletion, stopped runs, lookup backoff, restart, drain and
settled SQL quiet.
- Separate receiver processes: passed. A controlled HTTP provider
retains the resource while three independent receiver processes use two
empty web disks. Recovery creates one cleanup owner and never repeats
creation.
- Fresh PostgreSQL migration and two additional applications: passed.
The pending creation row survives unchanged.
- Server TypeScript and SDK/shared build checks pass after the
corrections. Migration numbering/safety checks passed. Exactly five
schema snapshots remain.
- Daytona adapter: all 330 tests passed, including stable-name lookup,
ownership checks, deletion and the updated manifest version. An earlier
run exposed the version expectation and it was corrected. Previous-head
CI passed.
- Live Daytona acceptance passed on `0b5cbb763a`: isolated real
PostgreSQL, normal runtime acquisition through independent receiver
processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu
24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert
failure retained the committed operation; a fresh receiver with empty
disk found the same sandbox, repeated recovery retained one cleanup
owner, and the actual claimed cleanup sweep expired the lease and
resolved the operation. Independent SDK lookup returned typed not-found;
ordinary recorded-ID inspection returned destroyed. This successful
attempt made one create, one recovery lookup, and one teardown. Two
earlier failed harness/provider-confirmation attempts are disclosed in
the [live acceptance
record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054).
The coordinator independently queried all three exact resource IDs
through the SDK and confirmed typed not-found for each.
- Complete CI-affected heartbeat/plugin warm-task suites passed 21/21
after test corrections. An unchanged workspace exposure case passed in
isolation after the prior CI shard reported an unhandled readiness
rejection. Complete previous-head CI and refreshed review passed.

## Risks

- Deleted companies retain an explicit manual reconciliation outcome.
Recovery does not retry missing credentials or insert a lease for a
deleted company.
- Only Daytona declares automatic creation lookup. Other uncertain
providers retain `needs_reconciliation`. A missing creation name alone
cannot prove that a late allocation will never appear.
- Pending operations use the existing process-local signal and startup
recovery model. This does not add cross-replica wake delivery or prove
deployed sleep.
- Legacy direct provider probes and custom-image setup use separate
lifecycles. They are outside this replacement of the run-lease
orphan-file fallback. The normal Test path that acquires a runtime lease
is covered.
- Existing orphan files still require the separate cutover inventory and
settlement. This change does not delete them.
- The schema change is additive. The new SQL can be reapplied.
Historical SQL and journal entries are preserved; only the oldest
generated snapshot is pruned.

## Model Used

OpenAI `gpt-6.1-sol`, with tool use, code execution and repository
review. The harness did not report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


The required branch status checks (`ci / verify`, `ci / e2e`) and all
test/build jobs passed on this head. The non-required Superagent
Contributor trust and Security Scan checks are still running without
findings; they are not claimed as passes. No required check or branch
rule is overridden.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 17:38:23 -05:00

1878 lines
80 KiB
TypeScript

import { createReusableLeaseHandoff, preserveReusableLeaseHandoff, withoutReusableLeaseHandoff } from "./environment-lease-handoff.js";
import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js";
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
import { randomUUID } from "node:crypto";
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "./environment-custom-image-runtime.js";
import { and, desc, eq, inArray, isNull, ne, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agents,
builtInManagedResources,
companies,
companySecretBindings,
environmentCustomImageSetupSessions,
environmentLeases,
sandboxCreationOperations,
environments,
executionWorkspaces,
heartbeatRuns,
instanceSettings,
issues,
projects,
} from "@paperclipai/db";
import {
ENVIRONMENT_DRIVERS,
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
ENVIRONMENT_LEASE_POLICIES,
ENVIRONMENT_LEASE_STATUSES,
ENVIRONMENT_STATUSES,
type CreateEnvironment,
type Environment,
type EnvironmentDeleteBlastRadius,
type EnvironmentDeleteBlockedReason,
type EnvironmentLease,
type EnvironmentLeaseCleanupStatus,
type EnvironmentLeasePolicy,
type EnvironmentLeaseStatus,
type UpdateEnvironment,
} from "@paperclipai/shared";
import { conflict, forbidden } from "../errors.js";
import { logActivity } from "./activity-log.js";
import { isCloudManagedInstance } from "./cloud-instance.js";
import {
resourceStatus,
stockHash,
type ManagedResourceStockStatus,
} from "./managed-resource-drift.js";
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
return db.transaction(async tx => {
await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
const [operation] = await tx.select().from(sandboxCreationOperations)
.where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update");
if (scope.heartbeatRunId) {
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"),
)).for("update");
if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed");
}
if (operation?.state === "attached" && operation.leaseId) {
const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId);
if (existing?.status === "active") return existing;
}
if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred");
const lease = await attach(tx as unknown as Db);
await notifyEnvironmentCleanup(tx, { companyId: scope.companyId });
await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
.where(eq(sandboxCreationOperations.id, id));
return lease;
});
}
export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {
const [environment] = await db.select({ id: environments.id }).from(environments)
.where(eq(environments.id, environmentId)).for("update");
if (!environment) throw new Error("Sandbox creation environment no longer exists");
const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources)
.where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId)));
if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) {
throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" });
}
}
type EnvironmentRow = typeof environments.$inferSelect;
type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
const DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION =
"Default execution environment for Paperclip runs on this machine.";
const DEFAULT_KUBERNETES_ENVIRONMENT_NAME = "Kubernetes Sandbox";
const DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION =
"Managed Kubernetes sandbox environment for hosted tenant execution.";
/** Provider key (== plugin driverKey) of the first-party Kubernetes sandbox provider. */
const KUBERNETES_PROVIDER_KEY = "kubernetes";
/** Metadata marker for the company's managed-by-config Kubernetes sandbox environment. */
const KUBERNETES_MANAGED_MARKER = "managedKubernetesSandbox";
const ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES = ["starting", "waiting_for_user", "capturing"] as const;
/**
* Configuration accepted by `ensureKubernetesEnvironment`. Mirrors the keys of
* the kubernetes sandbox-provider `configSchema` that an operator typically
* pins for a hosted cloud instance. Stored verbatim in `environment.config`
* (the plugin validates/defaults it via `kubernetesProviderConfigSchema` at
* lease time); `provider` is always forced to "kubernetes".
*/
export interface KubernetesEnvironmentConfigInput {
backend?: "sandbox-cr" | "job";
inCluster?: boolean;
runtimeClassName?: string;
egressMode?: "cilium" | "standard";
egressAllowFqdns?: string[];
egressAllowCidrs?: string[];
namespacePrefix?: string;
imageRegistry?: string;
adapterType?: string;
/**
* Sandbox lease RPC timeout in milliseconds. Read at lease time by
* `resolvePluginSandboxRpcTimeoutMs` to extend the worker-manager call
* timeout when acquiring a lease may take minutes (e.g. a cold node
* scale-up on an autoscale-to-zero pool). Stored verbatim in the
* environment config and validated by the sandbox config schema.
*/
timeoutMs?: number;
adapters?: import("@paperclipai/shared").AdapterRegistryEntry[];
[key: string]: unknown;
}
/**
* Input to `ensureManagedSandboxEnvironment`. Provider-agnostic: `provider`
* is the sandbox plugin's driver key and is forced into `config.provider`;
* the rest of `config` is stored verbatim for the plugin to validate at
* lease time.
*/
export interface ManagedSandboxEnvironmentInput {
/** Company whose managed-resource binding should be reconciled. Omit at instance boot to bind every company. */
companyId?: string;
name: string;
description?: string;
/** Sandbox provider key (the plugin's driverKey, e.g. "kubernetes", "daytona"). */
provider: string;
config?: Record<string, unknown>;
/**
* Extra metadata markers stamped on the managed row (e.g. the legacy
* kubernetes marker `managedKubernetesSandbox` that
* `findKubernetesEnvironment` keys on).
*/
extraMetadata?: Record<string, unknown>;
/** Version label recorded with the stock binding; hashes remain the drift authority. */
stockVersion?: string;
/**
* Asserts the caller's deployment gives no operator any path to hand-edit
* this row (currently only true for the PAPERCLIP_MANAGED_CONFIG applier,
* where `enableManagedSandboxOnly` removes the tenant's own environment
* choice entirely). When set, a plain content-hash mismatch against a real
* prior binding is treated as ordinary stock drift instead of an operator
* customization to protect — see the `operator_modified` handling below.
* Leave unset for any caller (self-hosted `kubernetes-execution-mode`
* bootstrap, tests, admin routes) where an operator could realistically
* have edited the row through the normal environments UI/API.
*/
platformFullyManaged?: boolean;
}
export type ManagedSandboxEnvironmentReconcileAction =
| "added"
| "updated"
| "unchanged"
| "skipped";
export interface ManagedSandboxEnvironmentReconcileResult {
environment: Environment;
action: ManagedSandboxEnvironmentReconcileAction;
/** Classification observed before this reconciliation wrote anything. */
stockStatus: ManagedResourceStockStatus;
/** True only when operator drift prevented the available stock update. */
updateAvailable: boolean;
stockHash: string;
}
function cloneRecord(value: unknown, fallback: Record<string, unknown> | null = null): Record<string, unknown> | null {
if (!value || typeof value !== "object" || Array.isArray(value)) return fallback;
return { ...(value as Record<string, unknown>) };
}
function readEnum<T extends string>(value: string | null, allowed: readonly T[], fieldName: string): T | null {
if (value === null) return null;
if ((allowed as readonly string[]).includes(value)) return value as T;
throw new Error(`Unexpected ${fieldName} value: ${value}`);
}
function hasConstraintName(error: unknown, constraintName: string): boolean {
if (typeof error !== "object" || error === null) return false;
const candidate = error as {
constraint?: unknown;
constraint_name?: unknown;
cause?: unknown;
};
return candidate.constraint === constraintName
|| candidate.constraint_name === constraintName
|| hasConstraintName(candidate.cause, constraintName);
}
function toEnvironment(row: EnvironmentRow): Environment {
return {
id: row.id,
name: row.name,
description: row.description ?? null,
driver: readEnum(row.driver, ENVIRONMENT_DRIVERS, "environment driver") ?? "local",
status: readEnum(row.status, ENVIRONMENT_STATUSES, "environment status") ?? "active",
config: cloneRecord(row.config, {}) ?? {},
envVars: cloneRecord(row.envVars, {}) ?? {},
metadata: cloneRecord(row.metadata),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
} as Environment;
}
type EnvironmentListFilters = {
status?: string;
driver?: string;
};
function resolveListFilters(
companyIdOrFilters?: string | EnvironmentListFilters,
maybeFilters?: EnvironmentListFilters,
): EnvironmentListFilters {
if (typeof companyIdOrFilters === "string") {
return maybeFilters ?? {};
}
return companyIdOrFilters ?? {};
}
function resolveCreateInput(
companyIdOrInput: string | CreateEnvironment,
maybeInput?: CreateEnvironment,
): CreateEnvironment {
if (typeof companyIdOrInput === "string") {
if (!maybeInput) throw new Error("Create environment input is required");
return maybeInput;
}
return companyIdOrInput;
}
function resolveKubernetesConfig(
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
maybeConfig?: KubernetesEnvironmentConfigInput,
): KubernetesEnvironmentConfigInput {
if (typeof companyIdOrConfig === "string") {
if (!maybeConfig) throw new Error("Kubernetes environment config is required");
return maybeConfig;
}
return companyIdOrConfig;
}
function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
return {
id: row.id,
companyId: row.companyId,
environmentId: row.environmentId,
executionWorkspaceId: row.executionWorkspaceId ?? null,
issueId: row.issueId ?? null,
heartbeatRunId: row.heartbeatRunId ?? null,
status: readEnum(row.status, ENVIRONMENT_LEASE_STATUSES, "environment lease status") ?? "active",
leasePolicy: readEnum(row.leasePolicy, ENVIRONMENT_LEASE_POLICIES, "environment lease policy") ?? "ephemeral",
provider: row.provider ?? null,
providerLeaseId: row.providerLeaseId ?? null,
acquiredAt: row.acquiredAt,
lastUsedAt: row.lastUsedAt,
expiresAt: row.expiresAt ?? null,
releasedAt: row.releasedAt ?? null,
failureReason: row.failureReason ?? null,
cleanupStatus: readEnum(
row.cleanupStatus,
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
"environment lease cleanup status",
),
metadata: cloneRecord(row.metadata),
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
}
function countFromRows(rows: Array<{ count: number | string | null | undefined }>): number {
return Number(rows[0]?.count ?? 0);
}
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
type EnvironmentWriteDb = Pick<Db | DbTransaction, "select" | "insert" | "update" | "delete">;
const MANAGED_ENVIRONMENT_BUNDLE_KEY = "managed-sandbox-environment";
const MANAGED_ENVIRONMENT_RESOURCE_KIND = "environment";
const MANAGED_ENVIRONMENT_RESOURCE_KEY = "managed-sandbox";
const MANAGED_ENVIRONMENT_STOCK_VERSION = "managed-environment-v1";
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY = "_paperclipManagedArchiveToken";
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY = "_paperclipManagedArchiveToken";
function managedEnvironmentBaselineDefaults(
defaultsJson: Record<string, unknown>,
): Record<string, unknown> {
const baseline = { ...defaultsJson };
delete baseline[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
return baseline;
}
function withoutManagedEnvironmentArchiveToken(
metadata: Record<string, unknown> | null,
): Record<string, unknown> | null {
if (!metadata) return null;
const next = { ...metadata };
delete next[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
return next;
}
function managedMetadataKeys(
desiredMetadata: Record<string, unknown>,
bindings: Array<{ defaultsJson: Record<string, unknown> }>,
): string[] {
const keys = new Set([
"managedByPaperclip",
"managedSandboxProvider",
KUBERNETES_MANAGED_MARKER,
...Object.keys(desiredMetadata),
]);
for (const binding of bindings) {
const metadata = cloneRecord(binding.defaultsJson.metadata);
for (const key of Object.keys(metadata ?? {})) keys.add(key);
}
return [...keys].sort((left, right) => left.localeCompare(right));
}
function managedEnvironmentStock(input: {
name: string;
description: string | null;
config: Record<string, unknown>;
metadata: Record<string, unknown> | null;
status: string;
}, metadataKeys: readonly string[]): Record<string, unknown> {
const metadata = input.metadata ?? {};
return {
name: input.name,
description: input.description,
config: input.config,
metadata: Object.fromEntries(
metadataKeys.map((key) => [key, Object.prototype.hasOwnProperty.call(metadata, key) ? metadata[key] : null]),
),
status: input.status,
};
}
function mergeManagedEnvironmentMetadata(
current: Record<string, unknown> | null,
desired: Record<string, unknown>,
keys: readonly string[],
): Record<string, unknown> {
const merged = { ...(current ?? {}) };
for (const key of keys) {
const value = Object.prototype.hasOwnProperty.call(desired, key) ? desired[key] : null;
if (value === null || value === undefined) delete merged[key];
else merged[key] = value;
}
return merged;
}
function preserveInvalidatedSnapshotProofs(replacement: SQL): SQL {
const proofs = sql`(select coalesce(jsonb_object_agg(entry.key, entry.value || '{"invalidated":true}'::jsonb), '{}'::jsonb)
from jsonb_each(coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)) as entry
where entry.value->>'v' = '2')`;
return sql`case when ${proofs} = '{}'::jsonb then ${replacement}
else coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${proofs}) end`;
}
export function environmentService(db: Db) {
/**
* Idempotently ensure THE Paperclip-managed sandbox environment for this
* instance, configured for an arbitrary sandbox provider plugin. Mirrors
* `ensureLocalEnvironment`; the partial unique index
* `environments_managed_sandbox_idx` enforces at most one managed sandbox
* row per instance, so this function owns that single slot regardless of
* provider:
*
* - A stock-controlled managed row advances to a new stock hash in the same
* transaction as its managed fields, including provider switches.
* - A stock-current row is returned without an environment write.
* - An operator-modified or previously unmanaged row is preserved and
* reported as skipped; its user-owned fields are never folded into stock.
*/
const ensureManagedSandboxEnvironment = async (
input: ManagedSandboxEnvironmentInput,
): Promise<ManagedSandboxEnvironmentReconcileResult> => {
const desiredConfig: Record<string, unknown> = {
...(input.config ?? {}),
provider: input.provider,
};
const desiredMetadata: Record<string, unknown> = {
managedByPaperclip: true,
managedSandboxProvider: input.provider,
...(input.extraMetadata ?? {}),
};
if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) {
desiredMetadata[KUBERNETES_MANAGED_MARKER] = null;
}
let activityCompanyIds: string[] = [];
let trackingInitialized = false;
let providerReactivated = false;
const reconciliation = await db.transaction(
async (tx): Promise<ManagedSandboxEnvironmentReconcileResult> => {
const companyIds = input.companyId
? [input.companyId]
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
activityCompanyIds = companyIds;
// Take the sandbox-row lock BEFORE reading the stock bindings. Two
// passes can reconcile the same slot concurrently (the boot ensure and
// the async provider-recovery reactivation, or two app builds during a
// rolling deploy). Concurrent passes serialize on this lock, and under
// READ COMMITTED each later statement sees a fresh snapshot — so a pass
// that blocks here then reads the bindings the winning pass committed,
// not a snapshot from before it waited. Reading bindings first left a
// window where a waiting pass compared the winner's fresh row against
// its own stale binding hash, misclassified the mismatch as
// `operator_modified`, and — once `platformFullyManaged` turns that
// into an update — rolled the row back to its own older stock.
const sandboxRows = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update");
const bindingConditions = and(
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
...(companyIds.length > 0 ? [inArray(builtInManagedResources.companyId, companyIds)] : []),
);
const bindings = companyIds.length > 0
? await tx.select().from(builtInManagedResources).where(bindingConditions)
: [];
trackingInitialized = bindings.length < companyIds.length;
const keys = managedMetadataKeys(desiredMetadata, bindings);
let row = sandboxRows.find(
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? sandboxRows.find((candidate) => candidate.name === input.name) ?? null;
const writeBindings = async (
environmentId: string,
stockVersion: string,
installedStockHash: string,
defaultsJson: Record<string, unknown>,
replace: boolean,
) => {
const targetCompanyIds = replace
? companyIds
: companyIds.filter(
(companyId) => !bindings.some((binding) => binding.companyId === companyId),
);
if (targetCompanyIds.length === 0) return;
const values = targetCompanyIds.map((companyId) => ({
companyId,
bundleKey: MANAGED_ENVIRONMENT_BUNDLE_KEY,
resourceKind: MANAGED_ENVIRONMENT_RESOURCE_KIND,
resourceKey: MANAGED_ENVIRONMENT_RESOURCE_KEY,
resourceId: environmentId,
stockVersion,
stockHash: installedStockHash,
defaultsJson,
}));
const insert = tx.insert(builtInManagedResources).values(values);
if (!replace) {
await insert.onConflictDoNothing({
target: [
builtInManagedResources.companyId,
builtInManagedResources.bundleKey,
builtInManagedResources.resourceKind,
builtInManagedResources.resourceKey,
],
});
return;
}
await insert.onConflictDoUpdate({
target: [
builtInManagedResources.companyId,
builtInManagedResources.bundleKey,
builtInManagedResources.resourceKind,
builtInManagedResources.resourceKey,
],
set: {
resourceId: environmentId,
stockVersion,
stockHash: installedStockHash,
defaultsJson,
updatedAt: new Date(),
},
});
};
if (!row) {
const nameOwner = await tx
.select()
.from(environments)
.where(eq(environments.name, input.name))
.then((rows) => rows[0] ?? null);
if (nameOwner && nameOwner.driver !== "sandbox") {
throw new Error(
`Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`,
);
}
const now = new Date();
const inserted = await tx
.insert(environments)
.values({
name: input.name,
description: input.description ?? null,
driver: "sandbox",
status: "active",
config: desiredConfig,
envVars: {},
metadata: mergeManagedEnvironmentMetadata(null, desiredMetadata, keys),
createdAt: now,
updatedAt: now,
})
// Either the managed-slot partial index or the global name index
// can select the concurrent winner. Treat both as convergence and
// reselect that winner under lock below.
.onConflictDoNothing()
.returning()
.then((rows) => rows[0] ?? null);
if (inserted) {
const stock = managedEnvironmentStock({
name: inserted.name,
description: inserted.description ?? null,
config: inserted.config,
metadata: inserted.metadata,
status: inserted.status,
}, keys);
const latestStockHash = stockHash(stock);
await writeBindings(
inserted.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
stock,
true,
);
return {
environment: toEnvironment(inserted),
action: "added",
stockStatus: "missing",
updateAvailable: false,
stockHash: latestStockHash,
};
}
row = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update")
.then(
(rows) => rows.find(
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? null,
);
if (!row) throw new Error("Failed to ensure managed sandbox environment");
}
const nameOwner = await tx
.select({ id: environments.id })
.from(environments)
.where(eq(environments.name, input.name))
.then((rows) => rows[0] ?? null);
const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name;
const desiredStock = managedEnvironmentStock({
name: desiredName,
description: input.description ?? null,
config: desiredConfig,
metadata: desiredMetadata,
status: "active",
}, keys);
const latestStockHash = stockHash(desiredStock);
const currentStock = managedEnvironmentStock({
name: row.name,
description: row.description ?? null,
config: row.config,
metadata: row.metadata,
status: row.status,
}, keys);
const currentHash = stockHash(currentStock);
const matchingBindings = bindings.filter((binding) => binding.resourceId === row!.id);
const rowArchiveToken = (row.metadata as Record<string, unknown> | null)
?.[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
let stockStatus = resourceStatus({
resourceId: row.id,
currentHash,
bindingStockHash: matchingBindings[0]?.stockHash ?? null,
latestStockHash,
});
if (stockStatus === "operator_modified") {
const stockControlledBinding = matchingBindings.find(
(binding) => resourceStatus({
resourceId: row!.id,
currentHash,
bindingStockHash: binding.stockHash,
latestStockHash,
}) === "stock_update_available",
);
if (stockControlledBinding) stockStatus = "stock_update_available";
}
const operatorReaffirmedArchive = row.status === "archived" && matchingBindings.some(
(binding) => {
const bindingToken = binding.defaultsJson[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
return binding.defaultsJson.status === "archived" &&
typeof bindingToken === "string" &&
bindingToken !== rowArchiveToken;
},
);
if (operatorReaffirmedArchive) stockStatus = "operator_modified";
// `platformFullyManaged` callers (currently: the PAPERCLIP_MANAGED_CONFIG
// applier) assert that nothing in their deployment can hand-edit this
// row — the product gives a cloud-harness tenant no path to it, unlike
// the general self-hosted contract this function otherwise protects
// (see "classifies operator drift" in environment-service.test.ts,
// which exercises a real operator edit and must keep winning). Under
// that assertion, a plain content-hash mismatch against a real prior
// binding can only be drift between two platform-driven reconciliation
// passes (e.g. a stock hash recorded by an older app build before a
// later stock field was added), never a customization to protect —
// apply it like any other stock-outdated row.
//
// Two things the bypass must never touch:
// - A row with NO matching binding. `row` can be a same-name sandbox
// row that was never Paperclip-managed (the fallback lookup above).
// It also reads as `operator_modified`, but there is no prior
// platform pass to have drifted from — adopting it would overwrite
// a tenant-created environment and stamp it managed. Require a
// binding for this exact row, so "prior binding" is enforced, not
// just documented.
// - Archive-reaffirmation. A `sandbox_image` update must never
// resurrect a row something else deliberately kept archived after
// Paperclip's own provider-unavailability archival, so that path
// still skips below regardless of this flag.
if (
input.platformFullyManaged &&
stockStatus === "operator_modified" &&
!operatorReaffirmedArchive &&
matchingBindings.length > 0
) {
stockStatus = "stock_update_available";
}
if (stockStatus === "operator_modified") {
const baseline = matchingBindings[0];
let baselineDefaults = baseline
? managedEnvironmentBaselineDefaults(baseline.defaultsJson)
: desiredStock;
let baselineHash = baseline?.stockHash ?? latestStockHash;
// Provider unavailability is an operational state transition, not
// an operator edit. If the binding records that Paperclip archived
// this row, restore only its availability status. Keep every other
// operator-modified field intact and leave the stock update pending.
// A manually archived row still has an active binding baseline, so
// it remains operator_modified and is not reactivated here.
const archivedByReconciler = row.status === "archived" &&
typeof rowArchiveToken === "string" &&
matchingBindings.some((binding) => {
const bindingDefaults = binding.defaultsJson;
return bindingDefaults.status === "archived" &&
bindingDefaults[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY] === rowArchiveToken;
});
if (archivedByReconciler) {
const reactivated = await tx
.update(environments)
.set({
status: "active",
metadata: withoutManagedEnvironmentArchiveToken(row.metadata),
updatedAt: new Date(),
})
.where(and(eq(environments.id, row.id), eq(environments.status, "archived")))
.returning()
.then((rows) => rows[0] ?? null);
if (!reactivated) {
throw new Error("Managed sandbox environment changed during reactivation");
}
row = reactivated;
providerReactivated = true;
for (const binding of matchingBindings) {
const reactivatedDefaults = {
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
status: "active",
};
const reactivatedHash = stockHash(reactivatedDefaults);
await tx
.update(builtInManagedResources)
.set({
stockHash: reactivatedHash,
defaultsJson: reactivatedDefaults,
updatedAt: new Date(),
})
.where(and(
eq(builtInManagedResources.id, binding.id),
eq(builtInManagedResources.resourceId, row.id),
));
if (binding.id === baseline?.id) {
baselineDefaults = reactivatedDefaults;
baselineHash = reactivatedHash;
}
}
}
await writeBindings(
row.id,
baseline?.stockVersion ?? input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
baselineHash,
baselineDefaults,
false,
);
return {
environment: toEnvironment(row),
action: "skipped",
stockStatus,
updateAvailable: true,
stockHash: latestStockHash,
};
}
if (stockStatus === "stock_current") {
await writeBindings(
row.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
desiredStock,
false,
);
return {
environment: toEnvironment(row),
action: "unchanged",
stockStatus,
updateAvailable: false,
stockHash: latestStockHash,
};
}
const updated = await tx
.update(environments)
.set({
name: desiredName,
description: input.description ?? null,
config: desiredConfig,
metadata: withoutManagedEnvironmentArchiveToken(
mergeManagedEnvironmentMetadata(row.metadata, desiredMetadata, keys),
),
status: "active",
updatedAt: new Date(),
})
.where(eq(environments.id, row.id))
.returning()
.then((rows) => rows[0] ?? null);
if (!updated) throw new Error("Managed sandbox environment changed during reconciliation");
await writeBindings(
updated.id,
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
latestStockHash,
desiredStock,
true,
);
return {
environment: toEnvironment(updated),
action: "updated",
stockStatus,
updateAvailable: false,
stockHash: latestStockHash,
};
},
);
if (reconciliation.action !== "unchanged" || trackingInitialized) {
const action = reconciliation.action === "added"
? "environment.managed_stock_added"
: reconciliation.action === "updated"
? "environment.managed_stock_updated"
: reconciliation.action === "skipped"
? "environment.managed_stock_skipped"
: "environment.managed_stock_tracking_initialized";
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
companyId,
actorType: "system",
actorId: "managed-environment-reconciler",
action,
entityType: "environment",
entityId: reconciliation.environment.id,
details: {
provider: input.provider,
reconciliationAction: reconciliation.action,
stockStatus: reconciliation.stockStatus,
updateAvailable: reconciliation.updateAvailable,
stockHash: reconciliation.stockHash,
providerReactivated,
},
})));
}
return reconciliation;
};
/**
* Archive the Paperclip-managed sandbox row when its provider became
* unavailable (plugin missing, not ready, or its worker not running), so
* run scheduling stops selecting an environment whose lease acquisition
* cannot succeed (`resolveEnvironment` rejects non-active rows).
*
* Scoped to the row provisioned for the SAME provider: a row that a
* provider switch left on a different provider is not touched (the ensure
* path adopts it once the new provider is healthy). Reactivation is
* automatic — the next successful `ensureManagedSandboxEnvironment` stamps
* the row `active` again.
*
* Returns the archived environment, or null when there is no active
* managed row for this provider.
*/
const archiveManagedSandboxEnvironment = async (
input: { provider: string; companyId?: string },
): Promise<Environment | null> => {
let activityCompanyIds: string[] = [];
const archived = await db.transaction(async (tx) => {
const existing = await tx
.select()
.from(environments)
.where(eq(environments.driver, "sandbox"))
.for("update")
.then(
(rows) => rows.find(
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
) ?? null,
);
if (!existing || existing.status !== "active") return null;
const rowProvider = (existing.metadata as Record<string, unknown> | null)
?.managedSandboxProvider;
if (rowProvider !== input.provider) return null;
const companyIds = input.companyId
? [input.companyId]
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
activityCompanyIds = companyIds;
const bindings = companyIds.length > 0
? await tx
.select()
.from(builtInManagedResources)
.where(and(
inArray(builtInManagedResources.companyId, companyIds),
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
eq(builtInManagedResources.resourceId, existing.id),
))
: [];
const archiveToken = randomUUID();
const archivedMetadata = {
...(existing.metadata ?? {}),
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]: archiveToken,
};
const archived = await tx
.update(environments)
.set({
status: "archived",
metadata: archivedMetadata,
updatedAt: new Date(),
})
.where(and(eq(environments.id, existing.id), eq(environments.status, "active")))
.returning()
.then((rows) => rows[0] ?? null);
if (!archived) return null;
// Archival is a Paperclip-owned availability transition. Record only
// that status change in each installed baseline. Deriving the new hash
// from defaultsJson keeps operator-modified row fields out of stock.
for (const binding of bindings) {
const archivedStock = {
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
status: "archived",
};
await tx
.update(builtInManagedResources)
.set({
stockHash: stockHash(archivedStock),
defaultsJson: {
...archivedStock,
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]: archiveToken,
},
updatedAt: new Date(),
})
.where(and(
eq(builtInManagedResources.id, binding.id),
eq(builtInManagedResources.resourceId, existing.id),
));
}
return toEnvironment(archived);
});
if (archived) {
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
companyId,
actorType: "system",
actorId: "managed-environment-reconciler",
action: "environment.managed_provider_unavailable_archived",
entityType: "environment",
entityId: archived.id,
details: { provider: input.provider },
})));
}
return archived;
};
return {
recordSandboxPlatformProbe: async (input: {
environmentId: string;
companyId: string;
config: Record<string, unknown>;
proof: Record<string, unknown> | null;
}): Promise<boolean> => {
const existing = sql`coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)`;
const previous = sql`${existing}->${input.companyId}`;
const probes = input.proof === null
? sql`case when ${previous}->>'v' = '2'
then jsonb_set(${existing}, ARRAY[${input.companyId}]::text[], ${previous} || '{"invalidated":true}'::jsonb, true)
else ${existing} - ${input.companyId} end`
: sql`${existing} || ${JSON.stringify({ [input.companyId]: input.proof })}::jsonb`;
const rows = await db.update(environments).set({
metadata: sql`jsonb_set(coalesce(${environments.metadata}, '{}'::jsonb), ARRAY[${SANDBOX_PLATFORM_PROBE_METADATA_KEY}]::text[], ${probes}, true)`,
}).where(and(eq(environments.id, input.environmentId), eq(environments.driver, "sandbox"), eq(environments.config, input.config))).returning({ id: environments.id });
return rows.length > 0;
},
list: async (
companyIdOrFilters?: string | EnvironmentListFilters,
maybeFilters?: EnvironmentListFilters,
): Promise<Environment[]> => {
const filters = resolveListFilters(companyIdOrFilters, maybeFilters);
const conditions = [];
if (filters.status) conditions.push(eq(environments.status, filters.status));
if (filters.driver) conditions.push(eq(environments.driver, filters.driver));
const rows = await db
.select()
.from(environments)
.where(conditions.length > 0 ? and(...conditions) : undefined)
.orderBy(desc(environments.updatedAt), desc(environments.createdAt));
return rows.map(toEnvironment);
},
getById: async (id: string): Promise<Environment | null> => {
const row = await db.select().from(environments).where(eq(environments.id, id)).then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
/**
* List the companies that own an environment through a built-in managed
* resource binding. A managed sandbox row binds to each company that the
* instance provisions it for. An operator-created environment has no
* binding, so this returns an empty list. The caller treats an empty list
* as an instance-global environment with no company owner.
*/
listBoundCompanyIds: async (environmentId: string): Promise<string[]> => {
const rows = await db
.select({ companyId: builtInManagedResources.companyId })
.from(builtInManagedResources)
.where(
and(
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
eq(builtInManagedResources.resourceId, environmentId),
),
);
return Array.from(new Set(rows.map((row) => row.companyId)));
},
getLeaseById: async (id: string): Promise<EnvironmentLease | null> => {
const row = await db
.select()
.from(environmentLeases)
.where(eq(environmentLeases.id, id))
.then((rows) => rows[0] ?? null);
return row ? toEnvironmentLease(row) : null;
},
/**
* Idempotently ensure THE local-driver environment row; the partial
* unique index `environments_local_driver_idx` enforces at most one per
* instance.
*
* On a cloud-managed instance an existing row is additionally ADOPTED —
* stamped `managedByPaperclip: true` (other metadata preserved) — so the
* single local slot is platform-owned there by construction, mirroring
* `ensureManagedSandboxEnvironment`'s adoption of the sandbox slot. This
* is what lets the environment-routes write floor treat a local row's
* platform markers as live state rather than a stale leftover: every
* caller (company creation, the heartbeat, run orchestration) converges
* the marker. Self-hosted instances keep the historical behavior:
* an existing row is returned untouched.
*/
ensureLocalEnvironment: async (_companyId?: string): Promise<Environment> => {
const now = new Date();
const insert = () =>
db
.insert(environments)
.values({
name: DEFAULT_LOCAL_ENVIRONMENT_NAME,
description: DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION,
driver: "local",
status: "active",
config: {},
envVars: {},
metadata: {
managedByPaperclip: true,
defaultForInstance: true,
},
createdAt: now,
updatedAt: now,
})
.onConflictDoNothing({
target: [environments.driver],
where: sql`${environments.driver} = 'local'`,
})
.returning()
.then((rows) => rows[0] ?? null);
const row = await insert().catch((error: unknown) => {
if (hasConstraintName(error, "environments_name_idx")) {
return null;
}
throw error;
});
if (row) return toEnvironment(row);
const existing = await db
.select()
.from(environments)
.where(eq(environments.driver, "local"))
.then((rows) => rows[0] ?? null);
if (!existing) {
throw new Error("Failed to ensure local environment");
}
const existingMetadata = (existing.metadata ?? {}) as Record<string, unknown>;
if (isCloudManagedInstance() && existingMetadata.managedByPaperclip !== true) {
const adopted = await db
.update(environments)
.set({
metadata: { ...existingMetadata, managedByPaperclip: true },
updatedAt: new Date(),
})
.where(eq(environments.id, existing.id))
.returning()
.then((rows) => rows[0] ?? existing);
return toEnvironment(adopted);
}
return toEnvironment(existing);
},
ensureManagedSandboxEnvironment,
archiveManagedSandboxEnvironment,
/**
* Idempotently ensure a managed Kubernetes sandbox environment exists for
* an instance, configured from instance/operator-supplied config. A thin
* wrapper over `ensureManagedSandboxEnvironment` that pins the provider to
* "kubernetes" and stamps the legacy marker `findKubernetesEnvironment`
* keys on. On subsequent calls stock-controlled config advances in place;
* operator modifications remain untouched for explicit review.
*/
ensureKubernetesEnvironment: async (
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
maybeConfig?: KubernetesEnvironmentConfigInput,
): Promise<Environment> => {
const config = resolveKubernetesConfig(companyIdOrConfig, maybeConfig);
return ensureManagedSandboxEnvironment({
companyId: typeof companyIdOrConfig === "string" ? companyIdOrConfig : undefined,
name: DEFAULT_KUBERNETES_ENVIRONMENT_NAME,
description: DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION,
provider: KUBERNETES_PROVIDER_KEY,
config,
extraMetadata: { [KUBERNETES_MANAGED_MARKER]: true },
}).then((result) => result.environment);
},
/**
* Find the active managed Kubernetes sandbox environment, if one
* exists. Read-only counterpart to `ensureKubernetesEnvironment` used by the
* per-run execution guard (which must not silently create config-less envs).
*/
findKubernetesEnvironment: async (_companyId?: string): Promise<Environment | null> => {
const rows = await db
.select()
.from(environments)
.where(
and(
eq(environments.driver, "sandbox"),
eq(environments.status, "active"),
),
)
.orderBy(desc(environments.updatedAt));
const match = rows.find(
(row) =>
(row.metadata as Record<string, unknown> | null)?.[KUBERNETES_MANAGED_MARKER] === true,
);
return match ? toEnvironment(match) : null;
},
/**
* Find the platform-managed sandbox environment (the single
* `managedByPaperclip`-marked slot row), if one exists. Read-only
* counterpart to `ensureManagedSandboxEnvironment`. The default
* (active-only) form serves the managed-sandbox-only run guard — which
* must fail closed rather than create a config-less environment when
* the slot is empty or archived (the provisioner archives it while its
* provider plugin is down). `includeArchived` serves reconciliation
* cleanup, which must find the row even after the provisioner archived
* it.
*/
findManagedSandboxEnvironment: async (
_companyId?: string,
options?: { includeArchived?: boolean },
): Promise<Environment | null> => {
const rows = await db
.select()
.from(environments)
.where(
options?.includeArchived === true
? eq(environments.driver, "sandbox")
: and(
eq(environments.driver, "sandbox"),
eq(environments.status, "active"),
),
)
.orderBy(desc(environments.updatedAt));
const match = rows.find(
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
);
return match ? toEnvironment(match) : null;
},
create: async (
companyIdOrInput: string | CreateEnvironment,
maybeInput?: CreateEnvironment,
options?: { db?: EnvironmentWriteDb },
): Promise<Environment> => {
const input = resolveCreateInput(companyIdOrInput, maybeInput);
const now = new Date();
const row = await (options?.db ?? db)
.insert(environments)
.values({
name: input.name,
description: input.description ?? null,
driver: input.driver,
status: input.status ?? "active",
config: input.config ?? {},
envVars: (input as CreateEnvironment & { envVars?: Record<string, unknown> }).envVars ?? {},
metadata: input.metadata ?? null,
createdAt: now,
updatedAt: now,
})
.returning()
.then((rows) => rows[0] ?? null)
.catch((error) => {
if (hasConstraintName(error, "environments_name_idx")) {
throw conflict(`An environment named "${input.name}" already exists for this instance.`);
}
if (hasConstraintName(error, "environments_local_driver_idx")) {
throw conflict("A local environment already exists for this instance.");
}
throw error;
});
if (!row) {
throw new Error("Failed to create environment");
}
return toEnvironment(row);
},
update: async (
id: string,
patch: UpdateEnvironment,
options?: { db?: EnvironmentWriteDb },
): Promise<Environment | null> => {
const writeDb = options?.db ?? db;
const values: Partial<typeof environments.$inferInsert> = {
updatedAt: new Date(),
};
if (patch.name !== undefined) values.name = patch.name;
if (patch.description !== undefined) values.description = patch.description ?? null;
if (patch.driver !== undefined) values.driver = patch.driver;
if (patch.status !== undefined) values.status = patch.status;
if (patch.config !== undefined) values.config = patch.config;
if ("envVars" in patch && patch.envVars !== undefined) {
values.envVars = (patch.envVars ?? {}) as Record<string, unknown>;
}
const changesTarget = patch.config !== undefined || patch.driver !== undefined;
let metadataUpdate: SQL | undefined;
if (patch.metadata !== undefined) {
const userMetadata = withoutManagedEnvironmentArchiveToken(patch.metadata ?? null);
if (userMetadata) delete userMetadata[SANDBOX_PLATFORM_PROBE_METADATA_KEY];
if (changesTarget) {
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
metadataUpdate = preserveInvalidatedSnapshotProofs(replacement);
} else {
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
metadataUpdate = sql`case when ${environments.metadata} ? ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}
then coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(
${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY})
else ${replacement} end`;
}
} else if (changesTarget || patch.status !== undefined) {
metadataUpdate = sql`${environments.metadata} - ${MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY}`;
if (changesTarget) metadataUpdate = preserveInvalidatedSnapshotProofs(sql`${metadataUpdate} - ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}`);
}
const row = await writeDb
.update(environments)
.set({ ...values, ...(metadataUpdate ? { metadata: metadataUpdate } : {}) })
.where(eq(environments.id, id))
.returning()
.then((rows) => rows[0] ?? null)
.catch((error) => {
if (hasConstraintName(error, "environments_name_idx")) {
throw conflict(`An environment named "${patch.name}" already exists for this instance.`);
}
if (hasConstraintName(error, "environments_local_driver_idx")) {
throw conflict("A local environment already exists for this instance.");
}
throw error;
});
return row ? toEnvironment(row) : null;
},
remove: async (id: string): Promise<Environment | null> => {
const row = await db
.delete(environments)
.where(eq(environments.id, id))
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
removeIfDeletable: async (id: string): Promise<Environment | null> => {
const row = await db
.delete(environments)
.where(
and(
eq(environments.id, id),
ne(environments.driver, "local"),
sql`not exists (
select 1 from ${instanceSettings}
where ${instanceSettings.defaultEnvironmentId} = ${environments.id}
)`,
// A `pending_cleanup` lease is the durable teardown reference for an
// orphan sandbox. The environment foreign key uses
// `on delete set null`, so a delete keeps the lease row but drops its
// environment reference. This predicate refuses the delete while such
// a lease exists, so the operator resolves the cleanup first and the
// lease keeps its environment link. It runs in the same statement as
// the delete, so it also closes the check-to-delete race.
sql`not exists (
select 1 from ${environmentLeases}
where ${environmentLeases.environmentId} = ${environments.id}
and ${environmentLeases.status} = 'pending_cleanup'
)`,
// A reusable lease keeps a provider sandbox after a run releases
// it, including when the run failed but release succeeded.
// Deleting the environment would set its reference to
// null, and both the normal release path and scoped reusable cleanup
// require that environment context. Refuse the delete atomically
// until the owning issue/workspace destroys the reusable sandbox.
sql`not exists (
select 1 from ${environmentLeases}
where ${environmentLeases.environmentId} = ${environments.id}
and ${environmentLeases.leasePolicy} = 'reuse_by_environment'
and ${environmentLeases.status} in ('active', 'released', 'retained', 'failed')
)`,
),
)
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironment(row) : null;
},
/**
* Return true when the environment has one or more leases in the terminal
* `pending_cleanup` state. Each such lease is the only durable provider
* reference for an orphan sandbox that a teardown retry must destroy. The
* delete guard and the provider-change guard call this to protect that
* reference.
*/
hasUnresolvedPendingCleanupLeases: async (environmentId: string): Promise<boolean> => {
const rows = await db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, environmentId),
eq(environmentLeases.status, "pending_cleanup"),
),
);
return countFromRows(rows) > 0;
},
getDeleteBlastRadius: async (id: string): Promise<EnvironmentDeleteBlastRadius | null> => {
const environment = await db
.select({
id: environments.id,
driver: environments.driver,
})
.from(environments)
.where(eq(environments.id, id))
.then((rows) => rows[0] ?? null);
if (!environment) return null;
const [
instanceDefaultRows,
agentDefaultRows,
executionWorkspaceRows,
issueRows,
projectRows,
secretBindingRows,
activeLeaseRows,
pendingCleanupLeaseRows,
reusableSandboxLeaseRows,
activeSetupRows,
] = await Promise.all([
db
.select({ count: sql<number>`count(*)::int` })
.from(instanceSettings)
.where(eq(instanceSettings.defaultEnvironmentId, id)),
db
.select({ count: sql<number>`count(*)::int` })
.from(agents)
.where(eq(agents.defaultEnvironmentId, id)),
db
.select({ count: sql<number>`count(*)::int` })
.from(executionWorkspaces)
.where(sql`${executionWorkspaces.metadata} -> 'config' ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(issues)
.where(sql`${issues.executionWorkspaceSettings} ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(projects)
.where(sql`${projects.executionWorkspacePolicy} ->> 'environmentId' = ${id}`),
db
.select({ count: sql<number>`count(*)::int` })
.from(companySecretBindings)
.where(
and(
eq(companySecretBindings.targetType, "environment"),
eq(companySecretBindings.targetId, id),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.status, "active"),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentLeases)
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.status, "pending_cleanup"),
),
),
db
.select({
leaseId: environmentLeases.id,
executionWorkspaceId: environmentLeases.executionWorkspaceId,
executionWorkspaceName: executionWorkspaces.name,
issueId: environmentLeases.issueId,
issueIdentifier: issues.identifier,
issueTitle: issues.title,
})
.from(environmentLeases)
.leftJoin(executionWorkspaces, eq(environmentLeases.executionWorkspaceId, executionWorkspaces.id))
.leftJoin(issues, eq(environmentLeases.issueId, issues.id))
.where(
and(
eq(environmentLeases.environmentId, id),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
),
),
db
.select({ count: sql<number>`count(*)::int` })
.from(environmentCustomImageSetupSessions)
.where(
and(
eq(environmentCustomImageSetupSessions.environmentId, id),
inArray(environmentCustomImageSetupSessions.status, [...ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES]),
),
),
]);
const isManagedLocal = environment.driver === "local";
const isInstanceDefault = countFromRows(instanceDefaultRows) > 0;
const pendingCleanupLeaseCount = countFromRows(pendingCleanupLeaseRows);
const reusableSandboxLeaseHolders = reusableSandboxLeaseRows.map((row) => ({
leaseId: row.leaseId,
executionWorkspaceId: row.executionWorkspaceId,
executionWorkspaceName: row.executionWorkspaceName,
issueId: row.issueId,
issueIdentifier: row.issueIdentifier,
issueTitle: row.issueTitle,
}));
const reusableSandboxLeaseCount = reusableSandboxLeaseHolders.length;
const deleteBlockedReasons: EnvironmentDeleteBlockedReason[] = [];
if (isManagedLocal) deleteBlockedReasons.push("managed_local");
if (isInstanceDefault) deleteBlockedReasons.push("instance_default");
// A `pending_cleanup` lease is the durable teardown reference for an orphan
// sandbox. The environment foreign key uses `on delete set null`, so a
// delete keeps the lease but drops its environment reference. Block the
// delete until the sweep resolves the lease, so the operator resolves the
// cleanup first and the lease keeps its environment link.
if (pendingCleanupLeaseCount > 0) deleteBlockedReasons.push("pending_sandbox_cleanup");
if (reusableSandboxLeaseCount > 0) deleteBlockedReasons.push("reusable_sandbox_lease");
const activeLeaseCount = countFromRows(activeLeaseRows);
const activeCustomImageSetupSessionCount = countFromRows(activeSetupRows);
return {
environmentId: id,
canDelete: deleteBlockedReasons.length === 0,
deleteBlockedReasons,
pendingCleanupLeaseCount,
reusableSandboxLeaseCount,
reusableSandboxLeaseHolders,
staticReferences: {
isManagedLocal,
isInstanceDefault,
agentDefaultCount: countFromRows(agentDefaultRows),
executionWorkspaceSelectionCount: countFromRows(executionWorkspaceRows),
issueSelectionCount: countFromRows(issueRows),
projectSelectionCount: countFromRows(projectRows),
secretBindingCount: countFromRows(secretBindingRows),
},
activeRuntimeUse: {
activeLeaseCount,
activeCustomImageSetupSessionCount,
hasActiveRuntimeUse: activeLeaseCount > 0 || activeCustomImageSetupSessionCount > 0,
},
};
},
listLeases: async (
environmentId: string,
filters: {
status?: string;
} = {},
): Promise<EnvironmentLease[]> => {
const conditions = [eq(environmentLeases.environmentId, environmentId)];
if (filters.status) conditions.push(eq(environmentLeases.status, filters.status));
const rows = await db
.select()
.from(environmentLeases)
.where(and(...conditions))
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
return rows.map(toEnvironmentLease);
},
acquireLease: async (input: {
creationOperationId?: string;
companyId: string;
environmentId: string;
executionWorkspaceId?: string | null;
issueId?: string | null;
heartbeatRunId?: string | null;
leasePolicy?: EnvironmentLeasePolicy;
provider?: string | null;
providerLeaseId?: string | null;
expiresAt?: Date | null;
metadata?: Record<string, unknown> | null;
/**
* Atomically retire the previous database ownership record when this
* acquisition reuses the same provider resource for a new run. The
* provider resume happens before this write, so a failed transaction
* leaves the prior retained row recoverable instead of publishing two
* reusable owners for one sandbox.
*/
replacesReusableLeaseId?: string | null;
/**
* Reactivate the exact lease row already owned by this heartbeat run.
* Native same-run recovery re-enters environment startup after the
* provider process is interrupted; it must not manufacture a second
* database owner for the same run and provider resource.
*/
reusesReusableLeaseId?: string | null;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login routes set this to close the check-to-lease
* race: managed reconciliation can bind a sandbox to another company
* between the route guard and this acquire. When the environment is
* bound to a company other than `companyId`, the insert throws the 403
* `environment_company_mismatch` and no lease row is created (fail
* closed). An unbound (instance-global) environment stays open to every
* member. All other callers keep the plain, non-transactional insert.
*/
assertCompanyBinding?: boolean;
}): Promise<EnvironmentLease> => {
if (input.creationOperationId) {
const { creationOperationId, ...leaseInput } = input;
return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId },
tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true }));
}
const now = new Date();
const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null;
const values = {
...(replacementLeaseId ? { id: replacementLeaseId } : {}),
companyId: input.companyId,
environmentId: input.environmentId,
executionWorkspaceId: input.executionWorkspaceId ?? null,
issueId: input.issueId ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
status: "active" as const,
leasePolicy: input.leasePolicy ?? "ephemeral",
provider: input.provider ?? null,
providerLeaseId: input.providerLeaseId ?? null,
acquiredAt: now,
lastUsedAt: now,
expiresAt: input.expiresAt ?? null,
releasedAt: null,
failureReason: null,
cleanupStatus: null,
metadata: withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata)),
createdAt: now,
updatedAt: now,
};
if (
(input.replacesReusableLeaseId || input.reusesReusableLeaseId) &&
((!input.executionWorkspaceId && (!input.issueId || typeof input.metadata?.agentId !== "string")) || !input.providerLeaseId)
) {
throw new Error(
"A reusable lease handoff requires a workspace or agent-bound task, and a provider lease id.",
);
}
if (input.reusesReusableLeaseId && !input.heartbeatRunId) {
throw new Error(
"A same-run reusable lease reacquisition requires a heartbeat run id.",
);
}
if (input.replacesReusableLeaseId && input.reusesReusableLeaseId) {
throw new Error(
"A reusable lease cannot be replaced and reacquired in the same operation.",
);
}
const row =
input.assertCompanyBinding ||
input.replacesReusableLeaseId ||
input.reusesReusableLeaseId
? await db.transaction(async (tx) => {
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
companyId: input.companyId, runId: input.heartbeatRunId,
});
if (input.assertCompanyBinding) {
await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId);
}
if (input.replacesReusableLeaseId) {
await notifyChatInboundWakeNotices(tx, { companyId: input.companyId });
const retired = await tx
.update(environmentLeases)
.set({
status: "expired",
releasedAt: now,
lastUsedAt: now,
updatedAt: now,
cleanupStatus: "success",
})
.where(
and(
eq(environmentLeases.id, input.replacesReusableLeaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.environmentId, input.environmentId),
sql`${environmentLeases.provider} is not distinct from ${input.provider ?? null}`,
input.executionWorkspaceId
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
: and(
isNull(environmentLeases.executionWorkspaceId),
eq(environmentLeases.issueId, input.issueId!),
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
eq(
environmentLeases.providerLeaseId,
input.providerLeaseId!,
),
inArray(environmentLeases.status, [
"released",
"retained",
]),
),
)
.returning();
if (retired.length !== 1) {
throw conflict(
"Reusable sandbox lease ownership changed during acquisition.",
);
}
const handoff = createReusableLeaseHandoff(retired[0]!, { ...values, id: replacementLeaseId! });
values.metadata = { ...values.metadata, reusableLeasePredecessor: handoff };
await tx.update(environmentLeases).set({ metadata: {
...retired[0]!.metadata, reusableLeaseSuccessor: handoff,
} }).where(eq(environmentLeases.id, retired[0]!.id));
}
if (input.reusesReusableLeaseId) {
const [previous] = await tx.select({ metadata: environmentLeases.metadata }).from(environmentLeases)
.where(and(eq(environmentLeases.id, input.reusesReusableLeaseId), eq(environmentLeases.companyId, input.companyId))).for("update");
const reacquired = await tx
.update(environmentLeases)
.set({
status: "active",
releasedAt: null,
lastUsedAt: now,
expiresAt: input.expiresAt ?? null,
failureReason: null,
cleanupStatus: null,
metadata: preserveReusableLeaseHandoff(previous?.metadata ?? null, withoutCleanupVerification(input.metadata)),
updatedAt: now,
})
.where(
and(
eq(environmentLeases.id, input.reusesReusableLeaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.environmentId, input.environmentId),
input.executionWorkspaceId
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
: and(
isNull(environmentLeases.executionWorkspaceId),
eq(environmentLeases.issueId, input.issueId!),
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
),
eq(
environmentLeases.heartbeatRunId,
input.heartbeatRunId!,
),
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
eq(
environmentLeases.providerLeaseId,
input.providerLeaseId!,
),
inArray(environmentLeases.status, [
"active",
"released",
"retained",
]),
),
)
.returning()
.then((rows) => rows[0] ?? null);
if (!reacquired) {
throw conflict(
"Reusable sandbox lease ownership changed during reacquisition.",
);
}
return reacquired;
}
return tx
.insert(environmentLeases)
.values(values)
.returning()
.then((rows) => rows[0] ?? null);
})
: await db.transaction(async tx => {
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
companyId: input.companyId, runId: input.heartbeatRunId,
});
return tx.insert(environmentLeases).values(values).returning().then(rows => rows[0] ?? null);
});
if (!row) {
throw new Error("Failed to acquire environment lease");
}
return toEnvironmentLease(row);
},
releaseLease: async (
id: string,
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed" | "retained" | "pending_cleanup"> = "released",
options?: {
expectedPendingCleanupAttemptId?: string;
failureReason?: string;
cleanupStatus?: EnvironmentLeaseCleanupStatus;
remoteExecutionTermination?: Record<string, unknown>;
},
) => {
const now = new Date();
const releaseMetadata = options?.remoteExecutionTermination
? sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ remoteExecutionTermination: options.remoteExecutionTermination })}::jsonb`
: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'`;
const cleanupSettled = options?.expectedPendingCleanupAttemptId && options.cleanupStatus === "success"
&& status !== "pending_cleanup";
const row = await db.transaction(async tx => {
const condition = and(eq(environmentLeases.id, id), options?.expectedPendingCleanupAttemptId
? and(eq(environmentLeases.status, "pending_cleanup"),
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.expectedPendingCleanupAttemptId}`)
: undefined);
const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId,
heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status,
cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt,
creationOperationId: sql<string | null>`${environmentLeases.metadata}->>'creationOperationId'`,
}).from(environmentLeases).where(condition).for("update");
if (!previous) return null;
await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined });
if (previous.heartbeatRunId && (previous.status !== status ||
(options?.cleanupStatus !== undefined && previous.cleanupStatus !== options.cleanupStatus) ||
(status !== "retained" && !previous.releasedAt))) {
await notifyChatInboundWakeNotices(tx, {
companyId: previous.companyId, issueId: previous.issueId ?? undefined,
});
}
if (previous.heartbeatRunId && ["active", "pending_cleanup"].includes(previous.status)
&& status !== "pending_cleanup") {
await notifyRetainedNativeSessionCleanup(tx, {
companyId: previous.companyId, runId: previous.heartbeatRunId,
});
}
if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
.where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId),
eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending")));
}
return tx.update(environmentLeases)
.set({
status,
releasedAt: status === "retained" ? null : now,
lastUsedAt: now,
updatedAt: now,
...(options?.failureReason !== undefined ? { failureReason: options.failureReason } : {}),
...(options?.cleanupStatus !== undefined ? { cleanupStatus: options.cleanupStatus } : {}),
// A later release without a receipt cannot reuse an earlier stop's
// authority (for example after a same-run lease resume).
metadata: cleanupSettled
? sql`coalesce(${releaseMetadata}, '{}'::jsonb) || '{"pendingCleanupInFlight":false,"pendingCleanupLeaseExpiresAtMs":0}'::jsonb`
: releaseMetadata,
})
.where(condition)
.returning()
.then((rows) => rows[0] ?? null);
});
return row ? toEnvironmentLease(row) : null;
},
/**
* Record a lease-less orphan sandbox directly in the terminal
* `pending_cleanup` state with one atomic insert. The sweep reads a row with
* status `pending_cleanup` and cleanup status `failed`, so this insert makes
* the orphan visible to recovery immediately. It never passes through the
* `active` state, so a process or database crash cannot strand the row in an
* intermediate state that no sweep finds. The insert skips the company
* binding assertion, so it records the orphan even for a foreign-bound
* environment.
*
* The insert runs in a transaction that first locks the environment row with
* `for update`. The delete path (`removeIfDeletable`) refuses a delete while a
* `pending_cleanup` lease exists, so this lock serializes the two writes:
*
* - The environment still exists: the insert records the orphan with the
* environment reference. A concurrent delete blocks on the lock, then its
* `not exists (pending_cleanup)` predicate fails, so the delete refuses.
* - The environment is already gone (a delete won the race before this
* insert): the insert records the orphan with a null environment
* reference. The row carries the immutable provider metadata the sweep
* needs, so the teardown still runs. This closes the window the finding
* describes, where the foreign-key insert fails after a delete.
*/
insertPendingCleanupLease: async (input: {
companyId: string;
environmentId: string;
executionWorkspaceId?: string | null;
issueId?: string | null;
heartbeatRunId?: string | null;
provider?: string | null;
providerLeaseId?: string | null;
metadata?: Record<string, unknown> | null;
failureReason: string;
}): Promise<EnvironmentLease> => {
const now = new Date();
const row = await db.transaction(async (tx) => {
// Lock the environment row so a concurrent delete cannot commit between
// this read and the insert. An absent row means a delete already removed
// the environment, so record the orphan with a null reference.
const environmentRows = await tx
.select({ id: environments.id })
.from(environments)
.where(eq(environments.id, input.environmentId))
.for("update");
const environmentIdForRow = environmentRows[0]?.id ?? null;
await notifyEnvironmentCleanup(tx, { companyId: input.companyId, runId: input.heartbeatRunId ?? undefined });
return tx
.insert(environmentLeases)
.values({
companyId: input.companyId,
environmentId: environmentIdForRow,
executionWorkspaceId: input.executionWorkspaceId ?? null,
issueId: input.issueId ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
status: "pending_cleanup" as const,
leasePolicy: "ephemeral",
provider: input.provider ?? null,
providerLeaseId: input.providerLeaseId ?? null,
acquiredAt: now,
lastUsedAt: now,
expiresAt: null,
releasedAt: now,
failureReason: input.failureReason,
cleanupStatus: "failed",
metadata: withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata)),
createdAt: now,
updatedAt: now,
})
.returning()
.then((rows) => rows[0] ?? null);
});
if (!row) {
throw new Error("Failed to record pending sandbox cleanup lease");
}
return toEnvironmentLease(row);
},
updateLeaseMetadata: async (
id: string,
metadata: Record<string, unknown> | null,
): Promise<EnvironmentLease | null> => {
const row = await db.transaction(async tx => {
const [lease] = await tx.select({ companyId: environmentLeases.companyId, heartbeatRunId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata })
.from(environmentLeases).where(eq(environmentLeases.id, id)).for("update");
if (!lease) return null;
await notifyEnvironmentCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined });
return tx.update(environmentLeases)
.set({
metadata: preserveReusableLeaseHandoff(lease.metadata, withoutCleanupVerification(metadata)),
lastUsedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(environmentLeases.id, id))
.returning()
.then((rows) => rows[0] ?? null);
});
return row ? toEnvironmentLease(row) : null;
},
releaseLeasesForRun: async (
heartbeatRunId: string,
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
): Promise<EnvironmentLease[]> => {
const now = new Date();
const rows = await db.transaction(async tx => {
const [run] = await tx.select({ companyId: heartbeatRuns.companyId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
if (run) {
await notifyEnvironmentCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
}
return tx.update(environmentLeases)
.set({
status,
releasedAt: now,
lastUsedAt: now,
updatedAt: now,
})
.where(
and(
eq(environmentLeases.heartbeatRunId, heartbeatRunId),
eq(environmentLeases.status, "active"),
),
)
.returning();
});
return rows.map(toEnvironmentLease);
},
};
}