mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
## Thinking Path > - Paperclip manages companies and the agents that do their work. > - Remote execution creates provider resources that need a durable owner. > - The server could create a sandbox before it saved its lease. > - A failed lease write could leave cleanup data only on the old web server disk. > - This change saves a company-scoped operation before creation and transfers ownership atomically. > - Recovery finds the original Daytona resource or records an explicit manual reconciliation outcome. ## Linked Issues or Issue Description **What happened?** A sandbox could exist without a lease after a database failure or company rebind. Failed cleanup recording could then create a local fallback file. Replacing that web server could lose the cleanup reference. **Expected behavior** A failed admission write makes no provider call. An interrupted creation retains a committed operation. Recovery must not create a second sandbox or destroy an attached lease. **Steps to reproduce** 1. Run the disposable PostgreSQL cases in `server/src/__tests__/sandbox-creation-operations.test.ts`. 2. Reject the lease insert after the controlled provider creates its resource. 3. Start a new receiver process with a different empty web disk. It must find the same resource and create one cleanup owner. Related ownership report: Refs #14672. This change addresses new allocations before lease persistence. It does not replace existing lease recovery. ## What Changed - Add an idempotent migration and a small creation-operation table. Retain original company, environment and run identities across deletion. - Commit creation intent before run, ad-hoc lease, and plugin task replacement calls. Fence duplicate requests and ambiguous lease commits. - Retain a known returned task resource ID in the existing operation before attachment. Preserve manual reconciliation and never delete an ambiguously attached resource. - Keep company and run authority checks and atomic lease transfer with the lease owner. - Reuse Daytona names, labels, read-only inspection and claimed cleanup. Keep provider payload handling in the provider. - Retain nonsecret credential identity hashes. Refuse recovery lookup under a changed credential value. - Record manual reconciliation for missing/deleted or explicitly inactive saved credentials, versions, and provider vaults. Keep temporary resolution failures retryable. Never reactivate a disabled credential. - Use transaction signals and concrete deadlines. Failed records back off independently. Manual reconciliation stays durable and quiet. - Remove new orphan-file fallback writes from the covered paths. Keep legacy file replay and its tests for cutover. - Replace obsolete fallback-producer assertions with normal-entrypoint durability tests. Keep cleanup, diagnostics and warm handoff coverage. ## Verification - [Independent current-head verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320) passed 190/190 combined tests, 40/40 incoming TestOps tests, full workspace typecheck, and build. PR head `958af986130f25e136065ad30d1fdd843c46255c` has tree `e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL, compiled Rust Runner and controlled object providers were used; external-S3 integration remains outside this local proof. - Current head `958af986130f25e136065ad30d1fdd843c46255c` normally merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole conflict was the maintenance test: a parameterized case now preserves both legacy-local `local_storage` and valid shared-S3 `provider_mismatch` checks, including the parent local receipt, exact version identity, missing-snapshot/no-seed refusal, and unchanged persisted row. The three master TestOps files remain intact. No production, schema, migration, or package changes were made after the fully qualified prior head `0b4d05505e`. Independent review approved the resolution; the full managed-skill-content file and Cloud guard controls pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge head, first attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412): 54 successful checks and three intended skips. The additional Paperclip review is neutral at 4/5; two external Superagent checks (Contributor trust and Security Scan) remain in progress with zero annotations and no assessment, so they are not passing scan evidence. Greptile reviewed exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final thread audit has zero unresolved threads. All completed evidence below retains its recorded revision scope. The additional Paperclip Review on `958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or should-fix item. Its optional observation is that a generic create rejection remains pending/manual even when an operator believes allocation did not occur. This change preserves uncertainty because the generic create-error contract does not prove provider absence; no new error classification or provider-specific compensation was added. - [Independent final-source verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247) passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full workspace typecheck, and build. The integration commit `c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as final PR head `0b4d05505e` (tree `bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider coverage passed; the separately guarded external-S3 suite was not run because the bounded local MinIO pull was unavailable. This includes the two final compatibility fixes. - The first final-head CI attempt exposed one E2E timing failure in `disconnect-write-recovery.spec.ts:181`: the title stayed unchanged through the 15-second session-recovery poll. The same original case passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all UI/session/E2E source is identical across those heads. The failure log and CI artifacts are retained; no cause is claimed. One failed-job-only retry is authorized on this evidence, without changing a timeout, assertion, or production behavior. [The single failed-job retry completed successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): all 35 E2E shard cases passed (6.6 minutes), including the original session-recovery case in 14.0 seconds. The initial failure remains disclosed and its cause remains unproven. - Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two independently reviewed integration fixes after the complete 7ec CI matrix exposed two deterministic failures. Server 9/12 failed legacy local-storage maintenance because the CLI reached the new Cloud server startup guard; server 5/12 found the Daytona package version did not match its bumped manifest. The failed run is retained at https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is not passing evidence and was not blindly retried. - Maintenance now constructs the existing explicit storage dependency after loading the required persisted Cloud snapshot, and passes it to both maintenance owners. It can verify legacy local storage before provider cutover while normal Cloud server startup remains guarded. The original case reproduced red locally, then the complete managed-skill-content and managed-cloud-prerequisites files passed 57/57 (32.05 seconds), including missing-snapshot refusal, unchanged stored configuration, and existing server guard controls. Daytona package version now matches manifest 0.1.10; the complete bundled package identity file passed 14/14 (3.98 seconds), and explicit manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42 seconds). An initial workspace-filter command matched no project and is not counted as pass evidence. The final diff changes no sandbox runtime or schema behavior. [Full CI is green on this exact final head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): 56 successful checks, three intended skips, one additional incomplete/neutral Paperclip Review, and no pending or failed checks. The neutral review is not a passing assessment. [Greptile reviewed the exact final head at 5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969), with no new actionable issues and all review threads resolved. - Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` includes master managed-skills migration 0417 and shared-storage prerequisites from `198218bf55` through normal merges. Sandbox creation moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL is byte-for-byte unchanged. The regenerated 0418 snapshot contains both sandbox creation operations and the master skill mutation receipts, and points to the master 0417 snapshot. Master SQL and journal history remain intact. Exactly five snapshots remain, 0414 through 0418. The only sandbox-specific changes since the approved credential policy are migration metadata/number and the matching test/document references; the sandbox runtime files are unchanged. - [Independent final combined verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026) passed on integration revision `e6f05b4cf0cd`, containing final sandbox head `7ec06f77b7` and actual master `198218bf55`: migration/declaration checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251 seconds), full workspace typecheck (81 seconds), and build (35 seconds). - Final combined migration proof: fresh real PostgreSQL has both tables. A separate original-0417 sandbox table with a retained pending row survives two applications of the renamed 0418 SQL unchanged, with the expected primary/index and foreign-key structure. Migration numbering/safety checks pass. The normal runtime migration reapply case passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first temporary qualification harness stopped before any database assertion because its external `.ts` file selected CommonJS; the `.mts` harness corrected that setup and passed. Independent review approved the exact metadata and reference diff. Earlier full suite and live-provider results below retain their stated revision scope. - Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd` handles the remaining inactive-credential review warning. The existing boundary accepts only concrete secret-owner 422 codes and exact no-details disabled/coming-soon vault messages. Unknown 422, authorization errors, provider wrappers, and temporary database/provider failures still propagate. Independent source review approved this three-file correction. No generic resolver, schema, provider protocol, retry cap, or activation behavior changed. - Creation/cleanup qualification at `c606d473b0`: all 39 cases pass against real PostgreSQL (81.27 seconds). The new cases use actual disabled/archived saved secrets, a valid disabled version, and provider vaults created ready before normal API updates disable them or mark them coming soon. Each remains manual through restart/repeated scans, with zero provider lookup and no retry deadline. The independent plugin idle blocker remains intact. The temporary 503 control retains retry ownership and later recovers one cleanup owner. Before this correction, all five inactive-state cases remained pending; that failed evidence is retained. The prior 18-case known-ID task proof below belongs to `237144fcf6`, whose helper and task path are unchanged by this correction. - Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also addresses the independent task-replacement reliability warning. The existing creation operation now saves only the known nonsecret returned resource ID before expiry validation and attachment. The write is scoped to operation/company/original environment and allows only a missing or matching ID. It changes no ownership state and copies no resolved configuration, credentials, or provider payload. Independent source review approved this four-file fix. - The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds). Rejected attachments retain the ID through manual reconciliation and repeated scans. A real committed attachment followed by an injected lost response retains the same active lease and attached operation, with no release or second create. Same-ID retries work; conflicting IDs and foreign companies cannot change it. The initial three regression cases failed because the ID was absent, before the fix. - The shipped Daytona manifest does not declare task support, so it cannot enter this generic replacement path. The dispatcher requires both declared task support and an environmentTask worker method. Generic drivers without creation lookup still use manual reconciliation, with a known returned ID retained when the database write can commit. No automatic lookup capability or unconditional compensation was added. - Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0` fixes the fresh review finding about missing credentials. Only known secret-owner 404 results at the recovery secret-resolution boundary select the existing manual outcome. Provider inspection and transient database/provider errors remain retryable. Independent source review approved this change. - Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass against isolated real PostgreSQL. The complete creation suite passes 16/16 (100.75 seconds); the complete cleanup coordinator suite passes 18/18 (99.83 seconds). The new cases use normal saved-secret creation and deletion, confirm zero provider lookup, restart/repeated scans, no automatic deadline, preserved independent plugin idle blocking, and no additional idle blocker after the controlled worker supplies its quiet proof. A temporary 503 resolution failure preserves pending ownership and later recovers one cleanup owner. No additional live provider resource was created. - Failed attempts remain part of the evidence. The initial red run reproduced the pending-operation bug and also exposed a numeric-deadline test harness mistake, which was corrected. A later combined run reported an existing replacement-receiver failure and a new idle-fixture assertion failure; it was interrupted while the next test file remained in imports. An isolated diagnostic confirmed that the manual state was correct but the installed fixture plugin correctly blocked sleep without its independent quiet proof. The corrected test retains that blocker assertion before supplying the controlled worker witness. The unchanged replacement-receiver case passes in the final complete 16-case run; its earlier failure is not counted as a pass or assigned a proven cause. No timeout or production idle guard was relaxed. - Greptile reviewed previous combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved review threads. Its CI produced the two concrete integration failures addressed above. The earlier CI and live evidence below remain historical evidence, not a claim that the current full gates have completed. - Master integration update: merged `dbff3747b8` and then latest master `76fde891cc` without rewriting branch history. Only `doc/DATABASE.md` conflicted; the sandbox ownership, object-backed run evidence, and cloud configuration sections all remain. The master merges changed no sandbox implementation, migration, or provider code. Migration numbering/safety checks pass and exactly five schema snapshots remain. At merge head `f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused creation/coordinator suites pass all 32 cases against isolated real PostgreSQL (64.57 seconds). Those merge-head checks precede the missing-credential fix below. The earlier live Daytona proof still covers the unchanged normal creation/lookup/teardown path; the new missing-credential branch is qualified with real PostgreSQL and controlled provider hooks. The optional Superagent Security Scan was inconclusive on prior head `7ec06f77b7`: its pinned PR merge-base checkout failed or timed out, with a neutral conclusion and zero annotations. An earlier scan could not start within 120 seconds; a bounded rerun request returned HTTP 404. These scanner failures are not passing security-scan evidence. On final head `0b4d05505e`, the additional Paperclip Review check is neutral: its assigned review execution did not complete and produced no validated assessment. Fresh Greptile 5/5 and independent source review are separate evidence. Previous-head required CI, independent review, and the live Daytona qualification below passed. - [Complete previous-head CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052) passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful checks and three intended skips, including typecheck, full tests, build and E2E. Greptile is 5/5 on this head and both original findings are resolved. - Initial head: 200 PostgreSQL runtime, creation and coordinator tests passed. The corrected head also passes all 32 creation and coordinator tests, including deletion before/during lookup, a failed lookup after deletion, restart, settled polling and actual idle-safety checks. Cases cover failed admission, failed lease writes, concurrent recovery, lost provider replies, ambiguous transfer retries, company rebind, environment deletion, stopped runs, lookup backoff, restart, drain and settled SQL quiet. - Separate receiver processes: passed. A controlled HTTP provider retains the resource while three independent receiver processes use two empty web disks. Recovery creates one cleanup owner and never repeats creation. - Fresh PostgreSQL migration and two additional applications: passed. The pending creation row survives unchanged. - Server TypeScript and SDK/shared build checks pass after the corrections. Migration numbering/safety checks passed. Exactly five schema snapshots remain. - Daytona adapter: all 330 tests passed, including stable-name lookup, ownership checks, deletion and the updated manifest version. An earlier run exposed the version expectation and it was corrected. Previous-head CI passed. - Live Daytona acceptance passed on `0b5cbb763a`: isolated real PostgreSQL, normal runtime acquisition through independent receiver processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu 24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert failure retained the committed operation; a fresh receiver with empty disk found the same sandbox, repeated recovery retained one cleanup owner, and the actual claimed cleanup sweep expired the lease and resolved the operation. Independent SDK lookup returned typed not-found; ordinary recorded-ID inspection returned destroyed. This successful attempt made one create, one recovery lookup, and one teardown. Two earlier failed harness/provider-confirmation attempts are disclosed in the [live acceptance record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054). The coordinator independently queried all three exact resource IDs through the SDK and confirmed typed not-found for each. - Complete CI-affected heartbeat/plugin warm-task suites passed 21/21 after test corrections. An unchanged workspace exposure case passed in isolation after the prior CI shard reported an unhandled readiness rejection. Complete previous-head CI and refreshed review passed. ## Risks - Deleted companies retain an explicit manual reconciliation outcome. Recovery does not retry missing credentials or insert a lease for a deleted company. - Only Daytona declares automatic creation lookup. Other uncertain providers retain `needs_reconciliation`. A missing creation name alone cannot prove that a late allocation will never appear. - Pending operations use the existing process-local signal and startup recovery model. This does not add cross-replica wake delivery or prove deployed sleep. - Legacy direct provider probes and custom-image setup use separate lifecycles. They are outside this replacement of the run-lease orphan-file fallback. The normal Test path that acquires a runtime lease is covered. - Existing orphan files still require the separate cutover inventory and settlement. This change does not delete them. - The schema change is additive. The new SQL can be reapplied. Historical SQL and journal entries are preserved; only the oldest generated snapshot is pruned. ## Model Used OpenAI `gpt-6.1-sol`, with tool use, code execution and repository review. The harness did not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The required branch status checks (`ci / verify`, `ci / e2e`) and all test/build jobs passed on this head. The non-required Superagent Contributor trust and Security Scan checks are still running without findings; they are not claimed as passes. No required check or branch rule is overridden. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1878 lines
80 KiB
TypeScript
1878 lines
80 KiB
TypeScript
import { createReusableLeaseHandoff, preserveReusableLeaseHandoff, withoutReusableLeaseHandoff } from "./environment-lease-handoff.js";
|
|
import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js";
|
|
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
|
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
|
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
|
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
|
|
import { randomUUID } from "node:crypto";
|
|
|
|
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "./environment-custom-image-runtime.js";
|
|
import { and, desc, eq, inArray, isNull, ne, sql, type SQL } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agents,
|
|
builtInManagedResources,
|
|
companies,
|
|
companySecretBindings,
|
|
environmentCustomImageSetupSessions,
|
|
environmentLeases,
|
|
sandboxCreationOperations,
|
|
environments,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
instanceSettings,
|
|
issues,
|
|
projects,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
ENVIRONMENT_DRIVERS,
|
|
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
|
|
ENVIRONMENT_LEASE_POLICIES,
|
|
ENVIRONMENT_LEASE_STATUSES,
|
|
ENVIRONMENT_STATUSES,
|
|
type CreateEnvironment,
|
|
type Environment,
|
|
type EnvironmentDeleteBlastRadius,
|
|
type EnvironmentDeleteBlockedReason,
|
|
type EnvironmentLease,
|
|
type EnvironmentLeaseCleanupStatus,
|
|
type EnvironmentLeasePolicy,
|
|
type EnvironmentLeaseStatus,
|
|
type UpdateEnvironment,
|
|
} from "@paperclipai/shared";
|
|
import { conflict, forbidden } from "../errors.js";
|
|
import { logActivity } from "./activity-log.js";
|
|
import { isCloudManagedInstance } from "./cloud-instance.js";
|
|
import {
|
|
resourceStatus,
|
|
stockHash,
|
|
type ManagedResourceStockStatus,
|
|
} from "./managed-resource-drift.js";
|
|
|
|
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
|
|
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
|
|
return db.transaction(async tx => {
|
|
await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
|
|
const [operation] = await tx.select().from(sandboxCreationOperations)
|
|
.where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
|
|
eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update");
|
|
if (scope.heartbeatRunId) {
|
|
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"),
|
|
)).for("update");
|
|
if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed");
|
|
}
|
|
if (operation?.state === "attached" && operation.leaseId) {
|
|
const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId);
|
|
if (existing?.status === "active") return existing;
|
|
}
|
|
if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred");
|
|
const lease = await attach(tx as unknown as Db);
|
|
await notifyEnvironmentCleanup(tx, { companyId: scope.companyId });
|
|
await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
|
|
.where(eq(sandboxCreationOperations.id, id));
|
|
return lease;
|
|
});
|
|
}
|
|
|
|
export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {
|
|
const [environment] = await db.select({ id: environments.id }).from(environments)
|
|
.where(eq(environments.id, environmentId)).for("update");
|
|
if (!environment) throw new Error("Sandbox creation environment no longer exists");
|
|
const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources)
|
|
.where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId)));
|
|
if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) {
|
|
throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" });
|
|
}
|
|
}
|
|
|
|
type EnvironmentRow = typeof environments.$inferSelect;
|
|
type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
|
|
const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
|
|
const DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION =
|
|
"Default execution environment for Paperclip runs on this machine.";
|
|
|
|
const DEFAULT_KUBERNETES_ENVIRONMENT_NAME = "Kubernetes Sandbox";
|
|
const DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION =
|
|
"Managed Kubernetes sandbox environment for hosted tenant execution.";
|
|
/** Provider key (== plugin driverKey) of the first-party Kubernetes sandbox provider. */
|
|
const KUBERNETES_PROVIDER_KEY = "kubernetes";
|
|
/** Metadata marker for the company's managed-by-config Kubernetes sandbox environment. */
|
|
const KUBERNETES_MANAGED_MARKER = "managedKubernetesSandbox";
|
|
const ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES = ["starting", "waiting_for_user", "capturing"] as const;
|
|
|
|
/**
|
|
* Configuration accepted by `ensureKubernetesEnvironment`. Mirrors the keys of
|
|
* the kubernetes sandbox-provider `configSchema` that an operator typically
|
|
* pins for a hosted cloud instance. Stored verbatim in `environment.config`
|
|
* (the plugin validates/defaults it via `kubernetesProviderConfigSchema` at
|
|
* lease time); `provider` is always forced to "kubernetes".
|
|
*/
|
|
export interface KubernetesEnvironmentConfigInput {
|
|
backend?: "sandbox-cr" | "job";
|
|
inCluster?: boolean;
|
|
runtimeClassName?: string;
|
|
egressMode?: "cilium" | "standard";
|
|
egressAllowFqdns?: string[];
|
|
egressAllowCidrs?: string[];
|
|
namespacePrefix?: string;
|
|
imageRegistry?: string;
|
|
adapterType?: string;
|
|
/**
|
|
* Sandbox lease RPC timeout in milliseconds. Read at lease time by
|
|
* `resolvePluginSandboxRpcTimeoutMs` to extend the worker-manager call
|
|
* timeout when acquiring a lease may take minutes (e.g. a cold node
|
|
* scale-up on an autoscale-to-zero pool). Stored verbatim in the
|
|
* environment config and validated by the sandbox config schema.
|
|
*/
|
|
timeoutMs?: number;
|
|
adapters?: import("@paperclipai/shared").AdapterRegistryEntry[];
|
|
[key: string]: unknown;
|
|
}
|
|
|
|
/**
|
|
* Input to `ensureManagedSandboxEnvironment`. Provider-agnostic: `provider`
|
|
* is the sandbox plugin's driver key and is forced into `config.provider`;
|
|
* the rest of `config` is stored verbatim for the plugin to validate at
|
|
* lease time.
|
|
*/
|
|
export interface ManagedSandboxEnvironmentInput {
|
|
/** Company whose managed-resource binding should be reconciled. Omit at instance boot to bind every company. */
|
|
companyId?: string;
|
|
name: string;
|
|
description?: string;
|
|
/** Sandbox provider key (the plugin's driverKey, e.g. "kubernetes", "daytona"). */
|
|
provider: string;
|
|
config?: Record<string, unknown>;
|
|
/**
|
|
* Extra metadata markers stamped on the managed row (e.g. the legacy
|
|
* kubernetes marker `managedKubernetesSandbox` that
|
|
* `findKubernetesEnvironment` keys on).
|
|
*/
|
|
extraMetadata?: Record<string, unknown>;
|
|
/** Version label recorded with the stock binding; hashes remain the drift authority. */
|
|
stockVersion?: string;
|
|
/**
|
|
* Asserts the caller's deployment gives no operator any path to hand-edit
|
|
* this row (currently only true for the PAPERCLIP_MANAGED_CONFIG applier,
|
|
* where `enableManagedSandboxOnly` removes the tenant's own environment
|
|
* choice entirely). When set, a plain content-hash mismatch against a real
|
|
* prior binding is treated as ordinary stock drift instead of an operator
|
|
* customization to protect — see the `operator_modified` handling below.
|
|
* Leave unset for any caller (self-hosted `kubernetes-execution-mode`
|
|
* bootstrap, tests, admin routes) where an operator could realistically
|
|
* have edited the row through the normal environments UI/API.
|
|
*/
|
|
platformFullyManaged?: boolean;
|
|
}
|
|
|
|
export type ManagedSandboxEnvironmentReconcileAction =
|
|
| "added"
|
|
| "updated"
|
|
| "unchanged"
|
|
| "skipped";
|
|
|
|
export interface ManagedSandboxEnvironmentReconcileResult {
|
|
environment: Environment;
|
|
action: ManagedSandboxEnvironmentReconcileAction;
|
|
/** Classification observed before this reconciliation wrote anything. */
|
|
stockStatus: ManagedResourceStockStatus;
|
|
/** True only when operator drift prevented the available stock update. */
|
|
updateAvailable: boolean;
|
|
stockHash: string;
|
|
}
|
|
|
|
function cloneRecord(value: unknown, fallback: Record<string, unknown> | null = null): Record<string, unknown> | null {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return fallback;
|
|
return { ...(value as Record<string, unknown>) };
|
|
}
|
|
|
|
function readEnum<T extends string>(value: string | null, allowed: readonly T[], fieldName: string): T | null {
|
|
if (value === null) return null;
|
|
if ((allowed as readonly string[]).includes(value)) return value as T;
|
|
throw new Error(`Unexpected ${fieldName} value: ${value}`);
|
|
}
|
|
|
|
function hasConstraintName(error: unknown, constraintName: string): boolean {
|
|
if (typeof error !== "object" || error === null) return false;
|
|
const candidate = error as {
|
|
constraint?: unknown;
|
|
constraint_name?: unknown;
|
|
cause?: unknown;
|
|
};
|
|
return candidate.constraint === constraintName
|
|
|| candidate.constraint_name === constraintName
|
|
|| hasConstraintName(candidate.cause, constraintName);
|
|
}
|
|
|
|
function toEnvironment(row: EnvironmentRow): Environment {
|
|
return {
|
|
id: row.id,
|
|
name: row.name,
|
|
description: row.description ?? null,
|
|
driver: readEnum(row.driver, ENVIRONMENT_DRIVERS, "environment driver") ?? "local",
|
|
status: readEnum(row.status, ENVIRONMENT_STATUSES, "environment status") ?? "active",
|
|
config: cloneRecord(row.config, {}) ?? {},
|
|
envVars: cloneRecord(row.envVars, {}) ?? {},
|
|
metadata: cloneRecord(row.metadata),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
} as Environment;
|
|
}
|
|
|
|
type EnvironmentListFilters = {
|
|
status?: string;
|
|
driver?: string;
|
|
};
|
|
|
|
function resolveListFilters(
|
|
companyIdOrFilters?: string | EnvironmentListFilters,
|
|
maybeFilters?: EnvironmentListFilters,
|
|
): EnvironmentListFilters {
|
|
if (typeof companyIdOrFilters === "string") {
|
|
return maybeFilters ?? {};
|
|
}
|
|
return companyIdOrFilters ?? {};
|
|
}
|
|
|
|
function resolveCreateInput(
|
|
companyIdOrInput: string | CreateEnvironment,
|
|
maybeInput?: CreateEnvironment,
|
|
): CreateEnvironment {
|
|
if (typeof companyIdOrInput === "string") {
|
|
if (!maybeInput) throw new Error("Create environment input is required");
|
|
return maybeInput;
|
|
}
|
|
return companyIdOrInput;
|
|
}
|
|
|
|
function resolveKubernetesConfig(
|
|
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
|
|
maybeConfig?: KubernetesEnvironmentConfigInput,
|
|
): KubernetesEnvironmentConfigInput {
|
|
if (typeof companyIdOrConfig === "string") {
|
|
if (!maybeConfig) throw new Error("Kubernetes environment config is required");
|
|
return maybeConfig;
|
|
}
|
|
return companyIdOrConfig;
|
|
}
|
|
|
|
function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
environmentId: row.environmentId,
|
|
executionWorkspaceId: row.executionWorkspaceId ?? null,
|
|
issueId: row.issueId ?? null,
|
|
heartbeatRunId: row.heartbeatRunId ?? null,
|
|
status: readEnum(row.status, ENVIRONMENT_LEASE_STATUSES, "environment lease status") ?? "active",
|
|
leasePolicy: readEnum(row.leasePolicy, ENVIRONMENT_LEASE_POLICIES, "environment lease policy") ?? "ephemeral",
|
|
provider: row.provider ?? null,
|
|
providerLeaseId: row.providerLeaseId ?? null,
|
|
acquiredAt: row.acquiredAt,
|
|
lastUsedAt: row.lastUsedAt,
|
|
expiresAt: row.expiresAt ?? null,
|
|
releasedAt: row.releasedAt ?? null,
|
|
failureReason: row.failureReason ?? null,
|
|
cleanupStatus: readEnum(
|
|
row.cleanupStatus,
|
|
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
|
|
"environment lease cleanup status",
|
|
),
|
|
metadata: cloneRecord(row.metadata),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
};
|
|
}
|
|
|
|
function countFromRows(rows: Array<{ count: number | string | null | undefined }>): number {
|
|
return Number(rows[0]?.count ?? 0);
|
|
}
|
|
|
|
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
type EnvironmentWriteDb = Pick<Db | DbTransaction, "select" | "insert" | "update" | "delete">;
|
|
|
|
const MANAGED_ENVIRONMENT_BUNDLE_KEY = "managed-sandbox-environment";
|
|
const MANAGED_ENVIRONMENT_RESOURCE_KIND = "environment";
|
|
const MANAGED_ENVIRONMENT_RESOURCE_KEY = "managed-sandbox";
|
|
const MANAGED_ENVIRONMENT_STOCK_VERSION = "managed-environment-v1";
|
|
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY = "_paperclipManagedArchiveToken";
|
|
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY = "_paperclipManagedArchiveToken";
|
|
|
|
function managedEnvironmentBaselineDefaults(
|
|
defaultsJson: Record<string, unknown>,
|
|
): Record<string, unknown> {
|
|
const baseline = { ...defaultsJson };
|
|
delete baseline[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
|
|
return baseline;
|
|
}
|
|
|
|
function withoutManagedEnvironmentArchiveToken(
|
|
metadata: Record<string, unknown> | null,
|
|
): Record<string, unknown> | null {
|
|
if (!metadata) return null;
|
|
const next = { ...metadata };
|
|
delete next[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
|
|
return next;
|
|
}
|
|
|
|
function managedMetadataKeys(
|
|
desiredMetadata: Record<string, unknown>,
|
|
bindings: Array<{ defaultsJson: Record<string, unknown> }>,
|
|
): string[] {
|
|
const keys = new Set([
|
|
"managedByPaperclip",
|
|
"managedSandboxProvider",
|
|
KUBERNETES_MANAGED_MARKER,
|
|
...Object.keys(desiredMetadata),
|
|
]);
|
|
for (const binding of bindings) {
|
|
const metadata = cloneRecord(binding.defaultsJson.metadata);
|
|
for (const key of Object.keys(metadata ?? {})) keys.add(key);
|
|
}
|
|
return [...keys].sort((left, right) => left.localeCompare(right));
|
|
}
|
|
|
|
function managedEnvironmentStock(input: {
|
|
name: string;
|
|
description: string | null;
|
|
config: Record<string, unknown>;
|
|
metadata: Record<string, unknown> | null;
|
|
status: string;
|
|
}, metadataKeys: readonly string[]): Record<string, unknown> {
|
|
const metadata = input.metadata ?? {};
|
|
return {
|
|
name: input.name,
|
|
description: input.description,
|
|
config: input.config,
|
|
metadata: Object.fromEntries(
|
|
metadataKeys.map((key) => [key, Object.prototype.hasOwnProperty.call(metadata, key) ? metadata[key] : null]),
|
|
),
|
|
status: input.status,
|
|
};
|
|
}
|
|
|
|
function mergeManagedEnvironmentMetadata(
|
|
current: Record<string, unknown> | null,
|
|
desired: Record<string, unknown>,
|
|
keys: readonly string[],
|
|
): Record<string, unknown> {
|
|
const merged = { ...(current ?? {}) };
|
|
for (const key of keys) {
|
|
const value = Object.prototype.hasOwnProperty.call(desired, key) ? desired[key] : null;
|
|
if (value === null || value === undefined) delete merged[key];
|
|
else merged[key] = value;
|
|
}
|
|
return merged;
|
|
}
|
|
|
|
function preserveInvalidatedSnapshotProofs(replacement: SQL): SQL {
|
|
const proofs = sql`(select coalesce(jsonb_object_agg(entry.key, entry.value || '{"invalidated":true}'::jsonb), '{}'::jsonb)
|
|
from jsonb_each(coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)) as entry
|
|
where entry.value->>'v' = '2')`;
|
|
return sql`case when ${proofs} = '{}'::jsonb then ${replacement}
|
|
else coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${proofs}) end`;
|
|
}
|
|
|
|
export function environmentService(db: Db) {
|
|
/**
|
|
* Idempotently ensure THE Paperclip-managed sandbox environment for this
|
|
* instance, configured for an arbitrary sandbox provider plugin. Mirrors
|
|
* `ensureLocalEnvironment`; the partial unique index
|
|
* `environments_managed_sandbox_idx` enforces at most one managed sandbox
|
|
* row per instance, so this function owns that single slot regardless of
|
|
* provider:
|
|
*
|
|
* - A stock-controlled managed row advances to a new stock hash in the same
|
|
* transaction as its managed fields, including provider switches.
|
|
* - A stock-current row is returned without an environment write.
|
|
* - An operator-modified or previously unmanaged row is preserved and
|
|
* reported as skipped; its user-owned fields are never folded into stock.
|
|
*/
|
|
const ensureManagedSandboxEnvironment = async (
|
|
input: ManagedSandboxEnvironmentInput,
|
|
): Promise<ManagedSandboxEnvironmentReconcileResult> => {
|
|
const desiredConfig: Record<string, unknown> = {
|
|
...(input.config ?? {}),
|
|
provider: input.provider,
|
|
};
|
|
const desiredMetadata: Record<string, unknown> = {
|
|
managedByPaperclip: true,
|
|
managedSandboxProvider: input.provider,
|
|
...(input.extraMetadata ?? {}),
|
|
};
|
|
if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) {
|
|
desiredMetadata[KUBERNETES_MANAGED_MARKER] = null;
|
|
}
|
|
|
|
let activityCompanyIds: string[] = [];
|
|
let trackingInitialized = false;
|
|
let providerReactivated = false;
|
|
const reconciliation = await db.transaction(
|
|
async (tx): Promise<ManagedSandboxEnvironmentReconcileResult> => {
|
|
const companyIds = input.companyId
|
|
? [input.companyId]
|
|
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
|
|
activityCompanyIds = companyIds;
|
|
|
|
// Take the sandbox-row lock BEFORE reading the stock bindings. Two
|
|
// passes can reconcile the same slot concurrently (the boot ensure and
|
|
// the async provider-recovery reactivation, or two app builds during a
|
|
// rolling deploy). Concurrent passes serialize on this lock, and under
|
|
// READ COMMITTED each later statement sees a fresh snapshot — so a pass
|
|
// that blocks here then reads the bindings the winning pass committed,
|
|
// not a snapshot from before it waited. Reading bindings first left a
|
|
// window where a waiting pass compared the winner's fresh row against
|
|
// its own stale binding hash, misclassified the mismatch as
|
|
// `operator_modified`, and — once `platformFullyManaged` turns that
|
|
// into an update — rolled the row back to its own older stock.
|
|
const sandboxRows = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update");
|
|
|
|
const bindingConditions = and(
|
|
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
|
|
...(companyIds.length > 0 ? [inArray(builtInManagedResources.companyId, companyIds)] : []),
|
|
);
|
|
const bindings = companyIds.length > 0
|
|
? await tx.select().from(builtInManagedResources).where(bindingConditions)
|
|
: [];
|
|
trackingInitialized = bindings.length < companyIds.length;
|
|
const keys = managedMetadataKeys(desiredMetadata, bindings);
|
|
|
|
let row = sandboxRows.find(
|
|
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? sandboxRows.find((candidate) => candidate.name === input.name) ?? null;
|
|
|
|
const writeBindings = async (
|
|
environmentId: string,
|
|
stockVersion: string,
|
|
installedStockHash: string,
|
|
defaultsJson: Record<string, unknown>,
|
|
replace: boolean,
|
|
) => {
|
|
const targetCompanyIds = replace
|
|
? companyIds
|
|
: companyIds.filter(
|
|
(companyId) => !bindings.some((binding) => binding.companyId === companyId),
|
|
);
|
|
if (targetCompanyIds.length === 0) return;
|
|
const values = targetCompanyIds.map((companyId) => ({
|
|
companyId,
|
|
bundleKey: MANAGED_ENVIRONMENT_BUNDLE_KEY,
|
|
resourceKind: MANAGED_ENVIRONMENT_RESOURCE_KIND,
|
|
resourceKey: MANAGED_ENVIRONMENT_RESOURCE_KEY,
|
|
resourceId: environmentId,
|
|
stockVersion,
|
|
stockHash: installedStockHash,
|
|
defaultsJson,
|
|
}));
|
|
const insert = tx.insert(builtInManagedResources).values(values);
|
|
if (!replace) {
|
|
await insert.onConflictDoNothing({
|
|
target: [
|
|
builtInManagedResources.companyId,
|
|
builtInManagedResources.bundleKey,
|
|
builtInManagedResources.resourceKind,
|
|
builtInManagedResources.resourceKey,
|
|
],
|
|
});
|
|
return;
|
|
}
|
|
await insert.onConflictDoUpdate({
|
|
target: [
|
|
builtInManagedResources.companyId,
|
|
builtInManagedResources.bundleKey,
|
|
builtInManagedResources.resourceKind,
|
|
builtInManagedResources.resourceKey,
|
|
],
|
|
set: {
|
|
resourceId: environmentId,
|
|
stockVersion,
|
|
stockHash: installedStockHash,
|
|
defaultsJson,
|
|
updatedAt: new Date(),
|
|
},
|
|
});
|
|
};
|
|
|
|
if (!row) {
|
|
const nameOwner = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.name, input.name))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (nameOwner && nameOwner.driver !== "sandbox") {
|
|
throw new Error(
|
|
`Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`,
|
|
);
|
|
}
|
|
const now = new Date();
|
|
const inserted = await tx
|
|
.insert(environments)
|
|
.values({
|
|
name: input.name,
|
|
description: input.description ?? null,
|
|
driver: "sandbox",
|
|
status: "active",
|
|
config: desiredConfig,
|
|
envVars: {},
|
|
metadata: mergeManagedEnvironmentMetadata(null, desiredMetadata, keys),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
// Either the managed-slot partial index or the global name index
|
|
// can select the concurrent winner. Treat both as convergence and
|
|
// reselect that winner under lock below.
|
|
.onConflictDoNothing()
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (inserted) {
|
|
const stock = managedEnvironmentStock({
|
|
name: inserted.name,
|
|
description: inserted.description ?? null,
|
|
config: inserted.config,
|
|
metadata: inserted.metadata,
|
|
status: inserted.status,
|
|
}, keys);
|
|
const latestStockHash = stockHash(stock);
|
|
await writeBindings(
|
|
inserted.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
stock,
|
|
true,
|
|
);
|
|
return {
|
|
environment: toEnvironment(inserted),
|
|
action: "added",
|
|
stockStatus: "missing",
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
row = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update")
|
|
.then(
|
|
(rows) => rows.find(
|
|
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? null,
|
|
);
|
|
if (!row) throw new Error("Failed to ensure managed sandbox environment");
|
|
}
|
|
|
|
const nameOwner = await tx
|
|
.select({ id: environments.id })
|
|
.from(environments)
|
|
.where(eq(environments.name, input.name))
|
|
.then((rows) => rows[0] ?? null);
|
|
const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name;
|
|
const desiredStock = managedEnvironmentStock({
|
|
name: desiredName,
|
|
description: input.description ?? null,
|
|
config: desiredConfig,
|
|
metadata: desiredMetadata,
|
|
status: "active",
|
|
}, keys);
|
|
const latestStockHash = stockHash(desiredStock);
|
|
const currentStock = managedEnvironmentStock({
|
|
name: row.name,
|
|
description: row.description ?? null,
|
|
config: row.config,
|
|
metadata: row.metadata,
|
|
status: row.status,
|
|
}, keys);
|
|
const currentHash = stockHash(currentStock);
|
|
const matchingBindings = bindings.filter((binding) => binding.resourceId === row!.id);
|
|
const rowArchiveToken = (row.metadata as Record<string, unknown> | null)
|
|
?.[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
|
|
let stockStatus = resourceStatus({
|
|
resourceId: row.id,
|
|
currentHash,
|
|
bindingStockHash: matchingBindings[0]?.stockHash ?? null,
|
|
latestStockHash,
|
|
});
|
|
if (stockStatus === "operator_modified") {
|
|
const stockControlledBinding = matchingBindings.find(
|
|
(binding) => resourceStatus({
|
|
resourceId: row!.id,
|
|
currentHash,
|
|
bindingStockHash: binding.stockHash,
|
|
latestStockHash,
|
|
}) === "stock_update_available",
|
|
);
|
|
if (stockControlledBinding) stockStatus = "stock_update_available";
|
|
}
|
|
const operatorReaffirmedArchive = row.status === "archived" && matchingBindings.some(
|
|
(binding) => {
|
|
const bindingToken = binding.defaultsJson[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
|
|
return binding.defaultsJson.status === "archived" &&
|
|
typeof bindingToken === "string" &&
|
|
bindingToken !== rowArchiveToken;
|
|
},
|
|
);
|
|
if (operatorReaffirmedArchive) stockStatus = "operator_modified";
|
|
|
|
// `platformFullyManaged` callers (currently: the PAPERCLIP_MANAGED_CONFIG
|
|
// applier) assert that nothing in their deployment can hand-edit this
|
|
// row — the product gives a cloud-harness tenant no path to it, unlike
|
|
// the general self-hosted contract this function otherwise protects
|
|
// (see "classifies operator drift" in environment-service.test.ts,
|
|
// which exercises a real operator edit and must keep winning). Under
|
|
// that assertion, a plain content-hash mismatch against a real prior
|
|
// binding can only be drift between two platform-driven reconciliation
|
|
// passes (e.g. a stock hash recorded by an older app build before a
|
|
// later stock field was added), never a customization to protect —
|
|
// apply it like any other stock-outdated row.
|
|
//
|
|
// Two things the bypass must never touch:
|
|
// - A row with NO matching binding. `row` can be a same-name sandbox
|
|
// row that was never Paperclip-managed (the fallback lookup above).
|
|
// It also reads as `operator_modified`, but there is no prior
|
|
// platform pass to have drifted from — adopting it would overwrite
|
|
// a tenant-created environment and stamp it managed. Require a
|
|
// binding for this exact row, so "prior binding" is enforced, not
|
|
// just documented.
|
|
// - Archive-reaffirmation. A `sandbox_image` update must never
|
|
// resurrect a row something else deliberately kept archived after
|
|
// Paperclip's own provider-unavailability archival, so that path
|
|
// still skips below regardless of this flag.
|
|
if (
|
|
input.platformFullyManaged &&
|
|
stockStatus === "operator_modified" &&
|
|
!operatorReaffirmedArchive &&
|
|
matchingBindings.length > 0
|
|
) {
|
|
stockStatus = "stock_update_available";
|
|
}
|
|
|
|
if (stockStatus === "operator_modified") {
|
|
const baseline = matchingBindings[0];
|
|
let baselineDefaults = baseline
|
|
? managedEnvironmentBaselineDefaults(baseline.defaultsJson)
|
|
: desiredStock;
|
|
let baselineHash = baseline?.stockHash ?? latestStockHash;
|
|
|
|
// Provider unavailability is an operational state transition, not
|
|
// an operator edit. If the binding records that Paperclip archived
|
|
// this row, restore only its availability status. Keep every other
|
|
// operator-modified field intact and leave the stock update pending.
|
|
// A manually archived row still has an active binding baseline, so
|
|
// it remains operator_modified and is not reactivated here.
|
|
const archivedByReconciler = row.status === "archived" &&
|
|
typeof rowArchiveToken === "string" &&
|
|
matchingBindings.some((binding) => {
|
|
const bindingDefaults = binding.defaultsJson;
|
|
return bindingDefaults.status === "archived" &&
|
|
bindingDefaults[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY] === rowArchiveToken;
|
|
});
|
|
if (archivedByReconciler) {
|
|
const reactivated = await tx
|
|
.update(environments)
|
|
.set({
|
|
status: "active",
|
|
metadata: withoutManagedEnvironmentArchiveToken(row.metadata),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(environments.id, row.id), eq(environments.status, "archived")))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!reactivated) {
|
|
throw new Error("Managed sandbox environment changed during reactivation");
|
|
}
|
|
row = reactivated;
|
|
providerReactivated = true;
|
|
|
|
for (const binding of matchingBindings) {
|
|
const reactivatedDefaults = {
|
|
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
|
|
status: "active",
|
|
};
|
|
const reactivatedHash = stockHash(reactivatedDefaults);
|
|
await tx
|
|
.update(builtInManagedResources)
|
|
.set({
|
|
stockHash: reactivatedHash,
|
|
defaultsJson: reactivatedDefaults,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(
|
|
eq(builtInManagedResources.id, binding.id),
|
|
eq(builtInManagedResources.resourceId, row.id),
|
|
));
|
|
if (binding.id === baseline?.id) {
|
|
baselineDefaults = reactivatedDefaults;
|
|
baselineHash = reactivatedHash;
|
|
}
|
|
}
|
|
}
|
|
await writeBindings(
|
|
row.id,
|
|
baseline?.stockVersion ?? input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
baselineHash,
|
|
baselineDefaults,
|
|
false,
|
|
);
|
|
return {
|
|
environment: toEnvironment(row),
|
|
action: "skipped",
|
|
stockStatus,
|
|
updateAvailable: true,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
|
|
if (stockStatus === "stock_current") {
|
|
await writeBindings(
|
|
row.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
desiredStock,
|
|
false,
|
|
);
|
|
return {
|
|
environment: toEnvironment(row),
|
|
action: "unchanged",
|
|
stockStatus,
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
|
|
const updated = await tx
|
|
.update(environments)
|
|
.set({
|
|
name: desiredName,
|
|
description: input.description ?? null,
|
|
config: desiredConfig,
|
|
metadata: withoutManagedEnvironmentArchiveToken(
|
|
mergeManagedEnvironmentMetadata(row.metadata, desiredMetadata, keys),
|
|
),
|
|
status: "active",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environments.id, row.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!updated) throw new Error("Managed sandbox environment changed during reconciliation");
|
|
await writeBindings(
|
|
updated.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
desiredStock,
|
|
true,
|
|
);
|
|
return {
|
|
environment: toEnvironment(updated),
|
|
action: "updated",
|
|
stockStatus,
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
},
|
|
);
|
|
if (reconciliation.action !== "unchanged" || trackingInitialized) {
|
|
const action = reconciliation.action === "added"
|
|
? "environment.managed_stock_added"
|
|
: reconciliation.action === "updated"
|
|
? "environment.managed_stock_updated"
|
|
: reconciliation.action === "skipped"
|
|
? "environment.managed_stock_skipped"
|
|
: "environment.managed_stock_tracking_initialized";
|
|
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
|
|
companyId,
|
|
actorType: "system",
|
|
actorId: "managed-environment-reconciler",
|
|
action,
|
|
entityType: "environment",
|
|
entityId: reconciliation.environment.id,
|
|
details: {
|
|
provider: input.provider,
|
|
reconciliationAction: reconciliation.action,
|
|
stockStatus: reconciliation.stockStatus,
|
|
updateAvailable: reconciliation.updateAvailable,
|
|
stockHash: reconciliation.stockHash,
|
|
providerReactivated,
|
|
},
|
|
})));
|
|
}
|
|
return reconciliation;
|
|
};
|
|
|
|
/**
|
|
* Archive the Paperclip-managed sandbox row when its provider became
|
|
* unavailable (plugin missing, not ready, or its worker not running), so
|
|
* run scheduling stops selecting an environment whose lease acquisition
|
|
* cannot succeed (`resolveEnvironment` rejects non-active rows).
|
|
*
|
|
* Scoped to the row provisioned for the SAME provider: a row that a
|
|
* provider switch left on a different provider is not touched (the ensure
|
|
* path adopts it once the new provider is healthy). Reactivation is
|
|
* automatic — the next successful `ensureManagedSandboxEnvironment` stamps
|
|
* the row `active` again.
|
|
*
|
|
* Returns the archived environment, or null when there is no active
|
|
* managed row for this provider.
|
|
*/
|
|
const archiveManagedSandboxEnvironment = async (
|
|
input: { provider: string; companyId?: string },
|
|
): Promise<Environment | null> => {
|
|
let activityCompanyIds: string[] = [];
|
|
const archived = await db.transaction(async (tx) => {
|
|
const existing = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update")
|
|
.then(
|
|
(rows) => rows.find(
|
|
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? null,
|
|
);
|
|
if (!existing || existing.status !== "active") return null;
|
|
const rowProvider = (existing.metadata as Record<string, unknown> | null)
|
|
?.managedSandboxProvider;
|
|
if (rowProvider !== input.provider) return null;
|
|
|
|
const companyIds = input.companyId
|
|
? [input.companyId]
|
|
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
|
|
activityCompanyIds = companyIds;
|
|
const bindings = companyIds.length > 0
|
|
? await tx
|
|
.select()
|
|
.from(builtInManagedResources)
|
|
.where(and(
|
|
inArray(builtInManagedResources.companyId, companyIds),
|
|
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
|
|
eq(builtInManagedResources.resourceId, existing.id),
|
|
))
|
|
: [];
|
|
const archiveToken = randomUUID();
|
|
const archivedMetadata = {
|
|
...(existing.metadata ?? {}),
|
|
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]: archiveToken,
|
|
};
|
|
const archived = await tx
|
|
.update(environments)
|
|
.set({
|
|
status: "archived",
|
|
metadata: archivedMetadata,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(environments.id, existing.id), eq(environments.status, "active")))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!archived) return null;
|
|
|
|
// Archival is a Paperclip-owned availability transition. Record only
|
|
// that status change in each installed baseline. Deriving the new hash
|
|
// from defaultsJson keeps operator-modified row fields out of stock.
|
|
for (const binding of bindings) {
|
|
const archivedStock = {
|
|
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
|
|
status: "archived",
|
|
};
|
|
await tx
|
|
.update(builtInManagedResources)
|
|
.set({
|
|
stockHash: stockHash(archivedStock),
|
|
defaultsJson: {
|
|
...archivedStock,
|
|
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]: archiveToken,
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(
|
|
eq(builtInManagedResources.id, binding.id),
|
|
eq(builtInManagedResources.resourceId, existing.id),
|
|
));
|
|
}
|
|
return toEnvironment(archived);
|
|
});
|
|
if (archived) {
|
|
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
|
|
companyId,
|
|
actorType: "system",
|
|
actorId: "managed-environment-reconciler",
|
|
action: "environment.managed_provider_unavailable_archived",
|
|
entityType: "environment",
|
|
entityId: archived.id,
|
|
details: { provider: input.provider },
|
|
})));
|
|
}
|
|
return archived;
|
|
};
|
|
|
|
return {
|
|
recordSandboxPlatformProbe: async (input: {
|
|
environmentId: string;
|
|
companyId: string;
|
|
config: Record<string, unknown>;
|
|
proof: Record<string, unknown> | null;
|
|
}): Promise<boolean> => {
|
|
const existing = sql`coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)`;
|
|
const previous = sql`${existing}->${input.companyId}`;
|
|
const probes = input.proof === null
|
|
? sql`case when ${previous}->>'v' = '2'
|
|
then jsonb_set(${existing}, ARRAY[${input.companyId}]::text[], ${previous} || '{"invalidated":true}'::jsonb, true)
|
|
else ${existing} - ${input.companyId} end`
|
|
: sql`${existing} || ${JSON.stringify({ [input.companyId]: input.proof })}::jsonb`;
|
|
const rows = await db.update(environments).set({
|
|
metadata: sql`jsonb_set(coalesce(${environments.metadata}, '{}'::jsonb), ARRAY[${SANDBOX_PLATFORM_PROBE_METADATA_KEY}]::text[], ${probes}, true)`,
|
|
}).where(and(eq(environments.id, input.environmentId), eq(environments.driver, "sandbox"), eq(environments.config, input.config))).returning({ id: environments.id });
|
|
return rows.length > 0;
|
|
},
|
|
list: async (
|
|
companyIdOrFilters?: string | EnvironmentListFilters,
|
|
maybeFilters?: EnvironmentListFilters,
|
|
): Promise<Environment[]> => {
|
|
const filters = resolveListFilters(companyIdOrFilters, maybeFilters);
|
|
const conditions = [];
|
|
if (filters.status) conditions.push(eq(environments.status, filters.status));
|
|
if (filters.driver) conditions.push(eq(environments.driver, filters.driver));
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(conditions.length > 0 ? and(...conditions) : undefined)
|
|
.orderBy(desc(environments.updatedAt), desc(environments.createdAt));
|
|
return rows.map(toEnvironment);
|
|
},
|
|
|
|
getById: async (id: string): Promise<Environment | null> => {
|
|
const row = await db.select().from(environments).where(eq(environments.id, id)).then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/**
|
|
* List the companies that own an environment through a built-in managed
|
|
* resource binding. A managed sandbox row binds to each company that the
|
|
* instance provisions it for. An operator-created environment has no
|
|
* binding, so this returns an empty list. The caller treats an empty list
|
|
* as an instance-global environment with no company owner.
|
|
*/
|
|
listBoundCompanyIds: async (environmentId: string): Promise<string[]> => {
|
|
const rows = await db
|
|
.select({ companyId: builtInManagedResources.companyId })
|
|
.from(builtInManagedResources)
|
|
.where(
|
|
and(
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceId, environmentId),
|
|
),
|
|
);
|
|
return Array.from(new Set(rows.map((row) => row.companyId)));
|
|
},
|
|
|
|
getLeaseById: async (id: string): Promise<EnvironmentLease | null> => {
|
|
const row = await db
|
|
.select()
|
|
.from(environmentLeases)
|
|
.where(eq(environmentLeases.id, id))
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Idempotently ensure THE local-driver environment row; the partial
|
|
* unique index `environments_local_driver_idx` enforces at most one per
|
|
* instance.
|
|
*
|
|
* On a cloud-managed instance an existing row is additionally ADOPTED —
|
|
* stamped `managedByPaperclip: true` (other metadata preserved) — so the
|
|
* single local slot is platform-owned there by construction, mirroring
|
|
* `ensureManagedSandboxEnvironment`'s adoption of the sandbox slot. This
|
|
* is what lets the environment-routes write floor treat a local row's
|
|
* platform markers as live state rather than a stale leftover: every
|
|
* caller (company creation, the heartbeat, run orchestration) converges
|
|
* the marker. Self-hosted instances keep the historical behavior:
|
|
* an existing row is returned untouched.
|
|
*/
|
|
ensureLocalEnvironment: async (_companyId?: string): Promise<Environment> => {
|
|
const now = new Date();
|
|
const insert = () =>
|
|
db
|
|
.insert(environments)
|
|
.values({
|
|
name: DEFAULT_LOCAL_ENVIRONMENT_NAME,
|
|
description: DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION,
|
|
driver: "local",
|
|
status: "active",
|
|
config: {},
|
|
envVars: {},
|
|
metadata: {
|
|
managedByPaperclip: true,
|
|
defaultForInstance: true,
|
|
},
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoNothing({
|
|
target: [environments.driver],
|
|
where: sql`${environments.driver} = 'local'`,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
const row = await insert().catch((error: unknown) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
return null;
|
|
}
|
|
throw error;
|
|
});
|
|
if (row) return toEnvironment(row);
|
|
|
|
const existing = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "local"))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!existing) {
|
|
throw new Error("Failed to ensure local environment");
|
|
}
|
|
const existingMetadata = (existing.metadata ?? {}) as Record<string, unknown>;
|
|
if (isCloudManagedInstance() && existingMetadata.managedByPaperclip !== true) {
|
|
const adopted = await db
|
|
.update(environments)
|
|
.set({
|
|
metadata: { ...existingMetadata, managedByPaperclip: true },
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environments.id, existing.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? existing);
|
|
return toEnvironment(adopted);
|
|
}
|
|
return toEnvironment(existing);
|
|
},
|
|
|
|
ensureManagedSandboxEnvironment,
|
|
|
|
archiveManagedSandboxEnvironment,
|
|
|
|
/**
|
|
* Idempotently ensure a managed Kubernetes sandbox environment exists for
|
|
* an instance, configured from instance/operator-supplied config. A thin
|
|
* wrapper over `ensureManagedSandboxEnvironment` that pins the provider to
|
|
* "kubernetes" and stamps the legacy marker `findKubernetesEnvironment`
|
|
* keys on. On subsequent calls stock-controlled config advances in place;
|
|
* operator modifications remain untouched for explicit review.
|
|
*/
|
|
ensureKubernetesEnvironment: async (
|
|
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
|
|
maybeConfig?: KubernetesEnvironmentConfigInput,
|
|
): Promise<Environment> => {
|
|
const config = resolveKubernetesConfig(companyIdOrConfig, maybeConfig);
|
|
return ensureManagedSandboxEnvironment({
|
|
companyId: typeof companyIdOrConfig === "string" ? companyIdOrConfig : undefined,
|
|
name: DEFAULT_KUBERNETES_ENVIRONMENT_NAME,
|
|
description: DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION,
|
|
provider: KUBERNETES_PROVIDER_KEY,
|
|
config,
|
|
extraMetadata: { [KUBERNETES_MANAGED_MARKER]: true },
|
|
}).then((result) => result.environment);
|
|
},
|
|
|
|
/**
|
|
* Find the active managed Kubernetes sandbox environment, if one
|
|
* exists. Read-only counterpart to `ensureKubernetesEnvironment` used by the
|
|
* per-run execution guard (which must not silently create config-less envs).
|
|
*/
|
|
findKubernetesEnvironment: async (_companyId?: string): Promise<Environment | null> => {
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(
|
|
and(
|
|
eq(environments.driver, "sandbox"),
|
|
eq(environments.status, "active"),
|
|
),
|
|
)
|
|
.orderBy(desc(environments.updatedAt));
|
|
const match = rows.find(
|
|
(row) =>
|
|
(row.metadata as Record<string, unknown> | null)?.[KUBERNETES_MANAGED_MARKER] === true,
|
|
);
|
|
return match ? toEnvironment(match) : null;
|
|
},
|
|
|
|
/**
|
|
* Find the platform-managed sandbox environment (the single
|
|
* `managedByPaperclip`-marked slot row), if one exists. Read-only
|
|
* counterpart to `ensureManagedSandboxEnvironment`. The default
|
|
* (active-only) form serves the managed-sandbox-only run guard — which
|
|
* must fail closed rather than create a config-less environment when
|
|
* the slot is empty or archived (the provisioner archives it while its
|
|
* provider plugin is down). `includeArchived` serves reconciliation
|
|
* cleanup, which must find the row even after the provisioner archived
|
|
* it.
|
|
*/
|
|
findManagedSandboxEnvironment: async (
|
|
_companyId?: string,
|
|
options?: { includeArchived?: boolean },
|
|
): Promise<Environment | null> => {
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(
|
|
options?.includeArchived === true
|
|
? eq(environments.driver, "sandbox")
|
|
: and(
|
|
eq(environments.driver, "sandbox"),
|
|
eq(environments.status, "active"),
|
|
),
|
|
)
|
|
.orderBy(desc(environments.updatedAt));
|
|
const match = rows.find(
|
|
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
);
|
|
return match ? toEnvironment(match) : null;
|
|
},
|
|
|
|
create: async (
|
|
companyIdOrInput: string | CreateEnvironment,
|
|
maybeInput?: CreateEnvironment,
|
|
options?: { db?: EnvironmentWriteDb },
|
|
): Promise<Environment> => {
|
|
const input = resolveCreateInput(companyIdOrInput, maybeInput);
|
|
const now = new Date();
|
|
const row = await (options?.db ?? db)
|
|
.insert(environments)
|
|
.values({
|
|
name: input.name,
|
|
description: input.description ?? null,
|
|
driver: input.driver,
|
|
status: input.status ?? "active",
|
|
config: input.config ?? {},
|
|
envVars: (input as CreateEnvironment & { envVars?: Record<string, unknown> }).envVars ?? {},
|
|
metadata: input.metadata ?? null,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null)
|
|
.catch((error) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
throw conflict(`An environment named "${input.name}" already exists for this instance.`);
|
|
}
|
|
if (hasConstraintName(error, "environments_local_driver_idx")) {
|
|
throw conflict("A local environment already exists for this instance.");
|
|
}
|
|
throw error;
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to create environment");
|
|
}
|
|
return toEnvironment(row);
|
|
},
|
|
|
|
update: async (
|
|
id: string,
|
|
patch: UpdateEnvironment,
|
|
options?: { db?: EnvironmentWriteDb },
|
|
): Promise<Environment | null> => {
|
|
const writeDb = options?.db ?? db;
|
|
const values: Partial<typeof environments.$inferInsert> = {
|
|
updatedAt: new Date(),
|
|
};
|
|
if (patch.name !== undefined) values.name = patch.name;
|
|
if (patch.description !== undefined) values.description = patch.description ?? null;
|
|
if (patch.driver !== undefined) values.driver = patch.driver;
|
|
if (patch.status !== undefined) values.status = patch.status;
|
|
if (patch.config !== undefined) values.config = patch.config;
|
|
if ("envVars" in patch && patch.envVars !== undefined) {
|
|
values.envVars = (patch.envVars ?? {}) as Record<string, unknown>;
|
|
}
|
|
const changesTarget = patch.config !== undefined || patch.driver !== undefined;
|
|
let metadataUpdate: SQL | undefined;
|
|
if (patch.metadata !== undefined) {
|
|
const userMetadata = withoutManagedEnvironmentArchiveToken(patch.metadata ?? null);
|
|
if (userMetadata) delete userMetadata[SANDBOX_PLATFORM_PROBE_METADATA_KEY];
|
|
if (changesTarget) {
|
|
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
|
|
metadataUpdate = preserveInvalidatedSnapshotProofs(replacement);
|
|
} else {
|
|
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
|
|
metadataUpdate = sql`case when ${environments.metadata} ? ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}
|
|
then coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(
|
|
${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY})
|
|
else ${replacement} end`;
|
|
}
|
|
} else if (changesTarget || patch.status !== undefined) {
|
|
metadataUpdate = sql`${environments.metadata} - ${MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY}`;
|
|
if (changesTarget) metadataUpdate = preserveInvalidatedSnapshotProofs(sql`${metadataUpdate} - ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}`);
|
|
}
|
|
|
|
const row = await writeDb
|
|
.update(environments)
|
|
.set({ ...values, ...(metadataUpdate ? { metadata: metadataUpdate } : {}) })
|
|
.where(eq(environments.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null)
|
|
.catch((error) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
throw conflict(`An environment named "${patch.name}" already exists for this instance.`);
|
|
}
|
|
if (hasConstraintName(error, "environments_local_driver_idx")) {
|
|
throw conflict("A local environment already exists for this instance.");
|
|
}
|
|
throw error;
|
|
});
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
remove: async (id: string): Promise<Environment | null> => {
|
|
const row = await db
|
|
.delete(environments)
|
|
.where(eq(environments.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
removeIfDeletable: async (id: string): Promise<Environment | null> => {
|
|
const row = await db
|
|
.delete(environments)
|
|
.where(
|
|
and(
|
|
eq(environments.id, id),
|
|
ne(environments.driver, "local"),
|
|
sql`not exists (
|
|
select 1 from ${instanceSettings}
|
|
where ${instanceSettings.defaultEnvironmentId} = ${environments.id}
|
|
)`,
|
|
// A `pending_cleanup` lease is the durable teardown reference for an
|
|
// orphan sandbox. The environment foreign key uses
|
|
// `on delete set null`, so a delete keeps the lease row but drops its
|
|
// environment reference. This predicate refuses the delete while such
|
|
// a lease exists, so the operator resolves the cleanup first and the
|
|
// lease keeps its environment link. It runs in the same statement as
|
|
// the delete, so it also closes the check-to-delete race.
|
|
sql`not exists (
|
|
select 1 from ${environmentLeases}
|
|
where ${environmentLeases.environmentId} = ${environments.id}
|
|
and ${environmentLeases.status} = 'pending_cleanup'
|
|
)`,
|
|
// A reusable lease keeps a provider sandbox after a run releases
|
|
// it, including when the run failed but release succeeded.
|
|
// Deleting the environment would set its reference to
|
|
// null, and both the normal release path and scoped reusable cleanup
|
|
// require that environment context. Refuse the delete atomically
|
|
// until the owning issue/workspace destroys the reusable sandbox.
|
|
sql`not exists (
|
|
select 1 from ${environmentLeases}
|
|
where ${environmentLeases.environmentId} = ${environments.id}
|
|
and ${environmentLeases.leasePolicy} = 'reuse_by_environment'
|
|
and ${environmentLeases.status} in ('active', 'released', 'retained', 'failed')
|
|
)`,
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Return true when the environment has one or more leases in the terminal
|
|
* `pending_cleanup` state. Each such lease is the only durable provider
|
|
* reference for an orphan sandbox that a teardown retry must destroy. The
|
|
* delete guard and the provider-change guard call this to protect that
|
|
* reference.
|
|
*/
|
|
hasUnresolvedPendingCleanupLeases: async (environmentId: string): Promise<boolean> => {
|
|
const rows = await db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, environmentId),
|
|
eq(environmentLeases.status, "pending_cleanup"),
|
|
),
|
|
);
|
|
return countFromRows(rows) > 0;
|
|
},
|
|
|
|
getDeleteBlastRadius: async (id: string): Promise<EnvironmentDeleteBlastRadius | null> => {
|
|
const environment = await db
|
|
.select({
|
|
id: environments.id,
|
|
driver: environments.driver,
|
|
})
|
|
.from(environments)
|
|
.where(eq(environments.id, id))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!environment) return null;
|
|
|
|
const [
|
|
instanceDefaultRows,
|
|
agentDefaultRows,
|
|
executionWorkspaceRows,
|
|
issueRows,
|
|
projectRows,
|
|
secretBindingRows,
|
|
activeLeaseRows,
|
|
pendingCleanupLeaseRows,
|
|
reusableSandboxLeaseRows,
|
|
activeSetupRows,
|
|
] = await Promise.all([
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.defaultEnvironmentId, id)),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(agents)
|
|
.where(eq(agents.defaultEnvironmentId, id)),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(executionWorkspaces)
|
|
.where(sql`${executionWorkspaces.metadata} -> 'config' ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(issues)
|
|
.where(sql`${issues.executionWorkspaceSettings} ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(projects)
|
|
.where(sql`${projects.executionWorkspacePolicy} ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(companySecretBindings)
|
|
.where(
|
|
and(
|
|
eq(companySecretBindings.targetType, "environment"),
|
|
eq(companySecretBindings.targetId, id),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
eq(environmentLeases.status, "active"),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
eq(environmentLeases.status, "pending_cleanup"),
|
|
),
|
|
),
|
|
db
|
|
.select({
|
|
leaseId: environmentLeases.id,
|
|
executionWorkspaceId: environmentLeases.executionWorkspaceId,
|
|
executionWorkspaceName: executionWorkspaces.name,
|
|
issueId: environmentLeases.issueId,
|
|
issueIdentifier: issues.identifier,
|
|
issueTitle: issues.title,
|
|
})
|
|
.from(environmentLeases)
|
|
.leftJoin(executionWorkspaces, eq(environmentLeases.executionWorkspaceId, executionWorkspaces.id))
|
|
.leftJoin(issues, eq(environmentLeases.issueId, issues.id))
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
|
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentCustomImageSetupSessions)
|
|
.where(
|
|
and(
|
|
eq(environmentCustomImageSetupSessions.environmentId, id),
|
|
inArray(environmentCustomImageSetupSessions.status, [...ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES]),
|
|
),
|
|
),
|
|
]);
|
|
|
|
const isManagedLocal = environment.driver === "local";
|
|
const isInstanceDefault = countFromRows(instanceDefaultRows) > 0;
|
|
const pendingCleanupLeaseCount = countFromRows(pendingCleanupLeaseRows);
|
|
const reusableSandboxLeaseHolders = reusableSandboxLeaseRows.map((row) => ({
|
|
leaseId: row.leaseId,
|
|
executionWorkspaceId: row.executionWorkspaceId,
|
|
executionWorkspaceName: row.executionWorkspaceName,
|
|
issueId: row.issueId,
|
|
issueIdentifier: row.issueIdentifier,
|
|
issueTitle: row.issueTitle,
|
|
}));
|
|
const reusableSandboxLeaseCount = reusableSandboxLeaseHolders.length;
|
|
const deleteBlockedReasons: EnvironmentDeleteBlockedReason[] = [];
|
|
if (isManagedLocal) deleteBlockedReasons.push("managed_local");
|
|
if (isInstanceDefault) deleteBlockedReasons.push("instance_default");
|
|
// A `pending_cleanup` lease is the durable teardown reference for an orphan
|
|
// sandbox. The environment foreign key uses `on delete set null`, so a
|
|
// delete keeps the lease but drops its environment reference. Block the
|
|
// delete until the sweep resolves the lease, so the operator resolves the
|
|
// cleanup first and the lease keeps its environment link.
|
|
if (pendingCleanupLeaseCount > 0) deleteBlockedReasons.push("pending_sandbox_cleanup");
|
|
if (reusableSandboxLeaseCount > 0) deleteBlockedReasons.push("reusable_sandbox_lease");
|
|
const activeLeaseCount = countFromRows(activeLeaseRows);
|
|
const activeCustomImageSetupSessionCount = countFromRows(activeSetupRows);
|
|
|
|
return {
|
|
environmentId: id,
|
|
canDelete: deleteBlockedReasons.length === 0,
|
|
deleteBlockedReasons,
|
|
pendingCleanupLeaseCount,
|
|
reusableSandboxLeaseCount,
|
|
reusableSandboxLeaseHolders,
|
|
staticReferences: {
|
|
isManagedLocal,
|
|
isInstanceDefault,
|
|
agentDefaultCount: countFromRows(agentDefaultRows),
|
|
executionWorkspaceSelectionCount: countFromRows(executionWorkspaceRows),
|
|
issueSelectionCount: countFromRows(issueRows),
|
|
projectSelectionCount: countFromRows(projectRows),
|
|
secretBindingCount: countFromRows(secretBindingRows),
|
|
},
|
|
activeRuntimeUse: {
|
|
activeLeaseCount,
|
|
activeCustomImageSetupSessionCount,
|
|
hasActiveRuntimeUse: activeLeaseCount > 0 || activeCustomImageSetupSessionCount > 0,
|
|
},
|
|
};
|
|
},
|
|
|
|
listLeases: async (
|
|
environmentId: string,
|
|
filters: {
|
|
status?: string;
|
|
} = {},
|
|
): Promise<EnvironmentLease[]> => {
|
|
const conditions = [eq(environmentLeases.environmentId, environmentId)];
|
|
if (filters.status) conditions.push(eq(environmentLeases.status, filters.status));
|
|
const rows = await db
|
|
.select()
|
|
.from(environmentLeases)
|
|
.where(and(...conditions))
|
|
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
|
|
return rows.map(toEnvironmentLease);
|
|
},
|
|
|
|
acquireLease: async (input: {
|
|
creationOperationId?: string;
|
|
companyId: string;
|
|
environmentId: string;
|
|
executionWorkspaceId?: string | null;
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
leasePolicy?: EnvironmentLeasePolicy;
|
|
provider?: string | null;
|
|
providerLeaseId?: string | null;
|
|
expiresAt?: Date | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
/**
|
|
* Atomically retire the previous database ownership record when this
|
|
* acquisition reuses the same provider resource for a new run. The
|
|
* provider resume happens before this write, so a failed transaction
|
|
* leaves the prior retained row recoverable instead of publishing two
|
|
* reusable owners for one sandbox.
|
|
*/
|
|
replacesReusableLeaseId?: string | null;
|
|
/**
|
|
* Reactivate the exact lease row already owned by this heartbeat run.
|
|
* Native same-run recovery re-enters environment startup after the
|
|
* provider process is interrupted; it must not manufacture a second
|
|
* database owner for the same run and provider resource.
|
|
*/
|
|
reusesReusableLeaseId?: string | null;
|
|
/**
|
|
* Re-check the environment company binding inside the lease insert
|
|
* transaction. The login routes set this to close the check-to-lease
|
|
* race: managed reconciliation can bind a sandbox to another company
|
|
* between the route guard and this acquire. When the environment is
|
|
* bound to a company other than `companyId`, the insert throws the 403
|
|
* `environment_company_mismatch` and no lease row is created (fail
|
|
* closed). An unbound (instance-global) environment stays open to every
|
|
* member. All other callers keep the plain, non-transactional insert.
|
|
*/
|
|
assertCompanyBinding?: boolean;
|
|
}): Promise<EnvironmentLease> => {
|
|
if (input.creationOperationId) {
|
|
const { creationOperationId, ...leaseInput } = input;
|
|
return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId },
|
|
tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true }));
|
|
}
|
|
const now = new Date();
|
|
const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null;
|
|
const values = {
|
|
...(replacementLeaseId ? { id: replacementLeaseId } : {}),
|
|
companyId: input.companyId,
|
|
environmentId: input.environmentId,
|
|
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
|
issueId: input.issueId ?? null,
|
|
heartbeatRunId: input.heartbeatRunId ?? null,
|
|
status: "active" as const,
|
|
leasePolicy: input.leasePolicy ?? "ephemeral",
|
|
provider: input.provider ?? null,
|
|
providerLeaseId: input.providerLeaseId ?? null,
|
|
acquiredAt: now,
|
|
lastUsedAt: now,
|
|
expiresAt: input.expiresAt ?? null,
|
|
releasedAt: null,
|
|
failureReason: null,
|
|
cleanupStatus: null,
|
|
metadata: withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata)),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
};
|
|
if (
|
|
(input.replacesReusableLeaseId || input.reusesReusableLeaseId) &&
|
|
((!input.executionWorkspaceId && (!input.issueId || typeof input.metadata?.agentId !== "string")) || !input.providerLeaseId)
|
|
) {
|
|
throw new Error(
|
|
"A reusable lease handoff requires a workspace or agent-bound task, and a provider lease id.",
|
|
);
|
|
}
|
|
if (input.reusesReusableLeaseId && !input.heartbeatRunId) {
|
|
throw new Error(
|
|
"A same-run reusable lease reacquisition requires a heartbeat run id.",
|
|
);
|
|
}
|
|
if (input.replacesReusableLeaseId && input.reusesReusableLeaseId) {
|
|
throw new Error(
|
|
"A reusable lease cannot be replaced and reacquired in the same operation.",
|
|
);
|
|
}
|
|
const row =
|
|
input.assertCompanyBinding ||
|
|
input.replacesReusableLeaseId ||
|
|
input.reusesReusableLeaseId
|
|
? await db.transaction(async (tx) => {
|
|
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
|
|
companyId: input.companyId, runId: input.heartbeatRunId,
|
|
});
|
|
if (input.assertCompanyBinding) {
|
|
await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId);
|
|
}
|
|
if (input.replacesReusableLeaseId) {
|
|
await notifyChatInboundWakeNotices(tx, { companyId: input.companyId });
|
|
const retired = await tx
|
|
.update(environmentLeases)
|
|
.set({
|
|
status: "expired",
|
|
releasedAt: now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
cleanupStatus: "success",
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.id, input.replacesReusableLeaseId),
|
|
eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.environmentId, input.environmentId),
|
|
sql`${environmentLeases.provider} is not distinct from ${input.provider ?? null}`,
|
|
input.executionWorkspaceId
|
|
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
|
|
: and(
|
|
isNull(environmentLeases.executionWorkspaceId),
|
|
eq(environmentLeases.issueId, input.issueId!),
|
|
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
|
|
),
|
|
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
|
eq(
|
|
environmentLeases.providerLeaseId,
|
|
input.providerLeaseId!,
|
|
),
|
|
inArray(environmentLeases.status, [
|
|
"released",
|
|
"retained",
|
|
]),
|
|
),
|
|
)
|
|
.returning();
|
|
if (retired.length !== 1) {
|
|
throw conflict(
|
|
"Reusable sandbox lease ownership changed during acquisition.",
|
|
);
|
|
}
|
|
const handoff = createReusableLeaseHandoff(retired[0]!, { ...values, id: replacementLeaseId! });
|
|
values.metadata = { ...values.metadata, reusableLeasePredecessor: handoff };
|
|
await tx.update(environmentLeases).set({ metadata: {
|
|
...retired[0]!.metadata, reusableLeaseSuccessor: handoff,
|
|
} }).where(eq(environmentLeases.id, retired[0]!.id));
|
|
}
|
|
if (input.reusesReusableLeaseId) {
|
|
const [previous] = await tx.select({ metadata: environmentLeases.metadata }).from(environmentLeases)
|
|
.where(and(eq(environmentLeases.id, input.reusesReusableLeaseId), eq(environmentLeases.companyId, input.companyId))).for("update");
|
|
const reacquired = await tx
|
|
.update(environmentLeases)
|
|
.set({
|
|
status: "active",
|
|
releasedAt: null,
|
|
lastUsedAt: now,
|
|
expiresAt: input.expiresAt ?? null,
|
|
failureReason: null,
|
|
cleanupStatus: null,
|
|
metadata: preserveReusableLeaseHandoff(previous?.metadata ?? null, withoutCleanupVerification(input.metadata)),
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.id, input.reusesReusableLeaseId),
|
|
eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.environmentId, input.environmentId),
|
|
input.executionWorkspaceId
|
|
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
|
|
: and(
|
|
isNull(environmentLeases.executionWorkspaceId),
|
|
eq(environmentLeases.issueId, input.issueId!),
|
|
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
|
|
),
|
|
eq(
|
|
environmentLeases.heartbeatRunId,
|
|
input.heartbeatRunId!,
|
|
),
|
|
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
|
eq(
|
|
environmentLeases.providerLeaseId,
|
|
input.providerLeaseId!,
|
|
),
|
|
inArray(environmentLeases.status, [
|
|
"active",
|
|
"released",
|
|
"retained",
|
|
]),
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!reacquired) {
|
|
throw conflict(
|
|
"Reusable sandbox lease ownership changed during reacquisition.",
|
|
);
|
|
}
|
|
return reacquired;
|
|
}
|
|
return tx
|
|
.insert(environmentLeases)
|
|
.values(values)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
})
|
|
: await db.transaction(async tx => {
|
|
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
|
|
companyId: input.companyId, runId: input.heartbeatRunId,
|
|
});
|
|
return tx.insert(environmentLeases).values(values).returning().then(rows => rows[0] ?? null);
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to acquire environment lease");
|
|
}
|
|
return toEnvironmentLease(row);
|
|
},
|
|
|
|
releaseLease: async (
|
|
id: string,
|
|
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed" | "retained" | "pending_cleanup"> = "released",
|
|
options?: {
|
|
expectedPendingCleanupAttemptId?: string;
|
|
failureReason?: string;
|
|
cleanupStatus?: EnvironmentLeaseCleanupStatus;
|
|
remoteExecutionTermination?: Record<string, unknown>;
|
|
},
|
|
) => {
|
|
const now = new Date();
|
|
const releaseMetadata = options?.remoteExecutionTermination
|
|
? sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ remoteExecutionTermination: options.remoteExecutionTermination })}::jsonb`
|
|
: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'`;
|
|
const cleanupSettled = options?.expectedPendingCleanupAttemptId && options.cleanupStatus === "success"
|
|
&& status !== "pending_cleanup";
|
|
const row = await db.transaction(async tx => {
|
|
const condition = and(eq(environmentLeases.id, id), options?.expectedPendingCleanupAttemptId
|
|
? and(eq(environmentLeases.status, "pending_cleanup"),
|
|
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.expectedPendingCleanupAttemptId}`)
|
|
: undefined);
|
|
const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId,
|
|
heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status,
|
|
cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt,
|
|
creationOperationId: sql<string | null>`${environmentLeases.metadata}->>'creationOperationId'`,
|
|
}).from(environmentLeases).where(condition).for("update");
|
|
if (!previous) return null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined });
|
|
if (previous.heartbeatRunId && (previous.status !== status ||
|
|
(options?.cleanupStatus !== undefined && previous.cleanupStatus !== options.cleanupStatus) ||
|
|
(status !== "retained" && !previous.releasedAt))) {
|
|
await notifyChatInboundWakeNotices(tx, {
|
|
companyId: previous.companyId, issueId: previous.issueId ?? undefined,
|
|
});
|
|
}
|
|
if (previous.heartbeatRunId && ["active", "pending_cleanup"].includes(previous.status)
|
|
&& status !== "pending_cleanup") {
|
|
await notifyRetainedNativeSessionCleanup(tx, {
|
|
companyId: previous.companyId, runId: previous.heartbeatRunId,
|
|
});
|
|
}
|
|
if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
|
|
await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
|
|
.where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId),
|
|
eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending")));
|
|
}
|
|
return tx.update(environmentLeases)
|
|
.set({
|
|
status,
|
|
releasedAt: status === "retained" ? null : now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
...(options?.failureReason !== undefined ? { failureReason: options.failureReason } : {}),
|
|
...(options?.cleanupStatus !== undefined ? { cleanupStatus: options.cleanupStatus } : {}),
|
|
// A later release without a receipt cannot reuse an earlier stop's
|
|
// authority (for example after a same-run lease resume).
|
|
metadata: cleanupSettled
|
|
? sql`coalesce(${releaseMetadata}, '{}'::jsonb) || '{"pendingCleanupInFlight":false,"pendingCleanupLeaseExpiresAtMs":0}'::jsonb`
|
|
: releaseMetadata,
|
|
})
|
|
.where(condition)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Record a lease-less orphan sandbox directly in the terminal
|
|
* `pending_cleanup` state with one atomic insert. The sweep reads a row with
|
|
* status `pending_cleanup` and cleanup status `failed`, so this insert makes
|
|
* the orphan visible to recovery immediately. It never passes through the
|
|
* `active` state, so a process or database crash cannot strand the row in an
|
|
* intermediate state that no sweep finds. The insert skips the company
|
|
* binding assertion, so it records the orphan even for a foreign-bound
|
|
* environment.
|
|
*
|
|
* The insert runs in a transaction that first locks the environment row with
|
|
* `for update`. The delete path (`removeIfDeletable`) refuses a delete while a
|
|
* `pending_cleanup` lease exists, so this lock serializes the two writes:
|
|
*
|
|
* - The environment still exists: the insert records the orphan with the
|
|
* environment reference. A concurrent delete blocks on the lock, then its
|
|
* `not exists (pending_cleanup)` predicate fails, so the delete refuses.
|
|
* - The environment is already gone (a delete won the race before this
|
|
* insert): the insert records the orphan with a null environment
|
|
* reference. The row carries the immutable provider metadata the sweep
|
|
* needs, so the teardown still runs. This closes the window the finding
|
|
* describes, where the foreign-key insert fails after a delete.
|
|
*/
|
|
insertPendingCleanupLease: async (input: {
|
|
companyId: string;
|
|
environmentId: string;
|
|
executionWorkspaceId?: string | null;
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
provider?: string | null;
|
|
providerLeaseId?: string | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
failureReason: string;
|
|
}): Promise<EnvironmentLease> => {
|
|
const now = new Date();
|
|
const row = await db.transaction(async (tx) => {
|
|
// Lock the environment row so a concurrent delete cannot commit between
|
|
// this read and the insert. An absent row means a delete already removed
|
|
// the environment, so record the orphan with a null reference.
|
|
const environmentRows = await tx
|
|
.select({ id: environments.id })
|
|
.from(environments)
|
|
.where(eq(environments.id, input.environmentId))
|
|
.for("update");
|
|
const environmentIdForRow = environmentRows[0]?.id ?? null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: input.companyId, runId: input.heartbeatRunId ?? undefined });
|
|
return tx
|
|
.insert(environmentLeases)
|
|
.values({
|
|
companyId: input.companyId,
|
|
environmentId: environmentIdForRow,
|
|
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
|
issueId: input.issueId ?? null,
|
|
heartbeatRunId: input.heartbeatRunId ?? null,
|
|
status: "pending_cleanup" as const,
|
|
leasePolicy: "ephemeral",
|
|
provider: input.provider ?? null,
|
|
providerLeaseId: input.providerLeaseId ?? null,
|
|
acquiredAt: now,
|
|
lastUsedAt: now,
|
|
expiresAt: null,
|
|
releasedAt: now,
|
|
failureReason: input.failureReason,
|
|
cleanupStatus: "failed",
|
|
metadata: withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata)),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to record pending sandbox cleanup lease");
|
|
}
|
|
return toEnvironmentLease(row);
|
|
},
|
|
|
|
updateLeaseMetadata: async (
|
|
id: string,
|
|
metadata: Record<string, unknown> | null,
|
|
): Promise<EnvironmentLease | null> => {
|
|
const row = await db.transaction(async tx => {
|
|
const [lease] = await tx.select({ companyId: environmentLeases.companyId, heartbeatRunId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata })
|
|
.from(environmentLeases).where(eq(environmentLeases.id, id)).for("update");
|
|
if (!lease) return null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined });
|
|
return tx.update(environmentLeases)
|
|
.set({
|
|
metadata: preserveReusableLeaseHandoff(lease.metadata, withoutCleanupVerification(metadata)),
|
|
lastUsedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environmentLeases.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
releaseLeasesForRun: async (
|
|
heartbeatRunId: string,
|
|
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
|
|
): Promise<EnvironmentLease[]> => {
|
|
const now = new Date();
|
|
const rows = await db.transaction(async tx => {
|
|
const [run] = await tx.select({ companyId: heartbeatRuns.companyId }).from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
|
|
if (run) {
|
|
await notifyEnvironmentCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
|
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
|
}
|
|
return tx.update(environmentLeases)
|
|
.set({
|
|
status,
|
|
releasedAt: now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.heartbeatRunId, heartbeatRunId),
|
|
eq(environmentLeases.status, "active"),
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
return rows.map(toEnvironmentLease);
|
|
},
|
|
};
|
|
}
|