mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. The `stranded-assignments` and `assigned-work-invariant` queues check assigned tasks that have no live run. One stranded check costs about 19 database statements. > - #21552 scoped run hints to the run's task graph. #21729 made the 5-minute backstop re-check only changed tasks. A hint with only a company id still makes the queue re-check every open task of the company, 25 per pass. > - Under load on staging (7 native runs), the stranded check ran about 6.7 times per second, and its statement bundle was more than a third of all statements. An activity fixture shows the cause: ordinary run writers send a company-only hint once per run, and one company hint costs about 850 checks. At 1,000 agents this work never stops. > - This pull request makes those writers name the run (or the task) they already know. A run that has no task now resolves to no tasks instead of the whole company. The changed-tasks backstop no longer counts an agent's own run heartbeat as a change to every task of that agent. > - The benefit: with 7 runs a minute plus one taskless run, stranded checks drop from 18.4 to 1.0 per second and stranded-check statements from 344 to 7 per second. Real company-wide changes (pause lifted, budget, company status, tree holds) keep the full company pass. ## Linked Issues or Issue Description Refs #21729, #21552, #21676, #21724 **What existing behavior does this improve?** The issue recovery task queues (`stranded-assignments`, `assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`) and the writers that wake them. **Subsystem affected** Server: issue recovery background work and the run lifecycle writers that publish recovery hints. **Current behavior** Many run lifecycle writers know the run but publish a company-only hint, for example `notifyExecutionRunWork(tx, { companyId })` in the native finalizer and projection, `settleNativeTerminalWake`, the claimed-wake settlement, and an environment lease release of a run without a task. A queued run without a task publishes a company hint. A graph hint for a run without a task resolves to "whole company". A comment sends a company hint to `assigned-work-invariant` and `resolved-dependencies`. Each of these re-checks all open tasks of the company. Also, the changed-tasks backstop treats any change to the assignee agent's row as a change to all of that agent's tasks, and every run start and finish writes that row. **Proposed behavior** A writer that knows the run sends the run's graph hint. A writer that knows the task sends the task hint. A run without a task affects no task queue entry. An agent row change that is only the agent's own run heartbeat does not mark its other tasks changed. Company-wide facts keep the company hint. **Reason and benefit** The work is now proportional to activity, not to (activity x history). See Verification. ## What Changed - `server/src/services/issue-recovery-notifications.ts`: - `notifyIssueRecovery` with a task sends the task hint to all four task queues (before: only to `stranded-assignments` and `stale-issue-locks`; `assigned-work-invariant` and `resolved-dependencies` got the company hint). Only comments (`issue-comment-writes.ts`) call it with a task. - `resolveIssueGraph` returns `[]` when a run-only graph has no task. A missing task seed still returns `null` (whole company). - `server/src/services/recovery/work.ts`: a load for an empty graph returns no rows. The `assigned-work-invariant` queue emits the same `check` debug event as `stranded-assignments` (with `PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load reason. - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent part of the changed-tasks condition skips an agent whose last write was its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or configuration write (no heartbeat stamp) still counts. - `server/src/services/heartbeat/queue.ts`: a queued run without a task gets its id before the insert and publishes a run hint, not a company hint. The daily-cap comment release publishes the run's graph hint. - `server/src/services/environments.ts`, `work-intent-outbox.ts`, `chat-inbound-wake-notice-notifications.ts`: a lease release passes its run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a run to that run's graph. `notifyChatInboundWakeNotices` accepts an optional `runId` for its settlement hint. - Native and recovery run writers now pass the run id they already have to `notifyExecutionRunWork`, `writeExecutionControl` or `notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3), `native-run-projection.ts` (2), `native-session-executor.ts` (3), `wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`, `native-safe-replacement.ts`, `native-restart-recovery.ts` (2), `native-failed-workspace-recovery.ts`, `native-recovery-preparation-failure.ts`, `execution-control-reconciliation.ts`, `execution-recovery-resolution.ts` (2). These are the native equivalents of the legacy run paths in `heartbeat/queue.ts`, which already pass the run id. - Tests: - `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new): staging-shaped company (780 in-review tasks with succeeded native runs, 40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, a fake clock, and real process-adapter runs. Each run is woken by a user comment through `heartbeat.wakeup`, starts, calls back to post an agent comment and move the task to `in_review`, and finishes. One (or five) taskless runs a minute run beside them. It counts checks by queue and reason, task-queue signals by caller, and statements by owner. It asserts the workload (runs started and succeeded, agent comments, run events, zero agent write failures, at most 10% failed runs), then no company check and no company-only task-queue signal. In CI it runs the 7-runs-a-minute workload for 4 minutes (about 1 minute of real time). With `RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length, attributes statements by stack, and writes the numbers below to that file. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests for each narrowed writer (see Verification). ### Publishers of company-only (or `*`) hints to the recovery task queues "Per" is how often the writer fires on a busy instance. "Known" says whether the run or task is known at the call site. Narrowed in this pull request: | Call site | Event | Per | Known | Now | | --- | --- | --- | --- | --- | | `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task (timer, manual, chat without a task) | each taskless run | run (id generated) | run graph (no tasks) | | `environments.ts:1780` `releaseLease` -> `notifyChatInboundWakeNotices` | environment lease released | each run without a task | run | run graph | | `environments.ts:1792` `releaseLease` -> `promoteDeferredWorkIntents(lock_released)` | lease released | each run without a task | run | run graph | | `environments.ts:1934` `releaseLeasesForRun` | leases of a run released | each run without a task | run | run graph | | `issue-comment-writes.ts:12` (`assigned-work-invariant`, `resolved-dependencies` only) | comment | each comment | task | task | | `native-runtime/native-run-projection.ts:54` `settleNativeTerminalWake` | native run's wake claim settled | each native run finish | run | run graph | | `native-runtime/native-run-projection.ts:177` | native run status projected | each native run finish | run | run graph | | `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native finalization terminal / failure / projection | each native run finish | run | run graph | | `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284` | native execution control deadline, cancelled retry, pre-result settlement | each native run end | run | run graph | | `recovery/wake-request-claimed-settlement.ts:67` | claimed wake settled after the run | each run whose claim is reconciled | run | run graph | | `heartbeat/queue.ts:590` | daily-cap comment key release | each run cancelled at the daily cap | run | run graph | | `native-runtime/native-superseded-retry.ts:42`, `native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`, `:1935`, `native-failed-workspace-recovery.ts:164`, `native-recovery-preparation-failure.ts:30`, `execution-control-reconciliation.ts:270`, `execution-recovery-resolution.ts:830`, `:965` | native recovery terminal writes | each recovered run | run | run graph | | `resolveIssueGraph` for any run-only hint of a run without a task (for example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle | each taskless run event | run | no tasks | Left with the company hint (company-wide fact, or the graph is not known): | Call site | Event | Per | Why kept | | --- | --- | --- | --- | | `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy changed, scope paused or resumed | operator or budget event | eligibility of every task can change | | `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`, `scheduling_resumed`, `reconciler_backstop`, and `accounting_settled` with an active hard stop | hold release | operator event; `accounting_settled` once per run only with a hard-stop policy | can admit any task; guarded by "keeps the company pass for hold releases that can admit any task" | | `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company created, paused, resumed, archived | operator | company-wide | | `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`, `agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent paused, resumed, configured, deleted | operator | can change many tasks, and other agents' review paths | | `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`, `board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and authority changes | operator, first sign-in | company authority | | `chat-channels/endpoints.ts:461`, `identity-links.ts:323`, `resource-reconciliation.ts:290`, `slack/setup-services.ts:50`, `chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50` | chat authority changes | operator | company authority | | `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting changed | operator | instance-wide | | `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold created or released | operator | many tasks | | `company-deletion.ts:83`, `company-portability.ts:6560` | company delete, import | rare | many tasks | | `issues.ts:8015` (blockers replaced), `:11165` (task created), `:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph changes | per task create or blocker edit | graph moves; #21552 kept these | | `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete and restore | rare | #21552 kept these | | `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552 kept these | | `system-recovery.ts:17`, `:70`, `task-retry.ts:294`, `workspace-recovery-capture-policy.ts:59`, `workspace-recovery-stop-proof.ts:122` | system and workspace recovery | per recovery | not shown to stay in one graph | | `agent-conversations.ts:607`, `chat-channels/telegram/attachment-failure.ts:83`, `native-runtime/chat-attachment-reuse.ts:1754`, `native-runner-file-handoff.ts:1317` | conversation and attachment paths | per failure | not shown to stay in one graph | | `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`, `routes/issues.ts:14552`, `issues.ts:12070`, `:12171`, `heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`, `environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) | saved notice and wake changes | per manual wake or project change | company admission state; not reached by run activity in the fixture | | `heartbeat/run-state.ts:1746`, `native-harness-backup-expiration.ts:356`, `:526`, `explicit-native-continuation.ts:705` | session goal clear, backup expiry, explicit continuation | rare | no run id at the call site | | `fast-responses.ts:201`, `:559`, `:672`, `:1053`, `chat-github-response-comments.ts:62`, `:229`, `:440`, `chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a delivery only) | chat response availability | per chat response | no task at the call site; not measured here (chat paths are not in the fixture) | Per-run-event path: appending run events, saving native checkpoints, renewing leases and streaming deltas publish no recovery task-queue hint. `heartbeat-run-events.ts` reaches only live-run recovery queues (`native.process_*`, `turn.*` events), chat progress, response presentation, and `notifyNativeStatusReassessment` for control-plane evidence events only. Checkpoints and lease renewals reach `notifyLiveRunRecovery`, which does not include the task queues. A run's start and finish publish its graph hint from several writers (about 20 task-queue signals per legacy run in the fixture, for example `notifyChatRunMilestones` in `run-identity.ts` activation, `claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce into about 3 checks of the run's own task per run. That is not one hint, but it is bounded by the run's graph. This pull request does not change it. ## Verification - Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per second by queue and reason, and statements per second. Each workload minute is exactly 60 s of fake-clock time, and the rates use the elapsed fake-clock time. "Before" is this branch's test against the `origin/master` service files. Both versions did the same work (7 runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620 to 622 succeeded, about 590 agent comments, about 5,550 run events). The test now asserts this workload and zero agent write failures before it accepts the counts. | | master | this branch | | --- | --- | --- | | 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30 company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company | | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 | | 7 runs/min, all statements/s | 577.5 | 220.4 | | 7 runs/min, company-only task-queue signals | 710 | 0 | | 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86 company + 0.31 hint | 1.05 hint, 0 company | | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 | | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 | | 50 runs/min, company-only task-queue signals | 2,864 | 0 | "Statements in stranded checks" counts statements whose stack includes `reconcileStrandedAssignedIssues`. The other statements are the runs themselves. Recovery load queries (graph lookups and hint loads) are 12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions. `stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both versions. - Without the taskless runs, master already showed the backstop degrade under activity: with 7 runs a minute, 2.46 backstop checks/s, because every run writes its agent's row and marks all of that agent's tasks changed. With this branch: 0.27/s. - New tests in `issue-recovery-deadlines.test.ts`, red on `origin/master` service files and green here: - "re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task": for `notifyRunDispatch`, `notifyExecutionRunWork`, `notifyExternalConversationSettlement`, `notifyChatInboundWakeNotices` and `promoteDeferredWorkIntents(lock_released)` with a run id, the run's graph (task, parent, dependent) is checked within one pass and none of 300 backlog tasks is. For a run without a task, no task is checked. A comment checks only its task in `stranded-assignments` and `assigned-work-invariant`. - "counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change". - "resolves a run without a task to no tasks and a missing task to the whole company". - Unchanged and passing: the #21729 pickup tests in `issue-recovery-idle-backstop.test.ts` (finished run, user comment, raised budget, resumed agent within a minute; missed hint within the backstop bound; outside change within the hourly pass), the #21552 tests in `issue-recovery-deadlines.test.ts` (including "keeps the company pass for hold releases that can admit any task" and "falls back to the company pass when a run affects more tasks than a hint names"). - Regression, embedded PostgreSQL: `issue-recovery-deadlines`, `issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`, `issue-recovery-work-lifecycle` (42 tests); `native-run-projection-work`, `native-recovery-notice-work`, `muse-terminal-settlement`, `heartbeat-held-comment-wake`, `superseded-workspace-export-recovery`, `native-recovery-preparation-failure`, `upgrade-readiness-post-result`, `chat-wake-conversation-batch`, `environment-service`, `retained-native-session-cleanup-work`, `slack-conversation-lifecycle`, `workspace-recovery-dependency-finalize`, `native-failed-run-owner-release`, `chat-run-milestone-work` (462 tests). - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **Recovery time bounds. The owner must see this.** No hint-driven pickup becomes slower for a task in the run's graph. Two missed-hint bounds change: - An agent change that sends no hint and is followed by a run heartbeat of the same agent inside the 5-minute window (for example a resume whose hint was lost, after which the agent ran another task) is now found by the hourly full pass, not the 5-minute changed-tasks pass. A normal resume sends `pause_lifted` and is checked at once (tested). - A task outside the run's graph whose recovery depended on the end of a taskless run, or on the native finish writers above, is now found by its own hint, the 5-minute changed-tasks pass (its own rows), or the hourly full pass. Before, any run finish re-checked the whole company. I found no recovery decision that waits on another run of the agent: concurrency waits are queued runs that dispatch resumes. - A run-only hint for a run row that does not exist resolves to no tasks (it was the whole company). A run row is not deleted while its tasks are open, except by company deletion, which sends the company hint. - Native finish writers now send run-scoped hints to the live-run recovery queues too (`notifyLiveRunRecovery`, `notifyLiveResourceRecovery`, attention retention). The legacy run paths already send these with the run id. - The fixture uses process-adapter runs, not native runs. The native writers are narrowed by code reading. Their effect on the hosted staging instance is not verified. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1957 lines
85 KiB
TypeScript
1957 lines
85 KiB
TypeScript
import { createReusableLeaseHandoff, preserveReusableLeaseHandoff, withoutReusableLeaseHandoff } from "./environment-lease-handoff.js";
|
|
import { promoteDeferredWorkIntents } from "./work-intent-outbox.js";
|
|
import { clearRemoteWorkspaceLocationForReleasedLease } from "./remote-workspace-snapshot-retention.js";
|
|
import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js";
|
|
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
|
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
|
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
|
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
|
|
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
|
|
import { withEnvironmentLeaseController } from "./environment-lease-controller.js";
|
|
import { randomUUID } from "node:crypto";
|
|
|
|
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "./environment-custom-image-runtime.js";
|
|
import { and, asc, desc, eq, inArray, isNull, ne, sql, type SQL } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agents,
|
|
builtInManagedResources,
|
|
companies,
|
|
companySecretBindings,
|
|
environmentCustomImageSetupSessions,
|
|
environmentLeases,
|
|
sandboxCreationOperations,
|
|
environments,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
instanceSettings,
|
|
issues,
|
|
projects,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
ENVIRONMENT_DRIVERS,
|
|
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
|
|
ENVIRONMENT_LEASE_POLICIES,
|
|
ENVIRONMENT_LEASE_STATUSES,
|
|
ENVIRONMENT_STATUSES,
|
|
type CreateEnvironment,
|
|
type Environment,
|
|
type EnvironmentDeleteBlastRadius,
|
|
type EnvironmentDeleteBlockedReason,
|
|
type EnvironmentLease,
|
|
type EnvironmentLeaseCleanupStatus,
|
|
type EnvironmentLeasePolicy,
|
|
type EnvironmentLeaseStatus,
|
|
type UpdateEnvironment,
|
|
} from "@paperclipai/shared";
|
|
import { conflict, forbidden } from "../errors.js";
|
|
import { logActivity } from "./activity-log.js";
|
|
import { isCloudManagedInstance } from "./cloud-instance.js";
|
|
import {
|
|
resourceStatus,
|
|
stockHash,
|
|
type ManagedResourceStockStatus,
|
|
} from "./managed-resource-drift.js";
|
|
import {
|
|
archiveManagedPluginEnvironment as archiveManagedPluginEnvironmentRow,
|
|
ensureManagedPluginEnvironment as ensureManagedPluginEnvironmentRow,
|
|
findManagedPluginEnvironment as findManagedPluginEnvironmentRow,
|
|
type ManagedPluginEnvironmentInput,
|
|
type ManagedPluginEnvironmentReconcileAction,
|
|
} from "./managed-plugin-environments.js";
|
|
|
|
/** Lease policies whose sandbox outlives its run: a reusable warm sandbox, and a
|
|
* retain-on-failure sandbox kept stopped as a workspace-restore source. The
|
|
* delete blast radius, the atomic delete guard and the consented teardown must
|
|
* all use this one list, or a delete can orphan a sandbox. */
|
|
export const KEPT_SANDBOX_LEASE_POLICIES = ["reuse_by_environment", "retain_on_failure"] as const;
|
|
|
|
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
|
|
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
|
|
return db.transaction(async tx => {
|
|
await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
|
|
const [operation] = await tx.select().from(sandboxCreationOperations)
|
|
.where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
|
|
eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update");
|
|
if (scope.heartbeatRunId) {
|
|
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"),
|
|
)).for("update");
|
|
if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed");
|
|
}
|
|
if (operation?.state === "attached" && operation.leaseId) {
|
|
const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId);
|
|
if (existing?.status === "active") return existing;
|
|
}
|
|
if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred");
|
|
const lease = await attach(tx as unknown as Db);
|
|
await notifyEnvironmentCleanup(tx, { companyId: scope.companyId, runId: operation.runId });
|
|
await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
|
|
.where(eq(sandboxCreationOperations.id, id));
|
|
return lease;
|
|
});
|
|
}
|
|
|
|
export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {
|
|
const [environment] = await db.select({ id: environments.id }).from(environments)
|
|
.where(eq(environments.id, environmentId)).for("update");
|
|
if (!environment) throw new Error("Sandbox creation environment no longer exists");
|
|
const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources)
|
|
.where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId)));
|
|
if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) {
|
|
throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" });
|
|
}
|
|
}
|
|
|
|
type EnvironmentRow = typeof environments.$inferSelect;
|
|
type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
|
|
const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
|
|
const DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION =
|
|
"Default execution environment for Paperclip runs on this machine.";
|
|
|
|
const DEFAULT_KUBERNETES_ENVIRONMENT_NAME = "Kubernetes Sandbox";
|
|
const DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION =
|
|
"Managed Kubernetes sandbox environment for hosted tenant execution.";
|
|
/** Provider key (== plugin driverKey) of the first-party Kubernetes sandbox provider. */
|
|
const KUBERNETES_PROVIDER_KEY = "kubernetes";
|
|
/** Metadata marker for the company's managed-by-config Kubernetes sandbox environment. */
|
|
const KUBERNETES_MANAGED_MARKER = "managedKubernetesSandbox";
|
|
const ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES = ["starting", "waiting_for_user", "capturing"] as const;
|
|
|
|
/**
|
|
* Configuration accepted by `ensureKubernetesEnvironment`. Mirrors the keys of
|
|
* the kubernetes sandbox-provider `configSchema` that an operator typically
|
|
* pins for a hosted cloud instance. Stored verbatim in `environment.config`
|
|
* (the plugin validates/defaults it via `kubernetesProviderConfigSchema` at
|
|
* lease time); `provider` is always forced to "kubernetes".
|
|
*/
|
|
export interface KubernetesEnvironmentConfigInput {
|
|
backend?: "sandbox-cr" | "job";
|
|
inCluster?: boolean;
|
|
runtimeClassName?: string;
|
|
egressMode?: "cilium" | "standard";
|
|
egressAllowFqdns?: string[];
|
|
egressAllowCidrs?: string[];
|
|
namespacePrefix?: string;
|
|
imageRegistry?: string;
|
|
adapterType?: string;
|
|
/**
|
|
* Sandbox lease RPC timeout in milliseconds. Read at lease time by
|
|
* `resolvePluginSandboxRpcTimeoutMs` to extend the worker-manager call
|
|
* timeout when acquiring a lease may take minutes (e.g. a cold node
|
|
* scale-up on an autoscale-to-zero pool). Stored verbatim in the
|
|
* environment config and validated by the sandbox config schema.
|
|
*/
|
|
timeoutMs?: number;
|
|
adapters?: import("@paperclipai/shared").AdapterRegistryEntry[];
|
|
[key: string]: unknown;
|
|
}
|
|
|
|
/**
|
|
* Input to `ensureManagedSandboxEnvironment`. Provider-agnostic: `provider`
|
|
* is the sandbox plugin's driver key and is forced into `config.provider`;
|
|
* the rest of `config` is stored verbatim for the plugin to validate at
|
|
* lease time.
|
|
*/
|
|
export interface ManagedSandboxEnvironmentInput {
|
|
/** Company whose managed-resource binding should be reconciled. Omit at instance boot to bind every company. */
|
|
companyId?: string;
|
|
name: string;
|
|
description?: string;
|
|
/** Sandbox provider key (the plugin's driverKey, e.g. "kubernetes", "daytona"). */
|
|
provider: string;
|
|
config?: Record<string, unknown>;
|
|
/**
|
|
* Extra metadata markers stamped on the managed row (e.g. the legacy
|
|
* kubernetes marker `managedKubernetesSandbox` that
|
|
* `findKubernetesEnvironment` keys on).
|
|
*/
|
|
extraMetadata?: Record<string, unknown>;
|
|
/** Version label recorded with the stock binding; hashes remain the drift authority. */
|
|
stockVersion?: string;
|
|
/**
|
|
* Asserts the caller's deployment gives no operator any path to hand-edit
|
|
* this row (currently only true for the PAPERCLIP_MANAGED_CONFIG applier,
|
|
* where `enableManagedSandboxOnly` removes the tenant's own environment
|
|
* choice entirely). When set, a plain content-hash mismatch against a real
|
|
* prior binding is treated as ordinary stock drift instead of an operator
|
|
* customization to protect — see the `operator_modified` handling below.
|
|
* Leave unset for any caller (self-hosted `kubernetes-execution-mode`
|
|
* bootstrap, tests, admin routes) where an operator could realistically
|
|
* have edited the row through the normal environments UI/API.
|
|
*/
|
|
platformFullyManaged?: boolean;
|
|
}
|
|
|
|
export type ManagedSandboxEnvironmentReconcileAction =
|
|
| "added"
|
|
| "updated"
|
|
| "unchanged"
|
|
| "skipped";
|
|
|
|
export interface ManagedSandboxEnvironmentReconcileResult {
|
|
environment: Environment;
|
|
action: ManagedSandboxEnvironmentReconcileAction;
|
|
/** Classification observed before this reconciliation wrote anything. */
|
|
stockStatus: ManagedResourceStockStatus;
|
|
/** True only when operator drift prevented the available stock update. */
|
|
updateAvailable: boolean;
|
|
stockHash: string;
|
|
}
|
|
|
|
function cloneRecord(value: unknown, fallback: Record<string, unknown> | null = null): Record<string, unknown> | null {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return fallback;
|
|
return { ...(value as Record<string, unknown>) };
|
|
}
|
|
|
|
function readEnum<T extends string>(value: string | null, allowed: readonly T[], fieldName: string): T | null {
|
|
if (value === null) return null;
|
|
if ((allowed as readonly string[]).includes(value)) return value as T;
|
|
throw new Error(`Unexpected ${fieldName} value: ${value}`);
|
|
}
|
|
|
|
function hasConstraintName(error: unknown, constraintName: string): boolean {
|
|
if (typeof error !== "object" || error === null) return false;
|
|
const candidate = error as {
|
|
constraint?: unknown;
|
|
constraint_name?: unknown;
|
|
cause?: unknown;
|
|
};
|
|
return candidate.constraint === constraintName
|
|
|| candidate.constraint_name === constraintName
|
|
|| hasConstraintName(candidate.cause, constraintName);
|
|
}
|
|
|
|
function toEnvironment(row: EnvironmentRow): Environment {
|
|
return {
|
|
id: row.id,
|
|
name: row.name,
|
|
description: row.description ?? null,
|
|
driver: readEnum(row.driver, ENVIRONMENT_DRIVERS, "environment driver") ?? "local",
|
|
status: readEnum(row.status, ENVIRONMENT_STATUSES, "environment status") ?? "active",
|
|
config: cloneRecord(row.config, {}) ?? {},
|
|
envVars: cloneRecord(row.envVars, {}) ?? {},
|
|
metadata: cloneRecord(row.metadata),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
} as Environment;
|
|
}
|
|
|
|
type EnvironmentListFilters = {
|
|
status?: string;
|
|
driver?: string;
|
|
};
|
|
|
|
function resolveListFilters(
|
|
companyIdOrFilters?: string | EnvironmentListFilters,
|
|
maybeFilters?: EnvironmentListFilters,
|
|
): EnvironmentListFilters {
|
|
if (typeof companyIdOrFilters === "string") {
|
|
return maybeFilters ?? {};
|
|
}
|
|
return companyIdOrFilters ?? {};
|
|
}
|
|
|
|
function resolveCreateInput(
|
|
companyIdOrInput: string | CreateEnvironment,
|
|
maybeInput?: CreateEnvironment,
|
|
): CreateEnvironment {
|
|
if (typeof companyIdOrInput === "string") {
|
|
if (!maybeInput) throw new Error("Create environment input is required");
|
|
return maybeInput;
|
|
}
|
|
return companyIdOrInput;
|
|
}
|
|
|
|
function resolveKubernetesConfig(
|
|
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
|
|
maybeConfig?: KubernetesEnvironmentConfigInput,
|
|
): KubernetesEnvironmentConfigInput {
|
|
if (typeof companyIdOrConfig === "string") {
|
|
if (!maybeConfig) throw new Error("Kubernetes environment config is required");
|
|
return maybeConfig;
|
|
}
|
|
return companyIdOrConfig;
|
|
}
|
|
|
|
export function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
environmentId: row.environmentId,
|
|
executionWorkspaceId: row.executionWorkspaceId ?? null,
|
|
issueId: row.issueId ?? null,
|
|
heartbeatRunId: row.heartbeatRunId ?? null,
|
|
status: readEnum(row.status, ENVIRONMENT_LEASE_STATUSES, "environment lease status") ?? "active",
|
|
leasePolicy: readEnum(row.leasePolicy, ENVIRONMENT_LEASE_POLICIES, "environment lease policy") ?? "ephemeral",
|
|
provider: row.provider ?? null,
|
|
providerLeaseId: row.providerLeaseId ?? null,
|
|
acquiredAt: row.acquiredAt,
|
|
lastUsedAt: row.lastUsedAt,
|
|
expiresAt: row.expiresAt ?? null,
|
|
releasedAt: row.releasedAt ?? null,
|
|
failureReason: row.failureReason ?? null,
|
|
cleanupStatus: readEnum(
|
|
row.cleanupStatus,
|
|
ENVIRONMENT_LEASE_CLEANUP_STATUSES,
|
|
"environment lease cleanup status",
|
|
),
|
|
metadata: cloneRecord(row.metadata),
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
};
|
|
}
|
|
|
|
function countFromRows(rows: Array<{ count: number | string | null | undefined }>): number {
|
|
return Number(rows[0]?.count ?? 0);
|
|
}
|
|
|
|
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
type EnvironmentWriteDb = Pick<Db | DbTransaction, "select" | "insert" | "update" | "delete">;
|
|
|
|
const MANAGED_ENVIRONMENT_BUNDLE_KEY = "managed-sandbox-environment";
|
|
const MANAGED_ENVIRONMENT_RESOURCE_KIND = "environment";
|
|
const MANAGED_ENVIRONMENT_RESOURCE_KEY = "managed-sandbox";
|
|
const MANAGED_ENVIRONMENT_STOCK_VERSION = "managed-environment-v1";
|
|
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY = "_paperclipManagedArchiveToken";
|
|
const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY = "_paperclipManagedArchiveToken";
|
|
|
|
function managedEnvironmentBaselineDefaults(
|
|
defaultsJson: Record<string, unknown>,
|
|
): Record<string, unknown> {
|
|
const baseline = { ...defaultsJson };
|
|
delete baseline[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
|
|
return baseline;
|
|
}
|
|
|
|
function withoutManagedEnvironmentArchiveToken(
|
|
metadata: Record<string, unknown> | null,
|
|
): Record<string, unknown> | null {
|
|
if (!metadata) return null;
|
|
const next = { ...metadata };
|
|
delete next[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
|
|
return next;
|
|
}
|
|
|
|
function managedMetadataKeys(
|
|
desiredMetadata: Record<string, unknown>,
|
|
bindings: Array<{ defaultsJson: Record<string, unknown> }>,
|
|
): string[] {
|
|
const keys = new Set([
|
|
"managedByPaperclip",
|
|
"managedSandboxProvider",
|
|
KUBERNETES_MANAGED_MARKER,
|
|
...Object.keys(desiredMetadata),
|
|
]);
|
|
for (const binding of bindings) {
|
|
const metadata = cloneRecord(binding.defaultsJson.metadata);
|
|
for (const key of Object.keys(metadata ?? {})) keys.add(key);
|
|
}
|
|
return [...keys].sort((left, right) => left.localeCompare(right));
|
|
}
|
|
|
|
function managedEnvironmentStock(input: {
|
|
name: string;
|
|
description: string | null;
|
|
config: Record<string, unknown>;
|
|
metadata: Record<string, unknown> | null;
|
|
status: string;
|
|
}, metadataKeys: readonly string[]): Record<string, unknown> {
|
|
const metadata = input.metadata ?? {};
|
|
return {
|
|
name: input.name,
|
|
description: input.description,
|
|
config: input.config,
|
|
metadata: Object.fromEntries(
|
|
metadataKeys.map((key) => [key, Object.prototype.hasOwnProperty.call(metadata, key) ? metadata[key] : null]),
|
|
),
|
|
status: input.status,
|
|
};
|
|
}
|
|
|
|
function mergeManagedEnvironmentMetadata(
|
|
current: Record<string, unknown> | null,
|
|
desired: Record<string, unknown>,
|
|
keys: readonly string[],
|
|
): Record<string, unknown> {
|
|
const merged = { ...(current ?? {}) };
|
|
for (const key of keys) {
|
|
const value = Object.prototype.hasOwnProperty.call(desired, key) ? desired[key] : null;
|
|
if (value === null || value === undefined) delete merged[key];
|
|
else merged[key] = value;
|
|
}
|
|
return merged;
|
|
}
|
|
|
|
function preserveInvalidatedSnapshotProofs(replacement: SQL): SQL {
|
|
const proofs = sql`(select coalesce(jsonb_object_agg(entry.key, entry.value || '{"invalidated":true}'::jsonb), '{}'::jsonb)
|
|
from jsonb_each(coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)) as entry
|
|
where entry.value->>'v' = '2')`;
|
|
return sql`case when ${proofs} = '{}'::jsonb then ${replacement}
|
|
else coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${proofs}) end`;
|
|
}
|
|
|
|
export function environmentService(db: Db) {
|
|
/**
|
|
* Idempotently ensure THE Paperclip-managed sandbox environment for this
|
|
* instance, configured for an arbitrary sandbox provider plugin. Mirrors
|
|
* `ensureLocalEnvironment`; the partial unique index
|
|
* `environments_managed_sandbox_idx` enforces at most one managed sandbox
|
|
* row per instance, so this function owns that single slot regardless of
|
|
* provider:
|
|
*
|
|
* - A stock-controlled managed row advances to a new stock hash in the same
|
|
* transaction as its managed fields, including provider switches.
|
|
* - A stock-current row is returned without an environment write.
|
|
* - An operator-modified or previously unmanaged row is preserved and
|
|
* reported as skipped; its user-owned fields are never folded into stock.
|
|
*/
|
|
const ensureManagedSandboxEnvironment = async (
|
|
input: ManagedSandboxEnvironmentInput,
|
|
): Promise<ManagedSandboxEnvironmentReconcileResult> => {
|
|
const desiredConfig: Record<string, unknown> = {
|
|
...(input.config ?? {}),
|
|
provider: input.provider,
|
|
};
|
|
const desiredMetadata: Record<string, unknown> = {
|
|
managedByPaperclip: true,
|
|
managedSandboxProvider: input.provider,
|
|
...(input.extraMetadata ?? {}),
|
|
};
|
|
if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) {
|
|
desiredMetadata[KUBERNETES_MANAGED_MARKER] = null;
|
|
}
|
|
|
|
let activityCompanyIds: string[] = [];
|
|
let trackingInitialized = false;
|
|
let providerReactivated = false;
|
|
const reconciliation = await db.transaction(
|
|
async (tx): Promise<ManagedSandboxEnvironmentReconcileResult> => {
|
|
const companyIds = input.companyId
|
|
? [input.companyId]
|
|
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
|
|
activityCompanyIds = companyIds;
|
|
|
|
// Take the sandbox-row lock BEFORE reading the stock bindings. Two
|
|
// passes can reconcile the same slot concurrently (the boot ensure and
|
|
// the async provider-recovery reactivation, or two app builds during a
|
|
// rolling deploy). Concurrent passes serialize on this lock, and under
|
|
// READ COMMITTED each later statement sees a fresh snapshot — so a pass
|
|
// that blocks here then reads the bindings the winning pass committed,
|
|
// not a snapshot from before it waited. Reading bindings first left a
|
|
// window where a waiting pass compared the winner's fresh row against
|
|
// its own stale binding hash, misclassified the mismatch as
|
|
// `operator_modified`, and — once `platformFullyManaged` turns that
|
|
// into an update — rolled the row back to its own older stock.
|
|
const sandboxRows = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update");
|
|
|
|
const bindingConditions = and(
|
|
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
|
|
...(companyIds.length > 0 ? [inArray(builtInManagedResources.companyId, companyIds)] : []),
|
|
);
|
|
const bindings = companyIds.length > 0
|
|
? await tx.select().from(builtInManagedResources).where(bindingConditions)
|
|
: [];
|
|
trackingInitialized = bindings.length < companyIds.length;
|
|
const keys = managedMetadataKeys(desiredMetadata, bindings);
|
|
|
|
let row = sandboxRows.find(
|
|
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? sandboxRows.find((candidate) => candidate.name === input.name) ?? null;
|
|
|
|
const writeBindings = async (
|
|
environmentId: string,
|
|
stockVersion: string,
|
|
installedStockHash: string,
|
|
defaultsJson: Record<string, unknown>,
|
|
replace: boolean,
|
|
) => {
|
|
const targetCompanyIds = replace
|
|
? companyIds
|
|
: companyIds.filter(
|
|
(companyId) => !bindings.some((binding) => binding.companyId === companyId),
|
|
);
|
|
if (targetCompanyIds.length === 0) return;
|
|
const values = targetCompanyIds.map((companyId) => ({
|
|
companyId,
|
|
bundleKey: MANAGED_ENVIRONMENT_BUNDLE_KEY,
|
|
resourceKind: MANAGED_ENVIRONMENT_RESOURCE_KIND,
|
|
resourceKey: MANAGED_ENVIRONMENT_RESOURCE_KEY,
|
|
resourceId: environmentId,
|
|
stockVersion,
|
|
stockHash: installedStockHash,
|
|
defaultsJson,
|
|
}));
|
|
const insert = tx.insert(builtInManagedResources).values(values);
|
|
if (!replace) {
|
|
await insert.onConflictDoNothing({
|
|
target: [
|
|
builtInManagedResources.companyId,
|
|
builtInManagedResources.bundleKey,
|
|
builtInManagedResources.resourceKind,
|
|
builtInManagedResources.resourceKey,
|
|
],
|
|
});
|
|
return;
|
|
}
|
|
await insert.onConflictDoUpdate({
|
|
target: [
|
|
builtInManagedResources.companyId,
|
|
builtInManagedResources.bundleKey,
|
|
builtInManagedResources.resourceKind,
|
|
builtInManagedResources.resourceKey,
|
|
],
|
|
set: {
|
|
resourceId: environmentId,
|
|
stockVersion,
|
|
stockHash: installedStockHash,
|
|
defaultsJson,
|
|
updatedAt: new Date(),
|
|
},
|
|
});
|
|
};
|
|
|
|
if (!row) {
|
|
const nameOwner = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.name, input.name))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (nameOwner && nameOwner.driver !== "sandbox") {
|
|
throw new Error(
|
|
`Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`,
|
|
);
|
|
}
|
|
const now = new Date();
|
|
const inserted = await tx
|
|
.insert(environments)
|
|
.values({
|
|
name: input.name,
|
|
description: input.description ?? null,
|
|
driver: "sandbox",
|
|
status: "active",
|
|
config: desiredConfig,
|
|
envVars: {},
|
|
metadata: mergeManagedEnvironmentMetadata(null, desiredMetadata, keys),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
// Either the managed-slot partial index or the global name index
|
|
// can select the concurrent winner. Treat both as convergence and
|
|
// reselect that winner under lock below.
|
|
.onConflictDoNothing()
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (inserted) {
|
|
const stock = managedEnvironmentStock({
|
|
name: inserted.name,
|
|
description: inserted.description ?? null,
|
|
config: inserted.config,
|
|
metadata: inserted.metadata,
|
|
status: inserted.status,
|
|
}, keys);
|
|
const latestStockHash = stockHash(stock);
|
|
await writeBindings(
|
|
inserted.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
stock,
|
|
true,
|
|
);
|
|
return {
|
|
environment: toEnvironment(inserted),
|
|
action: "added",
|
|
stockStatus: "missing",
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
row = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update")
|
|
.then(
|
|
(rows) => rows.find(
|
|
(candidate) => (candidate.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? null,
|
|
);
|
|
if (!row) throw new Error("Failed to ensure managed sandbox environment");
|
|
}
|
|
|
|
const nameOwner = await tx
|
|
.select({ id: environments.id })
|
|
.from(environments)
|
|
.where(eq(environments.name, input.name))
|
|
.then((rows) => rows[0] ?? null);
|
|
const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name;
|
|
const desiredStock = managedEnvironmentStock({
|
|
name: desiredName,
|
|
description: input.description ?? null,
|
|
config: desiredConfig,
|
|
metadata: desiredMetadata,
|
|
status: "active",
|
|
}, keys);
|
|
const latestStockHash = stockHash(desiredStock);
|
|
const currentStock = managedEnvironmentStock({
|
|
name: row.name,
|
|
description: row.description ?? null,
|
|
config: row.config,
|
|
metadata: row.metadata,
|
|
status: row.status,
|
|
}, keys);
|
|
const currentHash = stockHash(currentStock);
|
|
const matchingBindings = bindings.filter((binding) => binding.resourceId === row!.id);
|
|
const rowArchiveToken = (row.metadata as Record<string, unknown> | null)
|
|
?.[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY];
|
|
let stockStatus = resourceStatus({
|
|
resourceId: row.id,
|
|
currentHash,
|
|
bindingStockHash: matchingBindings[0]?.stockHash ?? null,
|
|
latestStockHash,
|
|
});
|
|
if (stockStatus === "operator_modified") {
|
|
const stockControlledBinding = matchingBindings.find(
|
|
(binding) => resourceStatus({
|
|
resourceId: row!.id,
|
|
currentHash,
|
|
bindingStockHash: binding.stockHash,
|
|
latestStockHash,
|
|
}) === "stock_update_available",
|
|
);
|
|
if (stockControlledBinding) stockStatus = "stock_update_available";
|
|
}
|
|
const operatorReaffirmedArchive = row.status === "archived" && matchingBindings.some(
|
|
(binding) => {
|
|
const bindingToken = binding.defaultsJson[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY];
|
|
return binding.defaultsJson.status === "archived" &&
|
|
typeof bindingToken === "string" &&
|
|
bindingToken !== rowArchiveToken;
|
|
},
|
|
);
|
|
if (operatorReaffirmedArchive) stockStatus = "operator_modified";
|
|
|
|
// `platformFullyManaged` callers (currently: the PAPERCLIP_MANAGED_CONFIG
|
|
// applier) assert that nothing in their deployment can hand-edit this
|
|
// row — the product gives a cloud-harness tenant no path to it, unlike
|
|
// the general self-hosted contract this function otherwise protects
|
|
// (see "classifies operator drift" in environment-service.test.ts,
|
|
// which exercises a real operator edit and must keep winning). Under
|
|
// that assertion, a plain content-hash mismatch against a real prior
|
|
// binding can only be drift between two platform-driven reconciliation
|
|
// passes (e.g. a stock hash recorded by an older app build before a
|
|
// later stock field was added), never a customization to protect —
|
|
// apply it like any other stock-outdated row.
|
|
//
|
|
// Two things the bypass must never touch:
|
|
// - A row with NO matching binding. `row` can be a same-name sandbox
|
|
// row that was never Paperclip-managed (the fallback lookup above).
|
|
// It also reads as `operator_modified`, but there is no prior
|
|
// platform pass to have drifted from — adopting it would overwrite
|
|
// a tenant-created environment and stamp it managed. Require a
|
|
// binding for this exact row, so "prior binding" is enforced, not
|
|
// just documented.
|
|
// - Archive-reaffirmation. A `sandbox_image` update must never
|
|
// resurrect a row something else deliberately kept archived after
|
|
// Paperclip's own provider-unavailability archival, so that path
|
|
// still skips below regardless of this flag.
|
|
if (
|
|
input.platformFullyManaged &&
|
|
stockStatus === "operator_modified" &&
|
|
!operatorReaffirmedArchive &&
|
|
matchingBindings.length > 0
|
|
) {
|
|
stockStatus = "stock_update_available";
|
|
}
|
|
|
|
if (stockStatus === "operator_modified") {
|
|
const baseline = matchingBindings[0];
|
|
let baselineDefaults = baseline
|
|
? managedEnvironmentBaselineDefaults(baseline.defaultsJson)
|
|
: desiredStock;
|
|
let baselineHash = baseline?.stockHash ?? latestStockHash;
|
|
|
|
// Provider unavailability is an operational state transition, not
|
|
// an operator edit. If the binding records that Paperclip archived
|
|
// this row, restore only its availability status. Keep every other
|
|
// operator-modified field intact and leave the stock update pending.
|
|
// A manually archived row still has an active binding baseline, so
|
|
// it remains operator_modified and is not reactivated here.
|
|
const archivedByReconciler = row.status === "archived" &&
|
|
typeof rowArchiveToken === "string" &&
|
|
matchingBindings.some((binding) => {
|
|
const bindingDefaults = binding.defaultsJson;
|
|
return bindingDefaults.status === "archived" &&
|
|
bindingDefaults[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY] === rowArchiveToken;
|
|
});
|
|
if (archivedByReconciler) {
|
|
const reactivated = await tx
|
|
.update(environments)
|
|
.set({
|
|
status: "active",
|
|
metadata: withoutManagedEnvironmentArchiveToken(row.metadata),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(environments.id, row.id), eq(environments.status, "archived")))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!reactivated) {
|
|
throw new Error("Managed sandbox environment changed during reactivation");
|
|
}
|
|
row = reactivated;
|
|
providerReactivated = true;
|
|
|
|
for (const binding of matchingBindings) {
|
|
const reactivatedDefaults = {
|
|
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
|
|
status: "active",
|
|
};
|
|
const reactivatedHash = stockHash(reactivatedDefaults);
|
|
await tx
|
|
.update(builtInManagedResources)
|
|
.set({
|
|
stockHash: reactivatedHash,
|
|
defaultsJson: reactivatedDefaults,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(
|
|
eq(builtInManagedResources.id, binding.id),
|
|
eq(builtInManagedResources.resourceId, row.id),
|
|
));
|
|
if (binding.id === baseline?.id) {
|
|
baselineDefaults = reactivatedDefaults;
|
|
baselineHash = reactivatedHash;
|
|
}
|
|
}
|
|
}
|
|
await writeBindings(
|
|
row.id,
|
|
baseline?.stockVersion ?? input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
baselineHash,
|
|
baselineDefaults,
|
|
false,
|
|
);
|
|
return {
|
|
environment: toEnvironment(row),
|
|
action: "skipped",
|
|
stockStatus,
|
|
updateAvailable: true,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
|
|
if (stockStatus === "stock_current") {
|
|
await writeBindings(
|
|
row.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
desiredStock,
|
|
false,
|
|
);
|
|
return {
|
|
environment: toEnvironment(row),
|
|
action: "unchanged",
|
|
stockStatus,
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
}
|
|
|
|
const updated = await tx
|
|
.update(environments)
|
|
.set({
|
|
name: desiredName,
|
|
description: input.description ?? null,
|
|
config: desiredConfig,
|
|
metadata: withoutManagedEnvironmentArchiveToken(
|
|
mergeManagedEnvironmentMetadata(row.metadata, desiredMetadata, keys),
|
|
),
|
|
status: "active",
|
|
updatedAt: row.status === "active" && stockHash(row.config) === stockHash(desiredConfig)
|
|
? row.updatedAt
|
|
: new Date(),
|
|
})
|
|
.where(eq(environments.id, row.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!updated) throw new Error("Managed sandbox environment changed during reconciliation");
|
|
await writeBindings(
|
|
updated.id,
|
|
input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION,
|
|
latestStockHash,
|
|
desiredStock,
|
|
true,
|
|
);
|
|
return {
|
|
environment: toEnvironment(updated),
|
|
action: "updated",
|
|
stockStatus,
|
|
updateAvailable: false,
|
|
stockHash: latestStockHash,
|
|
};
|
|
},
|
|
);
|
|
if (reconciliation.action !== "unchanged" || trackingInitialized) {
|
|
const action = reconciliation.action === "added"
|
|
? "environment.managed_stock_added"
|
|
: reconciliation.action === "updated"
|
|
? "environment.managed_stock_updated"
|
|
: reconciliation.action === "skipped"
|
|
? "environment.managed_stock_skipped"
|
|
: "environment.managed_stock_tracking_initialized";
|
|
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
|
|
companyId,
|
|
actorType: "system",
|
|
actorId: "managed-environment-reconciler",
|
|
action,
|
|
entityType: "environment",
|
|
entityId: reconciliation.environment.id,
|
|
details: {
|
|
provider: input.provider,
|
|
reconciliationAction: reconciliation.action,
|
|
stockStatus: reconciliation.stockStatus,
|
|
updateAvailable: reconciliation.updateAvailable,
|
|
stockHash: reconciliation.stockHash,
|
|
providerReactivated,
|
|
},
|
|
})));
|
|
}
|
|
return reconciliation;
|
|
};
|
|
|
|
/**
|
|
* Archive the Paperclip-managed sandbox row when its provider became
|
|
* unavailable (plugin missing, not ready, or its worker not running), so
|
|
* run scheduling stops selecting an environment whose lease acquisition
|
|
* cannot succeed (`resolveEnvironment` rejects non-active rows).
|
|
*
|
|
* Scoped to the row provisioned for the SAME provider: a row that a
|
|
* provider switch left on a different provider is not touched (the ensure
|
|
* path adopts it once the new provider is healthy). Reactivation is
|
|
* automatic — the next successful `ensureManagedSandboxEnvironment` stamps
|
|
* the row `active` again.
|
|
*
|
|
* Returns the archived environment, or null when there is no active
|
|
* managed row for this provider.
|
|
*/
|
|
const archiveManagedSandboxEnvironment = async (
|
|
input: { provider: string; companyId?: string },
|
|
): Promise<Environment | null> => {
|
|
let activityCompanyIds: string[] = [];
|
|
const archived = await db.transaction(async (tx) => {
|
|
const existing = await tx
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "sandbox"))
|
|
.for("update")
|
|
.then(
|
|
(rows) => rows.find(
|
|
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
) ?? null,
|
|
);
|
|
if (!existing || existing.status !== "active") return null;
|
|
const rowProvider = (existing.metadata as Record<string, unknown> | null)
|
|
?.managedSandboxProvider;
|
|
if (rowProvider !== input.provider) return null;
|
|
|
|
const companyIds = input.companyId
|
|
? [input.companyId]
|
|
: await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
|
|
activityCompanyIds = companyIds;
|
|
const bindings = companyIds.length > 0
|
|
? await tx
|
|
.select()
|
|
.from(builtInManagedResources)
|
|
.where(and(
|
|
inArray(builtInManagedResources.companyId, companyIds),
|
|
eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY),
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY),
|
|
eq(builtInManagedResources.resourceId, existing.id),
|
|
))
|
|
: [];
|
|
const archiveToken = randomUUID();
|
|
const archivedMetadata = {
|
|
...(existing.metadata ?? {}),
|
|
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]: archiveToken,
|
|
};
|
|
const archived = await tx
|
|
.update(environments)
|
|
.set({
|
|
status: "archived",
|
|
metadata: archivedMetadata,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(environments.id, existing.id), eq(environments.status, "active")))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!archived) return null;
|
|
|
|
// Archival is a Paperclip-owned availability transition. Record only
|
|
// that status change in each installed baseline. Deriving the new hash
|
|
// from defaultsJson keeps operator-modified row fields out of stock.
|
|
for (const binding of bindings) {
|
|
const archivedStock = {
|
|
...managedEnvironmentBaselineDefaults(binding.defaultsJson),
|
|
status: "archived",
|
|
};
|
|
await tx
|
|
.update(builtInManagedResources)
|
|
.set({
|
|
stockHash: stockHash(archivedStock),
|
|
defaultsJson: {
|
|
...archivedStock,
|
|
[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]: archiveToken,
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(
|
|
eq(builtInManagedResources.id, binding.id),
|
|
eq(builtInManagedResources.resourceId, existing.id),
|
|
));
|
|
}
|
|
return toEnvironment(archived);
|
|
});
|
|
if (archived) {
|
|
await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, {
|
|
companyId,
|
|
actorType: "system",
|
|
actorId: "managed-environment-reconciler",
|
|
action: "environment.managed_provider_unavailable_archived",
|
|
entityType: "environment",
|
|
entityId: archived.id,
|
|
details: { provider: input.provider },
|
|
})));
|
|
}
|
|
return archived;
|
|
};
|
|
|
|
return {
|
|
recordSandboxPlatformProbe: async (input: {
|
|
environmentId: string;
|
|
companyId: string;
|
|
config: Record<string, unknown>;
|
|
proof: Record<string, unknown> | null;
|
|
}): Promise<boolean> => {
|
|
const existing = sql`coalesce(${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY}, '{}'::jsonb)`;
|
|
const previous = sql`${existing}->${input.companyId}`;
|
|
const probes = input.proof === null
|
|
? sql`case when ${previous}->>'v' = '2'
|
|
then jsonb_set(${existing}, ARRAY[${input.companyId}]::text[], ${previous} || '{"invalidated":true}'::jsonb, true)
|
|
else ${existing} - ${input.companyId} end`
|
|
: sql`${existing} || ${JSON.stringify({ [input.companyId]: input.proof })}::jsonb`;
|
|
const rows = await db.update(environments).set({
|
|
metadata: sql`jsonb_set(coalesce(${environments.metadata}, '{}'::jsonb), ARRAY[${SANDBOX_PLATFORM_PROBE_METADATA_KEY}]::text[], ${probes}, true)`,
|
|
}).where(and(eq(environments.id, input.environmentId), eq(environments.driver, "sandbox"), eq(environments.config, input.config))).returning({ id: environments.id });
|
|
return rows.length > 0;
|
|
},
|
|
list: async (
|
|
companyIdOrFilters?: string | EnvironmentListFilters,
|
|
maybeFilters?: EnvironmentListFilters,
|
|
): Promise<Environment[]> => {
|
|
const filters = resolveListFilters(companyIdOrFilters, maybeFilters);
|
|
const conditions = [];
|
|
if (filters.status) conditions.push(eq(environments.status, filters.status));
|
|
if (filters.driver) conditions.push(eq(environments.driver, filters.driver));
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(conditions.length > 0 ? and(...conditions) : undefined)
|
|
.orderBy(desc(environments.updatedAt), desc(environments.createdAt));
|
|
return rows.map(toEnvironment);
|
|
},
|
|
|
|
getById: async (id: string): Promise<Environment | null> => {
|
|
const row = await db.select().from(environments).where(eq(environments.id, id)).then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/**
|
|
* List the companies that own an environment through a built-in managed
|
|
* resource binding. A managed sandbox row binds to each company that the
|
|
* instance provisions it for. An operator-created environment has no
|
|
* binding, so this returns an empty list. The caller treats an empty list
|
|
* as an instance-global environment with no company owner.
|
|
*/
|
|
listBoundCompanyIds: async (environmentId: string): Promise<string[]> => {
|
|
const rows = await db
|
|
.select({ companyId: builtInManagedResources.companyId })
|
|
.from(builtInManagedResources)
|
|
.where(
|
|
and(
|
|
eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND),
|
|
eq(builtInManagedResources.resourceId, environmentId),
|
|
),
|
|
);
|
|
return Array.from(new Set(rows.map((row) => row.companyId)));
|
|
},
|
|
|
|
getLeaseById: async (id: string): Promise<EnvironmentLease | null> => {
|
|
if (!id.trim()) return null;
|
|
const row = await db
|
|
.select()
|
|
.from(environmentLeases)
|
|
.where(eq(environmentLeases.id, id))
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Idempotently ensure THE local-driver environment row; the partial
|
|
* unique index `environments_local_driver_idx` enforces at most one per
|
|
* instance.
|
|
*
|
|
* On a cloud-managed instance an existing row is additionally ADOPTED —
|
|
* stamped `managedByPaperclip: true` (other metadata preserved) — so the
|
|
* single local slot is platform-owned there by construction, mirroring
|
|
* `ensureManagedSandboxEnvironment`'s adoption of the sandbox slot. This
|
|
* is what lets the environment-routes write floor treat a local row's
|
|
* platform markers as live state rather than a stale leftover: every
|
|
* caller (company creation, the heartbeat, run orchestration) converges
|
|
* the marker. Self-hosted instances keep the historical behavior:
|
|
* an existing row is returned untouched.
|
|
*/
|
|
ensureLocalEnvironment: async (_companyId?: string): Promise<Environment> => {
|
|
const now = new Date();
|
|
const insert = () =>
|
|
db
|
|
.insert(environments)
|
|
.values({
|
|
name: DEFAULT_LOCAL_ENVIRONMENT_NAME,
|
|
description: DEFAULT_LOCAL_ENVIRONMENT_DESCRIPTION,
|
|
driver: "local",
|
|
status: "active",
|
|
config: {},
|
|
envVars: {},
|
|
metadata: {
|
|
managedByPaperclip: true,
|
|
defaultForInstance: true,
|
|
},
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.onConflictDoNothing({
|
|
target: [environments.driver],
|
|
where: sql`${environments.driver} = 'local'`,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
const row = await insert().catch((error: unknown) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
return null;
|
|
}
|
|
throw error;
|
|
});
|
|
if (row) return toEnvironment(row);
|
|
|
|
const existing = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(eq(environments.driver, "local"))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!existing) {
|
|
throw new Error("Failed to ensure local environment");
|
|
}
|
|
const existingMetadata = (existing.metadata ?? {}) as Record<string, unknown>;
|
|
if (isCloudManagedInstance() && existingMetadata.managedByPaperclip !== true) {
|
|
const adopted = await db
|
|
.update(environments)
|
|
.set({
|
|
metadata: { ...existingMetadata, managedByPaperclip: true },
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environments.id, existing.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? existing);
|
|
return toEnvironment(adopted);
|
|
}
|
|
return toEnvironment(existing);
|
|
},
|
|
|
|
ensureManagedSandboxEnvironment,
|
|
|
|
archiveManagedSandboxEnvironment,
|
|
|
|
/**
|
|
* Ensure the instance-level row of one managed-config plugin entry,
|
|
* validating its driver config through the plugin environment validator.
|
|
* Adopts an unmarked row for the same plugin and driver (see
|
|
* `managed-plugin-environments.ts`).
|
|
*/
|
|
ensureManagedPluginEnvironment: async (
|
|
input: ManagedPluginEnvironmentInput,
|
|
): Promise<{ environment: Environment; action: ManagedPluginEnvironmentReconcileAction }> => {
|
|
const result = await ensureManagedPluginEnvironmentRow(db, input);
|
|
return { environment: toEnvironment(result.row), action: result.action };
|
|
},
|
|
|
|
/** Archive the active row of a managed plugin entry whose plugin is not running. */
|
|
archiveManagedPluginEnvironment: async (
|
|
input: { pluginKey: string; driverKey: string },
|
|
): Promise<Environment | null> => {
|
|
const row = await archiveManagedPluginEnvironmentRow(db, input);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/** The marked row of a managed plugin entry, archived or not. */
|
|
findManagedPluginEnvironment: async (
|
|
input: { pluginKey: string; driverKey: string },
|
|
): Promise<Environment | null> => {
|
|
const row = await findManagedPluginEnvironmentRow(db, input);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Idempotently ensure a managed Kubernetes sandbox environment exists for
|
|
* an instance, configured from instance/operator-supplied config. A thin
|
|
* wrapper over `ensureManagedSandboxEnvironment` that pins the provider to
|
|
* "kubernetes" and stamps the legacy marker `findKubernetesEnvironment`
|
|
* keys on. On subsequent calls stock-controlled config advances in place;
|
|
* operator modifications remain untouched for explicit review.
|
|
*/
|
|
ensureKubernetesEnvironment: async (
|
|
companyIdOrConfig: string | KubernetesEnvironmentConfigInput,
|
|
maybeConfig?: KubernetesEnvironmentConfigInput,
|
|
): Promise<Environment> => {
|
|
const config = resolveKubernetesConfig(companyIdOrConfig, maybeConfig);
|
|
return ensureManagedSandboxEnvironment({
|
|
companyId: typeof companyIdOrConfig === "string" ? companyIdOrConfig : undefined,
|
|
name: DEFAULT_KUBERNETES_ENVIRONMENT_NAME,
|
|
description: DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION,
|
|
provider: KUBERNETES_PROVIDER_KEY,
|
|
config,
|
|
extraMetadata: { [KUBERNETES_MANAGED_MARKER]: true },
|
|
}).then((result) => result.environment);
|
|
},
|
|
|
|
/**
|
|
* Find the active managed Kubernetes sandbox environment, if one
|
|
* exists. Read-only counterpart to `ensureKubernetesEnvironment` used by the
|
|
* per-run execution guard (which must not silently create config-less envs).
|
|
*/
|
|
findKubernetesEnvironment: async (_companyId?: string): Promise<Environment | null> => {
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(
|
|
and(
|
|
eq(environments.driver, "sandbox"),
|
|
eq(environments.status, "active"),
|
|
),
|
|
)
|
|
.orderBy(desc(environments.updatedAt));
|
|
const match = rows.find(
|
|
(row) =>
|
|
(row.metadata as Record<string, unknown> | null)?.[KUBERNETES_MANAGED_MARKER] === true,
|
|
);
|
|
return match ? toEnvironment(match) : null;
|
|
},
|
|
|
|
/**
|
|
* Find the platform-managed sandbox environment (the single
|
|
* `managedByPaperclip`-marked slot row), if one exists. Read-only
|
|
* counterpart to `ensureManagedSandboxEnvironment`. The default
|
|
* (active-only) form serves the managed-sandbox-only run guard — which
|
|
* must fail closed rather than create a config-less environment when
|
|
* the slot is empty or archived (the provisioner archives it while its
|
|
* provider plugin is down). `includeArchived` serves reconciliation
|
|
* cleanup, which must find the row even after the provisioner archived
|
|
* it.
|
|
*/
|
|
findManagedSandboxEnvironment: async (
|
|
_companyId?: string,
|
|
options?: { includeArchived?: boolean },
|
|
): Promise<Environment | null> => {
|
|
const rows = await db
|
|
.select()
|
|
.from(environments)
|
|
.where(
|
|
options?.includeArchived === true
|
|
? eq(environments.driver, "sandbox")
|
|
: and(
|
|
eq(environments.driver, "sandbox"),
|
|
eq(environments.status, "active"),
|
|
),
|
|
)
|
|
.orderBy(desc(environments.updatedAt));
|
|
const match = rows.find(
|
|
(row) => (row.metadata as Record<string, unknown> | null)?.managedByPaperclip === true,
|
|
);
|
|
return match ? toEnvironment(match) : null;
|
|
},
|
|
|
|
create: async (
|
|
companyIdOrInput: string | CreateEnvironment,
|
|
maybeInput?: CreateEnvironment,
|
|
options?: { db?: EnvironmentWriteDb },
|
|
): Promise<Environment> => {
|
|
const input = resolveCreateInput(companyIdOrInput, maybeInput);
|
|
const now = new Date();
|
|
const row = await (options?.db ?? db)
|
|
.insert(environments)
|
|
.values({
|
|
name: input.name,
|
|
description: input.description ?? null,
|
|
driver: input.driver,
|
|
status: input.status ?? "active",
|
|
config: input.config ?? {},
|
|
envVars: (input as CreateEnvironment & { envVars?: Record<string, unknown> }).envVars ?? {},
|
|
metadata: input.metadata ?? null,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null)
|
|
.catch((error) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
throw conflict(`An environment named "${input.name}" already exists for this instance.`);
|
|
}
|
|
if (hasConstraintName(error, "environments_local_driver_idx")) {
|
|
throw conflict("A local environment already exists for this instance.");
|
|
}
|
|
throw error;
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to create environment");
|
|
}
|
|
return toEnvironment(row);
|
|
},
|
|
|
|
update: async (
|
|
id: string,
|
|
patch: UpdateEnvironment,
|
|
options?: { db?: EnvironmentWriteDb },
|
|
): Promise<Environment | null> => {
|
|
const writeDb = options?.db ?? db;
|
|
const values: Partial<typeof environments.$inferInsert> = {
|
|
updatedAt: new Date(),
|
|
};
|
|
if (patch.name !== undefined) values.name = patch.name;
|
|
if (patch.description !== undefined) values.description = patch.description ?? null;
|
|
if (patch.driver !== undefined) values.driver = patch.driver;
|
|
if (patch.status !== undefined) values.status = patch.status;
|
|
if (patch.config !== undefined) values.config = patch.config;
|
|
if ("envVars" in patch && patch.envVars !== undefined) {
|
|
values.envVars = (patch.envVars ?? {}) as Record<string, unknown>;
|
|
}
|
|
const changesTarget = patch.config !== undefined || patch.driver !== undefined;
|
|
let metadataUpdate: SQL | undefined;
|
|
if (patch.metadata !== undefined) {
|
|
const userMetadata = withoutManagedEnvironmentArchiveToken(patch.metadata ?? null);
|
|
if (userMetadata) delete userMetadata[SANDBOX_PLATFORM_PROBE_METADATA_KEY];
|
|
if (changesTarget) {
|
|
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
|
|
metadataUpdate = preserveInvalidatedSnapshotProofs(replacement);
|
|
} else {
|
|
const replacement = userMetadata === null ? sql`null::jsonb` : sql`${JSON.stringify(userMetadata)}::jsonb`;
|
|
metadataUpdate = sql`case when ${environments.metadata} ? ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}
|
|
then coalesce(${replacement}, '{}'::jsonb) || jsonb_build_object(
|
|
${SANDBOX_PLATFORM_PROBE_METADATA_KEY}::text, ${environments.metadata}->${SANDBOX_PLATFORM_PROBE_METADATA_KEY})
|
|
else ${replacement} end`;
|
|
}
|
|
} else if (changesTarget || patch.status !== undefined) {
|
|
metadataUpdate = sql`${environments.metadata} - ${MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY}`;
|
|
if (changesTarget) metadataUpdate = preserveInvalidatedSnapshotProofs(sql`${metadataUpdate} - ${SANDBOX_PLATFORM_PROBE_METADATA_KEY}`);
|
|
}
|
|
|
|
const row = await writeDb
|
|
.update(environments)
|
|
.set({ ...values, ...(metadataUpdate ? { metadata: metadataUpdate } : {}) })
|
|
.where(eq(environments.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null)
|
|
.catch((error) => {
|
|
if (hasConstraintName(error, "environments_name_idx")) {
|
|
throw conflict(`An environment named "${patch.name}" already exists for this instance.`);
|
|
}
|
|
if (hasConstraintName(error, "environments_local_driver_idx")) {
|
|
throw conflict("A local environment already exists for this instance.");
|
|
}
|
|
throw error;
|
|
});
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
remove: async (id: string): Promise<Environment | null> => {
|
|
const row = await db
|
|
.delete(environments)
|
|
.where(eq(environments.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
removeIfDeletable: async (id: string): Promise<Environment | null> => {
|
|
const row = await db
|
|
.delete(environments)
|
|
.where(
|
|
and(
|
|
eq(environments.id, id),
|
|
ne(environments.driver, "local"),
|
|
sql`not exists (
|
|
select 1 from ${instanceSettings}
|
|
where ${instanceSettings.defaultEnvironmentId} = ${environments.id}
|
|
)`,
|
|
// A `pending_cleanup` lease is the durable teardown reference for an
|
|
// orphan sandbox. The environment foreign key uses
|
|
// `on delete set null`, so a delete keeps the lease row but drops its
|
|
// environment reference. This predicate refuses the delete while such
|
|
// a lease exists, so the operator resolves the cleanup first and the
|
|
// lease keeps its environment link. It runs in the same statement as
|
|
// the delete, so it also closes the check-to-delete race.
|
|
sql`not exists (
|
|
select 1 from ${environmentLeases}
|
|
where ${environmentLeases.environmentId} = ${environments.id}
|
|
and ${environmentLeases.status} = 'pending_cleanup'
|
|
)`,
|
|
// A reusable lease keeps a provider sandbox after a run releases
|
|
// it, including when the run failed but release succeeded.
|
|
// Deleting the environment would set its reference to
|
|
// null, and both the normal release path and scoped reusable cleanup
|
|
// require that environment context. Refuse the delete atomically
|
|
// until the owning issue/workspace destroys the reusable sandbox.
|
|
sql`not exists (
|
|
select 1 from ${environmentLeases}
|
|
where ${environmentLeases.environmentId} = ${environments.id}
|
|
and ${inArray(environmentLeases.leasePolicy, [...KEPT_SANDBOX_LEASE_POLICIES])}
|
|
and ${environmentLeases.status} in ('active', 'released', 'retained', 'failed')
|
|
)`,
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
return row ? toEnvironment(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Return true when the environment has one or more leases in the terminal
|
|
* `pending_cleanup` state. Each such lease is the only durable provider
|
|
* reference for an orphan sandbox that a teardown retry must destroy. The
|
|
* delete guard and the provider-change guard call this to protect that
|
|
* reference.
|
|
*/
|
|
hasUnresolvedPendingCleanupLeases: async (environmentId: string): Promise<boolean> => {
|
|
const rows = await db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, environmentId),
|
|
eq(environmentLeases.status, "pending_cleanup"),
|
|
),
|
|
);
|
|
return countFromRows(rows) > 0;
|
|
},
|
|
|
|
getDeleteBlastRadius: async (id: string): Promise<EnvironmentDeleteBlastRadius | null> => {
|
|
const environment = await db
|
|
.select({
|
|
id: environments.id,
|
|
driver: environments.driver,
|
|
})
|
|
.from(environments)
|
|
.where(eq(environments.id, id))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!environment) return null;
|
|
|
|
const [
|
|
instanceDefaultRows,
|
|
agentDefaultRows,
|
|
executionWorkspaceRows,
|
|
issueRows,
|
|
projectRows,
|
|
secretBindingRows,
|
|
activeLeaseRows,
|
|
pendingCleanupLeaseRows,
|
|
reusableSandboxLeaseRows,
|
|
activeSetupRows,
|
|
] = await Promise.all([
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(instanceSettings)
|
|
.where(eq(instanceSettings.defaultEnvironmentId, id)),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(agents)
|
|
.where(eq(agents.defaultEnvironmentId, id)),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(executionWorkspaces)
|
|
.where(sql`${executionWorkspaces.metadata} -> 'config' ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(issues)
|
|
.where(sql`${issues.executionWorkspaceSettings} ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(projects)
|
|
.where(sql`${projects.executionWorkspacePolicy} ->> 'environmentId' = ${id}`),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(companySecretBindings)
|
|
.where(
|
|
and(
|
|
eq(companySecretBindings.targetType, "environment"),
|
|
eq(companySecretBindings.targetId, id),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
eq(environmentLeases.status, "active"),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
eq(environmentLeases.status, "pending_cleanup"),
|
|
),
|
|
),
|
|
db
|
|
.select({
|
|
leaseId: environmentLeases.id,
|
|
leasePolicy: environmentLeases.leasePolicy,
|
|
executionWorkspaceId: environmentLeases.executionWorkspaceId,
|
|
executionWorkspaceName: executionWorkspaces.name,
|
|
issueId: environmentLeases.issueId,
|
|
issueIdentifier: issues.identifier,
|
|
issueTitle: issues.title,
|
|
})
|
|
.from(environmentLeases)
|
|
.leftJoin(executionWorkspaces, eq(environmentLeases.executionWorkspaceId, executionWorkspaces.id))
|
|
.leftJoin(issues, eq(environmentLeases.issueId, issues.id))
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.environmentId, id),
|
|
inArray(environmentLeases.leasePolicy, [...KEPT_SANDBOX_LEASE_POLICIES]),
|
|
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
|
|
),
|
|
),
|
|
db
|
|
.select({ count: sql<number>`count(*)::int` })
|
|
.from(environmentCustomImageSetupSessions)
|
|
.where(
|
|
and(
|
|
eq(environmentCustomImageSetupSessions.environmentId, id),
|
|
inArray(environmentCustomImageSetupSessions.status, [...ACTIVE_CUSTOM_IMAGE_SETUP_STATUSES]),
|
|
),
|
|
),
|
|
]);
|
|
|
|
const isManagedLocal = environment.driver === "local";
|
|
const isInstanceDefault = countFromRows(instanceDefaultRows) > 0;
|
|
const pendingCleanupLeaseCount = countFromRows(pendingCleanupLeaseRows);
|
|
const reusableSandboxLeaseHolders = reusableSandboxLeaseRows.map((row) => ({
|
|
leaseId: row.leaseId,
|
|
leasePolicy: row.leasePolicy,
|
|
executionWorkspaceId: row.executionWorkspaceId,
|
|
executionWorkspaceName: row.executionWorkspaceName,
|
|
issueId: row.issueId,
|
|
issueIdentifier: row.issueIdentifier,
|
|
issueTitle: row.issueTitle,
|
|
}));
|
|
const reusableSandboxLeaseCount = reusableSandboxLeaseHolders.length;
|
|
const deleteBlockedReasons: EnvironmentDeleteBlockedReason[] = [];
|
|
if (isManagedLocal) deleteBlockedReasons.push("managed_local");
|
|
if (isInstanceDefault) deleteBlockedReasons.push("instance_default");
|
|
// A `pending_cleanup` lease is the durable teardown reference for an orphan
|
|
// sandbox. The environment foreign key uses `on delete set null`, so a
|
|
// delete keeps the lease but drops its environment reference. Block the
|
|
// delete until the sweep resolves the lease, so the operator resolves the
|
|
// cleanup first and the lease keeps its environment link.
|
|
if (pendingCleanupLeaseCount > 0) deleteBlockedReasons.push("pending_sandbox_cleanup");
|
|
if (reusableSandboxLeaseCount > 0) deleteBlockedReasons.push("reusable_sandbox_lease");
|
|
const activeLeaseCount = countFromRows(activeLeaseRows);
|
|
const activeCustomImageSetupSessionCount = countFromRows(activeSetupRows);
|
|
|
|
return {
|
|
environmentId: id,
|
|
canDelete: deleteBlockedReasons.length === 0,
|
|
deleteBlockedReasons,
|
|
pendingCleanupLeaseCount,
|
|
reusableSandboxLeaseCount,
|
|
reusableSandboxLeaseHolders,
|
|
staticReferences: {
|
|
isManagedLocal,
|
|
isInstanceDefault,
|
|
agentDefaultCount: countFromRows(agentDefaultRows),
|
|
executionWorkspaceSelectionCount: countFromRows(executionWorkspaceRows),
|
|
issueSelectionCount: countFromRows(issueRows),
|
|
projectSelectionCount: countFromRows(projectRows),
|
|
secretBindingCount: countFromRows(secretBindingRows),
|
|
},
|
|
activeRuntimeUse: {
|
|
activeLeaseCount,
|
|
activeCustomImageSetupSessionCount,
|
|
hasActiveRuntimeUse: activeLeaseCount > 0 || activeCustomImageSetupSessionCount > 0,
|
|
},
|
|
};
|
|
},
|
|
|
|
listLeases: async (
|
|
environmentId: string,
|
|
filters: {
|
|
status?: string;
|
|
} = {},
|
|
): Promise<EnvironmentLease[]> => {
|
|
const conditions = [eq(environmentLeases.environmentId, environmentId)];
|
|
if (filters.status) conditions.push(eq(environmentLeases.status, filters.status));
|
|
const rows = await db
|
|
.select()
|
|
.from(environmentLeases)
|
|
.where(and(...conditions))
|
|
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
|
|
return rows.map(toEnvironmentLease);
|
|
},
|
|
|
|
acquireLease: async (input: {
|
|
creationOperationId?: string;
|
|
companyId: string;
|
|
environmentId: string;
|
|
executionWorkspaceId?: string | null;
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
leasePolicy?: EnvironmentLeasePolicy;
|
|
provider?: string | null;
|
|
providerLeaseId?: string | null;
|
|
expiresAt?: Date | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
/**
|
|
* Atomically retire the previous database ownership record when this
|
|
* acquisition reuses the same provider resource for a new run. The
|
|
* provider resume happens before this write, so a failed transaction
|
|
* leaves the prior retained row recoverable instead of publishing two
|
|
* reusable owners for one sandbox.
|
|
*/
|
|
replacesReusableLeaseId?: string | null;
|
|
/**
|
|
* Reactivate the exact lease row already owned by this heartbeat run.
|
|
* Native same-run recovery re-enters environment startup after the
|
|
* provider process is interrupted; it must not manufacture a second
|
|
* database owner for the same run and provider resource.
|
|
*/
|
|
reusesReusableLeaseId?: string | null;
|
|
/**
|
|
* Re-check the environment company binding inside the lease insert
|
|
* transaction. The login routes set this to close the check-to-lease
|
|
* race: managed reconciliation can bind a sandbox to another company
|
|
* between the route guard and this acquire. When the environment is
|
|
* bound to a company other than `companyId`, the insert throws the 403
|
|
* `environment_company_mismatch` and no lease row is created (fail
|
|
* closed). An unbound (instance-global) environment stays open to every
|
|
* member. All other callers keep the plain, non-transactional insert.
|
|
*/
|
|
assertCompanyBinding?: boolean;
|
|
}): Promise<EnvironmentLease> => {
|
|
if (input.creationOperationId) {
|
|
const { creationOperationId, ...leaseInput } = input;
|
|
return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId },
|
|
tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true }));
|
|
}
|
|
const now = new Date();
|
|
const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null;
|
|
const values = {
|
|
...(replacementLeaseId ? { id: replacementLeaseId } : {}),
|
|
companyId: input.companyId,
|
|
environmentId: input.environmentId,
|
|
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
|
issueId: input.issueId ?? null,
|
|
heartbeatRunId: input.heartbeatRunId ?? null,
|
|
status: "active" as const,
|
|
leasePolicy: input.leasePolicy ?? "ephemeral",
|
|
provider: input.provider ?? null,
|
|
providerLeaseId: input.providerLeaseId ?? null,
|
|
acquiredAt: now,
|
|
lastUsedAt: now,
|
|
expiresAt: input.expiresAt ?? null,
|
|
releasedAt: null,
|
|
failureReason: null,
|
|
cleanupStatus: null,
|
|
// The holder stamp is written with the row that makes the lease active.
|
|
metadata: withEnvironmentLeaseController(withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata))),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
};
|
|
if (
|
|
(input.replacesReusableLeaseId || input.reusesReusableLeaseId) &&
|
|
((!input.executionWorkspaceId && (!input.issueId || typeof input.metadata?.agentId !== "string")) || !input.providerLeaseId)
|
|
) {
|
|
throw new Error(
|
|
"A reusable lease handoff requires a workspace or agent-bound task, and a provider lease id.",
|
|
);
|
|
}
|
|
if (input.reusesReusableLeaseId && !input.heartbeatRunId) {
|
|
throw new Error(
|
|
"A same-run reusable lease reacquisition requires a heartbeat run id.",
|
|
);
|
|
}
|
|
if (input.replacesReusableLeaseId && input.reusesReusableLeaseId) {
|
|
throw new Error(
|
|
"A reusable lease cannot be replaced and reacquired in the same operation.",
|
|
);
|
|
}
|
|
const row =
|
|
input.assertCompanyBinding ||
|
|
input.replacesReusableLeaseId ||
|
|
input.reusesReusableLeaseId
|
|
? await db.transaction(async (tx) => {
|
|
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
|
|
companyId: input.companyId, runId: input.heartbeatRunId,
|
|
});
|
|
if (input.assertCompanyBinding) {
|
|
await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId);
|
|
}
|
|
if (input.replacesReusableLeaseId) {
|
|
await notifyChatInboundWakeNotices(tx, { companyId: input.companyId });
|
|
const retired = await tx
|
|
.update(environmentLeases)
|
|
.set({
|
|
status: "expired",
|
|
releasedAt: now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
cleanupStatus: "success",
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.id, input.replacesReusableLeaseId),
|
|
eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.environmentId, input.environmentId),
|
|
sql`${environmentLeases.provider} is not distinct from ${input.provider ?? null}`,
|
|
input.executionWorkspaceId
|
|
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
|
|
: and(
|
|
isNull(environmentLeases.executionWorkspaceId),
|
|
eq(environmentLeases.issueId, input.issueId!),
|
|
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
|
|
),
|
|
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
|
eq(
|
|
environmentLeases.providerLeaseId,
|
|
input.providerLeaseId!,
|
|
),
|
|
inArray(environmentLeases.status, [
|
|
"released",
|
|
"retained",
|
|
]),
|
|
// A source whose harness backup was restored into another
|
|
// lease holds older session state. That claim and this
|
|
// handoff compete on this row, so only one of them wins.
|
|
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'reusableSandboxLeaseSuperseded')`,
|
|
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'portableTaskSuccessor')`,
|
|
),
|
|
)
|
|
.returning();
|
|
if (retired.length !== 1) {
|
|
throw conflict(
|
|
"Reusable sandbox lease ownership changed during acquisition.",
|
|
);
|
|
}
|
|
const handoff = createReusableLeaseHandoff(retired[0]!, { ...values, id: replacementLeaseId! });
|
|
values.metadata = { ...values.metadata, reusableLeasePredecessor: handoff };
|
|
await tx.update(environmentLeases).set({ metadata: {
|
|
...retired[0]!.metadata, reusableLeaseSuccessor: handoff,
|
|
} }).where(eq(environmentLeases.id, retired[0]!.id));
|
|
}
|
|
if (input.reusesReusableLeaseId) {
|
|
const [previous] = await tx.select({ metadata: environmentLeases.metadata }).from(environmentLeases)
|
|
.where(and(eq(environmentLeases.id, input.reusesReusableLeaseId), eq(environmentLeases.companyId, input.companyId))).for("update");
|
|
const reacquired = await tx
|
|
.update(environmentLeases)
|
|
.set({
|
|
status: "active",
|
|
releasedAt: null,
|
|
lastUsedAt: now,
|
|
expiresAt: input.expiresAt ?? null,
|
|
failureReason: null,
|
|
cleanupStatus: null,
|
|
metadata: withEnvironmentLeaseController(
|
|
preserveReusableLeaseHandoff(previous?.metadata ?? null, withoutCleanupVerification(input.metadata))),
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.id, input.reusesReusableLeaseId),
|
|
eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.environmentId, input.environmentId),
|
|
input.executionWorkspaceId
|
|
? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
|
|
: and(
|
|
isNull(environmentLeases.executionWorkspaceId),
|
|
eq(environmentLeases.issueId, input.issueId!),
|
|
sql`${environmentLeases.metadata}->>'agentId' = ${input.metadata!.agentId}`,
|
|
),
|
|
eq(
|
|
environmentLeases.heartbeatRunId,
|
|
input.heartbeatRunId!,
|
|
),
|
|
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
|
eq(
|
|
environmentLeases.providerLeaseId,
|
|
input.providerLeaseId!,
|
|
),
|
|
inArray(environmentLeases.status, [
|
|
"active",
|
|
"released",
|
|
"retained",
|
|
]),
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!reacquired) {
|
|
throw conflict(
|
|
"Reusable sandbox lease ownership changed during reacquisition.",
|
|
);
|
|
}
|
|
return reacquired;
|
|
}
|
|
return tx
|
|
.insert(environmentLeases)
|
|
.values(values)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
})
|
|
: await db.transaction(async tx => {
|
|
if (input.heartbeatRunId) await notifyNativeEnvironmentSettlement(tx, {
|
|
companyId: input.companyId, runId: input.heartbeatRunId,
|
|
});
|
|
return tx.insert(environmentLeases).values(values).returning().then(rows => rows[0] ?? null);
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to acquire environment lease");
|
|
}
|
|
return toEnvironmentLease(row);
|
|
},
|
|
|
|
releaseLease: async (
|
|
id: string,
|
|
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed" | "retained" | "pending_cleanup"> = "released",
|
|
options?: {
|
|
expectedPendingCleanupAttemptId?: string;
|
|
failureReason?: string;
|
|
cleanupStatus?: EnvironmentLeaseCleanupStatus;
|
|
remoteExecutionTermination?: Record<string, unknown>;
|
|
},
|
|
) => {
|
|
const now = new Date();
|
|
const releaseMetadata = options?.remoteExecutionTermination
|
|
? sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ remoteExecutionTermination: options.remoteExecutionTermination })}::jsonb`
|
|
: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'`;
|
|
const cleanupSettled = options?.expectedPendingCleanupAttemptId && options.cleanupStatus === "success"
|
|
&& status !== "pending_cleanup";
|
|
const row = await db.transaction(async tx => {
|
|
if (options?.remoteExecutionTermination?.state === "destroyed") {
|
|
// Workspace preparation locks its authority before the lease. Keep
|
|
// that order when confirmed destruction clears the same authority.
|
|
const [identity] = await tx.select({ companyId: environmentLeases.companyId }).from(environmentLeases)
|
|
.where(eq(environmentLeases.id, id));
|
|
if (identity) await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, identity.companyId),
|
|
sql`${executionWorkspaces.metadata}->'remoteWorkspace'->'location'->>'leaseId' = ${id}`,
|
|
)).orderBy(asc(executionWorkspaces.id)).for("update");
|
|
}
|
|
const condition = and(eq(environmentLeases.id, id), options?.expectedPendingCleanupAttemptId
|
|
? and(eq(environmentLeases.status, "pending_cleanup"),
|
|
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.expectedPendingCleanupAttemptId}`)
|
|
: undefined);
|
|
const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId,
|
|
heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status,
|
|
cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt,
|
|
creationOperationId: sql<string | null>`${environmentLeases.metadata}->>'creationOperationId'`,
|
|
}).from(environmentLeases).where(condition).for("update");
|
|
if (!previous) return null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined, leaseId: id });
|
|
if (previous.heartbeatRunId && (previous.status !== status ||
|
|
(options?.cleanupStatus !== undefined && previous.cleanupStatus !== options.cleanupStatus) ||
|
|
(status !== "retained" && !previous.releasedAt))) {
|
|
await notifyChatInboundWakeNotices(tx, {
|
|
companyId: previous.companyId, issueId: previous.issueId ?? undefined, runId: previous.heartbeatRunId,
|
|
});
|
|
}
|
|
if (previous.heartbeatRunId && ["active", "pending_cleanup"].includes(previous.status)
|
|
&& status !== "pending_cleanup") {
|
|
await notifyRetainedNativeSessionCleanup(tx, {
|
|
companyId: previous.companyId, runId: previous.heartbeatRunId,
|
|
});
|
|
// Messages saved behind a finished run wait for its cleanup to
|
|
// settle. Reconsider them through normal deferred admission.
|
|
await notifyDeferredExecutionWork(tx, { companyId: previous.companyId });
|
|
await promoteDeferredWorkIntents(tx, { companyId: previous.companyId, trigger: "lock_released", issueId: previous.issueId,
|
|
runId: previous.heartbeatRunId });
|
|
}
|
|
if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
|
|
await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
|
|
.where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId),
|
|
eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending")));
|
|
}
|
|
const released = await tx.update(environmentLeases)
|
|
.set({
|
|
status,
|
|
releasedAt: status === "retained" ? null : now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
...(options?.failureReason !== undefined ? { failureReason: options.failureReason } : {}),
|
|
...(options?.cleanupStatus !== undefined ? { cleanupStatus: options.cleanupStatus } : {}),
|
|
// A later release without a receipt cannot reuse an earlier stop's
|
|
// authority (for example after a same-run lease resume).
|
|
metadata: cleanupSettled
|
|
? sql`coalesce(${releaseMetadata}, '{}'::jsonb) || '{"pendingCleanupInFlight":false,"pendingCleanupLeaseExpiresAtMs":0}'::jsonb`
|
|
: releaseMetadata,
|
|
})
|
|
.where(condition)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (released) await clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: toEnvironmentLease(released) });
|
|
return released;
|
|
});
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
/**
|
|
* Record a lease-less orphan sandbox directly in the terminal
|
|
* `pending_cleanup` state with one atomic insert. The sweep reads a row with
|
|
* status `pending_cleanup` and cleanup status `failed`, so this insert makes
|
|
* the orphan visible to recovery immediately. It never passes through the
|
|
* `active` state, so a process or database crash cannot strand the row in an
|
|
* intermediate state that no sweep finds. The insert skips the company
|
|
* binding assertion, so it records the orphan even for a foreign-bound
|
|
* environment.
|
|
*
|
|
* The insert runs in a transaction that first locks the environment row with
|
|
* `for update`. The delete path (`removeIfDeletable`) refuses a delete while a
|
|
* `pending_cleanup` lease exists, so this lock serializes the two writes:
|
|
*
|
|
* - The environment still exists: the insert records the orphan with the
|
|
* environment reference. A concurrent delete blocks on the lock, then its
|
|
* `not exists (pending_cleanup)` predicate fails, so the delete refuses.
|
|
* - The environment is already gone (a delete won the race before this
|
|
* insert): the insert records the orphan with a null environment
|
|
* reference. The row carries the immutable provider metadata the sweep
|
|
* needs, so the teardown still runs. This closes the window the finding
|
|
* describes, where the foreign-key insert fails after a delete.
|
|
*/
|
|
insertPendingCleanupLease: async (input: {
|
|
companyId: string;
|
|
environmentId: string;
|
|
executionWorkspaceId?: string | null;
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
provider?: string | null;
|
|
providerLeaseId?: string | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
failureReason: string;
|
|
}): Promise<EnvironmentLease> => {
|
|
const now = new Date();
|
|
const row = await db.transaction(async (tx) => {
|
|
// Lock the environment row so a concurrent delete cannot commit between
|
|
// this read and the insert. An absent row means a delete already removed
|
|
// the environment, so record the orphan with a null reference.
|
|
const environmentRows = await tx
|
|
.select({ id: environments.id })
|
|
.from(environments)
|
|
.where(eq(environments.id, input.environmentId))
|
|
.for("update");
|
|
const environmentIdForRow = environmentRows[0]?.id ?? null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: input.companyId, runId: input.heartbeatRunId ?? undefined });
|
|
return tx
|
|
.insert(environmentLeases)
|
|
.values({
|
|
companyId: input.companyId,
|
|
environmentId: environmentIdForRow,
|
|
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
|
issueId: input.issueId ?? null,
|
|
heartbeatRunId: input.heartbeatRunId ?? null,
|
|
status: "pending_cleanup" as const,
|
|
leasePolicy: "ephemeral",
|
|
provider: input.provider ?? null,
|
|
providerLeaseId: input.providerLeaseId ?? null,
|
|
acquiredAt: now,
|
|
lastUsedAt: now,
|
|
expiresAt: null,
|
|
releasedAt: now,
|
|
failureReason: input.failureReason,
|
|
cleanupStatus: "failed",
|
|
metadata: withoutReusableLeaseHandoff(withoutCleanupVerification(input.metadata)),
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
if (!row) {
|
|
throw new Error("Failed to record pending sandbox cleanup lease");
|
|
}
|
|
return toEnvironmentLease(row);
|
|
},
|
|
|
|
updateLeaseMetadata: async (
|
|
id: string,
|
|
metadata: Record<string, unknown> | null,
|
|
): Promise<EnvironmentLease | null> => {
|
|
const row = await db.transaction(async tx => {
|
|
const [lease] = await tx.select({ companyId: environmentLeases.companyId, heartbeatRunId: environmentLeases.heartbeatRunId, metadata: environmentLeases.metadata })
|
|
.from(environmentLeases).where(eq(environmentLeases.id, id)).for("update");
|
|
if (!lease) return null;
|
|
await notifyEnvironmentCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined, leaseId: id });
|
|
return tx.update(environmentLeases)
|
|
.set({
|
|
metadata: preserveReusableLeaseHandoff(lease.metadata, withoutCleanupVerification(metadata)),
|
|
lastUsedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environmentLeases.id, id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
return row ? toEnvironmentLease(row) : null;
|
|
},
|
|
|
|
releaseLeasesForRun: async (
|
|
heartbeatRunId: string,
|
|
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
|
|
): Promise<EnvironmentLease[]> => {
|
|
const now = new Date();
|
|
const rows = await db.transaction(async tx => {
|
|
const [run] = await tx.select({ companyId: heartbeatRuns.companyId,
|
|
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')` })
|
|
.from(heartbeatRuns).where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
|
|
if (run) {
|
|
await notifyEnvironmentCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
|
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
|
await notifyDeferredExecutionWork(tx, { companyId: run.companyId });
|
|
await promoteDeferredWorkIntents(tx, { companyId: run.companyId, trigger: "lock_released",
|
|
issueId: run.issueId && /^[0-9a-f-]{36}$/i.test(run.issueId) ? run.issueId : null, runId: heartbeatRunId });
|
|
}
|
|
return tx.update(environmentLeases)
|
|
.set({
|
|
status,
|
|
releasedAt: now,
|
|
lastUsedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.heartbeatRunId, heartbeatRunId),
|
|
eq(environmentLeases.status, "active"),
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
return rows.map(toEnvironmentLease);
|
|
},
|
|
};
|
|
}
|