diff --git a/doc/DATABASE.md b/doc/DATABASE.md index ac0d63bb728..bb4c964611d 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1545,6 +1545,23 @@ named `review_wait_benchmark...` can be supplied through `PAPERCLIP_TEST_DATABASE_URL` to compare another PostgreSQL major version. DataRow bytes exclude response metadata and transport overhead. + +### Sandbox creation ownership + +Migration 0418 adds `sandbox_creation_operations`. Admission commits company, +original environment, run, request key and provider identity before external +creation. Configuration contains nonsecret values and existing secret references. +The original identity survives environment, run, task and company deletion; only +the optional live environment FK uses SET NULL. Company/run deletion must not +erase an outstanding remote-resource obligation. + +The unique company/environment/run/request index fences duplicate creation and +ambiguous successful lease commits. Normal lease attachment and operation transfer +share a transaction. Recovery transfers one claimed cleanup lease after provider +ownership lookup. The pending deadline index drives the existing cleanup worker. +`needs_reconciliation` is a durable manual outcome, not an automatic retry or an +idle blocker. See [cleanup ownership](idle-environment-cleanup.md). + ## Object-backed run evidence Optional segmented run logs and raw provider traces store their payloads and diff --git a/doc/idle-environment-cleanup.md b/doc/idle-environment-cleanup.md index a8d2ecdff02..d999536cdfc 100644 --- a/doc/idle-environment-cleanup.md +++ b/doc/idle-environment-cleanup.md @@ -11,7 +11,7 @@ or an orphaned active lease. Explicit user retry keeps its existing direct path. | Duty | Truth | Trigger and next deadline | Quiet condition | | --- | --- | --- | --- | | Failed teardown | Existing `environment_leases` rows in `pending_cleanup` | Commit signal; later of row backoff and saved retry/ownership deadline | No pending lease or deferred orphan | -| Failed acquisition before a DB row exists | Existing local orphan spool and the producing runtime's buffer | Local buffer registration; five-minute retry while buffered; startup spool load | Durable row owns recovery and spool copy is removed | +| Interrupted acquisition | `sandbox_creation_operations` committed before provider creation | Transaction signal; saved acquisition deadline; five-minute backoff after recovery failure | Normal lease or one claimed cleanup lease owns the resource; unsupported uncertainty requires operator reconciliation | | Historical provider verification | Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof | Commit signal; five-minute retry while an unverified candidate remains | No unverified candidates, including retained verified history | `environment-cleanup-state.ts` owns the deadline query and the shared safe reader @@ -47,8 +47,9 @@ settlement and uncertainty handling through the shared DB implementation. - `heartbeat/recovery.ts` and `heartbeat/run-control.ts`: orphan-active recovery, retry settlement, and explicit recovery of historical leases. - `environment-task-runtime.ts`: unconfirmed plugin task termination and plugin - replacement orphans, including a registered deferred flusher when the row - cannot be written. The flusher retains an idle hold until the row is durable. + replacement creations. Replacement intent commits before the provider call. +- `sandbox-creation-operations.ts`: admission, atomic lease transfer, and bounded + recovery. An unavailable database stops admission before external creation. - `sandbox-stop-and-retain.ts`: persisted Stop intent and its settlement. - `native-workspace-export-resume.ts` and `native-workspace-export-retry.ts`: terminal-export cleanup, resumed-export cleanup claims, and settlement. @@ -64,7 +65,49 @@ already pending claim remain covered by its existing deadline. Deletion removes work; an already scheduled reconciliation observes that removal. Independent SQL writers and another server process are not covered by these process-local signals. -## Deferred orphan buffers +## Creation ownership + +A creation operation retains company, original environment, run, request identity, +provider, and nonsecret configuration references. Its environment FK uses SET NULL; +company/run/task identifiers remain after those records are removed. One operation +owns each company/environment/run/request. A retry of an attached operation returns +its active lease. Pending or unresolved ownership never authorizes another create. +Normal success transfers operation ownership and the lease in one transaction. +Current company binding and run authority are checked again at attachment. +Declared secret-reference fields retain only their references and nonsecret value +hashes. Recovery refuses lookup under a changed credential value; it records manual +reconciliation instead of treating a missing resource in a new account as proof. +A missing, deleted, disabled, or archived saved secret, a missing or inactive saved version, +or an explicitly disabled/coming-soon provider vault also records manual +reconciliation before provider lookup. Admin-disabled credentials require operator +reconciliation; recovery does not reactivate or use them. Temporary secret-provider +or database failures retain the pending operation and its retry deadline. + +Recovery never resumes partially prepared execution. It asks a capable provider to +inspect the stable creation identity, validates its ownership envelope, and transfers +one cleanup lease under the operation lock. The existing claimed cleanup worker +owns deletion and confirms resolution. Failed lookup/transfer records a bounded +retry and permits other operations to proceed. An attached operation cannot transfer +to cleanup. Pending creation blocks instance sleep until its obligation transfers. + +Daytona version 0.1.10 declares creationOperationVersion 1. It uses the committed +operation UUID in its existing creation name and labels. Recovery uses an explicit +creationOperation request on the existing read-only inspect hook and strict failed-creation deletion path. Normal lease +inspection still uses its recorded resource ID; creationOperationId metadata is +provenance only. A missing name alone cannot prove a late allocation will never appear. Missing identity or an +unsupported provider records needs_reconciliation and an operation-scoped warning. +A deleted company also records manual reconciliation before credential lookup or +cleanup transfer. A company lock fences deletion during the transfer transaction. +That manual diagnostic has no automatic timer and does not prevent idle. The row +must be reconciled before retrying the same request. Other providers do not yet +claim automatic creation lookup. Generic plugin operations retain identity only, including a returned task resource +ID before expiry checks or lease attachment. A failed or ambiguous attachment keeps +that known ID for reconciliation; it does not authorize unconditional release. +These operations do not copy credentials, resolved configuration, or provider +payloads into the operation record. A lost provider reply can still leave the +resource ID unknown. + +## Legacy deferred orphan buffers Runtimes that hold an unwritten orphan register their flusher in a registry scoped to the shared database handle. Registration wakes the worker without contacting @@ -75,8 +118,9 @@ runtime. The registry retains only runtimes with unresolved buffer ownership. Unreadable directories, records, and malformed records retain a retry; a missing spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and persisted leases to clean up, then fails the sweep so recovery stays scheduled. -Each concrete cleanup pass reloads the spool, including entries appended by -plugin task replacement after startup and entries beyond buffer capacity. +Each concrete cleanup pass reloads existing legacy spool entries, including entries +beyond buffer capacity. Normal acquisition and plugin task replacement do not append +new fallback files. Retained entries remain available for the separate cutover inventory. No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database, spool, and buffer-capacity failure remain; this is not a new durable queue. diff --git a/doc/skill-ephemeral-web-servers.md b/doc/skill-ephemeral-web-servers.md index 9de48505f96..f3c5b4e9a30 100644 --- a/doc/skill-ephemeral-web-servers.md +++ b/doc/skill-ephemeral-web-servers.md @@ -131,6 +131,13 @@ perform provider migration: `local_storage` and `provider_mismatch` are blocking results for cloud qualification, not instructions to switch configuration and retry. A directory of extracted caches is never a durable object backend. +The maintenance CLI reads the existing storage configuration directly so it can +verify legacy local-disk objects before cutover. On Cloud it still requires the +persisted runtime configuration and never seeds or changes that configuration. +This maintenance boundary does not start a server or relax Cloud server startup +requirements for external PostgreSQL, shared object storage and a stable secret +key. Apply operations retain their existing stopped-writer gates. + Conversion establishes the rollback floor. Afterward only compatible version-reading binaries may overlap or be rolled back against the same remote state. Reattaching an old volume does not roll back committed content. Filesystem-authoritative binaries diff --git a/packages/db/src/migrations/0418_curved_lady_mastermind.sql b/packages/db/src/migrations/0418_curved_lady_mastermind.sql new file mode 100644 index 00000000000..c0d7fe0d515 --- /dev/null +++ b/packages/db/src/migrations/0418_curved_lady_mastermind.sql @@ -0,0 +1,33 @@ +CREATE TABLE IF NOT EXISTS "sandbox_creation_operations" ( + "id" uuid PRIMARY KEY NOT NULL, + "company_id" uuid NOT NULL, + "environment_id" uuid, + "original_environment_id" uuid NOT NULL, + "run_id" uuid NOT NULL, + "request_key" text DEFAULT 'run' NOT NULL, + "issue_id" uuid, + "execution_workspace_id" uuid, + "provider" text NOT NULL, + "plugin_id" uuid, + "driver_key" text NOT NULL, + "state" text DEFAULT 'pending' NOT NULL, + "config" jsonb NOT NULL, + "metadata" jsonb NOT NULL, + "lease_id" uuid, + "next_attempt_at" timestamp with time zone NOT NULL, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +DO $$ BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_constraint + WHERE conname = 'sandbox_creation_operations_environment_id_environments_id_fk' + AND conrelid = 'sandbox_creation_operations'::regclass + ) THEN + ALTER TABLE "sandbox_creation_operations" ADD CONSTRAINT "sandbox_creation_operations_environment_id_environments_id_fk" FOREIGN KEY ("environment_id") REFERENCES "public"."environments"("id") ON DELETE set null ON UPDATE no action; + END IF; +END $$; +--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "sandbox_creation_operations_pending_deadline_idx" ON "sandbox_creation_operations" USING btree ("next_attempt_at") WHERE "sandbox_creation_operations"."state" = 'pending';--> statement-breakpoint +CREATE UNIQUE INDEX IF NOT EXISTS "sandbox_creation_operations_owner_idx" ON "sandbox_creation_operations" USING btree ("company_id","original_environment_id","run_id","request_key"); \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0413_snapshot.json b/packages/db/src/migrations/meta/0418_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0413_snapshot.json rename to packages/db/src/migrations/meta/0418_snapshot.json index c27c8a4f7b2..23ede87cb5d 100644 --- a/packages/db/src/migrations/meta/0413_snapshot.json +++ b/packages/db/src/migrations/meta/0418_snapshot.json @@ -1,6 +1,6 @@ { - "id": "5690a14a-f9cc-41a6-af51-c27f688f4ef6", - "prevId": "3b8bb508-028d-4154-abdd-af870f1f3ca6", + "id": "c777fa84-85a6-418f-bea1-0d60b56cf071", + "prevId": "c2c69a4a-8281-419c-9eeb-a7bcbc2beaae", "version": "7", "dialect": "postgresql", "tables": { @@ -1351,6 +1351,39 @@ "method": "btree", "with": {} }, + "activity_log_company_entity_created_idx": { + "name": "activity_log_company_entity_created_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "activity_log_company_user_issue_actor_idx": { "name": "activity_log_company_user_issue_actor_idx", "columns": [ @@ -19019,6 +19052,98 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.company_skill_mutation_receipts": { + "name": "company_skill_mutation_receipts", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "fingerprint": { + "name": "fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "skill_id": { + "name": "skill_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "version_id": { + "name": "version_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "response": { + "name": "response", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "company_skill_mutation_receipts_key_idx": { + "name": "company_skill_mutation_receipts_key_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "company_skill_mutation_receipts_company_id_companies_id_fk": { + "name": "company_skill_mutation_receipts_company_id_companies_id_fk", + "tableFrom": "company_skill_mutation_receipts", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_skill_repository_snapshots": { "name": "company_skill_repository_snapshots", "schema": "", @@ -19046,7 +19171,46 @@ "name": "file_inventory", "type": "jsonb", "primaryKey": false, - "notNull": true + "notNull": true, + "default": "'[]'::jsonb" + }, + "storage": { + "name": "storage", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "upload": { + "name": "upload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "restore": { + "name": "restore", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "maintenance_at": { + "name": "maintenance_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "restore_retry_at": { + "name": "restore_retry_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false }, "created_at": { "name": "created_at", @@ -19077,6 +19241,38 @@ "concurrently": false, "method": "btree", "with": {} + }, + "company_skill_snapshots_migration_idx": { + "name": "company_skill_snapshots_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_snapshots_maintenance_idx": { + "name": "company_skill_snapshots_maintenance_idx", + "columns": [ + { + "expression": "maintenance_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -19966,6 +20162,25 @@ "primaryKey": false, "notNull": false }, + "package_snapshot_id": { + "name": "package_snapshot_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "package_file_map": { + "name": "package_file_map", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, "revision_number": { "name": "revision_number", "type": "integer", @@ -20024,6 +20239,52 @@ } }, "indexes": { + "company_skill_versions_migration_idx": { + "name": "company_skill_versions_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_versions\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_package_snapshot_idx": { + "name": "company_skill_versions_package_snapshot_idx", + "columns": [ + { + "expression": "package_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_repository_snapshot_idx": { + "name": "company_skill_versions_repository_snapshot_idx", + "columns": [ + { + "expression": "repository_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "company_skill_versions_skill_revision_idx": { "name": "company_skill_versions_skill_revision_idx", "columns": [ @@ -20135,6 +20396,19 @@ "onDelete": "restrict", "onUpdate": "no action" }, + "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": { + "name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk", + "tableFrom": "company_skill_versions", + "tableTo": "company_skill_repository_snapshots", + "columnsFrom": [ + "package_snapshot_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + }, "company_skill_versions_author_agent_id_agents_id_fk": { "name": "company_skill_versions_author_agent_id_agents_id_fk", "tableFrom": "company_skill_versions", @@ -20580,6 +20854,78 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.skill_package_deletions": { + "name": "skill_package_deletions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "archive_key": { + "name": "archive_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manifest_key": { + "name": "manifest_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delete_after": { + "name": "delete_after", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "skill_package_deletions_due_idx": { + "name": "skill_package_deletions_due_idx", + "columns": [ + { + "expression": "delete_after", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_transfer_runs": { "name": "company_transfer_runs", "schema": "", @@ -33816,7 +34162,7 @@ } ], "isUnique": false, - "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE", + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE", "concurrently": false, "method": "btree", "with": {} @@ -51228,6 +51574,195 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.sandbox_creation_operations": { + "name": "sandbox_creation_operations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "environment_id": { + "name": "environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "original_environment_id": { + "name": "original_environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "request_key": { + "name": "request_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'run'" + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "execution_workspace_id": { + "name": "execution_workspace_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "plugin_id": { + "name": "plugin_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "driver_key": { + "name": "driver_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "config": { + "name": "config", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "lease_id": { + "name": "lease_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "next_attempt_at": { + "name": "next_attempt_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "sandbox_creation_operations_pending_deadline_idx": { + "name": "sandbox_creation_operations_pending_deadline_idx", + "columns": [ + { + "expression": "next_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"sandbox_creation_operations\".\"state\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "sandbox_creation_operations_owner_idx": { + "name": "sandbox_creation_operations_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "original_environment_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "request_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "sandbox_creation_operations_environment_id_environments_id_fk": { + "name": "sandbox_creation_operations_environment_id_environments_id_fk", + "tableFrom": "sandbox_creation_operations", + "tableTo": "environments", + "columnsFrom": [ + "environment_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.secret_access_events": { "name": "secret_access_events", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index e63e3338e21..47d9d449f1a 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2913,6 +2913,13 @@ "when": 1793371684483, "tag": "0417_chunky_red_hulk", "breakpoints": true + }, + { + "idx": 418, + "version": "7", + "when": 1793395364937, + "tag": "0418_curved_lady_mastermind", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 5c281a447c0..6d5bcc4ff30 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -39,6 +39,7 @@ export { } from "./execution_workspace_restore_points.js"; export { executionWorkspaceRuntimeLeases } from "./execution_workspace_runtime_leases.js"; export { environments } from "./environments.js"; +export { sandboxCreationOperations } from "./sandbox_creation_operations.js"; export { environmentLeases } from "./environment_leases.js"; export { environmentCustomImageTemplates } from "./environment_custom_image_templates.js"; export { environmentCustomImageSetupSessions } from "./environment_custom_image_setup_sessions.js"; diff --git a/packages/db/src/schema/sandbox_creation_operations.ts b/packages/db/src/schema/sandbox_creation_operations.ts new file mode 100644 index 00000000000..bb81e503793 --- /dev/null +++ b/packages/db/src/schema/sandbox_creation_operations.ts @@ -0,0 +1,27 @@ +import { index, jsonb, pgTable, text, timestamp, uuid, uniqueIndex } from "drizzle-orm/pg-core"; +import { sql } from "drizzle-orm"; +import { environments } from "./environments.js"; + +export const sandboxCreationOperations = pgTable("sandbox_creation_operations", { + id: uuid("id").primaryKey(), + companyId: uuid("company_id").notNull(), + environmentId: uuid("environment_id").references(() => environments.id, { onDelete: "set null" }), + originalEnvironmentId: uuid("original_environment_id").notNull(), + runId: uuid("run_id").notNull(), + requestKey: text("request_key").notNull().default("run"), + issueId: uuid("issue_id"), + executionWorkspaceId: uuid("execution_workspace_id"), + provider: text("provider").notNull(), + pluginId: uuid("plugin_id"), + driverKey: text("driver_key").notNull(), + state: text("state").notNull().default("pending"), + config: jsonb("config").$type>().notNull(), + metadata: jsonb("metadata").$type>().notNull(), + leaseId: uuid("lease_id"), + nextAttemptAt: timestamp("next_attempt_at", { withTimezone: true }).notNull(), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), +}, table => ({ + pendingDeadline: index("sandbox_creation_operations_pending_deadline_idx").on(table.nextAttemptAt).where(sql`${table.state} = 'pending'`), + owner: uniqueIndex("sandbox_creation_operations_owner_idx").on(table.companyId, table.originalEnvironmentId, table.runId, table.requestKey), +})); diff --git a/packages/plugins/sandbox-providers/daytona/README.md b/packages/plugins/sandbox-providers/daytona/README.md index 993b58f3c73..8f27d409e7e 100644 --- a/packages/plugins/sandbox-providers/daytona/README.md +++ b/packages/plugins/sandbox-providers/daytona/README.md @@ -63,3 +63,22 @@ second export after reconnect. The test creates and deletes one disposable - `src/manifest.ts` declares the sandbox-provider driver metadata - `src/plugin.ts` implements the environment lifecycle hooks - `paperclipPlugin.manifest` and `paperclipPlugin.worker` point the host at the built plugin entrypoints in `dist/` + + +### Interrupted run-lease creation + +The host commits a company-scoped creation operation before acquiring a new run +lease. The operation UUID becomes `paperclip-create-` and the +`paperclip-create-attempt` label. The existing inspection hook can resolve that +name and validate company, environment and run labels without starting a sandbox. +Recovery persists the observed provider ID before the existing cleanup worker +deletes it. A missing creation name is an uncertain outcome, not proof of removal. +The host retains that operation for manual reconciliation and never repeats the +creation request blindly. Credential identity must still match admission. + +This capability is declared as `creationOperationVersion: 1` in manifest version +0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement. +Legacy provider `environmentProbe` and custom-image interactive setup have separate +lifecycles and are outside this lease-orphan fallback replacement. The normal Test +path which acquires a runtime lease is covered. Unit fixtures exercise lookup and +deletion; live provider qualification requires an explicit disposable test account. diff --git a/packages/plugins/sandbox-providers/daytona/package.json b/packages/plugins/sandbox-providers/daytona/package.json index 84007584468..cbf36aedffb 100644 --- a/packages/plugins/sandbox-providers/daytona/package.json +++ b/packages/plugins/sandbox-providers/daytona/package.json @@ -1,6 +1,6 @@ { "name": "@paperclipai/plugin-daytona", - "version": "0.1.9", + "version": "0.1.10", "description": "Daytona sandbox provider plugin for Paperclip environments", "license": "MIT", "homepage": "https://github.com/paperclipai/paperclip", diff --git a/packages/plugins/sandbox-providers/daytona/src/manifest.ts b/packages/plugins/sandbox-providers/daytona/src/manifest.ts index fd3b668f544..a4bb68037de 100644 --- a/packages/plugins/sandbox-providers/daytona/src/manifest.ts +++ b/packages/plugins/sandbox-providers/daytona/src/manifest.ts @@ -15,7 +15,7 @@ export const DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS = 300_000; // 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner. // 0.1.7 exposes host-owned warm/cold runner lifecycle controls. // 0.1.8 declares the default provider acquisition budget to the host. -const PLUGIN_VERSION = "0.1.9"; +const PLUGIN_VERSION = "0.1.10"; const manifest: PaperclipPluginManifestV1 = { id: PLUGIN_ID, @@ -39,6 +39,7 @@ const manifest: PaperclipPluginManifestV1 = { description: "Provisions Daytona sandboxes with configurable image or snapshot selection, startup timeouts, and lease reuse.", supportsReusableLeases: true, + creationOperationVersion: 1, // Daytona keeps a persistent session and tails its callback log form, so it // emits incremental session output while the command runs. Declare the // opt-in capability so the host selects the session-output streaming path. diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 94838ec5c11..b7cd83089b9 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -51,6 +51,16 @@ describe("Daytona read-only resource inspection", () => { for (const method of [sandbox.start, sandbox.stop, sandbox.delete, sandbox.archive, sandbox.refreshData]) expect(method).not.toHaveBeenCalled(); expect(mockCreate).not.toHaveBeenCalled(); }); + it.each(["active", "destroyed"])("inspects a normal lease with creation provenance by recorded ID: %s", async state => { + const provenance = { creationOperationId: "16cd534e-27a1-45de-a4ad-5adbb6481361" }; + if (state === "active") mockGet.mockResolvedValue(createMockSandbox()); + else mockGet.mockRejectedValue(new MockDaytonaNotFoundError("gone")); + expect(await plugin.definition.onEnvironmentInspectLease!({ ...params, leaseMetadata: provenance })) + .toEqual({ providerLeaseId: "sandbox-123", state }); + expect(mockGet).toHaveBeenCalledExactlyOnceWith("sandbox-123"); + expect(mockCreate).not.toHaveBeenCalled(); + if (state === "destroyed") expect(await plugin.definition.onIdleDrain!(new AbortController().signal)).toBe("none"); + }); it("accepts only typed provider not-found as deletion proof", async () => { __resetDaytonaSandboxHandleCacheForTest(); mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not found")); @@ -80,6 +90,46 @@ describe("Daytona read-only resource inspection", () => { }); }); +describe("Daytona durable creation identity", () => { + const id = "16cd534e-27a1-45de-a4ad-5adbb6481361"; + const runId = "8cbd0194-e322-4b5b-9800-df4d5a682479"; + const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", + providerLeaseId: id, config: { apiKey: "fixture-not-a-key" }, + creationOperation: { id, runId } }; + beforeEach(() => { mockCreate.mockReset(); mockGet.mockReset(); __resetDaytonaSandboxHandleCacheForTest(); }); + afterEach(() => { vi.unstubAllEnvs(); __resetDaytonaSandboxHandleCacheForTest(); }); + function resource() { return { ...createMockSandbox({ name: `paperclip-create-${id}` }), labels: { + "paperclip-provider": "daytona", "paperclip-create-attempt": id, + "paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId, + "paperclip-run-id": runId, + } }; } + it("locates the exact resource by stable name without creation or lifecycle calls", async () => { + const sandbox = resource(); mockGet.mockResolvedValue(sandbox); + const observed = await plugin.definition.onEnvironmentInspectLease!(params); + expect(mockGet).toHaveBeenCalledWith(`paperclip-create-${id}`); + expect(observed).toMatchObject({ providerLeaseId: `paperclip-create-${id}`, state: "active", + creationCleanup: { observedProviderLeaseId: sandbox.id, attemptId: id, runId } }); + expect(mockCreate).not.toHaveBeenCalled(); expect(sandbox.delete).not.toHaveBeenCalled(); + await plugin.definition.onEnvironmentDestroyLease!({ ...params, providerLeaseId: observed.providerLeaseId, + leaseMetadata: { failedCreateCleanup: observed.creationCleanup } }); + expect(sandbox.delete).toHaveBeenCalledOnce(); + }); + it("retains uncertainty when a creation name is absent", async () => { + mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not yet visible")); + expect(await plugin.definition.onEnvironmentInspectLease!(params)).toEqual({ providerLeaseId: `paperclip-create-${id}`, state: "unknown" }); + expect(mockCreate).not.toHaveBeenCalled(); + }); + it("rejects a name or label owned by another creation", async () => { + mockGet.mockResolvedValue({ ...resource(), labels: { "paperclip-company-id": "foreign" } }); + await expect(plugin.definition.onEnvironmentInspectLease!(params)).rejects.toThrow("ownership does not match"); + }); + it("requires the recorded endpoint when ambient configuration changes", async () => { + vi.stubEnv("DAYTONA_API_URL", "https://changed.example.test"); + expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("unknown"); + expect(mockGet).not.toHaveBeenCalled(); + }); +}); + describe("Daytona idle eligibility", () => { const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", config: { apiKey: "fixture-not-a-key", reuseLease: true } }; @@ -537,7 +587,7 @@ describe("Daytona sandbox provider plugin", () => { // The bundled-plugin boot reconcile refreshes the stored manifest for an // existing install only when the version changes. The acquisition budget needs // the bump to reach an existing install. - expect(manifest.version).toBe("0.1.9"); + expect(manifest.version).toBe("0.1.10"); }); it.each([false, true])("closes duplex routes on lease release even when bridge drain hangs: %s", async (hangDrain) => { diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 77a3b618953..45ba1de804f 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -994,7 +994,8 @@ async function createSandbox( } // The SDK can create a sandbox and then throw while waiting for it to start. // Keep an attempt-specific provider name so that failure cannot lose ownership. - const attemptId = randomUUID(); + const attemptId = "creationOperationId" in params && params.creationOperationId ? params.creationOperationId : randomUUID(); + if (!/^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(attemptId)) throw new Error("Invalid creation operation identity"); const name = `paperclip-create-${attemptId}`; const labels = { ...buildSandboxLabels({ companyId: params.companyId, @@ -2988,6 +2989,32 @@ const plugin = definePlugin(trackDaytonaHooks({ if (!config.apiUrl && (process.env.DAYTONA_API_URL?.trim() || process.env.DAYTONA_SERVER_URL?.trim())) { return { providerLeaseId: params.providerLeaseId, state: "unknown" }; } + if (params.creationOperation !== undefined) { + const { id: attemptId, runId } = params.creationOperation; + if (![attemptId, runId].every(value => typeof value === "string" && /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(value))) { + throw new Error("Invalid creation operation identity"); + } + const name = `paperclip-create-${attemptId}`; + try { + const sandbox = await withLivenessTimeout("sandbox.creationLookup", 10_000, () => createDaytonaClient(config).get(name)); + const labels = { + "paperclip-provider": "daytona", "paperclip-create-attempt": attemptId, + "paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId, + "paperclip-run-id": runId, + }; + if (sandbox.name !== name || Object.entries(labels).some(([key, value]) => sandbox.labels?.[key] !== value)) { + throw new Error("Creation operation ownership does not match"); + } + return { providerLeaseId: name, state: "active", creationCleanup: { + providerLeaseId: name, observedProviderLeaseId: sandbox.id, companyId: params.companyId, + environmentId: params.environmentId, runId: labels["paperclip-run-id"], attemptId, + accountFingerprint: sandboxAccountDiscriminator(config), labels, + } }; + } catch (error) { + if (error instanceof DaytonaNotFoundError) return { providerLeaseId: name, state: "unknown" }; + throw error; + } + } const proof = idleActivity.proof(providerActivityKey(params, config, params.providerLeaseId)); // Read directly from the provider. The setup-status path and cached lease // reconnect path can start a stopped sandbox, so neither is an observation. diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index 464a7f3fec1..b308c1287c4 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -641,6 +641,7 @@ export interface PluginEnvironmentRunnerIngressEndpointParams export interface PluginEnvironmentAcquireLeaseParams extends PluginEnvironmentDriverBaseParams { runId: string; + creationOperationId?: string; /** Explicit platform probes may request a provider-owned immutable snapshot * identity. Supporting providers create by that ID and return snapshotIdentity * in lease metadata. Older providers may omit it; the host then cannot qualify @@ -701,11 +702,13 @@ export interface PluginEnvironmentDestroyLeaseParams extends PluginEnvironmentRe /** Observe this exact allocation without starting, stopping or deleting it. * Missing resources must be distinguished from permission/transport failures. */ export interface PluginEnvironmentInspectLeaseParams extends PluginEnvironmentDriverBaseParams { + creationOperation?: { id: string; runId: string }; providerLeaseId: string; leaseMetadata?: Record; } export interface PluginEnvironmentLeaseObservation { + creationCleanup?: import("./environment-creation-cleanup.js").PluginEnvironmentCreationCleanup; providerLeaseId: string; state: "active" | "stopped" | "destroyed" | "unknown"; } diff --git a/packages/shared/src/types/plugin.ts b/packages/shared/src/types/plugin.ts index 59001b3ffea..f5740487394 100644 --- a/packages/shared/src/types/plugin.ts +++ b/packages/shared/src/types/plugin.ts @@ -198,6 +198,7 @@ export interface PluginEnvironmentDriverDeclaration { /** Provides Runner execution, resource preparation, and PRP connections via environmentTask. */ supportsTasks?: boolean; freshWorkspaceVersion?: 1; + creationOperationVersion?: 1; /** Codex task retention and authorized run attachment, version 1. */ warmTaskVersion?: 1; /** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */ diff --git a/packages/shared/src/validators/plugin.ts b/packages/shared/src/validators/plugin.ts index 530889d91c7..1713f5ef38c 100644 --- a/packages/shared/src/validators/plugin.ts +++ b/packages/shared/src/validators/plugin.ts @@ -181,6 +181,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({ requiresAgent: z.boolean().optional(), supportsTasks: z.boolean().optional(), freshWorkspaceVersion: z.literal(1).optional(), + creationOperationVersion: z.literal(1).optional(), warmTaskVersion: z.literal(1).optional(), driverKey: z.string().min(1).regex( /^[a-z0-9][a-z0-9._-]*$/, diff --git a/server/scripts/migrate-skill-packages.ts b/server/scripts/migrate-skill-packages.ts index 6e566698085..a0fb8edaac3 100644 --- a/server/scripts/migrate-skill-packages.ts +++ b/server/scripts/migrate-skill-packages.ts @@ -1,5 +1,8 @@ /** Run from the server package with its configured storage and DATABASE_URL. */ import { createDb } from "@paperclipai/db"; +import { loadStorageConfig } from "../src/storage/config.js"; +import { createStorageProviderFromConfig } from "../src/storage/provider-registry.js"; +import { createStorageService } from "../src/storage/service.js"; import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js"; import { skillInstallationMaintenance } from "../src/services/skill-installation-maintenance.js"; import { isCloudManagedInstance } from "../src/services/cloud-instance.js"; @@ -19,8 +22,11 @@ try { if (!current) throw new Error("Initialize Cloud runtime configuration before running skill package maintenance"); setCloudRuntimeConfigSnapshot(current); } - const service = skillPackageMaintenance(db); - const installations = skillInstallationMaintenance(db); + // Maintenance must inspect the old provider before Cloud cutover. Server + // startup prerequisites still apply to the normal global storage service. + const storage = createStorageService(createStorageProviderFromConfig(loadStorageConfig())); + const service = skillPackageMaintenance(db, storage); + const installations = skillInstallationMaintenance(db, storage); const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions, orphans: service.collectOrphans, deletions: service.deleteObjects, installations: installations.installations, eligibility: installations.eligibility, verification: installations.verifyVersions }; diff --git a/server/src/__tests__/environment-cleanup-work.test.ts b/server/src/__tests__/environment-cleanup-work.test.ts index 5d71d2d4d42..29dbf953aae 100644 --- a/server/src/__tests__/environment-cleanup-work.test.ts +++ b/server/src/__tests__/environment-cleanup-work.test.ts @@ -1,3 +1,5 @@ +import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import { createOwnedSandbox } from "../services/sandbox-creation-operations.js"; import { mkdtemp, writeFile, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -7,7 +9,7 @@ import { idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; -import { companies, createDb, environmentLeases, environments, type Db } from "@paperclipai/db"; +import { companies, createDb, sandboxCreationOperations, environmentLeases, environments, type Db } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { registerEnvironmentCleanupWork } from "../services/environment-cleanup-work.js"; @@ -34,7 +36,7 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => { await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); }, 90_000); beforeEach(async () => { - await db.execute(sql`TRUNCATE companies, environments CASCADE`); + await db.execute(sql`TRUNCATE sandbox_creation_operations, companies, environments CASCADE`); companyId = randomUUID(); environmentId = randomUUID(); draining = false; failing = false; errors.length = 0; await db.insert(companies).values({ id: companyId, name: "Cleanup worker", issuePrefix: "CLN" }); await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" }); @@ -99,6 +101,64 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => { await settle(180_000); expect(inspections).toHaveBeenCalledTimes(3); await quiet(); }); + it("reconstructs a creation deadline after restart and leaves manual reconciliation quiet", async () => { + await start(); await quiet(); + await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} }, + async () => ({ providerLeaseId: "fixture-unattached" })); + await settle(); + expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now()); + worker!.stop(); await coordinator!.stop(); + await start(); + expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now()); + await settle(31 * 60_000); + await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([ + expect.objectContaining({ state: "needs_reconciliation" }), + ])); + await quiet(); + }); + it("settles a deleted-company creation as manual history across restart without idle polling", async () => { + await start(); await quiet(); + await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} }, + async () => ({ providerLeaseId: "fixture-deleted-company" })); + await db.delete(companies).where(eq(companies.id, companyId)); + await settle(31 * 60_000); + await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([ + expect.objectContaining({ state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }), + ])); + await quiet(); + worker!.stop(); await coordinator!.stop(); + await start(); await quiet(); + expect(releases).not.toHaveBeenCalled(); expect(inspections).not.toHaveBeenCalled(); + const ownerId = randomUUID(); + const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 }; + expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none")) + .toEqual({ version: 1, backgroundWork: "none" }); + await quiet(); + }); + it("verifies ordinary cleanup carrying creation provenance and stops provider polling", async () => { + const lease = await pending(); + await environmentService(db).updateLeaseMetadata(lease.id, { driver: "sandbox", creationOperationId: randomUUID() }); + await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" }); + await start(true); + expect(inspections).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ providerLeaseId: lease.providerLeaseId, + metadata: expect.objectContaining({ creationOperationId: expect.any(String) }) })); + await quiet(); + worker!.stop(); await coordinator!.stop(); + await start(true); expect(inspections).toHaveBeenCalledTimes(1); await quiet(); + }); + it("preserves a first creation signal during drain and disarms after atomic attachment", async () => { + await start(); await quiet(); draining = true; + const created = await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} }, + async () => ({ providerLeaseId: "fixture-attached" })); + const count = reads.mock.calls.length; + await settle(60_000, false); expect(reads).toHaveBeenCalledTimes(count); + draining = false; + await environmentService(db).acquireLease({ companyId, environmentId, provider: "fixture", providerLeaseId: created.providerLeaseId, + creationOperationId: String(created.metadata!.creationOperationId), metadata: created.metadata }); + await settle(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "attached" })]); + await quiet(); + }); it("wakes on a real orphan insert, waits for its backoff, then disarms after deletion", async () => { await start(); await quiet(); const lease = await pending(); await settle(); diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index a4b82d7b113..324a0eda702 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -25,6 +25,7 @@ import { companySecrets, createDb, environmentLeases, + sandboxCreationOperations, environments, executionWorkspaces, heartbeatRuns, @@ -41,7 +42,6 @@ import { SANDBOX_CAPABILITY_KEYS, environmentRuntimeService, findReusableSandboxLeaseId, - SandboxOrphanCleanupWriteError, } from "../services/environment-runtime.ts"; import * as sandboxProviderRuntime from "../services/sandbox-provider-runtime.ts"; import * as environmentsModule from "../services/environments.ts"; @@ -865,7 +865,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { providerLeaseId: `lifecycle-${++acquisitions}`, metadata: { remoteCwd: "/workspace" }, }; if (method === "environmentResumeLease") return { - providerLeaseId: warmProviderLeaseId, metadata: { remoteCwd: "/workspace" }, + providerLeaseId: warmProviderLeaseId, metadata: { remoteCwd: "/workspace", creationOperationId: randomUUID() }, }; if (method === "environmentDestroyLease" || method === "environmentStopLease" || method === "environmentReleaseLease") return { providerLeaseId: input.providerLeaseId, state: method === "environmentDestroyLease" ? "destroyed" : "stopped", @@ -896,6 +896,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" }); const next = await runtime.acquireRunLease({ ...input, environment: warmEnvironment, heartbeatRunId: nextRunId }); expect(next.lease.providerLeaseId).toBe(warmProviderLeaseId); + expect(next.lease.heartbeatRunId).toBe(nextRunId); + expect(next.lease.id).not.toBe(warm.lease.id); expect(acquisitions).toBe(2); expect((await environmentService(db).getById(environment.id))?.config.reuseLease).toBe(false); }); @@ -1192,32 +1194,12 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { expect(execution).toMatchObject({ environmentAcquisitionPhase: "shell", environmentAcquisitionElapsedMs: 300_000, environmentAcquisitionBudgetMs: 300_000 }); } else expect(execution).not.toHaveProperty("environmentAcquisitionPhase"); - if (mode === "inline") { - await expect(acquisition).rejects.toMatchObject({ - message: "Sandbox creation failed; allocated sandbox cleanup was confirmed.", cause: failure, - }); - } else { - await expect(acquisition).rejects.toBe(failure); - } + await expect(acquisition).rejects.toBe(failure); const rows = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, providerLeaseId)); - if (["foreign scope", "foreign environment", "foreign run", "malformed"].includes(mode)) { - expect(rows).toHaveLength(0); - expect(call.mock.calls.filter((c) => c[1] === "environmentDestroyLease")).toHaveLength(0); - return; - } - expect(rows).toHaveLength(1); - expect(rows[0]).toMatchObject({ status: mode === "inline" ? "expired" : "pending_cleanup", - cleanupStatus: mode === "inline" ? "success" : "failed", companyId: seeded.companyId, - metadata: { pluginId: seeded.pluginId, sandboxProviderPlugin: true, failedCreateCleanup: cleanup } }); - if (mode === "pending" || mode === "deleted environment") { - available = true; - if (mode === "deleted environment") { - await db.update(environmentLeases).set({ environmentId: null }).where(eq(environmentLeases.id, rows[0].id)); - } - const restarted = environmentRuntimeService(db, { pluginWorkerManager: worker }); - const persisted = await environmentService(db).getLeaseById(rows[0].id); - await expect(restarted.retryPendingSandboxTeardown({ environment: null, lease: persisted! })).resolves.toEqual({ providerLeaseId, state: "destroyed" }); - } + expect(rows).toHaveLength(0); + expect(call.mock.calls.filter(value => value[1] === "environmentDestroyLease")).toHaveLength(0); + expect(await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId, seeded.companyId))) + .toEqual([expect.objectContaining({ state: "pending", originalEnvironmentId: seeded.environment.id, runId: seeded.runId })]); }); it.each(["deleted receipt lost", "observation response lost", "journal write failed", "foreign observation"])( @@ -1270,8 +1252,13 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); const worker = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentDestroyLease"] }), call } as unknown as PluginWorkerManager; const runtime = environmentRuntimeService(db, { pluginWorkerManager: worker }); - await expect(runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, - issueId: null, heartbeatRunId: seeded.runId, persistedExecutionWorkspace: null })).rejects.toBe(failure); + const pending = await environmentService(db).insertPendingCleanupLease({ + companyId: seeded.companyId, environmentId: seeded.environment.id, heartbeatRunId: seeded.runId, + provider: "fake-plugin", providerLeaseId: cleanup.providerLeaseId, + metadata: { ...seeded.environment.config, driver: "sandbox", pluginId: seeded.pluginId, sandboxProviderPlugin: true, failedCreateCleanup: cleanup }, + failureReason: "interrupted_sandbox_creation", + }); + await runtime.retryPendingSandboxTeardown({ environment: null, lease: pending }).catch(() => {}); const rows = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, cleanup.providerLeaseId)); expect(rows).toHaveLength(1); expect(rows[0].status).toBe("pending_cleanup"); @@ -1766,732 +1753,6 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }, ); - it("releases the remote sandbox when the lease insert rejects a foreign-company binding", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox", - config: { - provider: "fake", - image: "ubuntu:24.04", - reuseLease: false, - }, - }); - - // A managed reconciliation binds the environment to another company after the - // route guard read an empty binding list. The lease insert then rejects the - // foreign-company binding with the 403 `environment_company_mismatch`. - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co", - issuePrefix: "OTA", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const destroySpy = vi.spyOn(sandboxProviderRuntime, "destroySandboxProviderLease"); - try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ - status: 403, - details: { code: "environment_company_mismatch" }, - }); - - // The acquire records the durable pending-cleanup row before the teardown, - // so the row lands while the database is proven reachable. The successful - // teardown then releases the row to the terminal `expired` state, so no - // active or pending_cleanup row remains for the orphan. - const leaseRows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(leaseRows).toHaveLength(1); - expect(leaseRows[0]?.status).toBe("expired"); - expect(leaseRows[0]?.cleanupStatus).toBe("success"); - - // The acquire already provisioned the remote sandbox, so it releases the - // sandbox on the rejection. Without this teardown the rejected insert leaks - // a live sandbox that no lease row tracks. The teardown carries the - // provisioned provider lease id. - expect(destroySpy).toHaveBeenCalledTimes(1); - expect(destroySpy.mock.calls[0]?.[0]?.providerLeaseId).toMatch( - new RegExp(`^sandbox://fake/${runId}/[0-9a-f-]{36}$`), - ); - } finally { - destroySpy.mockRestore(); - } - }); - - it("destroys the remote plugin sandbox when the lease insert rejects a foreign-company binding", async () => { - // The Claude login runs on a plugin-backed sandbox provider (Daytona), so this - // path is the one the login uses. It must also release the remote sandbox when - // the conditional lease insert rejects a foreign-company binding. - const pluginId = randomUUID(); - const { companyId, environment: baseEnvironment, runId } = await seedEnvironment(); - const pluginConfig = { provider: "fake-plugin", image: "fake:test", reuseLease: false }; - const environment = { - ...baseEnvironment, - name: "Foreign-bound Plugin Sandbox", - driver: "sandbox", - config: pluginConfig, - }; - await environmentService(db).update(environment.id, { - driver: "sandbox", - name: environment.name, - config: pluginConfig, - }); - await insertSandboxProviderPlugin(pluginId); - - // A managed reconciliation binds the environment to another company after the - // route guard read an empty binding list. The lease insert then rejects the - // foreign-company binding with the 403 `environment_company_mismatch`. - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co", - issuePrefix: "OTB", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const workerManager = { - isRunning: vi.fn((id: string) => id === pluginId), - getWorker: vi.fn(() => ({ supportedMethods: ["environmentDestroyLease"] })), - call: vi.fn(async (_pluginId: string, method: string) => { - if (method === "environmentAcquireLease") { - return { - providerLeaseId: "plugin-lease-1", - metadata: { provider: "fake-plugin", image: "fake:test", reuseLease: false }, - }; - } - if (method === "environmentDestroyLease") { - return { providerLeaseId: "plugin-lease-1", state: "destroyed" }; - } - throw new Error(`Unexpected plugin method: ${method}`); - }), - } as unknown as PluginWorkerManager; - const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); - - await expect( - runtimeWithPlugin.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ - status: 403, - details: { code: "environment_company_mismatch" }, - }); - - // The acquire records the durable pending-cleanup row before the teardown, - // so the row lands while the database is proven reachable. The successful - // teardown then releases the row to the terminal `expired` state, so no - // active or pending_cleanup row remains for the orphan. - const leaseRows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(leaseRows).toHaveLength(1); - expect(leaseRows[0]?.status).toBe("expired"); - expect(leaseRows[0]?.cleanupStatus).toBe("success"); - expect(leaseRows[0]?.metadata?.remoteExecutionTermination).toMatchObject({ - companyId, runId, leaseId: leaseRows[0]!.id, providerLeaseId: "plugin-lease-1", state: "destroyed", - }); - - // The acquire provisioned the remote plugin sandbox, so it destroys the - // sandbox on the rejection. Without this teardown the rejected insert leaks a - // live sandbox that no lease row tracks. - const destroyCalls = (workerManager.call as unknown as ReturnType).mock.calls.filter( - (callArgs) => callArgs[1] === "environmentDestroyLease", - ); - expect(destroyCalls).toHaveLength(1); - expect(destroyCalls[0]?.[2]).toMatchObject({ providerLeaseId: "plugin-lease-1" }); - - await environmentService(db).update(environment.id, { driver: "local", config: {} }); - }); - - it("records a durable pending-cleanup lease when the built-in teardown fails after a foreign-company rejection", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Teardown Fail", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Built-in", - issuePrefix: "OTC", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The remote teardown fails, so the acquire cannot release the sandbox - // directly. It must record a durable pending-cleanup row instead. - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); - - expect(destroySpy).toHaveBeenCalledTimes(1); - // The failed teardown left a durable pending-cleanup row that carries the - // provider lease id for a later sweep. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - expect(rows[0]?.cleanupStatus).toBe("failed"); - expect(rows[0]?.providerLeaseId).toMatch(new RegExp(`^sandbox://fake/${runId}/[0-9a-f-]{36}$`)); - } finally { - destroySpy.mockRestore(); - } - }); - - it("records the durable pending-cleanup row before it runs the inline teardown", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Write First", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Write First", - issuePrefix: "OWF", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The teardown reads the lease table when it runs. A durable pending_cleanup - // row must already exist, which proves the acquire records the row before the - // teardown. This ordering closes the window the finding describes: the durable - // write lands while the database is proven reachable, before the teardown RPC - // that a database outage could otherwise interrupt. - let pendingCleanupRowsAtTeardown = -1; - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockImplementation(async () => { - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - pendingCleanupRowsAtTeardown = rows.filter((row) => row.status === "pending_cleanup").length; - }); - try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); - - // The durable row already tracked the orphan when the teardown ran. - expect(pendingCleanupRowsAtTeardown).toBe(1); - expect(destroySpy).toHaveBeenCalledTimes(1); - - // The teardown succeeded, so the acquire released the row to the terminal - // `expired` state and left no pending_cleanup row. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("expired"); - expect(rows[0]?.cleanupStatus).toBe("success"); - } finally { - destroySpy.mockRestore(); - } - }); - - it("leaves no orphan and no error when the durable write fails but the teardown succeeds", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Write Fail Teardown Ok", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Write Fail", - issuePrefix: "OWK", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // Force every durable pending-cleanup write to fail, but let the teardown - // succeed. A successful teardown removes the orphan, so the acquire needs no - // durable row and raises no orphan-cleanup write error. The write-first order - // never turns a clean teardown into a failure. - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: () => - Promise.reject(new Error("pending-cleanup write failed; database down")), - }; - }); - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockResolvedValue(undefined as never); - try { - // Set the retry backoff to zero, so the failed write retries never slow the - // test. - const runtime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - const rejection = await runtime - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - - // The rejection is the original company mismatch, not an orphan-cleanup - // write error, because the teardown removed the orphan. - expect(rejection).not.toBeInstanceOf(SandboxOrphanCleanupWriteError); - expect(rejection).toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); - expect(destroySpy).toHaveBeenCalledTimes(1); - - // No orphan remains and no durable row was needed, so the lease table is - // empty for the environment. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(0); - - // The buffer holds no orphan, so a flush recovers nothing. - const flushed = await runtime.flushDeferredOrphanCleanups(); - expect(flushed).toEqual({ recovered: 0, pending: 0 }); - } finally { - destroySpy.mockRestore(); - factorySpy.mockRestore(); - } - }); - - it("records a durable pending-cleanup lease when the plugin teardown fails after a foreign-company rejection", async () => { - const pluginId = randomUUID(); - const { companyId, environment: baseEnvironment, runId } = await seedEnvironment(); - const pluginConfig = { provider: "fake-plugin", image: "fake:test", reuseLease: false }; - const environment = { - ...baseEnvironment, - name: "Foreign-bound Plugin Sandbox Teardown Fail", - driver: "sandbox", - config: pluginConfig, - }; - await environmentService(db).update(environment.id, { - driver: "sandbox", - name: environment.name, - config: pluginConfig, - }); - await insertSandboxProviderPlugin(pluginId); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Plugin", - issuePrefix: "OTD", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const workerManager = { - isRunning: vi.fn((id: string) => id === pluginId), - getWorker: vi.fn(() => ({ supportedMethods: ["environmentDestroyLease"] })), - call: vi.fn(async (_pluginId: string, method: string) => { - if (method === "environmentAcquireLease") { - return { - providerLeaseId: "plugin-lease-2", - metadata: { provider: "fake-plugin", image: "fake:test", reuseLease: false }, - }; - } - if (method === "environmentDestroyLease") { - throw new Error("destroy failed"); - } - throw new Error(`Unexpected plugin method: ${method}`); - }), - } as unknown as PluginWorkerManager; - const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); - - await expect( - runtimeWithPlugin.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); - - // The failed plugin teardown left a durable pending-cleanup row that carries - // the provider lease id for a later sweep. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - expect(rows[0]?.cleanupStatus).toBe("failed"); - expect(rows[0]?.providerLeaseId).toBe("plugin-lease-2"); - - await environmentService(db).update(environment.id, { driver: "local", config: {} }); - }); - - it("throws a SandboxOrphanCleanupWriteError when the built-in durable pending-cleanup write also fails", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Write Fail", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Write", - issuePrefix: "OCW", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The remote teardown fails, so the acquire tries to record a durable - // pending-cleanup row instead. - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - // Force the durable pending-cleanup write to fail on every attempt. The - // write retries a few times, so reject the insert each time and let the - // primary company-bound acquire reject for real. - const failingInsert = vi - .fn["insertPendingCleanupLease"]>, Promise>() - .mockRejectedValue(new Error("pending-cleanup write failed")); - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: failingInsert, - }; - }); - // The durable database write fails, so the only durable handle left is the - // error log. Capture it to prove the leaked sandbox stays discoverable. - const logSpy = vi.spyOn(logger, "error").mockImplementation(() => logger); - try { - // Set the retry backoff to zero, so the retries never slow the test. - const runtimeWithFailingCleanup = environmentRuntimeService(db, { - pendingCleanupWriteBackoffMs: 0, - }); - const rejection = await runtimeWithFailingCleanup - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - - // The failed durable write is not swallowed: the acquire throws a - // SandboxOrphanCleanupWriteError that keeps the original 403 rejection as - // its cause. - expect(rejection).toBeInstanceOf(SandboxOrphanCleanupWriteError); - expect(rejection).toMatchObject({ provider: "fake" }); - expect((rejection as SandboxOrphanCleanupWriteError).cause).toMatchObject({ - status: 403, - details: { code: "environment_company_mismatch" }, - }); - expect(destroySpy).toHaveBeenCalledTimes(1); - // The write retried the durable insert before it gave up, so more than one - // attempt ran. - expect(failingInsert.mock.calls.length).toBeGreaterThan(1); - // The durable write failed before it created a row, so no lease row exists. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(0); - // The durable log preserves the provider identifiers, so an operator keeps - // a handle to find and tear the leaked sandbox down by hand. - const cleanupLog = logSpy.mock.calls.find( - ([fields]) => - (fields as { errorKind?: string } | undefined)?.errorKind === - "sandbox_orphan_cleanup_write_failed", - ); - expect(cleanupLog).toBeDefined(); - expect(cleanupLog?.[0]).toMatchObject({ - provider: "fake", - companyId, - environmentId: environment.id, - }); - // The log never carries the caught exception, so no credential leaks. - expect(cleanupLog?.[0]).not.toHaveProperty("cause"); - expect(cleanupLog?.[0]).not.toHaveProperty("cleanupWriteError"); - } finally { - logSpy.mockRestore(); - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - - it("recovers cleanup state when a retry of the durable pending-cleanup write succeeds", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Write Retry", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Retry", - issuePrefix: "OCR", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The remote teardown fails, so the acquire tries to record a durable - // pending-cleanup row instead. - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - // The first durable write attempt fails, so the acquire retries. The second - // attempt delegates to the real insert, so a durable pending_cleanup row - // lands and a later sweep can find the orphan. - let insertAttempts = 0; - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: ( - input: Parameters[0], - ) => { - insertAttempts += 1; - if (insertAttempts === 1) { - return Promise.reject(new Error("pending-cleanup write failed once")); - } - return real.insertPendingCleanupLease(input); - }, - }; - }); - try { - // Set the retry backoff to zero, so the retry never slows the test. - const runtimeWithRetry = environmentRuntimeService(db, { - pendingCleanupWriteBackoffMs: 0, - }); - const rejection = await runtimeWithRetry - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - - // The retry landed the durable row, so the acquire rejects with the - // original company-mismatch rejection, not a SandboxOrphanCleanupWriteError. - expect(rejection).not.toBeInstanceOf(SandboxOrphanCleanupWriteError); - expect(rejection).toMatchObject({ - status: 403, - details: { code: "environment_company_mismatch" }, - }); - expect(destroySpy).toHaveBeenCalledTimes(1); - // The first attempt failed and the second succeeded, so the write ran twice. - expect(insertAttempts).toBe(2); - // A durable pending_cleanup lease row now tracks the orphan, so a sweep can - // find and release the leaked sandbox. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - expect(rows[0]?.cleanupStatus).toBe("failed"); - expect(rows[0]?.failureReason).toBe("acquire_rejected_teardown_failed"); - } finally { - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - - it("buffers the orphan in-process and a later sweep flush lands the durable row after the database recovers", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Buffer Flush", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Buffer", - issuePrefix: "OCB", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The remote teardown fails, so the acquire tries to record a durable - // pending-cleanup row instead. - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - // The database is down, so every synchronous pending-cleanup write rejects. - // The flag flips to false when the database recovers, so a later flush lands - // the durable row. - let databaseDown = true; - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: ( - input: Parameters[0], - ) => { - if (databaseDown) { - return Promise.reject(new Error("pending-cleanup write failed; database down")); - } - return real.insertPendingCleanupLease(input); - }, - }; - }); - const logSpy = vi.spyOn(logger, "error").mockImplementation(() => logger); - try { - // Set the retry backoff to zero, so the retries never slow the test. - const runtime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - const rejection = await runtime - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - - // The synchronous write failed on every attempt, so the acquire still - // throws the orphan-cleanup write error that keeps the original rejection. - expect(rejection).toBeInstanceOf(SandboxOrphanCleanupWriteError); - // The database was down, so no durable row exists yet. - let rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(0); - // The acquire buffered the orphan in-process, so the log records the buffer. - const bufferedLog = logSpy.mock.calls.find( - ([fields]) => - (fields as { errorKind?: string } | undefined)?.errorKind === - "sandbox_orphan_cleanup_write_failed", - ); - expect(bufferedLog?.[0]).toMatchObject({ buffered: true }); - - // The database recovers, so the next cleanup-sweep flush lands the row. - databaseDown = false; - const flushed = await runtime.flushDeferredOrphanCleanups(); - expect(flushed).toEqual({ recovered: 1, pending: 0 }); - - // A durable pending_cleanup row now tracks the orphan, so a sweep finds and - // releases the leaked sandbox. - rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - expect(rows[0]?.cleanupStatus).toBe("failed"); - expect(rows[0]?.failureReason).toBe("acquire_rejected_teardown_failed"); - expect(rows[0]?.providerLeaseId).toBeTruthy(); - - // The buffer is empty now, so a second flush inserts nothing. - const second = await runtime.flushDeferredOrphanCleanups(); - expect(second).toEqual({ recovered: 0, pending: 0 }); - const rowsAfter = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rowsAfter).toHaveLength(1); - } finally { - logSpy.mockRestore(); - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - it("ingests a replacement orphan appended after startup without duplicate concurrent cleanup", async () => { const {companyId,environment,runId}=await seedEnvironment({driver:"sandbox",name:"Late plugin orphan",config:{provider:"fake",image:"ubuntu:24.04",reuseLease:false}}); let release!:()=>void,entered!:()=>void; @@ -2520,430 +1781,6 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } finally {release();factory.mockRestore();} }); - it("persists the orphan to the durable spool so a restart recovers and lands the durable row", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Spool Restart", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Spool Restart", - issuePrefix: "OCS", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - // The remote teardown fails, so the acquire tries to record a durable - // pending-cleanup row instead. - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - // The database is down, so every synchronous pending-cleanup write rejects. - // The flag flips to false to simulate the database recovering after a - // restart. - let databaseDown = true; - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: ( - input: Parameters[0], - ) => { - if (databaseDown) { - return Promise.reject(new Error("pending-cleanup write failed; database down")); - } - return real.insertPendingCleanupLease(input); - }, - }; - }); - const logSpy = vi.spyOn(logger, "error").mockImplementation(() => logger); - const spoolDir = process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR!; - try { - // The first process acquires, fails every synchronous write, and persists - // the orphan to the durable spool. The retry backoff is zero, so the test - // never waits. - const firstProcessRuntime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - const rejection = await firstProcessRuntime - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - expect(rejection).toBeInstanceOf(SandboxOrphanCleanupWriteError); - - // The database was down, so no durable row exists yet. - let rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(0); - - // The acquire persisted the orphan to the durable spool, so a file waits on - // the disk and the log records the durable persist. - const spoolFiles = (await readdir(spoolDir)).filter((name) => name.endsWith(".json")); - expect(spoolFiles).toHaveLength(1); - const persistedLog = logSpy.mock.calls.find( - ([fields]) => - (fields as { errorKind?: string } | undefined)?.errorKind === - "sandbox_orphan_cleanup_write_failed", - ); - expect(persistedLog?.[0]).toMatchObject({ persisted: true }); - - // The process restarts. A fresh runtime keeps no in-process buffer, so only - // the durable spool carries the orphan. The database recovers, so the first - // cleanup-sweep flush loads the spool and lands the durable row. - databaseDown = false; - const restartedRuntime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - const flushed = await restartedRuntime.flushDeferredOrphanCleanups(); - expect(flushed).toEqual({ recovered: 1, pending: 0 }); - - rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - expect(rows[0]?.failureReason).toBe("acquire_rejected_teardown_failed"); - expect(rows[0]?.providerLeaseId).toBeTruthy(); - - // The flush removed the spooled copy after the row landed, so a second - // restart never re-inserts a duplicate. - const spoolFilesAfter = (await readdir(spoolDir)).filter((name) => name.endsWith(".json")); - expect(spoolFilesAfter).toHaveLength(0); - const secondRestartRuntime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - const secondFlush = await secondRestartRuntime.flushDeferredOrphanCleanups(); - expect(secondFlush).toEqual({ recovered: 0, pending: 0 }); - const rowsAfter = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rowsAfter).toHaveLength(1); - } finally { - logSpy.mockRestore(); - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - - it("buffers the orphan in-process when the durable spool write also fails", async () => { - const idleBefore = idleWorkSnapshot().active; - let releaseFlush!: () => void; - let enteredFlush!: () => void; - const flushGate = new Promise(resolve => { releaseFlush = resolve; }); - const flushEntered = new Promise(resolve => { enteredFlush = resolve; }); - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Spool Unwritable", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Spool Unwritable", - issuePrefix: "OCU", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - let databaseDown = true; - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: ( - input: Parameters[0], - ) => { - if (databaseDown) { - return Promise.reject(new Error("pending-cleanup write failed; database down")); - } - enteredFlush(); - return flushGate.then(() => real.insertPendingCleanupLease(input)); - }, - }; - }); - const logSpy = vi.spyOn(logger, "error").mockImplementation(() => logger); - try { - // Point the spool at an unwritable path, so the durable spool write fails - // like a full or read-only disk. The acquire must fall back to the - // in-process buffer and never lose the orphan silently. - const unwritableSpool = { - append: () => Promise.resolve(false), - load: () => Promise.resolve([]), - remove: () => Promise.resolve(), - }; - const runtime = environmentRuntimeService(db, { - pendingCleanupWriteBackoffMs: 0, - orphanCleanupSpool: unwritableSpool, - }); - await runtime - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - () => undefined, - ); - - // The spool write failed, so the log records the in-process-only fallback. - const fallbackLog = logSpy.mock.calls.find( - ([fields]) => - (fields as { errorKind?: string } | undefined)?.errorKind === - "sandbox_orphan_cleanup_write_failed", - ); - expect(fallbackLog?.[0]).toMatchObject({ persisted: false, buffered: true }); - expect(idleWorkSnapshot().active).toBe(idleBefore + 1); - - // The same runtime still keeps the orphan in-process, so a flush after the - // database recovers lands the durable row. - databaseDown = false; - const flushing = runtime.flushDeferredOrphanCleanups(); - await flushEntered; - // The batch was spliced out of the queue, but the DB does not own it yet. - expect(idleWorkSnapshot().active).toBe(idleBefore + 1); - releaseFlush(); - const flushed = await flushing; - expect(idleWorkSnapshot().active).toBe(idleBefore); - expect(flushed).toEqual({ recovered: 1, pending: 0 }); - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(1); - expect(rows[0]?.status).toBe("pending_cleanup"); - } finally { - releaseFlush(); - logSpy.mockRestore(); - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - - it("keeps the buffered orphan when the flush write still fails, then recovers on a later flush", async () => { - const { companyId, environment, runId } = await seedEnvironment({ - driver: "sandbox", - name: "Foreign-bound Fake Sandbox Cleanup Buffer Requeue", - config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, - }); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Cleanup Requeue", - issuePrefix: "OCQ", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const destroySpy = vi - .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") - .mockRejectedValue(new Error("teardown failed")); - let databaseDown = true; - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: ( - input: Parameters[0], - ) => { - if (databaseDown) { - return Promise.reject(new Error("pending-cleanup write failed; database down")); - } - return real.insertPendingCleanupLease(input); - }, - }; - }); - const errorLogSpy = vi.spyOn(logger, "error").mockImplementation(() => logger); - const warnLogSpy = vi.spyOn(logger, "warn").mockImplementation(() => logger); - try { - const runtime = environmentRuntimeService(db, { pendingCleanupWriteBackoffMs: 0 }); - await runtime - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - () => undefined, - ); - - // The database is still down, so the flush re-queues the orphan instead of - // losing it. The buffer keeps the record for a later tick. - const firstFlush = await runtime.flushDeferredOrphanCleanups(); - expect(firstFlush).toEqual({ recovered: 0, pending: 1 }); - const rowsWhileDown = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rowsWhileDown).toHaveLength(0); - // The flush logs the re-queue and never carries the caught write exception. - // The sync-retry path also warns with the same error kind, so match on the - // `requeued` field that only the flush warn carries. - const requeueLog = warnLogSpy.mock.calls.find( - ([fields]) => (fields as { requeued?: boolean } | undefined)?.requeued === true, - ); - expect(requeueLog).toBeDefined(); - expect(requeueLog?.[0]).toMatchObject({ - errorKind: "sandbox_orphan_cleanup_write_failed", - requeued: true, - }); - expect(requeueLog?.[0]).not.toHaveProperty("cause"); - expect(requeueLog?.[0]).not.toHaveProperty("cleanupWriteError"); - - // The database recovers, so the next flush lands the durable row exactly - // once from the still-buffered record. - databaseDown = false; - const secondFlush = await runtime.flushDeferredOrphanCleanups(); - expect(secondFlush).toEqual({ recovered: 1, pending: 0 }); - const rowsAfter = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rowsAfter).toHaveLength(1); - expect(rowsAfter[0]?.status).toBe("pending_cleanup"); - } finally { - warnLogSpy.mockRestore(); - errorLogSpy.mockRestore(); - factorySpy.mockRestore(); - destroySpy.mockRestore(); - } - }); - - it("throws a SandboxOrphanCleanupWriteError when the plugin durable pending-cleanup write also fails", async () => { - const pluginId = randomUUID(); - const { companyId, environment: baseEnvironment, runId } = await seedEnvironment(); - const pluginConfig = { provider: "fake-plugin", image: "fake:test", reuseLease: false }; - const environment = { - ...baseEnvironment, - name: "Foreign-bound Plugin Sandbox Cleanup Write Fail", - driver: "sandbox", - config: pluginConfig, - }; - await environmentService(db).update(environment.id, { - driver: "sandbox", - name: environment.name, - config: pluginConfig, - }); - await insertSandboxProviderPlugin(pluginId); - const otherCompanyId = randomUUID(); - await db.insert(companies).values({ - id: otherCompanyId, - name: "Other Co Plugin Cleanup Write", - issuePrefix: "OPW", - status: "active", - createdAt: new Date(), - updatedAt: new Date(), - }); - await insertForeignCompanyEnvironmentBinding(otherCompanyId, environment); - - const workerManager = { - isRunning: vi.fn((id: string) => id === pluginId), - getWorker: vi.fn(() => ({ supportedMethods: ["environmentDestroyLease"] })), - call: vi.fn(async (_pluginId: string, method: string) => { - if (method === "environmentAcquireLease") { - return { - providerLeaseId: "plugin-lease-write-fail", - metadata: { provider: "fake-plugin", image: "fake:test", reuseLease: false }, - }; - } - if (method === "environmentDestroyLease") { - throw new Error("destroy failed"); - } - throw new Error(`Unexpected plugin method: ${method}`); - }), - } as unknown as PluginWorkerManager; - - const realEnvironmentService = environmentsModule.environmentService; - const factorySpy = vi - .spyOn(environmentsModule, "environmentService") - .mockImplementation((database: Parameters[0]) => { - const real = realEnvironmentService(database); - return { - ...real, - insertPendingCleanupLease: () => - Promise.reject(new Error("pending-cleanup write failed")), - }; - }); - try { - const runtimeWithPlugin = environmentRuntimeService(db, { - pluginWorkerManager: workerManager, - pendingCleanupWriteBackoffMs: 0, - }); - const rejection = await runtimeWithPlugin - .acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }) - .then( - () => { - throw new Error("acquireRunLease resolved but must reject"); - }, - (error: unknown) => error, - ); - - expect(rejection).toBeInstanceOf(SandboxOrphanCleanupWriteError); - expect(rejection).toMatchObject({ provider: "fake-plugin", providerLeaseId: "plugin-lease-write-fail" }); - expect((rejection as SandboxOrphanCleanupWriteError).cause).toMatchObject({ - status: 403, - details: { code: "environment_company_mismatch" }, - }); - // The durable write failed before it created a row, so no lease row exists. - const rows = await db - .select() - .from(environmentLeases) - .where(eq(environmentLeases.environmentId, environment.id)); - expect(rows).toHaveLength(0); - } finally { - factorySpy.mockRestore(); - } - - await environmentService(db).update(environment.id, { driver: "local", config: {} }); - }); - it("records the orphan directly in the pending_cleanup state with one atomic insert", async () => { const { companyId, environment, runId } = await seedEnvironment({ driver: "sandbox", @@ -3006,16 +1843,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { .mockRejectedValueOnce(new Error("teardown failed")) .mockResolvedValue(undefined); try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); + const pending = await environmentService(db).insertPendingCleanupLease({ + companyId, environmentId: environment.id, heartbeatRunId: runId, provider: "fake", providerLeaseId: `sandbox://fake/${runId}`, + metadata: { ...environment.config, driver: "sandbox" }, failureReason: "interrupted_sandbox_creation", + }); + await runtime.retryPendingSandboxTeardown({ environment: null, lease: pending }).catch(() => {}); const orphan = await db .select() @@ -3059,16 +1891,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { .spyOn(sandboxProviderRuntime, "destroySandboxProviderLease") .mockRejectedValue(new Error("teardown failed")); try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); + const pending = await environmentService(db).insertPendingCleanupLease({ + companyId, environmentId: environment.id, heartbeatRunId: runId, provider: "fake", providerLeaseId: `sandbox://fake/${runId}`, + metadata: { ...environment.config, driver: "sandbox" }, failureReason: "interrupted_sandbox_creation", + }); + await runtime.retryPendingSandboxTeardown({ environment: null, lease: pending }).catch(() => {}); const orphan = await db .select() @@ -3124,16 +1951,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { .mockRejectedValueOnce(new Error("teardown failed")) .mockResolvedValue(undefined); try { - await expect( - runtime.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); + const pending = await environmentService(db).insertPendingCleanupLease({ + companyId, environmentId: environment.id, heartbeatRunId: runId, provider: "fake", providerLeaseId: `sandbox://fake/${runId}`, + metadata: { ...environment.config, driver: "sandbox" }, failureReason: "interrupted_sandbox_creation", + }); + await runtime.retryPendingSandboxTeardown({ environment: null, lease: pending }).catch(() => {}); const orphan = await db .select() @@ -3766,16 +2588,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); - await expect( - runtimeWithPlugin.acquireRunLease({ - companyId, - environment, - issueId: null, - heartbeatRunId: runId, - persistedExecutionWorkspace: null, - assertCompanyBinding: true, - }), - ).rejects.toMatchObject({ status: 403, details: { code: "environment_company_mismatch" } }); + const pending = await environmentService(db).insertPendingCleanupLease({ + companyId, environmentId: environment.id, heartbeatRunId: runId, provider: "fake-plugin", providerLeaseId: "plugin-lease-3", + metadata: { ...pluginConfig, driver: "sandbox", pluginId, sandboxProviderPlugin: true }, failureReason: "interrupted_sandbox_creation", + }); + await runtimeWithPlugin.retryPendingSandboxTeardown({ environment: null, lease: pending }).catch(() => {}); const orphan = await db .select() @@ -4434,6 +3251,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { heartbeatRunId: runId, persistedExecutionWorkspace: null, }); + const operations = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId, companyId)); + expect(JSON.stringify(operations)).not.toContain("resolved-provider-key"); + expect(operations[0]?.metadata.creationCredentialHashes).toEqual({ + apiKey: createHash("sha256").update("resolved-provider-key").digest("hex"), + }); expect(acquired.lease.metadata).toMatchObject({ provider: "secure-plugin", template: "base", @@ -7059,6 +5881,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", { + creationOperationId: expect.any(String), driverKey: "fake-plugin", companyId, environmentId: environment.id, diff --git a/server/src/__tests__/fixtures/sandbox-creation-receiver.ts b/server/src/__tests__/fixtures/sandbox-creation-receiver.ts new file mode 100644 index 00000000000..fc42f436492 --- /dev/null +++ b/server/src/__tests__/fixtures/sandbox-creation-receiver.ts @@ -0,0 +1,34 @@ +import { createDb } from "@paperclipai/db"; +import { environmentRuntimeService } from "../../services/environment-runtime.js"; +import { environmentService } from "../../services/environments.js"; +import type { PluginWorkerManager } from "../../services/plugin-worker-manager.js"; + +const db = createDb(process.env.CREATION_FIXTURE_DATABASE_URL!); +const workers = { + isRunning: () => true, + getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }), + call: async (_pluginId: string, method: string, input: unknown) => { + const response = await fetch(process.env.CREATION_FIXTURE_PROVIDER_URL!, { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ method, input }), + }); + if (!response.ok) throw new Error("Controlled provider unavailable"); + return response.json(); + }, +} as unknown as PluginWorkerManager; +try { + const runtime = environmentRuntimeService(db, { + pluginWorkerManager: workers, orphanCleanupSpoolDir: process.env.CREATION_FIXTURE_EMPTY_DISK, + }); + if (process.env.CREATION_FIXTURE_ACTION === "acquire") { + await runtime.acquireRunLease({ companyId: process.env.CREATION_FIXTURE_COMPANY_ID!, + environment: (await environmentService(db).getById(process.env.CREATION_FIXTURE_ENVIRONMENT_ID!))!, + heartbeatRunId: process.env.CREATION_FIXTURE_RUN_ID!, issueId: null, persistedExecutionWorkspace: null }); + } else { + await runtime.flushDeferredOrphanCleanups(); + } + process.stdout.write(JSON.stringify({ ok: true })); +} catch { + process.stdout.write(JSON.stringify({ ok: false })); +} finally { + await db.$client.end({ timeout: 1 }); +} diff --git a/server/src/__tests__/heartbeat-plugin-environment.test.ts b/server/src/__tests__/heartbeat-plugin-environment.test.ts index f41e8aea8cf..4fbedda7558 100644 --- a/server/src/__tests__/heartbeat-plugin-environment.test.ts +++ b/server/src/__tests__/heartbeat-plugin-environment.test.ts @@ -278,6 +278,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => { }, { timeout: 5_000 }); expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", { + creationOperationId: expect.any(String), driverKey: "sandbox", companyId, environmentId, @@ -745,6 +746,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => { }, { timeout: 5_000 }); expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", { + creationOperationId: expect.any(String), driverKey: "sandbox", companyId, environmentId: newEnvironmentId, diff --git a/server/src/__tests__/managed-skill-content.test.ts b/server/src/__tests__/managed-skill-content.test.ts index 677a3fbda65..210a88a8fd7 100644 --- a/server/src/__tests__/managed-skill-content.test.ts +++ b/server/src/__tests__/managed-skill-content.test.ts @@ -77,7 +77,12 @@ describe.skipIf(!support.supported)("managed skills use committed versions", () .toMatchObject({ content: "complete root guide" }); }); - it("loads persisted Cloud configuration for read-only maintenance without seeding it", async () => { + it.each([ + { storage: "legacy local", env: { PAPERCLIP_STORAGE_PROVIDER: "local_disk" }, expectedStatus: "local_storage" }, + { storage: "shared S3", env: { PAPERCLIP_STORAGE_PROVIDER: "s3", + PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture", PAPERCLIP_STORAGE_S3_REGION: "us-east-1", + PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" }, expectedStatus: "provider_mismatch" }, + ])("loads persisted Cloud configuration for read-only $storage maintenance without seeding it", async ({ env, expectedStatus }) => { const companyId = await company(); const skill = await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" }); const local = await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 }); @@ -89,8 +94,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", () `--phase=${phase}`, `--company=${companyId}`, "--limit=50", ], { env: { ...process.env, DATABASE_URL: database.connectionString, PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture", - PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture", - PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" }, + PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects"), ...env }, timeout: 60_000, maxBuffer: 1024 * 1024 }); await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration"); expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0); @@ -99,7 +103,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", () const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!); expect(eligibility.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "version_backed" })]); const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!); - expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "provider_mismatch" })]); + expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: expectedStatus })]); expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toEqual([persisted]); } finally { await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key)); diff --git a/server/src/__tests__/plugin-warm-tasks.integration.test.ts b/server/src/__tests__/plugin-warm-tasks.integration.test.ts index 6eb396b259e..f369988fca1 100644 --- a/server/src/__tests__/plugin-warm-tasks.integration.test.ts +++ b/server/src/__tests__/plugin-warm-tasks.integration.test.ts @@ -1,11 +1,13 @@ -import { flushDeferredEnvironmentCleanup, hasDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, subscribeDeferredEnvironmentCleanup } from "../services/environment-cleanup-notifications.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { mkdtemp, readdir, rm } from "node:fs/promises"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; import { randomUUID } from "node:crypto"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { eq } from "drizzle-orm"; -import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects, sandboxCreationOperations } from "@paperclipai/db"; import { environmentRuntimeService } from "../services/environment-runtime.js"; -import * as orphanCleanup from "../services/sandbox-orphan-cleanup-spool.js"; +import { recordSandboxCreationResource } from "../services/sandbox-creation-operations.js"; import { resolvePluginTaskResume } from "../services/environment-task-runtime.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -51,7 +53,7 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport if(p.operation.kind==="describe") return {kind:"description",taskId:p.taskId,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`},warmTaskVersion:1}; return {kind:"accepted",taskId:p.taskId}; }} as unknown as PluginWorkerManager; - const runtime=()=>environmentRuntimeService(db,{pluginWorkerManager:workers}); + const runtime=(orphanCleanupSpoolDir?:string)=>environmentRuntimeService(db,{pluginWorkerManager:workers,orphanCleanupSpoolDir}); const acquire=()=>runtime().acquireRunLease({companyId,environment:environment as never,issueId,agentId,heartbeatRunId:runId,persistedExecutionWorkspace:null,adapterType:"paperclip_runner",runnerTaskProjectIds:projectIds,requestedExpiresAt:deadline}); const first=await acquire(); const identity={runnerInstanceId:"runner",environmentLeaseId:first.lease.id,runId,normalizedSessionId:"session",turnId:"turn",itemId:"item"}; @@ -126,58 +128,64 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport await expect(resolvePluginTaskResume(db,{companyId:f.companyId,leaseId:next.lease.id,priorIdentity:f.identity})).resolves.toBe(f.first.lease.providerLeaseId); }); - it.each(["ownership-lost","database-error","cleanup-error","spool"] as const)("compensates an unowned replacement task: %s",async mode=>{ + it.each(["ownership-lost","database-error"] as const)("retains a durable operation for an unattached replacement task: %s",async mode=>{ const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped(); - if(mode==="cleanup-error" || mode==="spool") f.failRelease(); - const append=vi.fn(async()=>true); - let delayFlush=false,flushCalls=0,releaseFlush!:()=>void; - const flushGate=new Promise(resolve=>{releaseFlush=resolve;}); - const stopGate=mode==="spool" ? registerDeferredEnvironmentCleanup(db,async()=>{ - if(delayFlush && ++flushCalls===1) await flushGate; - return {recovered:0,pending:0}; - }) : undefined; - const wake=vi.fn(); - const unsubscribe=subscribeDeferredEnvironmentCleanup(db,wake); - const spool=mode==="spool" ? vi.spyOn(orphanCleanup,"createSandboxOrphanCleanupSpool").mockReturnValue({append,load:async()=>[],remove:async()=>{}}) : undefined; - let replacementFailed=false; - const transact=db.transaction.bind(db); - const transaction=mode==="spool" ? vi.spyOn(db,"transaction").mockImplementation((...args)=>replacementFailed - ? Promise.reject(Error("database unavailable")) : transact(...args)) : undefined; - const original=db.update.bind(db); - const update=vi.spyOn(db,"update").mockImplementation(((table:any)=>{ - const query=original(table),set=query.set.bind(query); - query.set=((values:any)=>values.providerLeaseId === "task-2" - ? {where:()=>({returning:async()=>{replacementFailed=true;if(mode!=="ownership-lost") throw Error("replacement write failed");return [];}})} - : set(values)) as never; - return query; - }) as never); + const disk=await mkdtemp(join(tmpdir(),"replacement-intent-")); + const runtime=()=>f.runtime(disk); + await db.execute(sql.raw(`create or replace function reject_task_replacement() returns trigger language plpgsql as $$ begin + if new.provider_lease_id = 'task-2' then ${mode === "ownership-lost" ? "return null;" : "raise exception 'replacement write failed';"} end if; return new; end $$`)); + await db.execute(sql`create trigger reject_task_replacement before update on environment_leases for each row execute function reject_task_replacement()`); try { - await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})) - .rejects.toThrow("reconcile the same task"); - if(mode==="spool") { - expect(wake).toHaveBeenCalledOnce(); - expect(hasDeferredEnvironmentCleanup(db)).toBe(true); - await expect(flushDeferredEnvironmentCleanup(db)).rejects.toThrow("Deferred orphan cleanup failed"); - expect(hasDeferredEnvironmentCleanup(db)).toBe(true); - } - } finally {update.mockRestore();transaction?.mockRestore();spool?.mockRestore();unsubscribe();} - expect(f.calls).toContain("environmentReleaseLease"); - if(mode==="spool") expect(append).toHaveBeenCalledWith(expect.objectContaining({providerLeaseId:"task-2",metadata:expect.objectContaining({driver:"plugin",runnerTaskDispatched:false})})); - const rows=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId)); - expect(rows.filter(row=>row.providerLeaseId === "task-2")).toEqual(mode==="cleanup-error" - ? [expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned",metadata:expect.objectContaining({runnerTaskDispatched:false})})] : []); - if(mode==="spool") { - delayFlush=true; - const delayed=flushDeferredEnvironmentCleanup(db); - try { expect(await flushDeferredEnvironmentCleanup(db)).toEqual({recovered:1,pending:0}); } - finally { releaseFlush(); stopGate!(); } - expect(await delayed).toEqual({recovered:1,pending:0}); - expect(hasDeferredEnvironmentCleanup(db)).toBe(false); - const recovered=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId)); - expect(recovered.filter(row=>row.providerLeaseId==="task-2")).toEqual([ - expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned"}), - ]); - } + await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task"); + } finally { await db.execute(sql`drop trigger reject_task_replacement on environment_leases`); } + expect(f.calls).not.toContain("environmentReleaseLease"); + const [lease]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id)); + expect(lease.providerLeaseId).toBe(next.lease.providerLeaseId); + const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId)); + const pending=operations.filter(row=>row.state === "pending"); + expect(pending).toHaveLength(1);expect(pending[0].requestKey).toBe(next.lease.providerLeaseId); + expect(pending[0].metadata.creationProviderLeaseId).toBe("task-2");expect(pending[0].config).toEqual({}); + expect(f.acquireRequests.at(-1)?.creationOperationId).toBe(pending[0].id); + const creates=f.acquireRequests.length; + await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task"); + expect(f.acquireRequests).toHaveLength(creates); + await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,pending[0].id)); + await runtime().flushDeferredOrphanCleanups(); + const [manual]=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,pending[0].id)); + expect(manual.state).toBe("needs_reconciliation");expect(manual.metadata.creationProviderLeaseId).toBe("task-2"); + await runtime().flushDeferredOrphanCleanups();expect(f.acquireRequests).toHaveLength(creates); + expect(f.calls).not.toContain("environmentReleaseLease"); + expect(await readdir(disk)).toEqual([]); + await rm(disk,{recursive:true,force:true}); + }); + + it("retains the known replacement identity after a committed attachment response is lost",async()=>{ + const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped(); + const transaction=db.transaction.bind(db);let lost=false; + const response=vi.spyOn(db,"transaction").mockImplementation((async(...args:any[])=>{ + const result=await (transaction as any)(...args); + if(!lost && result?.providerLeaseId === "task-2") {lost=true;throw new Error("Attachment commit response lost");} + return result; + }) as typeof db.transaction); + try { + await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task"); + } finally {response.mockRestore();} + expect(lost).toBe(true); + const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId)); + const replacement=operations.find(operation=>operation.requestKey === next.lease.providerLeaseId)!; + expect(replacement).toMatchObject({state:"attached",leaseId:next.lease.id,metadata:{creationProviderLeaseId:"task-2"}}); + const identity={id:replacement.id,companyId:f.companyId,environmentId:replacement.originalEnvironmentId,providerLeaseId:"task-2"}; + await recordSandboxCreationResource(db,identity); + await expect(recordSandboxCreationResource(db,{...identity,providerLeaseId:"different-resource"})).rejects.toThrow("identity changed"); + await expect(recordSandboxCreationResource(db,{...identity,companyId:randomUUID()})).rejects.toThrow("identity changed"); + expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,replacement.id)))[0]) + .toMatchObject({state:"attached",metadata:{creationProviderLeaseId:"task-2"}}); + const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id)); + expect(saved).toMatchObject({providerLeaseId:"task-2",status:"active"}); + await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,replacement.id)); + await f.runtime().flushDeferredOrphanCleanups(); + expect((await f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).taskId).toBe("task-2"); + expect(f.acquireRequests).toHaveLength(2);expect(f.calls).not.toContain("environmentReleaseLease"); }); it.each(["added","removed"] as const)("retires retained mounts when an authorized project is %s after controller restart",async mode=>{ @@ -203,7 +211,9 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport const replacing=f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}}); if(["violating","missing-bounded","invalid","expired"].includes(mode)) { await expect(replacing).rejects.toThrow("reconcile the same task"); - expect(f.calls).toContain("environmentReleaseLease"); + expect(f.calls).not.toContain("environmentReleaseLease"); + const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId)); + expect(operations.filter(row=>row.state === "pending")).toHaveLength(1); } else { const description=await replacing; const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id)); diff --git a/server/src/__tests__/sandbox-creation-operations.test.ts b/server/src/__tests__/sandbox-creation-operations.test.ts new file mode 100644 index 00000000000..7c5c1924ecb --- /dev/null +++ b/server/src/__tests__/sandbox-creation-operations.test.ts @@ -0,0 +1,333 @@ +import { execFile } from "node:child_process"; +import { createServer } from "node:http"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; +import { randomUUID } from "node:crypto"; +import { mkdtemp, readFile, readdir, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { agents, builtInManagedResources, companies, companySecretVersions, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { environmentRuntimeService } from "../services/environment-runtime.js"; +import { environmentService } from "../services/environments.js"; +import { HttpError } from "../errors.js"; +import * as secretOwner from "../services/secrets.js"; +import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; + +describe("durable sandbox creation through the runtime entry point", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string, environmentId: string, pluginId: string, runId: string, spool: string; + let creates: number, destroys: number, loseReply: boolean, failDestroy: boolean; + let beforeReply: (() => Promise) | undefined; + const resources = new Map(); + let workers: PluginWorkerManager; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("sandbox-creation-operations-"); + db = createDb(database.connectionString); + }, 90_000); + beforeEach(async () => { + await db.execute(sql`truncate sandbox_creation_operations, companies, environments, plugins cascade`); + companyId = randomUUID(); environmentId = randomUUID(); pluginId = randomUUID(); runId = randomUUID(); + const agentId = randomUUID(); creates = 0; destroys = 0; loseReply = false; failDestroy = false; beforeReply = undefined; resources.clear(); + spool = await mkdtemp(join(tmpdir(), "empty-sandbox-web-")); + await db.insert(companies).values({ id: companyId, name: "Creation fixture", issuePrefix: "CRT" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "codex_local" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", invocationSource: "manual" }); + await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox", config: { provider: "creation-fixture", image: "fixture", reuseLease: false } }); + await db.insert(plugins).values({ id: pluginId, pluginKey: "paperclip.creation-fixture", packageName: "creation-fixture", version: "1.0.0", apiVersion: 1, + status: "ready", categories: ["automation"], manifestJson: { id: "paperclip.creation-fixture", version: "1.0.0", apiVersion: 1, + displayName: "Creation fixture", description: "Controlled remote provider", author: "Paperclip", categories: ["automation"], + capabilities: ["environment.drivers.register"], entrypoints: { worker: "fixture.js" }, environmentDrivers: [{ + driverKey: "creation-fixture", kind: "sandbox_provider", displayName: "Fixture", creationOperationVersion: 1, configSchema: { type: "object" }, + }] } }); + workers = { + isRunning: () => true, + getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }), + call: vi.fn(async (_pluginId: string, method: string, input: any) => { + if (method === "environmentAcquireLease") { + expect(await db.select().from(sandboxCreationOperations)).toContainEqual(expect.objectContaining({ id: input.creationOperationId, state: "pending" })); + creates++; + const resource = `provider-${input.creationOperationId}`; + resources.set(input.creationOperationId, resource); + await beforeReply?.(); + if (loseReply) throw new Error("Lost provider reply"); + return { providerLeaseId: resource, metadata: { remoteCwd: "/workspace" } }; + } + if (method === "environmentInspectLease") { + const id = input.creationOperation.id; + const resource = resources.get(id); + if (!resource) return { providerLeaseId: id, state: "unknown" }; + return { providerLeaseId: resource, state: "active", creationCleanup: { providerLeaseId: resource, + observedProviderLeaseId: resource, companyId, environmentId, runId: input.creationOperation.runId, attemptId: id, + accountFingerprint: "a".repeat(64), labels: { "paperclip-create-attempt": id } } }; + } + if (method === "environmentDestroyLease") { + destroys++; + if (failDestroy) throw new Error("Fixture teardown failed"); + resources.delete(input.leaseMetadata.creationOperationId); + return { providerLeaseId: input.providerLeaseId, state: "destroyed" }; + } + throw new Error(`Unexpected fixture method ${method}`); + }), + } as unknown as PluginWorkerManager; + }); + afterEach(async () => { + await db.execute(sql`drop trigger if exists reject_fixture_lease on environment_leases`); + await db.execute(sql`drop trigger if exists reject_fixture_creation on sandbox_creation_operations`); + await rm(spool, { recursive: true, force: true }); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }); + function runtime() { return environmentRuntimeService(db, { pluginWorkerManager: workers, orphanCleanupSpoolDir: spool }); } + async function acquire() { + return runtime().acquireRunLease({ companyId, environment: (await environmentService(db).getById(environmentId))!, + heartbeatRunId: runId, issueId: null, persistedExecutionWorkspace: null, assertCompanyBinding: true }); + } + async function due() { await db.update(sandboxCreationOperations).set({ nextAttemptAt: new Date(0) }); } + async function rejectWrites(table: "environment_leases" | "sandbox_creation_operations", trigger: string) { + await db.execute(sql`create or replace function reject_fixture_write() returns trigger language plpgsql as $$ begin raise exception 'Fixture database write unavailable'; end $$`); + await db.execute(sql.raw(`create trigger ${trigger} before insert on ${table} for each row execute function reject_fixture_write()`)); + } + it("can reapply the creation migration without losing a committed intent", async () => { + loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply"); + const [before] = await db.select().from(sandboxCreationOperations); + const migration = await readFile(new URL("../../../packages/db/src/migrations/0418_curved_lady_mastermind.sql", import.meta.url), "utf8"); + for (let attempt = 0; attempt < 2; attempt++) { + for (const statement of migration.split("--> statement-breakpoint").map(value => value.trim()).filter(Boolean)) { + await db.execute(sql.raw(statement)); + } + } + expect(await db.select().from(sandboxCreationOperations)).toEqual([before]); + expect(creates).toBe(1); + }); + it("fails before the external action when the durable admission write fails", async () => { + await rejectWrites("sandbox_creation_operations", "reject_fixture_creation"); + await expect(acquire()).rejects.toThrow(); + expect(creates).toBe(0); expect(resources.size).toBe(0); expect(await readdir(spool)).toEqual([]); + }); + it("attaches once and reuses the committed transfer after a lost response", async () => { + const first = await acquire(); + const retry = await acquire(); + expect(retry.lease.id).toBe(first.lease.id); expect(creates).toBe(1); expect(destroys).toBe(0); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(environmentLeases)).toHaveLength(1); + expect(await nextEnvironmentCleanupAt(db)).toBeNull(); + }); + it("recovers a failed lease write with a fresh receiver and empty web disk", async () => { + await rejectWrites("environment_leases", "reject_fixture_lease"); + await expect(acquire()).rejects.toThrow(); + expect(creates).toBe(1); expect(await readdir(spool)).toEqual([]); + expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "pending" })]); + await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`); + await due(); + await Promise.all([runtime().flushDeferredOrphanCleanups(), runtime().flushDeferredOrphanCleanups()]); + const leases = await db.select().from(environmentLeases); + expect(leases).toEqual([expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] })]); + expect(creates).toBe(1); + failDestroy = true; + await expect(runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! })).rejects.toThrow("Fixture teardown failed"); + expect(resources.size).toBe(1); + failDestroy = false; + await runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! }); + expect(resources.size).toBe(0); expect(destroys).toBe(2); + }); + it("reconstructs ownership in a replacement receiver process with a different empty web disk", async () => { + const provider = createServer(async (request, response) => { + try { + let body = ""; for await (const chunk of request) body += chunk.toString(); + const { method, input } = JSON.parse(body); + response.setHeader("content-type", "application/json"); + response.end(JSON.stringify(await (workers.call as any)(pluginId, method, input))); + } catch { response.statusCode = 500; response.end("{}"); } + }); + await new Promise(resolve => provider.listen(0, "127.0.0.1", resolve)); + const address = provider.address() as { port: number }; + const replacementDisk = await mkdtemp(join(tmpdir(), "replacement-sandbox-web-")); + const fixture = fileURLToPath(new URL("./fixtures/sandbox-creation-receiver.ts", import.meta.url)); + async function receiver(action: string, disk: string) { + const result = await promisify(execFile)(process.execPath, ["--import", fileURLToPath(import.meta.resolve("tsx")), fixture], { + cwd: fileURLToPath(new URL("../../..", import.meta.url)), timeout: 60_000, + env: { ...process.env, CREATION_FIXTURE_ACTION: action, CREATION_FIXTURE_EMPTY_DISK: disk, + CREATION_FIXTURE_DATABASE_URL: database.connectionString, CREATION_FIXTURE_PROVIDER_URL: `http://127.0.0.1:${address.port}`, + CREATION_FIXTURE_COMPANY_ID: companyId, CREATION_FIXTURE_ENVIRONMENT_ID: environmentId, CREATION_FIXTURE_RUN_ID: runId }, + }); + return result.stdout; + } + try { + await rejectWrites("environment_leases", "reject_fixture_lease"); + expect(await receiver("acquire", spool)).toContain('"ok":false'); + expect(creates).toBe(1); expect(resources.size).toBe(1); expect(await readdir(spool)).toEqual([]); + await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`); + await due(); + expect(await receiver("recover", replacementDisk)).toContain('"ok":true'); + expect(await receiver("recover", replacementDisk)).toContain('"ok":true'); + expect(await db.select().from(environmentLeases)).toEqual([ + expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] }), + ]); + expect(creates).toBe(1); expect(await readdir(replacementDisk)).toEqual([]); + } finally { + await new Promise((resolve, reject) => provider.close(error => error ? reject(error) : resolve())); + await rm(replacementDisk, { recursive: true, force: true }); + } + }, 180_000); + it("never repeats creation after a lost provider reply", async () => { + loseReply = true; + await expect(acquire()).rejects.toThrow("Lost provider reply"); + await expect(acquire()).rejects.toThrow("requires reconciliation"); + expect(creates).toBe(1); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]); + }); + it("retains cleanup ownership when the environment is deleted during creation", async () => { + beforeReply = async () => { await environmentService(db).remove(environmentId); }; + await expect(acquire()).rejects.toThrow(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ environmentId: null, originalEnvironmentId: environmentId, state: "pending" })]); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ environmentId: null, status: "pending_cleanup" })]); + }); + it("rejects attachment after a company rebind and retains cleanup", async () => { + beforeReply = async () => { + const foreign = randomUUID(); await db.insert(companies).values({ id: foreign, name: "Foreign", issuePrefix: "FOR" }); + await db.insert(builtInManagedResources).values({ companyId: foreign, bundleKey: "managed-environment", resourceKind: "environment", + resourceKey: "sandbox", resourceId: environmentId, stockVersion: "1", stockHash: "fixture" }); + }; + await expect(acquire()).rejects.toMatchObject({ status: 403 }); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ companyId, status: "pending_cleanup" })]); + }); + it("backs off a failed lookup and recovers unrelated creations in the same pass", async () => { + loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply"); + const [first] = await db.select().from(sandboxCreationOperations); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + runId = randomUUID(); await db.insert(heartbeatRuns).values({ ...run!, id: runId }); + await expect(acquire()).rejects.toThrow("Lost provider reply"); + const originalCall = workers.call; + (workers as any).call = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => { + if (method === "environmentInspectLease" && params.creationOperation.id === first!.id) throw new Error("Transient fixture lookup"); + return originalCall(plugin, method as any, params, timeout); + }); + await due(); await runtime().flushDeferredOrphanCleanups(); + const rows = await db.select().from(sandboxCreationOperations); + expect(rows.find(value => value.id === first!.id)).toMatchObject({ state: "pending" }); + expect(rows.find(value => value.id === first!.id)!.nextAttemptAt.getTime()).toBeGreaterThan(Date.now()); + expect(rows.filter(value => value.state === "cleanup_pending")).toHaveLength(1); + }); + it("does not attach a sandbox to a run stopped during creation", async () => { + beforeReply = async () => { await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); }; + await expect(acquire()).rejects.toThrow("run authority changed"); + expect(await db.select().from(environmentLeases)).toHaveLength(0); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]); + }); + it.each(["before lookup", "during lookup", "lookup fails after deletion"])("records manual reconciliation when the company is deleted %s", async (deletion) => { + loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply"); + async function deleteCompany() { + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + await db.delete(agents).where(eq(agents.companyId, companyId)); + await db.delete(companies).where(eq(companies.id, companyId)); + } + if (deletion === "before lookup") await deleteCompany(); + const originalCall = workers.call; + const calls = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => { + if (method === "environmentInspectLease" && deletion !== "before lookup") { + await deleteCompany(); + if (deletion === "lookup fails after deletion") throw new Error("Company credential no longer exists"); + } + return originalCall(plugin, method as any, params, timeout); + }); + (workers as any).call = calls; + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([ + expect.objectContaining({ companyId, state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }), + ]); + expect(await db.select().from(environmentLeases)).toEqual([]); + expect(calls.mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(deletion === "before lookup" ? 0 : 1); + const count = calls.mock.calls.length; + await runtime().flushDeferredOrphanCleanups(); await runtime().flushDeferredOrphanCleanups(); + expect(calls).toHaveBeenCalledTimes(count); + expect(await nextEnvironmentCleanupAt(db)).toBeNull(); expect(creates).toBe(1); + expect(await readdir(spool)).toEqual([]); + }); + async function savedCreationCredential(providerConfigId?: string) { + const secrets = secretOwner.secretService(db); + const secret = await secrets.create(companyId, { + name: `creation-key-${randomUUID()}`, provider: "local_encrypted", providerConfigId, value: "fixture-private-key", + }); + await secrets.createBinding({ companyId, secretId: secret.id, targetType: "environment", targetId: environmentId, configPath: "apiKey" }); + await db.update(plugins).set({ manifestJson: sql`jsonb_set(${plugins.manifestJson}, '{environmentDrivers,0,configSchema}', + '{"type":"object","properties":{"apiKey":{"type":"string","format":"secret-ref"}}}'::jsonb)` }).where(eq(plugins.id, pluginId)); + await db.update(environments).set({ config: { provider: "creation-fixture", image: "fixture", reuseLease: false, apiKey: secret.id } }) + .where(eq(environments.id, environmentId)); + loseReply = true; + await expect(acquire()).rejects.toThrow("Lost provider reply"); + return secret; + } + it.each(["deleted", "disabled", "archived", "version-disabled", "vault-disabled", "vault-coming-soon"] as const)( + "retains a %s creation credential as quiet manual reconciliation across restart", async state => { + const secrets = secretOwner.secretService(db); + const vault = state.startsWith("vault-") ? await secrets.createProviderConfig(companyId, { + provider: "local_encrypted", displayName: "Creation vault", status: "ready", config: {}, + }) : undefined; + const secret = await savedCreationCredential(vault?.id); + if (state === "deleted") await secrets.remove(secret.id); + else if (state === "disabled" || state === "archived") await secrets.update(secret.id, { status: state }); + else if (state === "version-disabled") await db.update(companySecretVersions).set({ status: "disabled" }) + .where(eq(companySecretVersions.secretId, secret.id)); + else await secrets.updateProviderConfig(vault!.id, { status: state === "vault-disabled" ? "disabled" : "coming_soon" }); + const inspectCalls = () => vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease").length; + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([ + expect.objectContaining({ state: "needs_reconciliation" }), + ]); + expect(await db.select().from(environmentLeases)).toEqual([]); + expect(inspectCalls()).toBe(0); + expect(await nextEnvironmentCleanupAt(db)).toBeNull(); + const restarted = runtime(); + await restarted.flushDeferredOrphanCleanups(); await restarted.flushDeferredOrphanCleanups(); + expect(inspectCalls()).toBe(0); expect(creates).toBe(1); expect(destroys).toBe(0); + expect(await nextEnvironmentCleanupAt(db)).toBeNull(); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId)); + const ownerId = randomUUID(); + const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 }; + // An installed plugin is a separate blocker until its worker proves quiet. + expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none")) + .toEqual({ version: 1, backgroundWork: "present" }); + expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none", + async () => ({ backgroundWork: "none", pluginIds: [pluginId] }))) + .toEqual({ version: 1, backgroundWork: "none" }); + expect(await readdir(spool)).toEqual([]); + }); + it("retries a temporary creation credential resolution failure without losing ownership", async () => { + await savedCreationCredential(); + const original = secretOwner.secretService; + const resolution = vi.fn(async () => { throw new HttpError(503, "Temporary credential provider failure"); }); + const owner = vi.spyOn(secretOwner, "secretService").mockImplementation(client => ({ + ...original(client), resolveSecretValueForSandboxCleanup: resolution, + })); + try { + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(resolution).toHaveBeenCalledTimes(1); + expect(await db.select().from(sandboxCreationOperations)).toEqual([ + expect.objectContaining({ state: "pending", nextAttemptAt: expect.any(Date) }), + ]); + expect(await nextEnvironmentCleanupAt(db)).toBeGreaterThan(Date.now()); + expect(vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(0); + } finally { owner.mockRestore(); } + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "cleanup_pending" })]); + expect(await db.select().from(environmentLeases)).toHaveLength(1); + expect(creates).toBe(1); expect(destroys).toBe(0); expect(await readdir(spool)).toEqual([]); + }); + it("makes unsupported uncertain outcomes explicit without another creation", async () => { + loseReply = true; await expect(acquire()).rejects.toThrow(); + await db.update(plugins).set({ manifestJson: sql`${plugins.manifestJson} #- '{environmentDrivers,0,creationOperationVersion}'` }).where(eq(plugins.id, pluginId)); + await due(); await runtime().flushDeferredOrphanCleanups(); + expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "needs_reconciliation" })]); + expect(await nextEnvironmentCleanupAt(db)).toBeNull(); + await expect(acquire()).rejects.toThrow("requires reconciliation"); expect(creates).toBe(1); + }); +}); diff --git a/server/src/services/environment-cleanup-state.ts b/server/src/services/environment-cleanup-state.ts index 177ec9d1906..eae1d13fb4e 100644 --- a/server/src/services/environment-cleanup-state.ts +++ b/server/src/services/environment-cleanup-state.ts @@ -1,5 +1,5 @@ import { eq, sql } from "drizzle-orm"; -import { environmentLeases, type Db } from "@paperclipai/db"; +import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db"; import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js"; export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000; @@ -20,7 +20,10 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C const [row] = await db.select({ next: sql`min(greatest( extract(epoch from ${environmentLeases.updatedAt}) * 1000 + ${backoffMs}, ${pendingCleanupDeadlineSql()}))` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup")); - const saved = row?.next == null ? null : Number(row.next); + const [creation] = await db.select({ next: sql`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` }) + .from(sandboxCreationOperations).where(eq(sandboxCreationOperations.state, "pending")); + const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number); + const saved = deadlines.length ? Math.min(...deadlines) : null; const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null; return saved === null ? buffered : buffered === null ? saved : Math.min(saved, buffered); } diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 8db16c3209d..277a5a0254c 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -1,3 +1,6 @@ +import { HttpError } from "../errors.js"; +import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; +import { createOwnedSandbox, recoverSandboxCreations } from "./sandbox-creation-operations.js"; import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js"; import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js"; @@ -11,7 +14,6 @@ import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/nati import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js"; import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk"; import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js"; -import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import { hasNativeWorkspaceExportResume, releaseCompletedNativeWorkspaceExportRetention } from "./native-runtime/native-workspace-export-resume.js"; import { createHash, randomUUID } from "node:crypto"; @@ -93,7 +95,7 @@ import { resolvePluginExecuteRpcTimeoutMs, resumePluginEnvironmentLease, } from "./plugin-environment-driver.js"; -import { collectSecretRefPaths } from "./json-schema-secret-refs.js"; +import { collectSecretRefPaths, readConfigValueAtPath } from "./json-schema-secret-refs.js"; import { buildWorkspaceRealizationRecordFromDriverInput } from "./workspace-realization.js"; import { createSandboxOrphanCleanupSpool, @@ -698,39 +700,6 @@ export interface EnvironmentRuntimeDriver { flushDeferredOrphanCleanups?(): Promise<{ recovered: number; pending: number }>; } -/** - * The acquire provisioned a remote sandbox, the lease insert rejected it, the - * compensating teardown failed, and every retry of the durable pending-cleanup - * write also failed. A live sandbox now has no lease row and no cleanup record, - * so no sweep can find it. The acquire throws this error so the failure stays visible - * and never resolves to a clean rejection. The `cause` field holds the original - * lease insert rejection. The `cleanupWriteError` field holds the write failure. - */ -export class SandboxOrphanCleanupWriteError extends Error { - readonly provider: string; - readonly providerLeaseId: string | null; - readonly cleanupWriteError: unknown; - - constructor(input: { - provider: string; - providerLeaseId: string | null; - cause: unknown; - cleanupWriteError: unknown; - }) { - super( - `Sandbox provider "${input.provider}" leaked lease ` + - `"${input.providerLeaseId ?? "unknown"}": the acquire could not tear it down ` + - "and could not record a durable pending-cleanup row. The live sandbox has no " + - "durable cleanup state and needs a manual teardown.", - { cause: input.cause }, - ); - this.name = "SandboxOrphanCleanupWriteError"; - this.provider = input.provider; - this.providerLeaseId = input.providerLeaseId; - this.cleanupWriteError = input.cleanupWriteError; - } -} - /** A reusable sandbox could not be resumed, but has not been proven lost. */ export class ReusableSandboxResumeError extends Error { readonly provider: string; @@ -785,17 +754,6 @@ export type ProviderResourceDisposition = const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000; const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100; -// The durable pending-cleanup write is the only automated way to find a leaked -// sandbox. A single insert can lose to a transient database fault (a dropped -// connection, a lock timeout, a serialization conflict). So the acquire retries -// the write a few times with a short backoff before it gives up. A later -// attempt can still land the durable row that a sweep finds. -const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS = 3; -const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS = 100; - -// A prolonged database outage after a failed teardown can buffer many orphan -// records in-process. The bound stops that buffer from growing without limit. A -// full buffer keeps the error log as the last durable handle for a new orphan. const DEFAULT_DEFERRED_ORPHAN_CLEANUP_BUFFER_LIMIT = 256; function delay(ms: number): Promise { @@ -1311,8 +1269,6 @@ function createSandboxEnvironmentDriver( pluginWorkerManager?: PluginWorkerManager; pluginWorkerReadyTimeoutMs?: number; pluginWorkerReadyPollMs?: number; - pendingCleanupWriteAttempts?: number; - pendingCleanupWriteBackoffMs?: number; deferredOrphanCleanupBufferLimit?: number; orphanCleanupSpool?: SandboxOrphanCleanupSpool; orphanCleanupSpoolDir?: string; @@ -1321,16 +1277,6 @@ function createSandboxEnvironmentDriver( const pluginWorkerManager = options.pluginWorkerManager; const pluginWorkerReadyTimeoutMs = options.pluginWorkerReadyTimeoutMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS; const pluginWorkerReadyPollMs = options.pluginWorkerReadyPollMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS; - // The retry count is at least one attempt, so a zero or negative override - // never skips the durable write. The backoff is at least zero milliseconds. - const pendingCleanupWriteAttempts = Math.max( - 1, - options.pendingCleanupWriteAttempts ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS, - ); - const pendingCleanupWriteBackoffMs = Math.max( - 0, - options.pendingCleanupWriteBackoffMs ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS, - ); // The buffer holds at least one orphan, so a zero or negative override never // drops every buffered record. const deferredOrphanCleanupBufferLimit = Math.max( @@ -1339,15 +1285,6 @@ function createSandboxEnvironmentDriver( ); const environmentsSvc = environmentService(db); - // A live sandbox whose teardown failed needs a durable `pending_cleanup` row, - // so a sweep can find and release it. When every synchronous write attempt - // fails, the database is down but the process still runs. So the driver keeps - // the orphan record in this in-process buffer, and a later cleanup sweep - // flushes it back to the database. The buffer closes the common window where - // the database recovers after the synchronous attempts but while the process - // still runs. The `pending_cleanup` failure reason matches the synchronous - // write, so a flushed row and a synchronous row are indistinguishable to the - // sweep. const deferredOrphanCleanups: DeferredOrphanCleanupRecord[] = []; // Keep the token through queue splices, failed flushes and buffer overflow. // Release only when the database durably owns cleanup of this exact orphan. @@ -1363,16 +1300,6 @@ function createSandboxEnvironmentDriver( }; const DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON = "acquire_rejected_teardown_failed"; - // The in-process buffer alone loses an orphan on a restart. So the driver also - // persists each buffered orphan to this durable local spool. The spool keeps - // the record on the local disk, next to the other durable server state, so a - // restart still finds it. The first flush after a restart loads the spool back - // into the buffer, so the sweep re-inserts the durable row. The spool holds no - // secret value: it persists the same sanitized metadata the database row - // holds, so a restart-recovered record still authorizes the teardown but never - // exposes a credential. Only a crash during the outage before the durable - // spool write lands loses the record; the error log stays the last handle for - // that narrow case. const orphanCleanupSpool = options.orphanCleanupSpool ?? createSandboxOrphanCleanupSpool(options.orphanCleanupSpoolDir); @@ -1460,143 +1387,6 @@ function createSandboxEnvironmentDriver( return orphanFlush; }; - // Build the safe diagnostic fields for an orphan record. The provider - // identifiers are the only fields any diagnostic log emits, and they carry no - // secret. A caught write exception never enters a log: it can carry a - // credential in its name, code, message, cause, or stack. - const orphanDiagnosticFields = (record: DeferredOrphanCleanupRecord) => ({ - errorKind: SANDBOX_ORPHAN_CLEANUP_WRITE_ERROR_KIND, - provider: record.provider, - providerLeaseId: record.providerLeaseId, - companyId: record.companyId, - environmentId: record.environmentId, - executionWorkspaceId: record.executionWorkspaceId, - issueId: record.issueId, - heartbeatRunId: record.heartbeatRunId, - }); - - // Try to write the durable `pending_cleanup` row for an orphan sandbox, with a - // few retries. Report the new lease id on success, or a null lease id and the - // last write error on total failure. This function never buffers, never logs - // at error level, and never throws. The caller decides whether a total failure - // is fatal, because a later successful teardown removes the orphan and needs - // no durable row. - // - // The row carries the provider lease id and the same resolution metadata a - // normal lease holds, so a later cleanup sweep finds and releases the orphan. - // One atomic insert writes the row directly in the terminal `pending_cleanup` - // state, and the insert skips the company-binding assertion, so it records the - // orphan even for a foreign-bound environment. A transient database fault (a - // dropped connection, a lock timeout, a serialization conflict) can reject one - // insert but clear on the next, so the write retries a few times with a short - // backoff. - const tryWriteDurablePendingCleanup = async ( - record: DeferredOrphanCleanupRecord, - ): Promise<{ leaseId: string | null; lastError: unknown }> => { - const diagnosticFields = orphanDiagnosticFields(record); - let lastCleanupWriteError: unknown; - for (let attempt = 1; attempt <= pendingCleanupWriteAttempts; attempt += 1) { - try { - const lease = await environmentsSvc.insertPendingCleanupLease({ - companyId: record.companyId, - environmentId: record.environmentId, - executionWorkspaceId: record.executionWorkspaceId, - issueId: record.issueId, - heartbeatRunId: record.heartbeatRunId, - provider: record.provider, - providerLeaseId: record.providerLeaseId, - metadata: record.metadata, - failureReason: DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON, - }); - // The durable row exists now, so a sweep can find and release the orphan. - return { leaseId: lease.id, lastError: undefined }; - } catch (cleanupWriteError) { - lastCleanupWriteError = cleanupWriteError; - if (attempt < pendingCleanupWriteAttempts) { - // The write failed, but attempts remain. Log the retry at warn level - // and back off, then try again. A later attempt can still land the - // durable row. - logger.warn( - { ...diagnosticFields, attempt, maxAttempts: pendingCleanupWriteAttempts }, - "sandbox orphan cleanup write failed; retrying the durable pending-cleanup write", - ); - await delay(pendingCleanupWriteBackoffMs * attempt); - } - } - } - return { leaseId: null, lastError: lastCleanupWriteError }; - }; - - // Release the durable `pending_cleanup` row after a successful inline teardown. - // The teardown removed the orphan sandbox, so the row is no longer needed. The - // release moves the row to the terminal `expired` state with a `success` - // cleanup status, the same state the sweep records for a successful teardown. - // - // The release is best-effort. If it fails, the row stays `pending_cleanup`, so - // a later sweep runs the idempotent teardown on the already-gone sandbox and - // releases the row itself. A failed release never loses or leaks the orphan. - const releaseCleanedUpOrphanRow = async ( - leaseId: string, - diagnosticFields: Record, - receipt: unknown, - ): Promise => { - try { - const lease = await environmentsSvc.getLeaseById(leaseId); - await environmentsSvc.releaseLease(leaseId, "expired", { - ...(lease ? { remoteExecutionTermination: remoteTerminationReceipt(lease, receipt) } : {}), - cleanupStatus: "success", - failureReason: "acquire_rejected_teardown_succeeded", - }); - } catch { - // The caught release exception never enters a log: it can carry a - // credential in its name, code, message, cause, or stack. - logger.warn( - diagnosticFields, - "could not release the pending-cleanup row after a successful orphan teardown; a later sweep reconciles the released row", - ); - } - }; - - // Neither the durable write nor the inline teardown worked, so the orphan - // sandbox is live with no durable row. Persist the orphan to the durable spool - // and buffer it in-process, so a later cleanup-sweep flush lands the durable - // row. The spool survives a restart, so the flush recovers the orphan even - // after a crash. Then log the plain provider identifiers at error level, so an - // operator keeps a handle to tear the sandbox down by hand. Only a crash before - // the spool write lands loses the record, so the error log is the last handle - // for that narrow case. Throw `SandboxOrphanCleanupWriteError`, which the - // caller propagates. The error keeps the original insert rejection as its - // `cause`, so the real reason the acquire failed stays visible in the error - // chain. - const escalateUnwrittenOrphan = async ( - record: DeferredOrphanCleanupRecord, - cause: unknown, - cleanupWriteError: unknown, - ): Promise => { - trackOrphanIdleWork(record); - const persisted = await orphanCleanupSpool.append(record); - const buffered = enqueueDeferredOrphanCleanup(record); - logger.error( - { - ...orphanDiagnosticFields(record), - persisted, - buffered, - deferredBufferSize: deferredOrphanCleanups.length, - }, - persisted - ? "sandbox orphan cleanup write failed; persisted the orphan to the durable spool for a later cleanup-sweep flush" - : buffered - ? "sandbox orphan cleanup write failed and the durable spool write failed; buffered the orphan in-process only for a later cleanup-sweep flush" - : "sandbox orphan cleanup write failed and neither the durable spool nor the in-process buffer kept the orphan; live sandbox needs a manual teardown", - ); - throw new SandboxOrphanCleanupWriteError({ - provider: record.provider, - providerLeaseId: record.providerLeaseId, - cause, - cleanupWriteError, - }); - }; - // Resolve an uncertain creation to a verified provider ID, journal it, then // authorize deletion. This makes a later missing-ID response conclusive even // when the destructive RPC reply or the final lease update was lost. @@ -1628,68 +1418,7 @@ function createSandboxEnvironmentDriver( } }; - // Clean up an orphan after a rejected insert or uncertain provider creation. - // The acquire provisioned a remote sandbox, then the insert rejected (a - // foreign-company binding), so no lease row tracks the live sandbox. This - // handler records the durable `pending_cleanup` row FIRST, before the inline - // teardown, then tears the sandbox down. - // - // The write goes first on purpose. The conditional insert just returned a - // definitive rejection, so the database is reachable at this instant. An - // inline teardown can run long, and the database can drop during it. So a - // write after the teardown can fail and leave the orphan only in memory, which - // a restart loses. With the write first, the durable row lands while the - // database is proven up, so a crash after it still leaves a row that a sweep - // finds and releases. - // - // The teardown then runs. On success the orphan is gone, so the handler - // releases the durable row. On a failed teardown the row stays for the sweep. - // If the durable write itself failed and the teardown also failed, the handler - // buffers the orphan and throws `SandboxOrphanCleanupWriteError`. A successful - // teardown after a failed write leaves no orphan, so it needs no error. - const cleanUpRejectedOrphanSandbox = async (input: { - record: DeferredOrphanCleanupRecord; - cause: unknown; - canTeardown: boolean; - teardown: (pendingLeaseId: string | null) => Promise; - }): Promise => { - const durable = await tryWriteDurablePendingCleanup(input.record); - let teardownFailed = !input.canTeardown; - let receipt: unknown; - if (!teardownFailed) { - try { - receipt = await input.teardown(durable.leaseId); - } catch { - teardownFailed = true; - } - } - if (!teardownFailed) { - // The teardown removed the orphan, so drop the durable row if we wrote one. - if (durable.leaseId !== null) { - await releaseCleanedUpOrphanRow(durable.leaseId, orphanDiagnosticFields(input.record), receipt); - } - return true; - } - // The teardown failed, so the orphan sandbox is still live. - if (durable.leaseId !== null) { - // The durable row already tracks the orphan, so a sweep finds and releases - // it. The caller rethrows the original insert rejection. - return false; - } - return await escalateUnwrittenOrphan(input.record, input.cause, durable.lastError); - }; - // The run-time exec parent context, held per lease id. A plugin sandbox - // provider can open a persistent session on the first command and delete it - // on lease release. The session open runs inside `execute`, under the run - // parent (the same context the `sandbox.exec` span reads). The session delete - // runs inside the lease-release RPC, which the run orchestrator calls after - // the run, outside that scope. Without a parent the host mints no - // `traceparent` and drops the provider `session.teardown` span. So `execute` - // records the exec parent here, and the release paths replay it around the - // release RPC. The host still mints and validates the `traceparent` itself; - // the value only widens which host calls carry a run parent. An entry is - // removed on release, so the map holds at most one context per live lease. const runExecParentByLeaseId = new Map(); // Run a lease-release RPC under the lease's recorded exec parent context, and @@ -2128,12 +1857,19 @@ function createSandboxEnvironmentDriver( }); } } - const acquisitionRunId = input.heartbeatRunId ?? randomUUID(); - const acquiredLease = providerLease ?? await (async () => { + const acquiredLease = providerLease ?? await createOwnedSandbox(db, { + companyId: input.companyId, environmentId: input.environment.id, + heartbeatRunId: input.heartbeatRunId, issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId, + provider: parsed.config.provider, pluginId: pluginProvider.resolved.plugin.id, driverKey: parsed.config.provider, + resolvedConfig: { ...parsed.config }, + configSchema: pluginProvider.resolved.driver.configSchema as Record | undefined, + config: sandboxConfigForLeaseMetadata(storedConfig), requestKey: createHash("sha256").update(JSON.stringify(storedConfig)).digest("hex"), + metadata: { ...sandboxConfigForLeaseMetadata(storedConfig), driver: input.environment.driver, + pluginId: pluginProvider.resolved.plugin.id, pluginKey: pluginProvider.resolved.plugin.pluginKey, sandboxProviderPlugin: true }, + timeoutMs: resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs), + }, async identity => { try { - return await pluginWorkerManager.call( - pluginProvider.resolved.plugin.id, - "environmentAcquireLease", + return await pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentAcquireLease", { driverKey: parsed.config.provider, companyId: input.companyId, @@ -2143,7 +1879,8 @@ function createSandboxEnvironmentDriver( // Plugin SDK requires a string; ad-hoc test leases use a fresh // UUID so providers that validate or persist the runId still see // a well-formed identifier. - runId: acquisitionRunId, + runId: identity.runId, + creationOperationId: identity.id, workspaceMode: input.executionWorkspaceMode ?? undefined, agentId: input.agentId ?? undefined, executionWorkspaceId: input.executionWorkspaceId ?? undefined, @@ -2162,62 +1899,18 @@ function createSandboxEnvironmentDriver( ? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) } : {}), }, - resolvePluginSandboxRpcTimeoutMs( - workerConfig, - pluginProvider.resolved.driver.defaultAcquireTimeoutMs, - ), - ); + resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs)); } catch (error) { - captureEnvironmentAcquisitionDiagnostic(error, { - companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId, - }, null); const cleanup = readEnvironmentCreationCleanupError(error); - // The authenticated worker may identify its uncertain allocation, - // but cannot redirect cleanup to another company, environment, or run. - if (cleanup && cleanup.companyId === input.companyId && - cleanup.environmentId === input.environment.id && cleanup.runId === acquisitionRunId) { - if (error instanceof JsonRpcCallError) captureEnvironmentAcquisitionDiagnostic(error, { - companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId, - }, readEnvironmentAcquisitionDiagnostic(error)); - const cleanupMetadata = { - ...sandboxConfigForLeaseMetadata(storedConfig), - failedCreateCleanup: cleanup, - driver: input.environment.driver, - pluginId: pluginProvider.resolved.plugin.id, - pluginKey: pluginProvider.resolved.plugin.pluginKey, - sandboxProviderPlugin: true, - }; - // Record before retrying the provider. Existing pending-cleanup - // recovery (including its durable spool) survives controller loss. - const cleanupConfirmed = await cleanUpRejectedOrphanSandbox({ - record: { - companyId: input.companyId, environmentId: input.environment.id, - executionWorkspaceId: input.executionWorkspaceId ?? null, - issueId: input.issueId ?? null, heartbeatRunId: input.heartbeatRunId ?? null, - provider: parsed.config.provider, providerLeaseId: cleanup.providerLeaseId, - metadata: cleanupMetadata, - }, - cause: error, - canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id), - teardown: (pendingLeaseId) => destroyWithCreationObservation({ - leaseId: pendingLeaseId, metadata: cleanupMetadata, - teardown: (metadata) => pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentDestroyLease", { - driverKey: parsed.config.provider, companyId: input.companyId, - environmentId: input.environment.id, issueId: input.issueId, - config: workerConfig, providerLeaseId: cleanup.providerLeaseId, - leaseMetadata: metadata, - }, resolvePluginSandboxRpcTimeoutMs(workerConfig)), - }), - }); - if (cleanupConfirmed) { - throw new Error("Sandbox creation failed; allocated sandbox cleanup was confirmed.", { cause: error }); - } - } - // A cleanup record is not a usable lease. Keep acquisition failed, - // even when its compensating deletion succeeds on this attempt. + const diagnostic = error instanceof JsonRpcCallError && cleanup?.companyId === input.companyId && + cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId + ? readEnvironmentAcquisitionDiagnostic(error) : null; + captureEnvironmentAcquisitionDiagnostic(error, { + companyId: input.companyId, environmentId: input.environment.id, runId: identity.runId, + }, diagnostic); throw error; } - })(); + }); // Ad-hoc test leases are never publishable for reuse: storing them // as `reuse_by_environment` would let a concurrent heartbeat resume @@ -2275,82 +1968,34 @@ function createSandboxEnvironmentDriver( : {}), ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}), }; - try { - return await environmentsSvc.acquireLease({ - companyId: input.companyId, - environmentId: input.environment.id, - executionWorkspaceId: input.executionWorkspaceId, - issueId: input.issueId, - heartbeatRunId: input.heartbeatRunId, - assertCompanyBinding: input.assertCompanyBinding, - leasePolicy: resolvedLeasePolicy, - provider: parsed.config.provider, - providerLeaseId: acquiredLease.providerLeaseId, - expiresAt: providerAttestedLeaseExpiry( - input.requestedExpiresAt, - acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined, - ), - metadata: pluginLeaseMetadata, - reusesReusableLeaseId: - providerLease && - reusableLease?.heartbeatRunId === input.heartbeatRunId - ? reusableLease.id - : null, - replacesReusableLeaseId: - providerLease && - reusableLease?.heartbeatRunId !== input.heartbeatRunId - ? reusableLease?.id - : null, - }); - } catch (error) { - // The conditional lease insert rejected, so no lease row exists. A - // managed reconciliation can bind the environment to another company - // between the route guard and this insert, so the insert fails closed - // with `environment_company_mismatch`. This call already provisioned the - // remote plugin sandbox above, so tear it down now. Without this step - // the rejected insert leaks a live sandbox that no lease row tracks. The - // Claude login runs on a plugin-backed sandbox, so this path is the one - // the login uses. Do not tear down a reused sandbox that an earlier - // lease still owns. - if (!reusableLease || acquiredLease.providerLeaseId !== reusableLease.providerLeaseId) { - // Record the durable pending-cleanup row before the teardown, then - // tear the sandbox down. On a successful teardown the handler releases - // the row. If the durable write and the teardown both fail, the - // handler throws a `SandboxOrphanCleanupWriteError` that carries the - // original rejection as its cause. - await cleanUpRejectedOrphanSandbox({ - record: { - companyId: input.companyId, - environmentId: input.environment.id, - executionWorkspaceId: input.executionWorkspaceId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - provider: parsed.config.provider, - providerLeaseId: acquiredLease.providerLeaseId, - metadata: pluginLeaseMetadata, - }, - cause: error, - canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id), - teardown: async () => { - return await pluginWorkerManager.call( - pluginProvider.resolved.plugin.id, - "environmentDestroyLease", - { - driverKey: parsed.config.provider, - companyId: input.companyId, - environmentId: input.environment.id, - issueId: input.issueId, - config: workerConfig, - providerLeaseId: acquiredLease.providerLeaseId, - leaseMetadata: acquiredLease.metadata ?? undefined, - }, - resolvePluginSandboxRpcTimeoutMs(workerConfig), - ); - }, - }); - } - throw error; - } + return await environmentsSvc.acquireLease({ + creationOperationId: providerLease ? undefined : acquiredLease.metadata?.creationOperationId as string | undefined, + companyId: input.companyId, + environmentId: input.environment.id, + executionWorkspaceId: input.executionWorkspaceId, + issueId: input.issueId, + heartbeatRunId: input.heartbeatRunId, + assertCompanyBinding: input.assertCompanyBinding, + leasePolicy: resolvedLeasePolicy, + provider: parsed.config.provider, + providerLeaseId: acquiredLease.providerLeaseId, + expiresAt: providerAttestedLeaseExpiry( + input.requestedExpiresAt, + acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined, + ), + metadata: pluginLeaseMetadata, + reusesReusableLeaseId: + providerLease && + reusableLease?.heartbeatRunId === input.heartbeatRunId + ? reusableLease.id + : null, + replacesReusableLeaseId: + providerLease && + reusableLease?.heartbeatRunId !== input.heartbeatRunId + ? reusableLease?.id + : null, + }); + } // Built-in sandbox provider path. Same guard as the plugin-backed path: @@ -2441,10 +2086,16 @@ function createSandboxEnvironmentDriver( let providerLease; try { - providerLease = await acquireSandboxProviderLease({ + providerLease = await createOwnedSandbox(db, { + companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId, + issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId, provider: parsed.config.provider, + driverKey: parsed.config.provider, config: sandboxConfigForLeaseMetadata(parsed.config), + requestKey: createHash("sha256").update(JSON.stringify(parsed.config)).digest("hex"), + metadata: { ...sandboxConfigForLeaseMetadata(parsed.config), driver: input.environment.driver }, + }, identity => acquireSandboxProviderLease({ config: parsed.config, environmentId: input.environment.id, - heartbeatRunId: input.heartbeatRunId ?? randomUUID(), + heartbeatRunId: identity.runId, issueId: input.issueId, agentId: input.agentId, executionWorkspaceId: input.executionWorkspaceId, @@ -2452,7 +2103,7 @@ function createSandboxEnvironmentDriver( // Forward a caller deadline so the provider configures a provider-side // expiry at or before it and returns the real provider expiry. requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt), - }); + })); } catch (error) { if (reusableLease) { throw new ReusableSandboxResumeError({ @@ -2521,72 +2172,35 @@ function createSandboxEnvironmentDriver( : {}), ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}), }; - try { - return await environmentsSvc.acquireLease({ - companyId: input.companyId, - environmentId: input.environment.id, - executionWorkspaceId: input.executionWorkspaceId, - issueId: input.issueId, - heartbeatRunId: input.heartbeatRunId, - assertCompanyBinding: input.assertCompanyBinding, - leasePolicy: resolvedLeasePolicy, - provider: parsed.config.provider, - providerLeaseId: providerLease.providerLeaseId, - expiresAt: providerAttestedLeaseExpiry( - input.requestedExpiresAt, - providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined, - ), - metadata: builtinLeaseMetadata, - reusesReusableLeaseId: - reusableLease && - providerLease.providerLeaseId === reusableLease.providerLeaseId && - reusableLease.heartbeatRunId === input.heartbeatRunId - ? reusableLease.id - : null, - replacesReusableLeaseId: - reusableLease && - providerLease.providerLeaseId === reusableLease.providerLeaseId && - reusableLease.heartbeatRunId !== input.heartbeatRunId - ? reusableLease.id - : null, - }); - } catch (error) { - // The conditional lease insert rejected, so no lease row exists. A managed - // reconciliation can bind the environment to another company between the - // route guard and this insert, so the insert fails closed with - // `environment_company_mismatch`. This call already provisioned the remote - // sandbox above, so release it now. Without this teardown the rejected - // insert leaks a live sandbox that no lease row tracks. Do not tear down a - // reused sandbox that an earlier lease still owns. - if (!reusableLease || providerLease.providerLeaseId !== reusableLease.providerLeaseId) { - // Record the durable pending-cleanup row before the teardown, then - // release the remote sandbox. On a successful teardown the handler - // releases the row. If the durable write and the teardown both fail, - // the handler throws a `SandboxOrphanCleanupWriteError` that carries the - // original rejection as its cause. - await cleanUpRejectedOrphanSandbox({ - record: { - companyId: input.companyId, - environmentId: input.environment.id, - executionWorkspaceId: input.executionWorkspaceId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - provider: parsed.config.provider, - providerLeaseId: providerLease.providerLeaseId, - metadata: builtinLeaseMetadata, - }, - cause: error, - canTeardown: true, - teardown: async () => { - await destroySandboxProviderLease({ - config: parsed.config, - providerLeaseId: providerLease.providerLeaseId, - }); - }, - }); - } - throw error; - } + return await environmentsSvc.acquireLease({ + creationOperationId: providerLease.metadata?.creationOperationId as string | undefined, + companyId: input.companyId, + environmentId: input.environment.id, + executionWorkspaceId: input.executionWorkspaceId, + issueId: input.issueId, + heartbeatRunId: input.heartbeatRunId, + assertCompanyBinding: input.assertCompanyBinding, + leasePolicy: resolvedLeasePolicy, + provider: parsed.config.provider, + providerLeaseId: providerLease.providerLeaseId, + expiresAt: providerAttestedLeaseExpiry( + input.requestedExpiresAt, + providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined, + ), + metadata: builtinLeaseMetadata, + reusesReusableLeaseId: + reusableLease && + providerLease.providerLeaseId === reusableLease.providerLeaseId && + reusableLease.heartbeatRunId === input.heartbeatRunId + ? reusableLease.id + : null, + replacesReusableLeaseId: + reusableLease && + providerLease.providerLeaseId === reusableLease.providerLeaseId && + reusableLease.heartbeatRunId !== input.heartbeatRunId + ? reusableLease.id + : null, + }); }, async releaseRunLease(input) { @@ -2719,7 +2333,7 @@ function createSandboxEnvironmentDriver( `Sandbox provider "${recordedProvider}" needs a plugin worker manager for cleanup, but none is available.`, ); } - const pinnedPluginId = exportResume ? readString(input.lease.metadata?.pluginId) : null; + const pinnedPluginId = exportResume || input.lease.metadata?.creationOperationId ? readString(input.lease.metadata?.pluginId) : null; if (exportResume && !pinnedPluginId) throw new Error("Workspace export cleanup has no recorded provider plugin."); const pluginProvider = await resolveSandboxProviderPlugin({ provider: recordedProvider, ...(pinnedPluginId ? { pluginId: pinnedPluginId } : {}) }); if (pluginProvider.state !== "running") { @@ -3534,6 +3148,7 @@ const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([ "remoteCwd", "shellCommand", "sandboxProviderPlugin", + "creationOperationId", "sandboxLeaseAcquisition", "nativeHarnessBackup", "nativeWorkspaceSync", @@ -3698,13 +3313,20 @@ function createPluginEnvironmentDriver( resumed = undefined; } } - const providerLease = resumed ?? await workerManager.call(plugin.id, "environmentAcquireLease", { + const providerLease = resumed ?? await createOwnedSandbox(db, { + companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId, + issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId, + provider: `plugin:${parsed.config.pluginKey}:${parsed.config.driverKey}`, pluginId: plugin.id, driverKey: parsed.config.driverKey, + config: {}, timeoutMs: rpcTimeoutMs, requestKey: fingerprint, + metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: parsed.config.pluginKey, driverKey: parsed.config.driverKey }, + }, identity => workerManager.call(plugin.id, "environmentAcquireLease", { driverKey: parsed.config.driverKey, companyId: input.companyId, environmentId: input.environment.id, issueId: input.issueId, config: parsed.config.driverConfig, - runId: input.heartbeatRunId ?? randomUUID(), + runId: identity.runId, + creationOperationId: identity.id, workspaceMode: input.executionWorkspaceMode ?? undefined, agentId: input.agentId ?? undefined, executionWorkspaceId: input.executionWorkspaceId ?? undefined, @@ -3715,9 +3337,10 @@ function createPluginEnvironmentDriver( ...(requestedExpiresAtParam(input.requestedExpiresAt) !== undefined ? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) } : {}), - } as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride); + } as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride)); return await environmentsSvc.acquireLease({ + creationOperationId: resumed ? undefined : providerLease.metadata?.creationOperationId as string | undefined, companyId: input.companyId, environmentId: input.environment.id, executionWorkspaceId: input.executionWorkspaceId, @@ -3964,14 +3587,13 @@ export function environmentRuntimeService( pluginWorkerManager?: PluginWorkerManager; pluginWorkerReadyTimeoutMs?: number; pluginWorkerReadyPollMs?: number; - pendingCleanupWriteAttempts?: number; - pendingCleanupWriteBackoffMs?: number; deferredOrphanCleanupBufferLimit?: number; orphanCleanupSpool?: SandboxOrphanCleanupSpool; orphanCleanupSpoolDir?: string; } = {}, ) { const environmentsSvc = environmentService(db); + let orphanRecoveryFlush: Promise<{ recovered: number; pending: number }> | null = null; const drivers = new Map(); const defaultDrivers = [ @@ -3981,8 +3603,6 @@ export function environmentRuntimeService( pluginWorkerManager: options.pluginWorkerManager, pluginWorkerReadyTimeoutMs: options.pluginWorkerReadyTimeoutMs, pluginWorkerReadyPollMs: options.pluginWorkerReadyPollMs, - pendingCleanupWriteAttempts: options.pendingCleanupWriteAttempts, - pendingCleanupWriteBackoffMs: options.pendingCleanupWriteBackoffMs, deferredOrphanCleanupBufferLimit: options.deferredOrphanCleanupBufferLimit, orphanCleanupSpool: options.orphanCleanupSpool, orphanCleanupSpoolDir: options.orphanCleanupSpoolDir, @@ -4439,14 +4059,53 @@ export function environmentRuntimeService( return driver.isPendingCleanupWorkerReady(input); }, - // Flush the in-process orphan-cleanup buffers of every driver that keeps one. - // The cleanup sweep calls this each tick, so a buffered orphan lands a durable - // `pending_cleanup` row once the database recovers, and the same sweep tears - // it down. The dispatcher sums the recovered and pending counts across all - // drivers. A driver with no buffer contributes nothing. + // Recover committed creation operations before replaying legacy cleanup inputs. + // New creation failures retain their database operation and do not make files. + // Legacy buffers and files remain until their cleanup rows are durable. async flushDeferredOrphanCleanups(): Promise<{ recovered: number; pending: number }> { - return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values()) - .flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : [])); + orphanRecoveryFlush ??= (async () => { + await recoverSandboxCreations(db, async operation => { + if (!operation.pluginId || operation.metadata.sandboxProviderPlugin !== true) return null; + if (!options.pluginWorkerManager) throw new Error("Sandbox creation recovery worker is unavailable"); + const plugin = await pluginRegistryService(db).getById(operation.pluginId); + const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === operation.driverKey); + if (driver?.creationOperationVersion !== 1) return null; + if (!options.pluginWorkerManager.isRunning(operation.pluginId) || + !options.pluginWorkerManager.getWorker(operation.pluginId)?.supportedMethods.includes("environmentInspectLease")) { + throw new Error("Sandbox creation recovery worker is unavailable"); + } + let resolved: SandboxEnvironmentConfig; + try { + resolved = await resolveSandboxCleanupConfigSecrets(db, operation.companyId, + operation.config as SandboxEnvironmentConfig, { issueId: operation.issueId }); + } catch (error) { + // Missing or explicitly inactive saved credentials need an operator. + // Temporary database/provider failures still retain retry ownership. + const code = error instanceof HttpError ? (error.details as { code?: string } | undefined)?.code : undefined; + if (error instanceof HttpError && ( + (error.status === 404 && (code === "secret_deleted" || code === "version_missing" || + (error.details === undefined && error.message === "Secret not found"))) || + (error.status === 422 && (code === "secret_inactive" || code === "version_inactive" || + (error.details === undefined && (error.message === "Provider vault is disabled" || + error.message === "Provider vault is locked while coming soon")))) + )) return null; + throw error; + } + const credentialHashes = operation.metadata.creationCredentialHashes as Record | undefined; + for (const [path, expected] of Object.entries(credentialHashes ?? {})) { + const actual = readConfigValueAtPath({ ...resolved }, path); + if (typeof actual !== "string" || createHash("sha256").update(actual).digest("hex") !== expected) return null; + } + return options.pluginWorkerManager.call(operation.pluginId, "environmentInspectLease", { + driverKey: operation.driverKey, companyId: operation.companyId, environmentId: operation.originalEnvironmentId, + config: stripSandboxProviderEnvelope(resolved), providerLeaseId: operation.id, + creationOperation: { id: operation.id, runId: operation.runId }, + }); + }); + return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values()) + .flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : [])); + })().finally(() => { orphanRecoveryFlush = null; }); + return orphanRecoveryFlush; }, async destroyReusableSandboxLeases(input: { diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts index 5abe356bda3..3595d00b430 100644 --- a/server/src/services/environment-task-runtime.ts +++ b/server/src/services/environment-task-runtime.ts @@ -1,8 +1,6 @@ -import { beginIdleTrackedWork } from "./task-admission.js"; -import { registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; -import { environmentService } from "./environments.js"; -import { createSandboxOrphanCleanupSpool, type DeferredOrphanCleanupRecord } from "./sandbox-orphan-cleanup-spool.js"; -import { logger } from "../middleware/logger.js"; +import { attachSandboxCreation, environmentService } from "./environments.js"; +import { createOwnedSandbox, recordSandboxCreationResource } from "./sandbox-creation-operations.js"; +import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js"; import { readWorkspaceProviderBinding } from "@paperclipai/shared"; import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js"; @@ -234,59 +232,38 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({runnerTaskExecutionStopped:true, runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`, }).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId))); - const fresh = await workers.call(plugin.id, "environmentAcquireLease", {driverKey:metadata.driverKey, + const replacementDriverKey = metadata.driverKey; + const fresh = await createOwnedSandbox(db, { + companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId, + issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!, + pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId, + metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey, runnerTaskLease: true }, + }, identity => workers.call(plugin.id, "environmentAcquireLease", {driverKey:replacementDriverKey, companyId:input.companyId,environmentId:environment!.id,issueId:lease.issueId,config:driverConfig, - runId:run!.id,agentId:run!.agentId,adapterType:"paperclip_runner", - ...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})}); + runId:identity.runId,creationOperationId:identity.id,agentId:run!.agentId,adapterType:"paperclip_runner", + ...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})})); if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable"); + await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId, + environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId }); const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff, runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata; const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false, runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false}; - try { + { const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null; const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null; if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) || (deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline))) throw new Error("Plugin replacement task expiry is not bounded"); - const replacement = await db.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt, - metadata:replacementMetadata, + await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => { + const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt, + metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId}, }).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId), eq(environmentLeases.status,"active"))).returning({id:environmentLeases.id}); - if (replacement.length !== 1) throw new Error("Plugin task replacement ownership changed"); - } catch (error) { - try { - await workers.call(plugin.id,"environmentReleaseLease",{driverKey:metadata.driverKey,companyId:input.companyId, - environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,providerLeaseId:fresh.providerLeaseId, - leaseMetadata:replacementMetadata}); - } catch { - const orphan:DeferredOrphanCleanupRecord={companyId:input.companyId,environmentId:environment!.id, - executionWorkspaceId:lease.executionWorkspaceId,issueId:lease.issueId,heartbeatRunId:lease.heartbeatRunId, - provider:lease.provider!,providerLeaseId:fresh.providerLeaseId,metadata:replacementMetadata}; - try { - await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"}); - } catch { - // The existing orphan spool recovers cleanup even when the database is unavailable. - const spool = createSandboxOrphanCleanupSpool(); - const persisted = await spool.append(orphan); - const finish = beginIdleTrackedWork(); - let flushing: Promise<{ recovered: number; pending: number }> | null = null; - const unregister = registerDeferredEnvironmentCleanup(db, () => { - flushing ??= (async () => { - await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"}); - await spool.remove(orphan); - unregister(); - finish(); - return { recovered: 1, pending: 0 }; - })().catch(error => { flushing = null; throw error; }); - // Keep successful completion for a concurrent sweep that already - // captured this callback before it was unregistered. - return flushing; - }); - logger.error({providerLeaseId:fresh.providerLeaseId,persisted},"Plugin replacement cleanup requires recovery"); - } - } - throw error; + if (!updated) throw new Error("Plugin task replacement ownership changed"); + return (await environmentService(tx).getLeaseById(updated.id))!; + }); + } return executeEnvironmentTask(db, workers, input); } diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index ea72243e464..e46df62ed7d 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -16,6 +16,7 @@ import { companySecretBindings, environmentCustomImageSetupSessions, environmentLeases, + sandboxCreationOperations, environments, executionWorkspaces, heartbeatRuns, @@ -48,6 +49,43 @@ import { type ManagedResourceStockStatus, } from "./managed-resource-drift.js"; +export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null }, + attach: (tx: Db) => Promise): Promise { + return db.transaction(async tx => { + await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId); + const [operation] = await tx.select().from(sandboxCreationOperations) + .where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId), + eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update"); + if (scope.heartbeatRunId) { + const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"), + )).for("update"); + if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed"); + } + if (operation?.state === "attached" && operation.leaseId) { + const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId); + if (existing?.status === "active") return existing; + } + if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred"); + const lease = await attach(tx as unknown as Db); + await notifyEnvironmentCleanup(tx, { companyId: scope.companyId }); + await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() }) + .where(eq(sandboxCreationOperations.id, id)); + return lease; + }); +} + +export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise { + const [environment] = await db.select({ id: environments.id }).from(environments) + .where(eq(environments.id, environmentId)).for("update"); + if (!environment) throw new Error("Sandbox creation environment no longer exists"); + const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources) + .where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId))); + if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) { + throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" }); + } +} + type EnvironmentRow = typeof environments.$inferSelect; type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect; const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local"; @@ -1428,6 +1466,7 @@ export function environmentService(db: Db) { }, acquireLease: async (input: { + creationOperationId?: string; companyId: string; environmentId: string; executionWorkspaceId?: string | null; @@ -1465,6 +1504,11 @@ export function environmentService(db: Db) { */ assertCompanyBinding?: boolean; }): Promise => { + if (input.creationOperationId) { + const { creationOperationId, ...leaseInput } = input; + return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId }, + tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true })); + } const now = new Date(); const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null; const values = { @@ -1515,48 +1559,7 @@ export function environmentService(db: Db) { companyId: input.companyId, runId: input.heartbeatRunId, }); if (input.assertCompanyBinding) { - // Lock the environment row first. Managed reconciliation locks the - // same sandbox environment rows with `for update` before it writes a - // company binding, so this lock serializes the two transactions on - // this row and closes the time-of-check to time-of-use window. - await tx - .select({ id: environments.id }) - .from(environments) - .where(eq(environments.id, input.environmentId)) - .for("update"); - // Re-read the company binding inside the locked transaction. A - // binding a reconciliation committed after the route guard now - // appears here. Reject a foreign-company environment before the - // insert, so the login holds no lease. - const boundRows = await tx - .select({ companyId: builtInManagedResources.companyId }) - .from(builtInManagedResources) - .where( - and( - eq( - builtInManagedResources.resourceKind, - MANAGED_ENVIRONMENT_RESOURCE_KIND, - ), - eq( - builtInManagedResources.resourceId, - input.environmentId, - ), - ), - ); - const boundCompanyIds = Array.from( - new Set(boundRows.map((boundRow) => boundRow.companyId)), - ); - if ( - boundCompanyIds.length > 0 && - !boundCompanyIds.includes(input.companyId) - ) { - throw forbidden( - "The selected environment belongs to another company.", - { - code: "environment_company_mismatch", - }, - ); - } + await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId); } if (input.replacesReusableLeaseId) { await notifyChatInboundWakeNotices(tx, { companyId: input.companyId }); @@ -1699,6 +1702,7 @@ export function environmentService(db: Db) { const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId, heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status, cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt, + creationOperationId: sql`${environmentLeases.metadata}->>'creationOperationId'`, }).from(environmentLeases).where(condition).for("update"); if (!previous) return null; await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined }); @@ -1715,6 +1719,11 @@ export function environmentService(db: Db) { companyId: previous.companyId, runId: previous.heartbeatRunId, }); } + if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") { + await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now }) + .where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId), + eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending"))); + } return tx.update(environmentLeases) .set({ status, diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index a43a3b2f089..c0d58a64bdd 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -54,6 +54,7 @@ const WORK_CHECKS = [ state NOT IN ('saved', 'unchanged', 'resolved', 'unavailable', 'conflict') OR process_stopped_at IS NULL OR next_attempt_at IS NOT NULL OR receipt ? 'baseline' OR receipt->>'cleanupPending' = 'true'`, + `SELECT 1 FROM sandbox_creation_operations WHERE state = 'pending'`, `SELECT 1 FROM budget_reservations WHERE state NOT IN ('settled', 'released')`, `SELECT 1 FROM budget_policies WHERE enforcement_version > enforcement_delivered_version`, `SELECT 1 FROM agents WHERE lifecycle_state IN diff --git a/server/src/services/sandbox-creation-operations.ts b/server/src/services/sandbox-creation-operations.ts new file mode 100644 index 00000000000..a56c6615284 --- /dev/null +++ b/server/src/services/sandbox-creation-operations.ts @@ -0,0 +1,150 @@ +import { collectSecretRefPaths, isUuidSecretRef, parseSecretRefBindingObject, readConfigValueAtPath } from "./json-schema-secret-refs.js"; +import { createHash, randomUUID } from "node:crypto"; +import { and, eq, lte, or, sql } from "drizzle-orm"; +import { companies, environments, sandboxCreationOperations, type Db } from "@paperclipai/db"; +import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk"; +import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js"; +import { ENVIRONMENT_CLEANUP_RETRY_MS } from "./environment-cleanup-state.js"; +import { logger } from "../middleware/logger.js"; +import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js"; + +export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect; + +type CreationOwner = { + companyId: string; + environmentId: string; + executionWorkspaceId?: string | null; + issueId?: string | null; + heartbeatRunId?: string | null; + provider: string; + pluginId?: string; + driverKey: string; + config: Record; + metadata: Record; + timeoutMs?: number; + requestKey?: string; + configSchema?: Record | null; + resolvedConfig?: Record; +}; + +export async function createOwnedSandbox( + db: Db, owner: CreationOwner, + create: (identity: { id: string; runId: string }) => Promise, +): Promise { + const credentialHashes: Record = {}; + for (const path of collectSecretRefPaths(owner.configSchema)) { + const value = readConfigValueAtPath(owner.config, path); + if (value != null && value !== "" && !(typeof value === "string" && isUuidSecretRef(value)) && !parseSecretRefBindingObject(value)) { + throw new Error("Sandbox creation requires saved secret references"); + } + if (value != null && value !== "") { + const resolved = readConfigValueAtPath(owner.resolvedConfig ?? {}, path); + if (typeof resolved !== "string") throw new Error("Sandbox creation credential identity is unavailable"); + credentialHashes[path] = createHash("sha256").update(resolved).digest("hex"); + } + } + const id = randomUUID(); + const runId = owner.heartbeatRunId ?? randomUUID(); + const operation = await db.transaction(async tx => { + await notifyEnvironmentCleanup(tx, { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined }); + await assertEnvironmentCompanyBinding(tx as unknown as Db, owner.environmentId, owner.companyId); + const [inserted] = await tx.insert(sandboxCreationOperations).values({ + id, companyId: owner.companyId, environmentId: owner.environmentId, originalEnvironmentId: owner.environmentId, + runId, requestKey: owner.requestKey ?? "run", issueId: owner.issueId, executionWorkspaceId: owner.executionWorkspaceId, + provider: owner.provider, pluginId: owner.pluginId, driverKey: owner.driverKey, + config: owner.config, metadata: { ...owner.metadata, creationCredentialHashes: credentialHashes }, + nextAttemptAt: new Date(Date.now() + Math.max(30 * 60_000, (owner.timeoutMs ?? 0) + 60_000)), + }).onConflictDoNothing().returning(); + if (inserted) return inserted; + const [existing] = await tx.select().from(sandboxCreationOperations).where(and( + eq(sandboxCreationOperations.companyId, owner.companyId), eq(sandboxCreationOperations.originalEnvironmentId, owner.environmentId), + eq(sandboxCreationOperations.runId, runId), eq(sandboxCreationOperations.requestKey, owner.requestKey ?? "run"), + )); + if (!existing) throw new Error("Sandbox creation ownership was not recorded"); + return existing; + }); + if (operation.id !== id) { + if (operation.state !== "attached" || !operation.leaseId) throw new Error("Sandbox creation requires reconciliation before another attempt"); + const existing = await environmentService(db).getLeaseById(operation.leaseId); + if (!existing || existing.status !== "active" || !existing.providerLeaseId) throw new Error("Sandbox creation has already transferred ownership"); + return { providerLeaseId: existing.providerLeaseId, expiresAt: existing.expiresAt?.toISOString(), + metadata: { ...existing.metadata, creationOperationId: operation.id } }; + } + const lease = await create({ id, runId }); + return { ...lease, metadata: { ...lease.metadata, creationOperationId: id } }; +} + +/** Keep an already returned resource identity even when task attachment fails. + * This enriches the existing owner; it never transfers or reopens ownership. */ +export async function recordSandboxCreationResource(db: Db, input: { + id: string; companyId: string; environmentId: string; providerLeaseId: string; +}): Promise { + const [recorded] = await db.update(sandboxCreationOperations).set({ + metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId })}::jsonb`, + updatedAt: new Date(), + }).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId), + eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId), + or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`, + sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`), + )).returning({ id: sandboxCreationOperations.id }); + if (!recorded) throw new Error("Sandbox creation resource identity changed"); +} + +export async function recoverSandboxCreations(db: Db, + inspect: (operation: SandboxCreationOperation) => Promise, +): Promise { + const due = await db.select().from(sandboxCreationOperations).where(and( + eq(sandboxCreationOperations.state, "pending"), lte(sandboxCreationOperations.nextAttemptAt, new Date()), + )).limit(100); + for (const candidate of due) { + try { + const [existingCompany] = await db.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).limit(1); + const observation = existingCompany ? await inspect(candidate) : null; + const cleanup = observation?.creationCleanup ? readEnvironmentCreationCleanupError({ data: { + schema: "paperclip/environment-creation-cleanup/v1", cleanup: observation.creationCleanup, + } }) : null; + if (cleanup && (cleanup.companyId !== candidate.companyId || cleanup.environmentId !== candidate.originalEnvironmentId || + cleanup.runId !== candidate.runId || cleanup.attemptId !== candidate.id || cleanup.providerLeaseId !== observation!.providerLeaseId)) { + throw new Error("Sandbox creation recovery returned foreign ownership"); + } + await db.transaction(async tx => { + const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share"); + await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, candidate.originalEnvironmentId)).for("update"); + const [operation] = await tx.select().from(sandboxCreationOperations) + .where(eq(sandboxCreationOperations.id, candidate.id)).for("update"); + if (!operation || operation.state !== "pending") return; + await notifyEnvironmentCleanup(tx, { companyId: operation.companyId }); + if (!company || !cleanup) { + logger.warn({ errorKind: "sandbox_creation_needs_reconciliation", operationId: operation.id, companyId: operation.companyId }, + "Sandbox creation could not be identified; operator reconciliation is required"); + await tx.update(sandboxCreationOperations).set({ state: "needs_reconciliation", updatedAt: new Date(), + metadata: { ...operation.metadata, creationReconciliationReason: company ? "creation_identity_unavailable" : "company_deleted" }, + }) + .where(eq(sandboxCreationOperations.id, operation.id)); + return; + } + const lease = await environmentService(tx as unknown as Db).insertPendingCleanupLease({ + companyId: operation.companyId, environmentId: operation.originalEnvironmentId, + provider: operation.provider, providerLeaseId: cleanup.providerLeaseId, + metadata: { ...operation.metadata, creationOperationId: operation.id, failedCreateCleanup: cleanup }, + failureReason: "interrupted_sandbox_creation", + }); + await tx.update(sandboxCreationOperations).set({ state: "cleanup_pending", leaseId: lease.id, updatedAt: new Date() }) + .where(eq(sandboxCreationOperations.id, operation.id)); + }); + } catch { + const retry = await db.transaction(async tx => { + const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share"); + await notifyEnvironmentCleanup(tx, { companyId: candidate.companyId }); + await tx.update(sandboxCreationOperations).set(company + ? { nextAttemptAt: new Date(Date.now() + ENVIRONMENT_CLEANUP_RETRY_MS), updatedAt: new Date() } + : { state: "needs_reconciliation", updatedAt: new Date(), + metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || '{"creationReconciliationReason":"company_deleted"}'::jsonb` }) + .where(and(eq(sandboxCreationOperations.id, candidate.id), eq(sandboxCreationOperations.state, "pending"))); + return Boolean(company); + }); + logger.warn({ errorKind: retry ? "sandbox_creation_recovery_failed" : "sandbox_creation_needs_reconciliation", operationId: candidate.id, companyId: candidate.companyId }, + retry ? "Sandbox creation recovery will retry at its saved deadline" : "Sandbox creation company was deleted; operator reconciliation is required"); + } + } +}