mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
fix(environments): persist sandbox ownership before creation (#18357)
## Thinking Path > - Paperclip manages companies and the agents that do their work. > - Remote execution creates provider resources that need a durable owner. > - The server could create a sandbox before it saved its lease. > - A failed lease write could leave cleanup data only on the old web server disk. > - This change saves a company-scoped operation before creation and transfers ownership atomically. > - Recovery finds the original Daytona resource or records an explicit manual reconciliation outcome. ## Linked Issues or Issue Description **What happened?** A sandbox could exist without a lease after a database failure or company rebind. Failed cleanup recording could then create a local fallback file. Replacing that web server could lose the cleanup reference. **Expected behavior** A failed admission write makes no provider call. An interrupted creation retains a committed operation. Recovery must not create a second sandbox or destroy an attached lease. **Steps to reproduce** 1. Run the disposable PostgreSQL cases in `server/src/__tests__/sandbox-creation-operations.test.ts`. 2. Reject the lease insert after the controlled provider creates its resource. 3. Start a new receiver process with a different empty web disk. It must find the same resource and create one cleanup owner. Related ownership report: Refs #14672. This change addresses new allocations before lease persistence. It does not replace existing lease recovery. ## What Changed - Add an idempotent migration and a small creation-operation table. Retain original company, environment and run identities across deletion. - Commit creation intent before run, ad-hoc lease, and plugin task replacement calls. Fence duplicate requests and ambiguous lease commits. - Retain a known returned task resource ID in the existing operation before attachment. Preserve manual reconciliation and never delete an ambiguously attached resource. - Keep company and run authority checks and atomic lease transfer with the lease owner. - Reuse Daytona names, labels, read-only inspection and claimed cleanup. Keep provider payload handling in the provider. - Retain nonsecret credential identity hashes. Refuse recovery lookup under a changed credential value. - Record manual reconciliation for missing/deleted or explicitly inactive saved credentials, versions, and provider vaults. Keep temporary resolution failures retryable. Never reactivate a disabled credential. - Use transaction signals and concrete deadlines. Failed records back off independently. Manual reconciliation stays durable and quiet. - Remove new orphan-file fallback writes from the covered paths. Keep legacy file replay and its tests for cutover. - Replace obsolete fallback-producer assertions with normal-entrypoint durability tests. Keep cleanup, diagnostics and warm handoff coverage. ## Verification - [Independent current-head verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320) passed 190/190 combined tests, 40/40 incoming TestOps tests, full workspace typecheck, and build. PR head `958af986130f25e136065ad30d1fdd843c46255c` has tree `e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL, compiled Rust Runner and controlled object providers were used; external-S3 integration remains outside this local proof. - Current head `958af986130f25e136065ad30d1fdd843c46255c` normally merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole conflict was the maintenance test: a parameterized case now preserves both legacy-local `local_storage` and valid shared-S3 `provider_mismatch` checks, including the parent local receipt, exact version identity, missing-snapshot/no-seed refusal, and unchanged persisted row. The three master TestOps files remain intact. No production, schema, migration, or package changes were made after the fully qualified prior head `0b4d05505e`. Independent review approved the resolution; the full managed-skill-content file and Cloud guard controls pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge head, first attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412): 54 successful checks and three intended skips. The additional Paperclip review is neutral at 4/5; two external Superagent checks (Contributor trust and Security Scan) remain in progress with zero annotations and no assessment, so they are not passing scan evidence. Greptile reviewed exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final thread audit has zero unresolved threads. All completed evidence below retains its recorded revision scope. The additional Paperclip Review on `958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or should-fix item. Its optional observation is that a generic create rejection remains pending/manual even when an operator believes allocation did not occur. This change preserves uncertainty because the generic create-error contract does not prove provider absence; no new error classification or provider-specific compensation was added. - [Independent final-source verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247) passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full workspace typecheck, and build. The integration commit `c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as final PR head `0b4d05505e` (tree `bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider coverage passed; the separately guarded external-S3 suite was not run because the bounded local MinIO pull was unavailable. This includes the two final compatibility fixes. - The first final-head CI attempt exposed one E2E timing failure in `disconnect-write-recovery.spec.ts:181`: the title stayed unchanged through the 15-second session-recovery poll. The same original case passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all UI/session/E2E source is identical across those heads. The failure log and CI artifacts are retained; no cause is claimed. One failed-job-only retry is authorized on this evidence, without changing a timeout, assertion, or production behavior. [The single failed-job retry completed successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): all 35 E2E shard cases passed (6.6 minutes), including the original session-recovery case in 14.0 seconds. The initial failure remains disclosed and its cause remains unproven. - Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two independently reviewed integration fixes after the complete 7ec CI matrix exposed two deterministic failures. Server 9/12 failed legacy local-storage maintenance because the CLI reached the new Cloud server startup guard; server 5/12 found the Daytona package version did not match its bumped manifest. The failed run is retained at https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is not passing evidence and was not blindly retried. - Maintenance now constructs the existing explicit storage dependency after loading the required persisted Cloud snapshot, and passes it to both maintenance owners. It can verify legacy local storage before provider cutover while normal Cloud server startup remains guarded. The original case reproduced red locally, then the complete managed-skill-content and managed-cloud-prerequisites files passed 57/57 (32.05 seconds), including missing-snapshot refusal, unchanged stored configuration, and existing server guard controls. Daytona package version now matches manifest 0.1.10; the complete bundled package identity file passed 14/14 (3.98 seconds), and explicit manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42 seconds). An initial workspace-filter command matched no project and is not counted as pass evidence. The final diff changes no sandbox runtime or schema behavior. [Full CI is green on this exact final head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): 56 successful checks, three intended skips, one additional incomplete/neutral Paperclip Review, and no pending or failed checks. The neutral review is not a passing assessment. [Greptile reviewed the exact final head at 5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969), with no new actionable issues and all review threads resolved. - Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` includes master managed-skills migration 0417 and shared-storage prerequisites from `198218bf55` through normal merges. Sandbox creation moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL is byte-for-byte unchanged. The regenerated 0418 snapshot contains both sandbox creation operations and the master skill mutation receipts, and points to the master 0417 snapshot. Master SQL and journal history remain intact. Exactly five snapshots remain, 0414 through 0418. The only sandbox-specific changes since the approved credential policy are migration metadata/number and the matching test/document references; the sandbox runtime files are unchanged. - [Independent final combined verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026) passed on integration revision `e6f05b4cf0cd`, containing final sandbox head `7ec06f77b7` and actual master `198218bf55`: migration/declaration checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251 seconds), full workspace typecheck (81 seconds), and build (35 seconds). - Final combined migration proof: fresh real PostgreSQL has both tables. A separate original-0417 sandbox table with a retained pending row survives two applications of the renamed 0418 SQL unchanged, with the expected primary/index and foreign-key structure. Migration numbering/safety checks pass. The normal runtime migration reapply case passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first temporary qualification harness stopped before any database assertion because its external `.ts` file selected CommonJS; the `.mts` harness corrected that setup and passed. Independent review approved the exact metadata and reference diff. Earlier full suite and live-provider results below retain their stated revision scope. - Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd` handles the remaining inactive-credential review warning. The existing boundary accepts only concrete secret-owner 422 codes and exact no-details disabled/coming-soon vault messages. Unknown 422, authorization errors, provider wrappers, and temporary database/provider failures still propagate. Independent source review approved this three-file correction. No generic resolver, schema, provider protocol, retry cap, or activation behavior changed. - Creation/cleanup qualification at `c606d473b0`: all 39 cases pass against real PostgreSQL (81.27 seconds). The new cases use actual disabled/archived saved secrets, a valid disabled version, and provider vaults created ready before normal API updates disable them or mark them coming soon. Each remains manual through restart/repeated scans, with zero provider lookup and no retry deadline. The independent plugin idle blocker remains intact. The temporary 503 control retains retry ownership and later recovers one cleanup owner. Before this correction, all five inactive-state cases remained pending; that failed evidence is retained. The prior 18-case known-ID task proof below belongs to `237144fcf6`, whose helper and task path are unchanged by this correction. - Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also addresses the independent task-replacement reliability warning. The existing creation operation now saves only the known nonsecret returned resource ID before expiry validation and attachment. The write is scoped to operation/company/original environment and allows only a missing or matching ID. It changes no ownership state and copies no resolved configuration, credentials, or provider payload. Independent source review approved this four-file fix. - The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds). Rejected attachments retain the ID through manual reconciliation and repeated scans. A real committed attachment followed by an injected lost response retains the same active lease and attached operation, with no release or second create. Same-ID retries work; conflicting IDs and foreign companies cannot change it. The initial three regression cases failed because the ID was absent, before the fix. - The shipped Daytona manifest does not declare task support, so it cannot enter this generic replacement path. The dispatcher requires both declared task support and an environmentTask worker method. Generic drivers without creation lookup still use manual reconciliation, with a known returned ID retained when the database write can commit. No automatic lookup capability or unconditional compensation was added. - Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0` fixes the fresh review finding about missing credentials. Only known secret-owner 404 results at the recovery secret-resolution boundary select the existing manual outcome. Provider inspection and transient database/provider errors remain retryable. Independent source review approved this change. - Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass against isolated real PostgreSQL. The complete creation suite passes 16/16 (100.75 seconds); the complete cleanup coordinator suite passes 18/18 (99.83 seconds). The new cases use normal saved-secret creation and deletion, confirm zero provider lookup, restart/repeated scans, no automatic deadline, preserved independent plugin idle blocking, and no additional idle blocker after the controlled worker supplies its quiet proof. A temporary 503 resolution failure preserves pending ownership and later recovers one cleanup owner. No additional live provider resource was created. - Failed attempts remain part of the evidence. The initial red run reproduced the pending-operation bug and also exposed a numeric-deadline test harness mistake, which was corrected. A later combined run reported an existing replacement-receiver failure and a new idle-fixture assertion failure; it was interrupted while the next test file remained in imports. An isolated diagnostic confirmed that the manual state was correct but the installed fixture plugin correctly blocked sleep without its independent quiet proof. The corrected test retains that blocker assertion before supplying the controlled worker witness. The unchanged replacement-receiver case passes in the final complete 16-case run; its earlier failure is not counted as a pass or assigned a proven cause. No timeout or production idle guard was relaxed. - Greptile reviewed previous combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved review threads. Its CI produced the two concrete integration failures addressed above. The earlier CI and live evidence below remain historical evidence, not a claim that the current full gates have completed. - Master integration update: merged `dbff3747b8` and then latest master `76fde891cc` without rewriting branch history. Only `doc/DATABASE.md` conflicted; the sandbox ownership, object-backed run evidence, and cloud configuration sections all remain. The master merges changed no sandbox implementation, migration, or provider code. Migration numbering/safety checks pass and exactly five schema snapshots remain. At merge head `f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused creation/coordinator suites pass all 32 cases against isolated real PostgreSQL (64.57 seconds). Those merge-head checks precede the missing-credential fix below. The earlier live Daytona proof still covers the unchanged normal creation/lookup/teardown path; the new missing-credential branch is qualified with real PostgreSQL and controlled provider hooks. The optional Superagent Security Scan was inconclusive on prior head `7ec06f77b7`: its pinned PR merge-base checkout failed or timed out, with a neutral conclusion and zero annotations. An earlier scan could not start within 120 seconds; a bounded rerun request returned HTTP 404. These scanner failures are not passing security-scan evidence. On final head `0b4d05505e`, the additional Paperclip Review check is neutral: its assigned review execution did not complete and produced no validated assessment. Fresh Greptile 5/5 and independent source review are separate evidence. Previous-head required CI, independent review, and the live Daytona qualification below passed. - [Complete previous-head CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052) passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful checks and three intended skips, including typecheck, full tests, build and E2E. Greptile is 5/5 on this head and both original findings are resolved. - Initial head: 200 PostgreSQL runtime, creation and coordinator tests passed. The corrected head also passes all 32 creation and coordinator tests, including deletion before/during lookup, a failed lookup after deletion, restart, settled polling and actual idle-safety checks. Cases cover failed admission, failed lease writes, concurrent recovery, lost provider replies, ambiguous transfer retries, company rebind, environment deletion, stopped runs, lookup backoff, restart, drain and settled SQL quiet. - Separate receiver processes: passed. A controlled HTTP provider retains the resource while three independent receiver processes use two empty web disks. Recovery creates one cleanup owner and never repeats creation. - Fresh PostgreSQL migration and two additional applications: passed. The pending creation row survives unchanged. - Server TypeScript and SDK/shared build checks pass after the corrections. Migration numbering/safety checks passed. Exactly five schema snapshots remain. - Daytona adapter: all 330 tests passed, including stable-name lookup, ownership checks, deletion and the updated manifest version. An earlier run exposed the version expectation and it was corrected. Previous-head CI passed. - Live Daytona acceptance passed on `0b5cbb763a`: isolated real PostgreSQL, normal runtime acquisition through independent receiver processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu 24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert failure retained the committed operation; a fresh receiver with empty disk found the same sandbox, repeated recovery retained one cleanup owner, and the actual claimed cleanup sweep expired the lease and resolved the operation. Independent SDK lookup returned typed not-found; ordinary recorded-ID inspection returned destroyed. This successful attempt made one create, one recovery lookup, and one teardown. Two earlier failed harness/provider-confirmation attempts are disclosed in the [live acceptance record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054). The coordinator independently queried all three exact resource IDs through the SDK and confirmed typed not-found for each. - Complete CI-affected heartbeat/plugin warm-task suites passed 21/21 after test corrections. An unchanged workspace exposure case passed in isolation after the prior CI shard reported an unhandled readiness rejection. Complete previous-head CI and refreshed review passed. ## Risks - Deleted companies retain an explicit manual reconciliation outcome. Recovery does not retry missing credentials or insert a lease for a deleted company. - Only Daytona declares automatic creation lookup. Other uncertain providers retain `needs_reconciliation`. A missing creation name alone cannot prove that a late allocation will never appear. - Pending operations use the existing process-local signal and startup recovery model. This does not add cross-replica wake delivery or prove deployed sleep. - Legacy direct provider probes and custom-image setup use separate lifecycles. They are outside this replacement of the run-lease orphan-file fallback. The normal Test path that acquires a runtime lease is covered. - Existing orphan files still require the separate cutover inventory and settlement. This change does not delete them. - The schema change is additive. The new SQL can be reapplied. Historical SQL and journal entries are preserved; only the oldest generated snapshot is pruned. ## Model Used OpenAI `gpt-6.1-sol`, with tool use, code execution and repository review. The harness did not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The required branch status checks (`ci / verify`, `ci / e2e`) and all test/build jobs passed on this head. The non-required Superagent Contributor trust and Security Scan checks are still running without findings; they are not claimed as passes. No required check or branch rule is overridden. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
81ecaf0654
commit
45a3a92e6d
30 files changed
+1723
-1879
No files matched your search
@@ -1545,6 +1545,23 @@ named `review_wait_benchmark...` can be supplied through
|
||||
`PAPERCLIP_TEST_DATABASE_URL` to compare another PostgreSQL major version.
|
||||
DataRow bytes exclude response metadata and transport overhead.
|
||||
|
||||
|
||||
### Sandbox creation ownership
|
||||
|
||||
Migration 0418 adds `sandbox_creation_operations`. Admission commits company,
|
||||
original environment, run, request key and provider identity before external
|
||||
creation. Configuration contains nonsecret values and existing secret references.
|
||||
The original identity survives environment, run, task and company deletion; only
|
||||
the optional live environment FK uses SET NULL. Company/run deletion must not
|
||||
erase an outstanding remote-resource obligation.
|
||||
|
||||
The unique company/environment/run/request index fences duplicate creation and
|
||||
ambiguous successful lease commits. Normal lease attachment and operation transfer
|
||||
share a transaction. Recovery transfers one claimed cleanup lease after provider
|
||||
ownership lookup. The pending deadline index drives the existing cleanup worker.
|
||||
`needs_reconciliation` is a durable manual outcome, not an automatic retry or an
|
||||
idle blocker. See [cleanup ownership](idle-environment-cleanup.md).
|
||||
|
||||
## Object-backed run evidence
|
||||
|
||||
Optional segmented run logs and raw provider traces store their payloads and
|
||||
|
||||
@@ -11,7 +11,7 @@ or an orphaned active lease. Explicit user retry keeps its existing direct path.
|
||||
| Duty | Truth | Trigger and next deadline | Quiet condition |
|
||||
| --- | --- | --- | --- |
|
||||
| Failed teardown | Existing `environment_leases` rows in `pending_cleanup` | Commit signal; later of row backoff and saved retry/ownership deadline | No pending lease or deferred orphan |
|
||||
| Failed acquisition before a DB row exists | Existing local orphan spool and the producing runtime's buffer | Local buffer registration; five-minute retry while buffered; startup spool load | Durable row owns recovery and spool copy is removed |
|
||||
| Interrupted acquisition | `sandbox_creation_operations` committed before provider creation | Transaction signal; saved acquisition deadline; five-minute backoff after recovery failure | Normal lease or one claimed cleanup lease owns the resource; unsupported uncertainty requires operator reconciliation |
|
||||
| Historical provider verification | Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof | Commit signal; five-minute retry while an unverified candidate remains | No unverified candidates, including retained verified history |
|
||||
|
||||
`environment-cleanup-state.ts` owns the deadline query and the shared safe reader
|
||||
@@ -47,8 +47,9 @@ settlement and uncertainty handling through the shared DB implementation.
|
||||
- `heartbeat/recovery.ts` and `heartbeat/run-control.ts`: orphan-active recovery,
|
||||
retry settlement, and explicit recovery of historical leases.
|
||||
- `environment-task-runtime.ts`: unconfirmed plugin task termination and plugin
|
||||
replacement orphans, including a registered deferred flusher when the row
|
||||
cannot be written. The flusher retains an idle hold until the row is durable.
|
||||
replacement creations. Replacement intent commits before the provider call.
|
||||
- `sandbox-creation-operations.ts`: admission, atomic lease transfer, and bounded
|
||||
recovery. An unavailable database stops admission before external creation.
|
||||
- `sandbox-stop-and-retain.ts`: persisted Stop intent and its settlement.
|
||||
- `native-workspace-export-resume.ts` and `native-workspace-export-retry.ts`:
|
||||
terminal-export cleanup, resumed-export cleanup claims, and settlement.
|
||||
@@ -64,7 +65,49 @@ already pending claim remain covered by its existing deadline. Deletion removes
|
||||
work; an already scheduled reconciliation observes that removal. Independent SQL
|
||||
writers and another server process are not covered by these process-local signals.
|
||||
|
||||
## Deferred orphan buffers
|
||||
## Creation ownership
|
||||
|
||||
A creation operation retains company, original environment, run, request identity,
|
||||
provider, and nonsecret configuration references. Its environment FK uses SET NULL;
|
||||
company/run/task identifiers remain after those records are removed. One operation
|
||||
owns each company/environment/run/request. A retry of an attached operation returns
|
||||
its active lease. Pending or unresolved ownership never authorizes another create.
|
||||
Normal success transfers operation ownership and the lease in one transaction.
|
||||
Current company binding and run authority are checked again at attachment.
|
||||
Declared secret-reference fields retain only their references and nonsecret value
|
||||
hashes. Recovery refuses lookup under a changed credential value; it records manual
|
||||
reconciliation instead of treating a missing resource in a new account as proof.
|
||||
A missing, deleted, disabled, or archived saved secret, a missing or inactive saved version,
|
||||
or an explicitly disabled/coming-soon provider vault also records manual
|
||||
reconciliation before provider lookup. Admin-disabled credentials require operator
|
||||
reconciliation; recovery does not reactivate or use them. Temporary secret-provider
|
||||
or database failures retain the pending operation and its retry deadline.
|
||||
|
||||
Recovery never resumes partially prepared execution. It asks a capable provider to
|
||||
inspect the stable creation identity, validates its ownership envelope, and transfers
|
||||
one cleanup lease under the operation lock. The existing claimed cleanup worker
|
||||
owns deletion and confirms resolution. Failed lookup/transfer records a bounded
|
||||
retry and permits other operations to proceed. An attached operation cannot transfer
|
||||
to cleanup. Pending creation blocks instance sleep until its obligation transfers.
|
||||
|
||||
Daytona version 0.1.10 declares creationOperationVersion 1. It uses the committed
|
||||
operation UUID in its existing creation name and labels. Recovery uses an explicit
|
||||
creationOperation request on the existing read-only inspect hook and strict failed-creation deletion path. Normal lease
|
||||
inspection still uses its recorded resource ID; creationOperationId metadata is
|
||||
provenance only. A missing name alone cannot prove a late allocation will never appear. Missing identity or an
|
||||
unsupported provider records needs_reconciliation and an operation-scoped warning.
|
||||
A deleted company also records manual reconciliation before credential lookup or
|
||||
cleanup transfer. A company lock fences deletion during the transfer transaction.
|
||||
That manual diagnostic has no automatic timer and does not prevent idle. The row
|
||||
must be reconciled before retrying the same request. Other providers do not yet
|
||||
claim automatic creation lookup. Generic plugin operations retain identity only, including a returned task resource
|
||||
ID before expiry checks or lease attachment. A failed or ambiguous attachment keeps
|
||||
that known ID for reconciliation; it does not authorize unconditional release.
|
||||
These operations do not copy credentials, resolved configuration, or provider
|
||||
payloads into the operation record. A lost provider reply can still leave the
|
||||
resource ID unknown.
|
||||
|
||||
## Legacy deferred orphan buffers
|
||||
|
||||
Runtimes that hold an unwritten orphan register their flusher in a registry scoped
|
||||
to the shared database handle. Registration wakes the worker without contacting
|
||||
@@ -75,8 +118,9 @@ runtime. The registry retains only runtimes with unresolved buffer ownership.
|
||||
Unreadable directories, records, and malformed records retain a retry; a missing
|
||||
spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and
|
||||
persisted leases to clean up, then fails the sweep so recovery stays scheduled.
|
||||
Each concrete cleanup pass reloads the spool, including entries appended by
|
||||
plugin task replacement after startup and entries beyond buffer capacity.
|
||||
Each concrete cleanup pass reloads existing legacy spool entries, including entries
|
||||
beyond buffer capacity. Normal acquisition and plugin task replacement do not append
|
||||
new fallback files. Retained entries remain available for the separate cutover inventory.
|
||||
No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database,
|
||||
spool, and buffer-capacity failure remain; this is not a new durable queue.
|
||||
|
||||
|
||||
@@ -131,6 +131,13 @@ perform provider migration: `local_storage` and `provider_mismatch` are blocking
|
||||
results for cloud qualification, not instructions to switch configuration and retry.
|
||||
A directory of extracted caches is never a durable object backend.
|
||||
|
||||
The maintenance CLI reads the existing storage configuration directly so it can
|
||||
verify legacy local-disk objects before cutover. On Cloud it still requires the
|
||||
persisted runtime configuration and never seeds or changes that configuration.
|
||||
This maintenance boundary does not start a server or relax Cloud server startup
|
||||
requirements for external PostgreSQL, shared object storage and a stable secret
|
||||
key. Apply operations retain their existing stopped-writer gates.
|
||||
|
||||
Conversion establishes the rollback floor. Afterward only compatible version-reading
|
||||
binaries may overlap or be rolled back against the same remote state. Reattaching an
|
||||
old volume does not roll back committed content. Filesystem-authoritative binaries
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
CREATE TABLE IF NOT EXISTS "sandbox_creation_operations" (
|
||||
"id" uuid PRIMARY KEY NOT NULL,
|
||||
"company_id" uuid NOT NULL,
|
||||
"environment_id" uuid,
|
||||
"original_environment_id" uuid NOT NULL,
|
||||
"run_id" uuid NOT NULL,
|
||||
"request_key" text DEFAULT 'run' NOT NULL,
|
||||
"issue_id" uuid,
|
||||
"execution_workspace_id" uuid,
|
||||
"provider" text NOT NULL,
|
||||
"plugin_id" uuid,
|
||||
"driver_key" text NOT NULL,
|
||||
"state" text DEFAULT 'pending' NOT NULL,
|
||||
"config" jsonb NOT NULL,
|
||||
"metadata" jsonb NOT NULL,
|
||||
"lease_id" uuid,
|
||||
"next_attempt_at" timestamp with time zone NOT NULL,
|
||||
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
|
||||
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
DO $$ BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conname = 'sandbox_creation_operations_environment_id_environments_id_fk'
|
||||
AND conrelid = 'sandbox_creation_operations'::regclass
|
||||
) THEN
|
||||
ALTER TABLE "sandbox_creation_operations" ADD CONSTRAINT "sandbox_creation_operations_environment_id_environments_id_fk" FOREIGN KEY ("environment_id") REFERENCES "public"."environments"("id") ON DELETE set null ON UPDATE no action;
|
||||
END IF;
|
||||
END $$;
|
||||
--> statement-breakpoint
|
||||
CREATE INDEX IF NOT EXISTS "sandbox_creation_operations_pending_deadline_idx" ON "sandbox_creation_operations" USING btree ("next_attempt_at") WHERE "sandbox_creation_operations"."state" = 'pending';--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS "sandbox_creation_operations_owner_idx" ON "sandbox_creation_operations" USING btree ("company_id","original_environment_id","run_id","request_key");
|
||||
+539
-4
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "5690a14a-f9cc-41a6-af51-c27f688f4ef6",
|
||||
"prevId": "3b8bb508-028d-4154-abdd-af870f1f3ca6",
|
||||
"id": "c777fa84-85a6-418f-bea1-0d60b56cf071",
|
||||
"prevId": "c2c69a4a-8281-419c-9eeb-a7bcbc2beaae",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -1351,6 +1351,39 @@
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_entity_created_idx": {
|
||||
"name": "activity_log_company_entity_created_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_type",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "created_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
@@ -19019,6 +19052,98 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.company_skill_mutation_receipts": {
|
||||
"name": "company_skill_mutation_receipts",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"key": {
|
||||
"name": "key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"fingerprint": {
|
||||
"name": "fingerprint",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"skill_id": {
|
||||
"name": "skill_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"version_id": {
|
||||
"name": "version_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"response": {
|
||||
"name": "response",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"company_skill_mutation_receipts_key_idx": {
|
||||
"name": "company_skill_mutation_receipts_key_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "key",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"company_skill_mutation_receipts_company_id_companies_id_fk": {
|
||||
"name": "company_skill_mutation_receipts_company_id_companies_id_fk",
|
||||
"tableFrom": "company_skill_mutation_receipts",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.company_skill_repository_snapshots": {
|
||||
"name": "company_skill_repository_snapshots",
|
||||
"schema": "",
|
||||
@@ -19046,7 +19171,46 @@
|
||||
"name": "file_inventory",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
"notNull": true,
|
||||
"default": "'[]'::jsonb"
|
||||
},
|
||||
"storage": {
|
||||
"name": "storage",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"upload": {
|
||||
"name": "upload",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"restore": {
|
||||
"name": "restore",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"migration_at": {
|
||||
"name": "migration_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"default": "now()"
|
||||
},
|
||||
"maintenance_at": {
|
||||
"name": "maintenance_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"default": "now()"
|
||||
},
|
||||
"restore_retry_at": {
|
||||
"name": "restore_retry_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
@@ -19077,6 +19241,38 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"company_skill_snapshots_migration_idx": {
|
||||
"name": "company_skill_snapshots_migration_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "migration_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"company_skill_snapshots_maintenance_idx": {
|
||||
"name": "company_skill_snapshots_maintenance_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "maintenance_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -19966,6 +20162,25 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"package_snapshot_id": {
|
||||
"name": "package_snapshot_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"package_file_map": {
|
||||
"name": "package_file_map",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"migration_at": {
|
||||
"name": "migration_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"default": "now()"
|
||||
},
|
||||
"revision_number": {
|
||||
"name": "revision_number",
|
||||
"type": "integer",
|
||||
@@ -20024,6 +20239,52 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"company_skill_versions_migration_idx": {
|
||||
"name": "company_skill_versions_migration_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "migration_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"company_skill_versions\".\"migration_at\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"company_skill_versions_package_snapshot_idx": {
|
||||
"name": "company_skill_versions_package_snapshot_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "package_snapshot_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"company_skill_versions_repository_snapshot_idx": {
|
||||
"name": "company_skill_versions_repository_snapshot_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "repository_snapshot_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"company_skill_versions_skill_revision_idx": {
|
||||
"name": "company_skill_versions_skill_revision_idx",
|
||||
"columns": [
|
||||
@@ -20135,6 +20396,19 @@
|
||||
"onDelete": "restrict",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": {
|
||||
"name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk",
|
||||
"tableFrom": "company_skill_versions",
|
||||
"tableTo": "company_skill_repository_snapshots",
|
||||
"columnsFrom": [
|
||||
"package_snapshot_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "restrict",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"company_skill_versions_author_agent_id_agents_id_fk": {
|
||||
"name": "company_skill_versions_author_agent_id_agents_id_fk",
|
||||
"tableFrom": "company_skill_versions",
|
||||
@@ -20580,6 +20854,78 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.skill_package_deletions": {
|
||||
"name": "skill_package_deletions",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"provider": {
|
||||
"name": "provider",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"archive_key": {
|
||||
"name": "archive_key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"manifest_key": {
|
||||
"name": "manifest_key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"delete_after": {
|
||||
"name": "delete_after",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"skill_package_deletions_due_idx": {
|
||||
"name": "skill_package_deletions_due_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "delete_after",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.company_transfer_runs": {
|
||||
"name": "company_transfer_runs",
|
||||
"schema": "",
|
||||
@@ -33816,7 +34162,7 @@
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE",
|
||||
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
@@ -51228,6 +51574,195 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.sandbox_creation_operations": {
|
||||
"name": "sandbox_creation_operations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"environment_id": {
|
||||
"name": "environment_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"original_environment_id": {
|
||||
"name": "original_environment_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"request_key": {
|
||||
"name": "request_key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'run'"
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"execution_workspace_id": {
|
||||
"name": "execution_workspace_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"provider": {
|
||||
"name": "provider",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"plugin_id": {
|
||||
"name": "plugin_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"driver_key": {
|
||||
"name": "driver_key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"state": {
|
||||
"name": "state",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'pending'"
|
||||
},
|
||||
"config": {
|
||||
"name": "config",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"metadata": {
|
||||
"name": "metadata",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"lease_id": {
|
||||
"name": "lease_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"next_attempt_at": {
|
||||
"name": "next_attempt_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"sandbox_creation_operations_pending_deadline_idx": {
|
||||
"name": "sandbox_creation_operations_pending_deadline_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "next_attempt_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"sandbox_creation_operations\".\"state\" = 'pending'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"sandbox_creation_operations_owner_idx": {
|
||||
"name": "sandbox_creation_operations_owner_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "original_environment_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "request_key",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"sandbox_creation_operations_environment_id_environments_id_fk": {
|
||||
"name": "sandbox_creation_operations_environment_id_environments_id_fk",
|
||||
"tableFrom": "sandbox_creation_operations",
|
||||
"tableTo": "environments",
|
||||
"columnsFrom": [
|
||||
"environment_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.secret_access_events": {
|
||||
"name": "secret_access_events",
|
||||
"schema": "",
|
||||
+7
@@ -2913,6 +2913,13 @@
|
||||
"when": 1793371684483,
|
||||
"tag": "0417_chunky_red_hulk",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 418,
|
||||
"version": "7",
|
||||
"when": 1793395364937,
|
||||
"tag": "0418_curved_lady_mastermind",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -39,6 +39,7 @@ export {
|
||||
} from "./execution_workspace_restore_points.js";
|
||||
export { executionWorkspaceRuntimeLeases } from "./execution_workspace_runtime_leases.js";
|
||||
export { environments } from "./environments.js";
|
||||
export { sandboxCreationOperations } from "./sandbox_creation_operations.js";
|
||||
export { environmentLeases } from "./environment_leases.js";
|
||||
export { environmentCustomImageTemplates } from "./environment_custom_image_templates.js";
|
||||
export { environmentCustomImageSetupSessions } from "./environment_custom_image_setup_sessions.js";
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import { index, jsonb, pgTable, text, timestamp, uuid, uniqueIndex } from "drizzle-orm/pg-core";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { environments } from "./environments.js";
|
||||
|
||||
export const sandboxCreationOperations = pgTable("sandbox_creation_operations", {
|
||||
id: uuid("id").primaryKey(),
|
||||
companyId: uuid("company_id").notNull(),
|
||||
environmentId: uuid("environment_id").references(() => environments.id, { onDelete: "set null" }),
|
||||
originalEnvironmentId: uuid("original_environment_id").notNull(),
|
||||
runId: uuid("run_id").notNull(),
|
||||
requestKey: text("request_key").notNull().default("run"),
|
||||
issueId: uuid("issue_id"),
|
||||
executionWorkspaceId: uuid("execution_workspace_id"),
|
||||
provider: text("provider").notNull(),
|
||||
pluginId: uuid("plugin_id"),
|
||||
driverKey: text("driver_key").notNull(),
|
||||
state: text("state").notNull().default("pending"),
|
||||
config: jsonb("config").$type<Record<string, unknown>>().notNull(),
|
||||
metadata: jsonb("metadata").$type<Record<string, unknown>>().notNull(),
|
||||
leaseId: uuid("lease_id"),
|
||||
nextAttemptAt: timestamp("next_attempt_at", { withTimezone: true }).notNull(),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
}, table => ({
|
||||
pendingDeadline: index("sandbox_creation_operations_pending_deadline_idx").on(table.nextAttemptAt).where(sql`${table.state} = 'pending'`),
|
||||
owner: uniqueIndex("sandbox_creation_operations_owner_idx").on(table.companyId, table.originalEnvironmentId, table.runId, table.requestKey),
|
||||
}));
|
||||
@@ -63,3 +63,22 @@ second export after reconnect. The test creates and deletes one disposable
|
||||
- `src/manifest.ts` declares the sandbox-provider driver metadata
|
||||
- `src/plugin.ts` implements the environment lifecycle hooks
|
||||
- `paperclipPlugin.manifest` and `paperclipPlugin.worker` point the host at the built plugin entrypoints in `dist/`
|
||||
|
||||
|
||||
### Interrupted run-lease creation
|
||||
|
||||
The host commits a company-scoped creation operation before acquiring a new run
|
||||
lease. The operation UUID becomes `paperclip-create-<uuid>` and the
|
||||
`paperclip-create-attempt` label. The existing inspection hook can resolve that
|
||||
name and validate company, environment and run labels without starting a sandbox.
|
||||
Recovery persists the observed provider ID before the existing cleanup worker
|
||||
deletes it. A missing creation name is an uncertain outcome, not proof of removal.
|
||||
The host retains that operation for manual reconciliation and never repeats the
|
||||
creation request blindly. Credential identity must still match admission.
|
||||
|
||||
This capability is declared as `creationOperationVersion: 1` in manifest version
|
||||
0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement.
|
||||
Legacy provider `environmentProbe` and custom-image interactive setup have separate
|
||||
lifecycles and are outside this lease-orphan fallback replacement. The normal Test
|
||||
path which acquires a runtime lease is covered. Unit fixtures exercise lookup and
|
||||
deletion; live provider qualification requires an explicit disposable test account.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@paperclipai/plugin-daytona",
|
||||
"version": "0.1.9",
|
||||
"version": "0.1.10",
|
||||
"description": "Daytona sandbox provider plugin for Paperclip environments",
|
||||
"license": "MIT",
|
||||
"homepage": "https://github.com/paperclipai/paperclip",
|
||||
|
||||
@@ -15,7 +15,7 @@ export const DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS = 300_000;
|
||||
// 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner.
|
||||
// 0.1.7 exposes host-owned warm/cold runner lifecycle controls.
|
||||
// 0.1.8 declares the default provider acquisition budget to the host.
|
||||
const PLUGIN_VERSION = "0.1.9";
|
||||
const PLUGIN_VERSION = "0.1.10";
|
||||
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
id: PLUGIN_ID,
|
||||
@@ -39,6 +39,7 @@ const manifest: PaperclipPluginManifestV1 = {
|
||||
description:
|
||||
"Provisions Daytona sandboxes with configurable image or snapshot selection, startup timeouts, and lease reuse.",
|
||||
supportsReusableLeases: true,
|
||||
creationOperationVersion: 1,
|
||||
// Daytona keeps a persistent session and tails its callback log form, so it
|
||||
// emits incremental session output while the command runs. Declare the
|
||||
// opt-in capability so the host selects the session-output streaming path.
|
||||
|
||||
@@ -51,6 +51,16 @@ describe("Daytona read-only resource inspection", () => {
|
||||
for (const method of [sandbox.start, sandbox.stop, sandbox.delete, sandbox.archive, sandbox.refreshData]) expect(method).not.toHaveBeenCalled();
|
||||
expect(mockCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
it.each(["active", "destroyed"])("inspects a normal lease with creation provenance by recorded ID: %s", async state => {
|
||||
const provenance = { creationOperationId: "16cd534e-27a1-45de-a4ad-5adbb6481361" };
|
||||
if (state === "active") mockGet.mockResolvedValue(createMockSandbox());
|
||||
else mockGet.mockRejectedValue(new MockDaytonaNotFoundError("gone"));
|
||||
expect(await plugin.definition.onEnvironmentInspectLease!({ ...params, leaseMetadata: provenance }))
|
||||
.toEqual({ providerLeaseId: "sandbox-123", state });
|
||||
expect(mockGet).toHaveBeenCalledExactlyOnceWith("sandbox-123");
|
||||
expect(mockCreate).not.toHaveBeenCalled();
|
||||
if (state === "destroyed") expect(await plugin.definition.onIdleDrain!(new AbortController().signal)).toBe("none");
|
||||
});
|
||||
it("accepts only typed provider not-found as deletion proof", async () => {
|
||||
__resetDaytonaSandboxHandleCacheForTest();
|
||||
mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not found"));
|
||||
@@ -80,6 +90,46 @@ describe("Daytona read-only resource inspection", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("Daytona durable creation identity", () => {
|
||||
const id = "16cd534e-27a1-45de-a4ad-5adbb6481361";
|
||||
const runId = "8cbd0194-e322-4b5b-9800-df4d5a682479";
|
||||
const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
|
||||
providerLeaseId: id, config: { apiKey: "fixture-not-a-key" },
|
||||
creationOperation: { id, runId } };
|
||||
beforeEach(() => { mockCreate.mockReset(); mockGet.mockReset(); __resetDaytonaSandboxHandleCacheForTest(); });
|
||||
afterEach(() => { vi.unstubAllEnvs(); __resetDaytonaSandboxHandleCacheForTest(); });
|
||||
function resource() { return { ...createMockSandbox({ name: `paperclip-create-${id}` }), labels: {
|
||||
"paperclip-provider": "daytona", "paperclip-create-attempt": id,
|
||||
"paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId,
|
||||
"paperclip-run-id": runId,
|
||||
} }; }
|
||||
it("locates the exact resource by stable name without creation or lifecycle calls", async () => {
|
||||
const sandbox = resource(); mockGet.mockResolvedValue(sandbox);
|
||||
const observed = await plugin.definition.onEnvironmentInspectLease!(params);
|
||||
expect(mockGet).toHaveBeenCalledWith(`paperclip-create-${id}`);
|
||||
expect(observed).toMatchObject({ providerLeaseId: `paperclip-create-${id}`, state: "active",
|
||||
creationCleanup: { observedProviderLeaseId: sandbox.id, attemptId: id, runId } });
|
||||
expect(mockCreate).not.toHaveBeenCalled(); expect(sandbox.delete).not.toHaveBeenCalled();
|
||||
await plugin.definition.onEnvironmentDestroyLease!({ ...params, providerLeaseId: observed.providerLeaseId,
|
||||
leaseMetadata: { failedCreateCleanup: observed.creationCleanup } });
|
||||
expect(sandbox.delete).toHaveBeenCalledOnce();
|
||||
});
|
||||
it("retains uncertainty when a creation name is absent", async () => {
|
||||
mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not yet visible"));
|
||||
expect(await plugin.definition.onEnvironmentInspectLease!(params)).toEqual({ providerLeaseId: `paperclip-create-${id}`, state: "unknown" });
|
||||
expect(mockCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
it("rejects a name or label owned by another creation", async () => {
|
||||
mockGet.mockResolvedValue({ ...resource(), labels: { "paperclip-company-id": "foreign" } });
|
||||
await expect(plugin.definition.onEnvironmentInspectLease!(params)).rejects.toThrow("ownership does not match");
|
||||
});
|
||||
it("requires the recorded endpoint when ambient configuration changes", async () => {
|
||||
vi.stubEnv("DAYTONA_API_URL", "https://changed.example.test");
|
||||
expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("unknown");
|
||||
expect(mockGet).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("Daytona idle eligibility", () => {
|
||||
const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
|
||||
config: { apiKey: "fixture-not-a-key", reuseLease: true } };
|
||||
@@ -537,7 +587,7 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
// The bundled-plugin boot reconcile refreshes the stored manifest for an
|
||||
// existing install only when the version changes. The acquisition budget needs
|
||||
// the bump to reach an existing install.
|
||||
expect(manifest.version).toBe("0.1.9");
|
||||
expect(manifest.version).toBe("0.1.10");
|
||||
});
|
||||
|
||||
it.each([false, true])("closes duplex routes on lease release even when bridge drain hangs: %s", async (hangDrain) => {
|
||||
|
||||
@@ -994,7 +994,8 @@ async function createSandbox(
|
||||
}
|
||||
// The SDK can create a sandbox and then throw while waiting for it to start.
|
||||
// Keep an attempt-specific provider name so that failure cannot lose ownership.
|
||||
const attemptId = randomUUID();
|
||||
const attemptId = "creationOperationId" in params && params.creationOperationId ? params.creationOperationId : randomUUID();
|
||||
if (!/^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(attemptId)) throw new Error("Invalid creation operation identity");
|
||||
const name = `paperclip-create-${attemptId}`;
|
||||
const labels = { ...buildSandboxLabels({
|
||||
companyId: params.companyId,
|
||||
@@ -2988,6 +2989,32 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
if (!config.apiUrl && (process.env.DAYTONA_API_URL?.trim() || process.env.DAYTONA_SERVER_URL?.trim())) {
|
||||
return { providerLeaseId: params.providerLeaseId, state: "unknown" };
|
||||
}
|
||||
if (params.creationOperation !== undefined) {
|
||||
const { id: attemptId, runId } = params.creationOperation;
|
||||
if (![attemptId, runId].every(value => typeof value === "string" && /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(value))) {
|
||||
throw new Error("Invalid creation operation identity");
|
||||
}
|
||||
const name = `paperclip-create-${attemptId}`;
|
||||
try {
|
||||
const sandbox = await withLivenessTimeout("sandbox.creationLookup", 10_000, () => createDaytonaClient(config).get(name));
|
||||
const labels = {
|
||||
"paperclip-provider": "daytona", "paperclip-create-attempt": attemptId,
|
||||
"paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId,
|
||||
"paperclip-run-id": runId,
|
||||
};
|
||||
if (sandbox.name !== name || Object.entries(labels).some(([key, value]) => sandbox.labels?.[key] !== value)) {
|
||||
throw new Error("Creation operation ownership does not match");
|
||||
}
|
||||
return { providerLeaseId: name, state: "active", creationCleanup: {
|
||||
providerLeaseId: name, observedProviderLeaseId: sandbox.id, companyId: params.companyId,
|
||||
environmentId: params.environmentId, runId: labels["paperclip-run-id"], attemptId,
|
||||
accountFingerprint: sandboxAccountDiscriminator(config), labels,
|
||||
} };
|
||||
} catch (error) {
|
||||
if (error instanceof DaytonaNotFoundError) return { providerLeaseId: name, state: "unknown" };
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
const proof = idleActivity.proof(providerActivityKey(params, config, params.providerLeaseId));
|
||||
// Read directly from the provider. The setup-status path and cached lease
|
||||
// reconnect path can start a stopped sandbox, so neither is an observation.
|
||||
|
||||
@@ -641,6 +641,7 @@ export interface PluginEnvironmentRunnerIngressEndpointParams
|
||||
|
||||
export interface PluginEnvironmentAcquireLeaseParams extends PluginEnvironmentDriverBaseParams {
|
||||
runId: string;
|
||||
creationOperationId?: string;
|
||||
/** Explicit platform probes may request a provider-owned immutable snapshot
|
||||
* identity. Supporting providers create by that ID and return snapshotIdentity
|
||||
* in lease metadata. Older providers may omit it; the host then cannot qualify
|
||||
@@ -701,11 +702,13 @@ export interface PluginEnvironmentDestroyLeaseParams extends PluginEnvironmentRe
|
||||
/** Observe this exact allocation without starting, stopping or deleting it.
|
||||
* Missing resources must be distinguished from permission/transport failures. */
|
||||
export interface PluginEnvironmentInspectLeaseParams extends PluginEnvironmentDriverBaseParams {
|
||||
creationOperation?: { id: string; runId: string };
|
||||
providerLeaseId: string;
|
||||
leaseMetadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface PluginEnvironmentLeaseObservation {
|
||||
creationCleanup?: import("./environment-creation-cleanup.js").PluginEnvironmentCreationCleanup;
|
||||
providerLeaseId: string;
|
||||
state: "active" | "stopped" | "destroyed" | "unknown";
|
||||
}
|
||||
|
||||
@@ -198,6 +198,7 @@ export interface PluginEnvironmentDriverDeclaration {
|
||||
/** Provides Runner execution, resource preparation, and PRP connections via environmentTask. */
|
||||
supportsTasks?: boolean;
|
||||
freshWorkspaceVersion?: 1;
|
||||
creationOperationVersion?: 1;
|
||||
/** Codex task retention and authorized run attachment, version 1. */
|
||||
warmTaskVersion?: 1;
|
||||
/** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */
|
||||
|
||||
@@ -181,6 +181,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({
|
||||
requiresAgent: z.boolean().optional(),
|
||||
supportsTasks: z.boolean().optional(),
|
||||
freshWorkspaceVersion: z.literal(1).optional(),
|
||||
creationOperationVersion: z.literal(1).optional(),
|
||||
warmTaskVersion: z.literal(1).optional(),
|
||||
driverKey: z.string().min(1).regex(
|
||||
/^[a-z0-9][a-z0-9._-]*$/,
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
/** Run from the server package with its configured storage and DATABASE_URL. */
|
||||
import { createDb } from "@paperclipai/db";
|
||||
import { loadStorageConfig } from "../src/storage/config.js";
|
||||
import { createStorageProviderFromConfig } from "../src/storage/provider-registry.js";
|
||||
import { createStorageService } from "../src/storage/service.js";
|
||||
import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js";
|
||||
import { skillInstallationMaintenance } from "../src/services/skill-installation-maintenance.js";
|
||||
import { isCloudManagedInstance } from "../src/services/cloud-instance.js";
|
||||
@@ -19,8 +22,11 @@ try {
|
||||
if (!current) throw new Error("Initialize Cloud runtime configuration before running skill package maintenance");
|
||||
setCloudRuntimeConfigSnapshot(current);
|
||||
}
|
||||
const service = skillPackageMaintenance(db);
|
||||
const installations = skillInstallationMaintenance(db);
|
||||
// Maintenance must inspect the old provider before Cloud cutover. Server
|
||||
// startup prerequisites still apply to the normal global storage service.
|
||||
const storage = createStorageService(createStorageProviderFromConfig(loadStorageConfig()));
|
||||
const service = skillPackageMaintenance(db, storage);
|
||||
const installations = skillInstallationMaintenance(db, storage);
|
||||
const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions,
|
||||
orphans: service.collectOrphans, deletions: service.deleteObjects,
|
||||
installations: installations.installations, eligibility: installations.eligibility, verification: installations.verifyVersions };
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
import { createOwnedSandbox } from "../services/sandbox-creation-operations.js";
|
||||
import { mkdtemp, writeFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
@@ -7,7 +9,7 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { companies, createDb, environmentLeases, environments, type Db } from "@paperclipai/db";
|
||||
import { companies, createDb, sandboxCreationOperations, environmentLeases, environments, type Db } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { registerEnvironmentCleanupWork } from "../services/environment-cleanup-work.js";
|
||||
@@ -34,7 +36,7 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => {
|
||||
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
|
||||
}, 90_000);
|
||||
beforeEach(async () => {
|
||||
await db.execute(sql`TRUNCATE companies, environments CASCADE`);
|
||||
await db.execute(sql`TRUNCATE sandbox_creation_operations, companies, environments CASCADE`);
|
||||
companyId = randomUUID(); environmentId = randomUUID(); draining = false; failing = false; errors.length = 0;
|
||||
await db.insert(companies).values({ id: companyId, name: "Cleanup worker", issuePrefix: "CLN" });
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
@@ -99,6 +101,64 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => {
|
||||
await settle(180_000); expect(inspections).toHaveBeenCalledTimes(3);
|
||||
await quiet();
|
||||
});
|
||||
it("reconstructs a creation deadline after restart and leaves manual reconciliation quiet", async () => {
|
||||
await start(); await quiet();
|
||||
await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
|
||||
async () => ({ providerLeaseId: "fixture-unattached" }));
|
||||
await settle();
|
||||
expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now());
|
||||
worker!.stop(); await coordinator!.stop();
|
||||
await start();
|
||||
expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now());
|
||||
await settle(31 * 60_000);
|
||||
await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ state: "needs_reconciliation" }),
|
||||
]));
|
||||
await quiet();
|
||||
});
|
||||
it("settles a deleted-company creation as manual history across restart without idle polling", async () => {
|
||||
await start(); await quiet();
|
||||
await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
|
||||
async () => ({ providerLeaseId: "fixture-deleted-company" }));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await settle(31 * 60_000);
|
||||
await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }),
|
||||
]));
|
||||
await quiet();
|
||||
worker!.stop(); await coordinator!.stop();
|
||||
await start(); await quiet();
|
||||
expect(releases).not.toHaveBeenCalled(); expect(inspections).not.toHaveBeenCalled();
|
||||
const ownerId = randomUUID();
|
||||
const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 };
|
||||
expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none"))
|
||||
.toEqual({ version: 1, backgroundWork: "none" });
|
||||
await quiet();
|
||||
});
|
||||
it("verifies ordinary cleanup carrying creation provenance and stops provider polling", async () => {
|
||||
const lease = await pending();
|
||||
await environmentService(db).updateLeaseMetadata(lease.id, { driver: "sandbox", creationOperationId: randomUUID() });
|
||||
await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" });
|
||||
await start(true);
|
||||
expect(inspections).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ providerLeaseId: lease.providerLeaseId,
|
||||
metadata: expect.objectContaining({ creationOperationId: expect.any(String) }) }));
|
||||
await quiet();
|
||||
worker!.stop(); await coordinator!.stop();
|
||||
await start(true); expect(inspections).toHaveBeenCalledTimes(1); await quiet();
|
||||
});
|
||||
it("preserves a first creation signal during drain and disarms after atomic attachment", async () => {
|
||||
await start(); await quiet(); draining = true;
|
||||
const created = await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
|
||||
async () => ({ providerLeaseId: "fixture-attached" }));
|
||||
const count = reads.mock.calls.length;
|
||||
await settle(60_000, false); expect(reads).toHaveBeenCalledTimes(count);
|
||||
draining = false;
|
||||
await environmentService(db).acquireLease({ companyId, environmentId, provider: "fixture", providerLeaseId: created.providerLeaseId,
|
||||
creationOperationId: String(created.metadata!.creationOperationId), metadata: created.metadata });
|
||||
await settle();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "attached" })]);
|
||||
await quiet();
|
||||
});
|
||||
it("wakes on a real orphan insert, waits for its backoff, then disarms after deletion", async () => {
|
||||
await start(); await quiet();
|
||||
const lease = await pending(); await settle();
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,34 @@
|
||||
import { createDb } from "@paperclipai/db";
|
||||
import { environmentRuntimeService } from "../../services/environment-runtime.js";
|
||||
import { environmentService } from "../../services/environments.js";
|
||||
import type { PluginWorkerManager } from "../../services/plugin-worker-manager.js";
|
||||
|
||||
const db = createDb(process.env.CREATION_FIXTURE_DATABASE_URL!);
|
||||
const workers = {
|
||||
isRunning: () => true,
|
||||
getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }),
|
||||
call: async (_pluginId: string, method: string, input: unknown) => {
|
||||
const response = await fetch(process.env.CREATION_FIXTURE_PROVIDER_URL!, {
|
||||
method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ method, input }),
|
||||
});
|
||||
if (!response.ok) throw new Error("Controlled provider unavailable");
|
||||
return response.json();
|
||||
},
|
||||
} as unknown as PluginWorkerManager;
|
||||
try {
|
||||
const runtime = environmentRuntimeService(db, {
|
||||
pluginWorkerManager: workers, orphanCleanupSpoolDir: process.env.CREATION_FIXTURE_EMPTY_DISK,
|
||||
});
|
||||
if (process.env.CREATION_FIXTURE_ACTION === "acquire") {
|
||||
await runtime.acquireRunLease({ companyId: process.env.CREATION_FIXTURE_COMPANY_ID!,
|
||||
environment: (await environmentService(db).getById(process.env.CREATION_FIXTURE_ENVIRONMENT_ID!))!,
|
||||
heartbeatRunId: process.env.CREATION_FIXTURE_RUN_ID!, issueId: null, persistedExecutionWorkspace: null });
|
||||
} else {
|
||||
await runtime.flushDeferredOrphanCleanups();
|
||||
}
|
||||
process.stdout.write(JSON.stringify({ ok: true }));
|
||||
} catch {
|
||||
process.stdout.write(JSON.stringify({ ok: false }));
|
||||
} finally {
|
||||
await db.$client.end({ timeout: 1 });
|
||||
}
|
||||
@@ -278,6 +278,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
|
||||
}, { timeout: 5_000 });
|
||||
|
||||
expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", {
|
||||
creationOperationId: expect.any(String),
|
||||
driverKey: "sandbox",
|
||||
companyId,
|
||||
environmentId,
|
||||
@@ -745,6 +746,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
|
||||
}, { timeout: 5_000 });
|
||||
|
||||
expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", {
|
||||
creationOperationId: expect.any(String),
|
||||
driverKey: "sandbox",
|
||||
companyId,
|
||||
environmentId: newEnvironmentId,
|
||||
|
||||
@@ -77,7 +77,12 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
|
||||
.toMatchObject({ content: "complete root guide" });
|
||||
});
|
||||
|
||||
it("loads persisted Cloud configuration for read-only maintenance without seeding it", async () => {
|
||||
it.each([
|
||||
{ storage: "legacy local", env: { PAPERCLIP_STORAGE_PROVIDER: "local_disk" }, expectedStatus: "local_storage" },
|
||||
{ storage: "shared S3", env: { PAPERCLIP_STORAGE_PROVIDER: "s3",
|
||||
PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture", PAPERCLIP_STORAGE_S3_REGION: "us-east-1",
|
||||
PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" }, expectedStatus: "provider_mismatch" },
|
||||
])("loads persisted Cloud configuration for read-only $storage maintenance without seeding it", async ({ env, expectedStatus }) => {
|
||||
const companyId = await company();
|
||||
const skill = await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
|
||||
const local = await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 });
|
||||
@@ -89,8 +94,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
|
||||
`--phase=${phase}`, `--company=${companyId}`, "--limit=50",
|
||||
], { env: { ...process.env, DATABASE_URL: database.connectionString,
|
||||
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture",
|
||||
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture",
|
||||
PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" },
|
||||
PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects"), ...env },
|
||||
timeout: 60_000, maxBuffer: 1024 * 1024 });
|
||||
await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration");
|
||||
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0);
|
||||
@@ -99,7 +103,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
|
||||
const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!);
|
||||
expect(eligibility.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "version_backed" })]);
|
||||
const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!);
|
||||
expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "provider_mismatch" })]);
|
||||
expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: expectedStatus })]);
|
||||
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toEqual([persisted]);
|
||||
} finally {
|
||||
await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key));
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import { flushDeferredEnvironmentCleanup, hasDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, subscribeDeferredEnvironmentCleanup } from "../services/environment-cleanup-notifications.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { mkdtemp, readdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects, sandboxCreationOperations } from "@paperclipai/db";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import * as orphanCleanup from "../services/sandbox-orphan-cleanup-spool.js";
|
||||
import { recordSandboxCreationResource } from "../services/sandbox-creation-operations.js";
|
||||
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
@@ -51,7 +53,7 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
|
||||
if(p.operation.kind==="describe") return {kind:"description",taskId:p.taskId,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`},warmTaskVersion:1};
|
||||
return {kind:"accepted",taskId:p.taskId};
|
||||
}} as unknown as PluginWorkerManager;
|
||||
const runtime=()=>environmentRuntimeService(db,{pluginWorkerManager:workers});
|
||||
const runtime=(orphanCleanupSpoolDir?:string)=>environmentRuntimeService(db,{pluginWorkerManager:workers,orphanCleanupSpoolDir});
|
||||
const acquire=()=>runtime().acquireRunLease({companyId,environment:environment as never,issueId,agentId,heartbeatRunId:runId,persistedExecutionWorkspace:null,adapterType:"paperclip_runner",runnerTaskProjectIds:projectIds,requestedExpiresAt:deadline});
|
||||
const first=await acquire();
|
||||
const identity={runnerInstanceId:"runner",environmentLeaseId:first.lease.id,runId,normalizedSessionId:"session",turnId:"turn",itemId:"item"};
|
||||
@@ -126,58 +128,64 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
|
||||
await expect(resolvePluginTaskResume(db,{companyId:f.companyId,leaseId:next.lease.id,priorIdentity:f.identity})).resolves.toBe(f.first.lease.providerLeaseId);
|
||||
});
|
||||
|
||||
it.each(["ownership-lost","database-error","cleanup-error","spool"] as const)("compensates an unowned replacement task: %s",async mode=>{
|
||||
it.each(["ownership-lost","database-error"] as const)("retains a durable operation for an unattached replacement task: %s",async mode=>{
|
||||
const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped();
|
||||
if(mode==="cleanup-error" || mode==="spool") f.failRelease();
|
||||
const append=vi.fn(async()=>true);
|
||||
let delayFlush=false,flushCalls=0,releaseFlush!:()=>void;
|
||||
const flushGate=new Promise<void>(resolve=>{releaseFlush=resolve;});
|
||||
const stopGate=mode==="spool" ? registerDeferredEnvironmentCleanup(db,async()=>{
|
||||
if(delayFlush && ++flushCalls===1) await flushGate;
|
||||
return {recovered:0,pending:0};
|
||||
}) : undefined;
|
||||
const wake=vi.fn();
|
||||
const unsubscribe=subscribeDeferredEnvironmentCleanup(db,wake);
|
||||
const spool=mode==="spool" ? vi.spyOn(orphanCleanup,"createSandboxOrphanCleanupSpool").mockReturnValue({append,load:async()=>[],remove:async()=>{}}) : undefined;
|
||||
let replacementFailed=false;
|
||||
const transact=db.transaction.bind(db);
|
||||
const transaction=mode==="spool" ? vi.spyOn(db,"transaction").mockImplementation((...args)=>replacementFailed
|
||||
? Promise.reject(Error("database unavailable")) : transact(...args)) : undefined;
|
||||
const original=db.update.bind(db);
|
||||
const update=vi.spyOn(db,"update").mockImplementation(((table:any)=>{
|
||||
const query=original(table),set=query.set.bind(query);
|
||||
query.set=((values:any)=>values.providerLeaseId === "task-2"
|
||||
? {where:()=>({returning:async()=>{replacementFailed=true;if(mode!=="ownership-lost") throw Error("replacement write failed");return [];}})}
|
||||
: set(values)) as never;
|
||||
return query;
|
||||
}) as never);
|
||||
const disk=await mkdtemp(join(tmpdir(),"replacement-intent-"));
|
||||
const runtime=()=>f.runtime(disk);
|
||||
await db.execute(sql.raw(`create or replace function reject_task_replacement() returns trigger language plpgsql as $$ begin
|
||||
if new.provider_lease_id = 'task-2' then ${mode === "ownership-lost" ? "return null;" : "raise exception 'replacement write failed';"} end if; return new; end $$`));
|
||||
await db.execute(sql`create trigger reject_task_replacement before update on environment_leases for each row execute function reject_task_replacement()`);
|
||||
try {
|
||||
await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}}))
|
||||
.rejects.toThrow("reconcile the same task");
|
||||
if(mode==="spool") {
|
||||
expect(wake).toHaveBeenCalledOnce();
|
||||
expect(hasDeferredEnvironmentCleanup(db)).toBe(true);
|
||||
await expect(flushDeferredEnvironmentCleanup(db)).rejects.toThrow("Deferred orphan cleanup failed");
|
||||
expect(hasDeferredEnvironmentCleanup(db)).toBe(true);
|
||||
}
|
||||
} finally {update.mockRestore();transaction?.mockRestore();spool?.mockRestore();unsubscribe();}
|
||||
expect(f.calls).toContain("environmentReleaseLease");
|
||||
if(mode==="spool") expect(append).toHaveBeenCalledWith(expect.objectContaining({providerLeaseId:"task-2",metadata:expect.objectContaining({driver:"plugin",runnerTaskDispatched:false})}));
|
||||
const rows=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId));
|
||||
expect(rows.filter(row=>row.providerLeaseId === "task-2")).toEqual(mode==="cleanup-error"
|
||||
? [expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned",metadata:expect.objectContaining({runnerTaskDispatched:false})})] : []);
|
||||
if(mode==="spool") {
|
||||
delayFlush=true;
|
||||
const delayed=flushDeferredEnvironmentCleanup(db);
|
||||
try { expect(await flushDeferredEnvironmentCleanup(db)).toEqual({recovered:1,pending:0}); }
|
||||
finally { releaseFlush(); stopGate!(); }
|
||||
expect(await delayed).toEqual({recovered:1,pending:0});
|
||||
expect(hasDeferredEnvironmentCleanup(db)).toBe(false);
|
||||
const recovered=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId));
|
||||
expect(recovered.filter(row=>row.providerLeaseId==="task-2")).toEqual([
|
||||
expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned"}),
|
||||
]);
|
||||
}
|
||||
await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
|
||||
} finally { await db.execute(sql`drop trigger reject_task_replacement on environment_leases`); }
|
||||
expect(f.calls).not.toContain("environmentReleaseLease");
|
||||
const [lease]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));
|
||||
expect(lease.providerLeaseId).toBe(next.lease.providerLeaseId);
|
||||
const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
|
||||
const pending=operations.filter(row=>row.state === "pending");
|
||||
expect(pending).toHaveLength(1);expect(pending[0].requestKey).toBe(next.lease.providerLeaseId);
|
||||
expect(pending[0].metadata.creationProviderLeaseId).toBe("task-2");expect(pending[0].config).toEqual({});
|
||||
expect(f.acquireRequests.at(-1)?.creationOperationId).toBe(pending[0].id);
|
||||
const creates=f.acquireRequests.length;
|
||||
await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
|
||||
expect(f.acquireRequests).toHaveLength(creates);
|
||||
await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,pending[0].id));
|
||||
await runtime().flushDeferredOrphanCleanups();
|
||||
const [manual]=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,pending[0].id));
|
||||
expect(manual.state).toBe("needs_reconciliation");expect(manual.metadata.creationProviderLeaseId).toBe("task-2");
|
||||
await runtime().flushDeferredOrphanCleanups();expect(f.acquireRequests).toHaveLength(creates);
|
||||
expect(f.calls).not.toContain("environmentReleaseLease");
|
||||
expect(await readdir(disk)).toEqual([]);
|
||||
await rm(disk,{recursive:true,force:true});
|
||||
});
|
||||
|
||||
it("retains the known replacement identity after a committed attachment response is lost",async()=>{
|
||||
const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped();
|
||||
const transaction=db.transaction.bind(db);let lost=false;
|
||||
const response=vi.spyOn(db,"transaction").mockImplementation((async(...args:any[])=>{
|
||||
const result=await (transaction as any)(...args);
|
||||
if(!lost && result?.providerLeaseId === "task-2") {lost=true;throw new Error("Attachment commit response lost");}
|
||||
return result;
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
|
||||
} finally {response.mockRestore();}
|
||||
expect(lost).toBe(true);
|
||||
const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
|
||||
const replacement=operations.find(operation=>operation.requestKey === next.lease.providerLeaseId)!;
|
||||
expect(replacement).toMatchObject({state:"attached",leaseId:next.lease.id,metadata:{creationProviderLeaseId:"task-2"}});
|
||||
const identity={id:replacement.id,companyId:f.companyId,environmentId:replacement.originalEnvironmentId,providerLeaseId:"task-2"};
|
||||
await recordSandboxCreationResource(db,identity);
|
||||
await expect(recordSandboxCreationResource(db,{...identity,providerLeaseId:"different-resource"})).rejects.toThrow("identity changed");
|
||||
await expect(recordSandboxCreationResource(db,{...identity,companyId:randomUUID()})).rejects.toThrow("identity changed");
|
||||
expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,replacement.id)))[0])
|
||||
.toMatchObject({state:"attached",metadata:{creationProviderLeaseId:"task-2"}});
|
||||
const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));
|
||||
expect(saved).toMatchObject({providerLeaseId:"task-2",status:"active"});
|
||||
await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,replacement.id));
|
||||
await f.runtime().flushDeferredOrphanCleanups();
|
||||
expect((await f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).taskId).toBe("task-2");
|
||||
expect(f.acquireRequests).toHaveLength(2);expect(f.calls).not.toContain("environmentReleaseLease");
|
||||
});
|
||||
|
||||
it.each(["added","removed"] as const)("retires retained mounts when an authorized project is %s after controller restart",async mode=>{
|
||||
@@ -203,7 +211,9 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
|
||||
const replacing=f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}});
|
||||
if(["violating","missing-bounded","invalid","expired"].includes(mode)) {
|
||||
await expect(replacing).rejects.toThrow("reconcile the same task");
|
||||
expect(f.calls).toContain("environmentReleaseLease");
|
||||
expect(f.calls).not.toContain("environmentReleaseLease");
|
||||
const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
|
||||
expect(operations.filter(row=>row.state === "pending")).toHaveLength(1);
|
||||
} else {
|
||||
const description=await replacing;
|
||||
const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));
|
||||
|
||||
@@ -0,0 +1,333 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createServer } from "node:http";
|
||||
import { promisify } from "node:util";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { agents, builtInManagedResources, companies, companySecretVersions, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { HttpError } from "../errors.js";
|
||||
import * as secretOwner from "../services/secrets.js";
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
|
||||
describe("durable sandbox creation through the runtime entry point", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let companyId: string, environmentId: string, pluginId: string, runId: string, spool: string;
|
||||
let creates: number, destroys: number, loseReply: boolean, failDestroy: boolean;
|
||||
let beforeReply: (() => Promise<void>) | undefined;
|
||||
const resources = new Map<string, string>();
|
||||
let workers: PluginWorkerManager;
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("sandbox-creation-operations-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
beforeEach(async () => {
|
||||
await db.execute(sql`truncate sandbox_creation_operations, companies, environments, plugins cascade`);
|
||||
companyId = randomUUID(); environmentId = randomUUID(); pluginId = randomUUID(); runId = randomUUID();
|
||||
const agentId = randomUUID(); creates = 0; destroys = 0; loseReply = false; failDestroy = false; beforeReply = undefined; resources.clear();
|
||||
spool = await mkdtemp(join(tmpdir(), "empty-sandbox-web-"));
|
||||
await db.insert(companies).values({ id: companyId, name: "Creation fixture", issuePrefix: "CRT" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "codex_local" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", invocationSource: "manual" });
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox", config: { provider: "creation-fixture", image: "fixture", reuseLease: false } });
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: "paperclip.creation-fixture", packageName: "creation-fixture", version: "1.0.0", apiVersion: 1,
|
||||
status: "ready", categories: ["automation"], manifestJson: { id: "paperclip.creation-fixture", version: "1.0.0", apiVersion: 1,
|
||||
displayName: "Creation fixture", description: "Controlled remote provider", author: "Paperclip", categories: ["automation"],
|
||||
capabilities: ["environment.drivers.register"], entrypoints: { worker: "fixture.js" }, environmentDrivers: [{
|
||||
driverKey: "creation-fixture", kind: "sandbox_provider", displayName: "Fixture", creationOperationVersion: 1, configSchema: { type: "object" },
|
||||
}] } });
|
||||
workers = {
|
||||
isRunning: () => true,
|
||||
getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }),
|
||||
call: vi.fn(async (_pluginId: string, method: string, input: any) => {
|
||||
if (method === "environmentAcquireLease") {
|
||||
expect(await db.select().from(sandboxCreationOperations)).toContainEqual(expect.objectContaining({ id: input.creationOperationId, state: "pending" }));
|
||||
creates++;
|
||||
const resource = `provider-${input.creationOperationId}`;
|
||||
resources.set(input.creationOperationId, resource);
|
||||
await beforeReply?.();
|
||||
if (loseReply) throw new Error("Lost provider reply");
|
||||
return { providerLeaseId: resource, metadata: { remoteCwd: "/workspace" } };
|
||||
}
|
||||
if (method === "environmentInspectLease") {
|
||||
const id = input.creationOperation.id;
|
||||
const resource = resources.get(id);
|
||||
if (!resource) return { providerLeaseId: id, state: "unknown" };
|
||||
return { providerLeaseId: resource, state: "active", creationCleanup: { providerLeaseId: resource,
|
||||
observedProviderLeaseId: resource, companyId, environmentId, runId: input.creationOperation.runId, attemptId: id,
|
||||
accountFingerprint: "a".repeat(64), labels: { "paperclip-create-attempt": id } } };
|
||||
}
|
||||
if (method === "environmentDestroyLease") {
|
||||
destroys++;
|
||||
if (failDestroy) throw new Error("Fixture teardown failed");
|
||||
resources.delete(input.leaseMetadata.creationOperationId);
|
||||
return { providerLeaseId: input.providerLeaseId, state: "destroyed" };
|
||||
}
|
||||
throw new Error(`Unexpected fixture method ${method}`);
|
||||
}),
|
||||
} as unknown as PluginWorkerManager;
|
||||
});
|
||||
afterEach(async () => {
|
||||
await db.execute(sql`drop trigger if exists reject_fixture_lease on environment_leases`);
|
||||
await db.execute(sql`drop trigger if exists reject_fixture_creation on sandbox_creation_operations`);
|
||||
await rm(spool, { recursive: true, force: true });
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
|
||||
function runtime() { return environmentRuntimeService(db, { pluginWorkerManager: workers, orphanCleanupSpoolDir: spool }); }
|
||||
async function acquire() {
|
||||
return runtime().acquireRunLease({ companyId, environment: (await environmentService(db).getById(environmentId))!,
|
||||
heartbeatRunId: runId, issueId: null, persistedExecutionWorkspace: null, assertCompanyBinding: true });
|
||||
}
|
||||
async function due() { await db.update(sandboxCreationOperations).set({ nextAttemptAt: new Date(0) }); }
|
||||
async function rejectWrites(table: "environment_leases" | "sandbox_creation_operations", trigger: string) {
|
||||
await db.execute(sql`create or replace function reject_fixture_write() returns trigger language plpgsql as $$ begin raise exception 'Fixture database write unavailable'; end $$`);
|
||||
await db.execute(sql.raw(`create trigger ${trigger} before insert on ${table} for each row execute function reject_fixture_write()`));
|
||||
}
|
||||
it("can reapply the creation migration without losing a committed intent", async () => {
|
||||
loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
const [before] = await db.select().from(sandboxCreationOperations);
|
||||
const migration = await readFile(new URL("../../../packages/db/src/migrations/0418_curved_lady_mastermind.sql", import.meta.url), "utf8");
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
for (const statement of migration.split("--> statement-breakpoint").map(value => value.trim()).filter(Boolean)) {
|
||||
await db.execute(sql.raw(statement));
|
||||
}
|
||||
}
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([before]);
|
||||
expect(creates).toBe(1);
|
||||
});
|
||||
it("fails before the external action when the durable admission write fails", async () => {
|
||||
await rejectWrites("sandbox_creation_operations", "reject_fixture_creation");
|
||||
await expect(acquire()).rejects.toThrow();
|
||||
expect(creates).toBe(0); expect(resources.size).toBe(0); expect(await readdir(spool)).toEqual([]);
|
||||
});
|
||||
it("attaches once and reuses the committed transfer after a lost response", async () => {
|
||||
const first = await acquire();
|
||||
const retry = await acquire();
|
||||
expect(retry.lease.id).toBe(first.lease.id); expect(creates).toBe(1); expect(destroys).toBe(0);
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(environmentLeases)).toHaveLength(1);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
|
||||
});
|
||||
it("recovers a failed lease write with a fresh receiver and empty web disk", async () => {
|
||||
await rejectWrites("environment_leases", "reject_fixture_lease");
|
||||
await expect(acquire()).rejects.toThrow();
|
||||
expect(creates).toBe(1); expect(await readdir(spool)).toEqual([]);
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "pending" })]);
|
||||
await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`);
|
||||
await due();
|
||||
await Promise.all([runtime().flushDeferredOrphanCleanups(), runtime().flushDeferredOrphanCleanups()]);
|
||||
const leases = await db.select().from(environmentLeases);
|
||||
expect(leases).toEqual([expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] })]);
|
||||
expect(creates).toBe(1);
|
||||
failDestroy = true;
|
||||
await expect(runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! })).rejects.toThrow("Fixture teardown failed");
|
||||
expect(resources.size).toBe(1);
|
||||
failDestroy = false;
|
||||
await runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! });
|
||||
expect(resources.size).toBe(0); expect(destroys).toBe(2);
|
||||
});
|
||||
it("reconstructs ownership in a replacement receiver process with a different empty web disk", async () => {
|
||||
const provider = createServer(async (request, response) => {
|
||||
try {
|
||||
let body = ""; for await (const chunk of request) body += chunk.toString();
|
||||
const { method, input } = JSON.parse(body);
|
||||
response.setHeader("content-type", "application/json");
|
||||
response.end(JSON.stringify(await (workers.call as any)(pluginId, method, input)));
|
||||
} catch { response.statusCode = 500; response.end("{}"); }
|
||||
});
|
||||
await new Promise<void>(resolve => provider.listen(0, "127.0.0.1", resolve));
|
||||
const address = provider.address() as { port: number };
|
||||
const replacementDisk = await mkdtemp(join(tmpdir(), "replacement-sandbox-web-"));
|
||||
const fixture = fileURLToPath(new URL("./fixtures/sandbox-creation-receiver.ts", import.meta.url));
|
||||
async function receiver(action: string, disk: string) {
|
||||
const result = await promisify(execFile)(process.execPath, ["--import", fileURLToPath(import.meta.resolve("tsx")), fixture], {
|
||||
cwd: fileURLToPath(new URL("../../..", import.meta.url)), timeout: 60_000,
|
||||
env: { ...process.env, CREATION_FIXTURE_ACTION: action, CREATION_FIXTURE_EMPTY_DISK: disk,
|
||||
CREATION_FIXTURE_DATABASE_URL: database.connectionString, CREATION_FIXTURE_PROVIDER_URL: `http://127.0.0.1:${address.port}`,
|
||||
CREATION_FIXTURE_COMPANY_ID: companyId, CREATION_FIXTURE_ENVIRONMENT_ID: environmentId, CREATION_FIXTURE_RUN_ID: runId },
|
||||
});
|
||||
return result.stdout;
|
||||
}
|
||||
try {
|
||||
await rejectWrites("environment_leases", "reject_fixture_lease");
|
||||
expect(await receiver("acquire", spool)).toContain('"ok":false');
|
||||
expect(creates).toBe(1); expect(resources.size).toBe(1); expect(await readdir(spool)).toEqual([]);
|
||||
await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`);
|
||||
await due();
|
||||
expect(await receiver("recover", replacementDisk)).toContain('"ok":true');
|
||||
expect(await receiver("recover", replacementDisk)).toContain('"ok":true');
|
||||
expect(await db.select().from(environmentLeases)).toEqual([
|
||||
expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] }),
|
||||
]);
|
||||
expect(creates).toBe(1); expect(await readdir(replacementDisk)).toEqual([]);
|
||||
} finally {
|
||||
await new Promise<void>((resolve, reject) => provider.close(error => error ? reject(error) : resolve()));
|
||||
await rm(replacementDisk, { recursive: true, force: true });
|
||||
}
|
||||
}, 180_000);
|
||||
it("never repeats creation after a lost provider reply", async () => {
|
||||
loseReply = true;
|
||||
await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
await expect(acquire()).rejects.toThrow("requires reconciliation");
|
||||
expect(creates).toBe(1);
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]);
|
||||
});
|
||||
it("retains cleanup ownership when the environment is deleted during creation", async () => {
|
||||
beforeReply = async () => { await environmentService(db).remove(environmentId); };
|
||||
await expect(acquire()).rejects.toThrow();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ environmentId: null, originalEnvironmentId: environmentId, state: "pending" })]);
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ environmentId: null, status: "pending_cleanup" })]);
|
||||
});
|
||||
it("rejects attachment after a company rebind and retains cleanup", async () => {
|
||||
beforeReply = async () => {
|
||||
const foreign = randomUUID(); await db.insert(companies).values({ id: foreign, name: "Foreign", issuePrefix: "FOR" });
|
||||
await db.insert(builtInManagedResources).values({ companyId: foreign, bundleKey: "managed-environment", resourceKind: "environment",
|
||||
resourceKey: "sandbox", resourceId: environmentId, stockVersion: "1", stockHash: "fixture" });
|
||||
};
|
||||
await expect(acquire()).rejects.toMatchObject({ status: 403 });
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ companyId, status: "pending_cleanup" })]);
|
||||
});
|
||||
it("backs off a failed lookup and recovers unrelated creations in the same pass", async () => {
|
||||
loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
const [first] = await db.select().from(sandboxCreationOperations);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
runId = randomUUID(); await db.insert(heartbeatRuns).values({ ...run!, id: runId });
|
||||
await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
const originalCall = workers.call;
|
||||
(workers as any).call = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => {
|
||||
if (method === "environmentInspectLease" && params.creationOperation.id === first!.id) throw new Error("Transient fixture lookup");
|
||||
return originalCall(plugin, method as any, params, timeout);
|
||||
});
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
const rows = await db.select().from(sandboxCreationOperations);
|
||||
expect(rows.find(value => value.id === first!.id)).toMatchObject({ state: "pending" });
|
||||
expect(rows.find(value => value.id === first!.id)!.nextAttemptAt.getTime()).toBeGreaterThan(Date.now());
|
||||
expect(rows.filter(value => value.state === "cleanup_pending")).toHaveLength(1);
|
||||
});
|
||||
it("does not attach a sandbox to a run stopped during creation", async () => {
|
||||
beforeReply = async () => { await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); };
|
||||
await expect(acquire()).rejects.toThrow("run authority changed");
|
||||
expect(await db.select().from(environmentLeases)).toHaveLength(0);
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]);
|
||||
});
|
||||
it.each(["before lookup", "during lookup", "lookup fails after deletion"])("records manual reconciliation when the company is deleted %s", async (deletion) => {
|
||||
loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
async function deleteCompany() {
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
}
|
||||
if (deletion === "before lookup") await deleteCompany();
|
||||
const originalCall = workers.call;
|
||||
const calls = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => {
|
||||
if (method === "environmentInspectLease" && deletion !== "before lookup") {
|
||||
await deleteCompany();
|
||||
if (deletion === "lookup fails after deletion") throw new Error("Company credential no longer exists");
|
||||
}
|
||||
return originalCall(plugin, method as any, params, timeout);
|
||||
});
|
||||
(workers as any).call = calls;
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ companyId, state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }),
|
||||
]);
|
||||
expect(await db.select().from(environmentLeases)).toEqual([]);
|
||||
expect(calls.mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(deletion === "before lookup" ? 0 : 1);
|
||||
const count = calls.mock.calls.length;
|
||||
await runtime().flushDeferredOrphanCleanups(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(calls).toHaveBeenCalledTimes(count);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull(); expect(creates).toBe(1);
|
||||
expect(await readdir(spool)).toEqual([]);
|
||||
});
|
||||
async function savedCreationCredential(providerConfigId?: string) {
|
||||
const secrets = secretOwner.secretService(db);
|
||||
const secret = await secrets.create(companyId, {
|
||||
name: `creation-key-${randomUUID()}`, provider: "local_encrypted", providerConfigId, value: "fixture-private-key",
|
||||
});
|
||||
await secrets.createBinding({ companyId, secretId: secret.id, targetType: "environment", targetId: environmentId, configPath: "apiKey" });
|
||||
await db.update(plugins).set({ manifestJson: sql`jsonb_set(${plugins.manifestJson}, '{environmentDrivers,0,configSchema}',
|
||||
'{"type":"object","properties":{"apiKey":{"type":"string","format":"secret-ref"}}}'::jsonb)` }).where(eq(plugins.id, pluginId));
|
||||
await db.update(environments).set({ config: { provider: "creation-fixture", image: "fixture", reuseLease: false, apiKey: secret.id } })
|
||||
.where(eq(environments.id, environmentId));
|
||||
loseReply = true;
|
||||
await expect(acquire()).rejects.toThrow("Lost provider reply");
|
||||
return secret;
|
||||
}
|
||||
it.each(["deleted", "disabled", "archived", "version-disabled", "vault-disabled", "vault-coming-soon"] as const)(
|
||||
"retains a %s creation credential as quiet manual reconciliation across restart", async state => {
|
||||
const secrets = secretOwner.secretService(db);
|
||||
const vault = state.startsWith("vault-") ? await secrets.createProviderConfig(companyId, {
|
||||
provider: "local_encrypted", displayName: "Creation vault", status: "ready", config: {},
|
||||
}) : undefined;
|
||||
const secret = await savedCreationCredential(vault?.id);
|
||||
if (state === "deleted") await secrets.remove(secret.id);
|
||||
else if (state === "disabled" || state === "archived") await secrets.update(secret.id, { status: state });
|
||||
else if (state === "version-disabled") await db.update(companySecretVersions).set({ status: "disabled" })
|
||||
.where(eq(companySecretVersions.secretId, secret.id));
|
||||
else await secrets.updateProviderConfig(vault!.id, { status: state === "vault-disabled" ? "disabled" : "coming_soon" });
|
||||
const inspectCalls = () => vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease").length;
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ state: "needs_reconciliation" }),
|
||||
]);
|
||||
expect(await db.select().from(environmentLeases)).toEqual([]);
|
||||
expect(inspectCalls()).toBe(0);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
|
||||
const restarted = runtime();
|
||||
await restarted.flushDeferredOrphanCleanups(); await restarted.flushDeferredOrphanCleanups();
|
||||
expect(inspectCalls()).toBe(0); expect(creates).toBe(1); expect(destroys).toBe(0);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
|
||||
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
||||
const ownerId = randomUUID();
|
||||
const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 };
|
||||
// An installed plugin is a separate blocker until its worker proves quiet.
|
||||
expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none"))
|
||||
.toEqual({ version: 1, backgroundWork: "present" });
|
||||
expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none",
|
||||
async () => ({ backgroundWork: "none", pluginIds: [pluginId] })))
|
||||
.toEqual({ version: 1, backgroundWork: "none" });
|
||||
expect(await readdir(spool)).toEqual([]);
|
||||
});
|
||||
it("retries a temporary creation credential resolution failure without losing ownership", async () => {
|
||||
await savedCreationCredential();
|
||||
const original = secretOwner.secretService;
|
||||
const resolution = vi.fn(async () => { throw new HttpError(503, "Temporary credential provider failure"); });
|
||||
const owner = vi.spyOn(secretOwner, "secretService").mockImplementation(client => ({
|
||||
...original(client), resolveSecretValueForSandboxCleanup: resolution,
|
||||
}));
|
||||
try {
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(resolution).toHaveBeenCalledTimes(1);
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ state: "pending", nextAttemptAt: expect.any(Date) }),
|
||||
]);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeGreaterThan(Date.now());
|
||||
expect(vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(0);
|
||||
} finally { owner.mockRestore(); }
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "cleanup_pending" })]);
|
||||
expect(await db.select().from(environmentLeases)).toHaveLength(1);
|
||||
expect(creates).toBe(1); expect(destroys).toBe(0); expect(await readdir(spool)).toEqual([]);
|
||||
});
|
||||
it("makes unsupported uncertain outcomes explicit without another creation", async () => {
|
||||
loseReply = true; await expect(acquire()).rejects.toThrow();
|
||||
await db.update(plugins).set({ manifestJson: sql`${plugins.manifestJson} #- '{environmentDrivers,0,creationOperationVersion}'` }).where(eq(plugins.id, pluginId));
|
||||
await due(); await runtime().flushDeferredOrphanCleanups();
|
||||
expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "needs_reconciliation" })]);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
|
||||
await expect(acquire()).rejects.toThrow("requires reconciliation"); expect(creates).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,5 @@
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
|
||||
@@ -20,7 +20,10 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
|
||||
const [row] = await db.select({ next: sql<string | null>`min(greatest(
|
||||
extract(epoch from ${environmentLeases.updatedAt}) * 1000 + ${backoffMs},
|
||||
${pendingCleanupDeadlineSql()}))` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
|
||||
const saved = row?.next == null ? null : Number(row.next);
|
||||
const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
|
||||
.from(sandboxCreationOperations).where(eq(sandboxCreationOperations.state, "pending"));
|
||||
const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
|
||||
const saved = deadlines.length ? Math.min(...deadlines) : null;
|
||||
const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
|
||||
return saved === null ? buffered : buffered === null ? saved : Math.min(saved, buffered);
|
||||
}
|
||||
@@ -1,3 +1,6 @@
|
||||
import { HttpError } from "../errors.js";
|
||||
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
import { createOwnedSandbox, recoverSandboxCreations } from "./sandbox-creation-operations.js";
|
||||
import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js";
|
||||
import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
|
||||
@@ -11,7 +14,6 @@ import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/nati
|
||||
import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
|
||||
import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
|
||||
import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
|
||||
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { hasNativeWorkspaceExportResume, releaseCompletedNativeWorkspaceExportRetention } from "./native-runtime/native-workspace-export-resume.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
@@ -93,7 +95,7 @@ import {
|
||||
resolvePluginExecuteRpcTimeoutMs,
|
||||
resumePluginEnvironmentLease,
|
||||
} from "./plugin-environment-driver.js";
|
||||
import { collectSecretRefPaths } from "./json-schema-secret-refs.js";
|
||||
import { collectSecretRefPaths, readConfigValueAtPath } from "./json-schema-secret-refs.js";
|
||||
import { buildWorkspaceRealizationRecordFromDriverInput } from "./workspace-realization.js";
|
||||
import {
|
||||
createSandboxOrphanCleanupSpool,
|
||||
@@ -698,39 +700,6 @@ export interface EnvironmentRuntimeDriver {
|
||||
flushDeferredOrphanCleanups?(): Promise<{ recovered: number; pending: number }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The acquire provisioned a remote sandbox, the lease insert rejected it, the
|
||||
* compensating teardown failed, and every retry of the durable pending-cleanup
|
||||
* write also failed. A live sandbox now has no lease row and no cleanup record,
|
||||
* so no sweep can find it. The acquire throws this error so the failure stays visible
|
||||
* and never resolves to a clean rejection. The `cause` field holds the original
|
||||
* lease insert rejection. The `cleanupWriteError` field holds the write failure.
|
||||
*/
|
||||
export class SandboxOrphanCleanupWriteError extends Error {
|
||||
readonly provider: string;
|
||||
readonly providerLeaseId: string | null;
|
||||
readonly cleanupWriteError: unknown;
|
||||
|
||||
constructor(input: {
|
||||
provider: string;
|
||||
providerLeaseId: string | null;
|
||||
cause: unknown;
|
||||
cleanupWriteError: unknown;
|
||||
}) {
|
||||
super(
|
||||
`Sandbox provider "${input.provider}" leaked lease ` +
|
||||
`"${input.providerLeaseId ?? "unknown"}": the acquire could not tear it down ` +
|
||||
"and could not record a durable pending-cleanup row. The live sandbox has no " +
|
||||
"durable cleanup state and needs a manual teardown.",
|
||||
{ cause: input.cause },
|
||||
);
|
||||
this.name = "SandboxOrphanCleanupWriteError";
|
||||
this.provider = input.provider;
|
||||
this.providerLeaseId = input.providerLeaseId;
|
||||
this.cleanupWriteError = input.cleanupWriteError;
|
||||
}
|
||||
}
|
||||
|
||||
/** A reusable sandbox could not be resumed, but has not been proven lost. */
|
||||
export class ReusableSandboxResumeError extends Error {
|
||||
readonly provider: string;
|
||||
@@ -785,17 +754,6 @@ export type ProviderResourceDisposition =
|
||||
const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000;
|
||||
const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100;
|
||||
|
||||
// The durable pending-cleanup write is the only automated way to find a leaked
|
||||
// sandbox. A single insert can lose to a transient database fault (a dropped
|
||||
// connection, a lock timeout, a serialization conflict). So the acquire retries
|
||||
// the write a few times with a short backoff before it gives up. A later
|
||||
// attempt can still land the durable row that a sweep finds.
|
||||
const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS = 3;
|
||||
const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS = 100;
|
||||
|
||||
// A prolonged database outage after a failed teardown can buffer many orphan
|
||||
// records in-process. The bound stops that buffer from growing without limit. A
|
||||
// full buffer keeps the error log as the last durable handle for a new orphan.
|
||||
const DEFAULT_DEFERRED_ORPHAN_CLEANUP_BUFFER_LIMIT = 256;
|
||||
|
||||
function delay(ms: number): Promise<void> {
|
||||
@@ -1311,8 +1269,6 @@ function createSandboxEnvironmentDriver(
|
||||
pluginWorkerManager?: PluginWorkerManager;
|
||||
pluginWorkerReadyTimeoutMs?: number;
|
||||
pluginWorkerReadyPollMs?: number;
|
||||
pendingCleanupWriteAttempts?: number;
|
||||
pendingCleanupWriteBackoffMs?: number;
|
||||
deferredOrphanCleanupBufferLimit?: number;
|
||||
orphanCleanupSpool?: SandboxOrphanCleanupSpool;
|
||||
orphanCleanupSpoolDir?: string;
|
||||
@@ -1321,16 +1277,6 @@ function createSandboxEnvironmentDriver(
|
||||
const pluginWorkerManager = options.pluginWorkerManager;
|
||||
const pluginWorkerReadyTimeoutMs = options.pluginWorkerReadyTimeoutMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS;
|
||||
const pluginWorkerReadyPollMs = options.pluginWorkerReadyPollMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS;
|
||||
// The retry count is at least one attempt, so a zero or negative override
|
||||
// never skips the durable write. The backoff is at least zero milliseconds.
|
||||
const pendingCleanupWriteAttempts = Math.max(
|
||||
1,
|
||||
options.pendingCleanupWriteAttempts ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS,
|
||||
);
|
||||
const pendingCleanupWriteBackoffMs = Math.max(
|
||||
0,
|
||||
options.pendingCleanupWriteBackoffMs ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS,
|
||||
);
|
||||
// The buffer holds at least one orphan, so a zero or negative override never
|
||||
// drops every buffered record.
|
||||
const deferredOrphanCleanupBufferLimit = Math.max(
|
||||
@@ -1339,15 +1285,6 @@ function createSandboxEnvironmentDriver(
|
||||
);
|
||||
const environmentsSvc = environmentService(db);
|
||||
|
||||
// A live sandbox whose teardown failed needs a durable `pending_cleanup` row,
|
||||
// so a sweep can find and release it. When every synchronous write attempt
|
||||
// fails, the database is down but the process still runs. So the driver keeps
|
||||
// the orphan record in this in-process buffer, and a later cleanup sweep
|
||||
// flushes it back to the database. The buffer closes the common window where
|
||||
// the database recovers after the synchronous attempts but while the process
|
||||
// still runs. The `pending_cleanup` failure reason matches the synchronous
|
||||
// write, so a flushed row and a synchronous row are indistinguishable to the
|
||||
// sweep.
|
||||
const deferredOrphanCleanups: DeferredOrphanCleanupRecord[] = [];
|
||||
// Keep the token through queue splices, failed flushes and buffer overflow.
|
||||
// Release only when the database durably owns cleanup of this exact orphan.
|
||||
@@ -1363,16 +1300,6 @@ function createSandboxEnvironmentDriver(
|
||||
};
|
||||
const DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON = "acquire_rejected_teardown_failed";
|
||||
|
||||
// The in-process buffer alone loses an orphan on a restart. So the driver also
|
||||
// persists each buffered orphan to this durable local spool. The spool keeps
|
||||
// the record on the local disk, next to the other durable server state, so a
|
||||
// restart still finds it. The first flush after a restart loads the spool back
|
||||
// into the buffer, so the sweep re-inserts the durable row. The spool holds no
|
||||
// secret value: it persists the same sanitized metadata the database row
|
||||
// holds, so a restart-recovered record still authorizes the teardown but never
|
||||
// exposes a credential. Only a crash during the outage before the durable
|
||||
// spool write lands loses the record; the error log stays the last handle for
|
||||
// that narrow case.
|
||||
const orphanCleanupSpool =
|
||||
options.orphanCleanupSpool ?? createSandboxOrphanCleanupSpool(options.orphanCleanupSpoolDir);
|
||||
|
||||
@@ -1460,143 +1387,6 @@ function createSandboxEnvironmentDriver(
|
||||
return orphanFlush;
|
||||
};
|
||||
|
||||
// Build the safe diagnostic fields for an orphan record. The provider
|
||||
// identifiers are the only fields any diagnostic log emits, and they carry no
|
||||
// secret. A caught write exception never enters a log: it can carry a
|
||||
// credential in its name, code, message, cause, or stack.
|
||||
const orphanDiagnosticFields = (record: DeferredOrphanCleanupRecord) => ({
|
||||
errorKind: SANDBOX_ORPHAN_CLEANUP_WRITE_ERROR_KIND,
|
||||
provider: record.provider,
|
||||
providerLeaseId: record.providerLeaseId,
|
||||
companyId: record.companyId,
|
||||
environmentId: record.environmentId,
|
||||
executionWorkspaceId: record.executionWorkspaceId,
|
||||
issueId: record.issueId,
|
||||
heartbeatRunId: record.heartbeatRunId,
|
||||
});
|
||||
|
||||
// Try to write the durable `pending_cleanup` row for an orphan sandbox, with a
|
||||
// few retries. Report the new lease id on success, or a null lease id and the
|
||||
// last write error on total failure. This function never buffers, never logs
|
||||
// at error level, and never throws. The caller decides whether a total failure
|
||||
// is fatal, because a later successful teardown removes the orphan and needs
|
||||
// no durable row.
|
||||
//
|
||||
// The row carries the provider lease id and the same resolution metadata a
|
||||
// normal lease holds, so a later cleanup sweep finds and releases the orphan.
|
||||
// One atomic insert writes the row directly in the terminal `pending_cleanup`
|
||||
// state, and the insert skips the company-binding assertion, so it records the
|
||||
// orphan even for a foreign-bound environment. A transient database fault (a
|
||||
// dropped connection, a lock timeout, a serialization conflict) can reject one
|
||||
// insert but clear on the next, so the write retries a few times with a short
|
||||
// backoff.
|
||||
const tryWriteDurablePendingCleanup = async (
|
||||
record: DeferredOrphanCleanupRecord,
|
||||
): Promise<{ leaseId: string | null; lastError: unknown }> => {
|
||||
const diagnosticFields = orphanDiagnosticFields(record);
|
||||
let lastCleanupWriteError: unknown;
|
||||
for (let attempt = 1; attempt <= pendingCleanupWriteAttempts; attempt += 1) {
|
||||
try {
|
||||
const lease = await environmentsSvc.insertPendingCleanupLease({
|
||||
companyId: record.companyId,
|
||||
environmentId: record.environmentId,
|
||||
executionWorkspaceId: record.executionWorkspaceId,
|
||||
issueId: record.issueId,
|
||||
heartbeatRunId: record.heartbeatRunId,
|
||||
provider: record.provider,
|
||||
providerLeaseId: record.providerLeaseId,
|
||||
metadata: record.metadata,
|
||||
failureReason: DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON,
|
||||
});
|
||||
// The durable row exists now, so a sweep can find and release the orphan.
|
||||
return { leaseId: lease.id, lastError: undefined };
|
||||
} catch (cleanupWriteError) {
|
||||
lastCleanupWriteError = cleanupWriteError;
|
||||
if (attempt < pendingCleanupWriteAttempts) {
|
||||
// The write failed, but attempts remain. Log the retry at warn level
|
||||
// and back off, then try again. A later attempt can still land the
|
||||
// durable row.
|
||||
logger.warn(
|
||||
{ ...diagnosticFields, attempt, maxAttempts: pendingCleanupWriteAttempts },
|
||||
"sandbox orphan cleanup write failed; retrying the durable pending-cleanup write",
|
||||
);
|
||||
await delay(pendingCleanupWriteBackoffMs * attempt);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { leaseId: null, lastError: lastCleanupWriteError };
|
||||
};
|
||||
|
||||
// Release the durable `pending_cleanup` row after a successful inline teardown.
|
||||
// The teardown removed the orphan sandbox, so the row is no longer needed. The
|
||||
// release moves the row to the terminal `expired` state with a `success`
|
||||
// cleanup status, the same state the sweep records for a successful teardown.
|
||||
//
|
||||
// The release is best-effort. If it fails, the row stays `pending_cleanup`, so
|
||||
// a later sweep runs the idempotent teardown on the already-gone sandbox and
|
||||
// releases the row itself. A failed release never loses or leaks the orphan.
|
||||
const releaseCleanedUpOrphanRow = async (
|
||||
leaseId: string,
|
||||
diagnosticFields: Record<string, unknown>,
|
||||
receipt: unknown,
|
||||
): Promise<void> => {
|
||||
try {
|
||||
const lease = await environmentsSvc.getLeaseById(leaseId);
|
||||
await environmentsSvc.releaseLease(leaseId, "expired", {
|
||||
...(lease ? { remoteExecutionTermination: remoteTerminationReceipt(lease, receipt) } : {}),
|
||||
cleanupStatus: "success",
|
||||
failureReason: "acquire_rejected_teardown_succeeded",
|
||||
});
|
||||
} catch {
|
||||
// The caught release exception never enters a log: it can carry a
|
||||
// credential in its name, code, message, cause, or stack.
|
||||
logger.warn(
|
||||
diagnosticFields,
|
||||
"could not release the pending-cleanup row after a successful orphan teardown; a later sweep reconciles the released row",
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Neither the durable write nor the inline teardown worked, so the orphan
|
||||
// sandbox is live with no durable row. Persist the orphan to the durable spool
|
||||
// and buffer it in-process, so a later cleanup-sweep flush lands the durable
|
||||
// row. The spool survives a restart, so the flush recovers the orphan even
|
||||
// after a crash. Then log the plain provider identifiers at error level, so an
|
||||
// operator keeps a handle to tear the sandbox down by hand. Only a crash before
|
||||
// the spool write lands loses the record, so the error log is the last handle
|
||||
// for that narrow case. Throw `SandboxOrphanCleanupWriteError`, which the
|
||||
// caller propagates. The error keeps the original insert rejection as its
|
||||
// `cause`, so the real reason the acquire failed stays visible in the error
|
||||
// chain.
|
||||
const escalateUnwrittenOrphan = async (
|
||||
record: DeferredOrphanCleanupRecord,
|
||||
cause: unknown,
|
||||
cleanupWriteError: unknown,
|
||||
): Promise<never> => {
|
||||
trackOrphanIdleWork(record);
|
||||
const persisted = await orphanCleanupSpool.append(record);
|
||||
const buffered = enqueueDeferredOrphanCleanup(record);
|
||||
logger.error(
|
||||
{
|
||||
...orphanDiagnosticFields(record),
|
||||
persisted,
|
||||
buffered,
|
||||
deferredBufferSize: deferredOrphanCleanups.length,
|
||||
},
|
||||
persisted
|
||||
? "sandbox orphan cleanup write failed; persisted the orphan to the durable spool for a later cleanup-sweep flush"
|
||||
: buffered
|
||||
? "sandbox orphan cleanup write failed and the durable spool write failed; buffered the orphan in-process only for a later cleanup-sweep flush"
|
||||
: "sandbox orphan cleanup write failed and neither the durable spool nor the in-process buffer kept the orphan; live sandbox needs a manual teardown",
|
||||
);
|
||||
throw new SandboxOrphanCleanupWriteError({
|
||||
provider: record.provider,
|
||||
providerLeaseId: record.providerLeaseId,
|
||||
cause,
|
||||
cleanupWriteError,
|
||||
});
|
||||
};
|
||||
|
||||
// Resolve an uncertain creation to a verified provider ID, journal it, then
|
||||
// authorize deletion. This makes a later missing-ID response conclusive even
|
||||
// when the destructive RPC reply or the final lease update was lost.
|
||||
@@ -1628,68 +1418,7 @@ function createSandboxEnvironmentDriver(
|
||||
}
|
||||
};
|
||||
|
||||
// Clean up an orphan after a rejected insert or uncertain provider creation.
|
||||
// The acquire provisioned a remote sandbox, then the insert rejected (a
|
||||
// foreign-company binding), so no lease row tracks the live sandbox. This
|
||||
// handler records the durable `pending_cleanup` row FIRST, before the inline
|
||||
// teardown, then tears the sandbox down.
|
||||
//
|
||||
// The write goes first on purpose. The conditional insert just returned a
|
||||
// definitive rejection, so the database is reachable at this instant. An
|
||||
// inline teardown can run long, and the database can drop during it. So a
|
||||
// write after the teardown can fail and leave the orphan only in memory, which
|
||||
// a restart loses. With the write first, the durable row lands while the
|
||||
// database is proven up, so a crash after it still leaves a row that a sweep
|
||||
// finds and releases.
|
||||
//
|
||||
// The teardown then runs. On success the orphan is gone, so the handler
|
||||
// releases the durable row. On a failed teardown the row stays for the sweep.
|
||||
// If the durable write itself failed and the teardown also failed, the handler
|
||||
// buffers the orphan and throws `SandboxOrphanCleanupWriteError`. A successful
|
||||
// teardown after a failed write leaves no orphan, so it needs no error.
|
||||
const cleanUpRejectedOrphanSandbox = async (input: {
|
||||
record: DeferredOrphanCleanupRecord;
|
||||
cause: unknown;
|
||||
canTeardown: boolean;
|
||||
teardown: (pendingLeaseId: string | null) => Promise<unknown>;
|
||||
}): Promise<boolean> => {
|
||||
const durable = await tryWriteDurablePendingCleanup(input.record);
|
||||
let teardownFailed = !input.canTeardown;
|
||||
let receipt: unknown;
|
||||
if (!teardownFailed) {
|
||||
try {
|
||||
receipt = await input.teardown(durable.leaseId);
|
||||
} catch {
|
||||
teardownFailed = true;
|
||||
}
|
||||
}
|
||||
if (!teardownFailed) {
|
||||
// The teardown removed the orphan, so drop the durable row if we wrote one.
|
||||
if (durable.leaseId !== null) {
|
||||
await releaseCleanedUpOrphanRow(durable.leaseId, orphanDiagnosticFields(input.record), receipt);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
// The teardown failed, so the orphan sandbox is still live.
|
||||
if (durable.leaseId !== null) {
|
||||
// The durable row already tracks the orphan, so a sweep finds and releases
|
||||
// it. The caller rethrows the original insert rejection.
|
||||
return false;
|
||||
}
|
||||
return await escalateUnwrittenOrphan(input.record, input.cause, durable.lastError);
|
||||
};
|
||||
|
||||
// The run-time exec parent context, held per lease id. A plugin sandbox
|
||||
// provider can open a persistent session on the first command and delete it
|
||||
// on lease release. The session open runs inside `execute`, under the run
|
||||
// parent (the same context the `sandbox.exec` span reads). The session delete
|
||||
// runs inside the lease-release RPC, which the run orchestrator calls after
|
||||
// the run, outside that scope. Without a parent the host mints no
|
||||
// `traceparent` and drops the provider `session.teardown` span. So `execute`
|
||||
// records the exec parent here, and the release paths replay it around the
|
||||
// release RPC. The host still mints and validates the `traceparent` itself;
|
||||
// the value only widens which host calls carry a run parent. An entry is
|
||||
// removed on release, so the map holds at most one context per live lease.
|
||||
const runExecParentByLeaseId = new Map<string, StartupSpanContext>();
|
||||
|
||||
// Run a lease-release RPC under the lease's recorded exec parent context, and
|
||||
@@ -2128,12 +1857,19 @@ function createSandboxEnvironmentDriver(
|
||||
});
|
||||
}
|
||||
}
|
||||
const acquisitionRunId = input.heartbeatRunId ?? randomUUID();
|
||||
const acquiredLease = providerLease ?? await (async () => {
|
||||
const acquiredLease = providerLease ?? await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: input.environment.id,
|
||||
heartbeatRunId: input.heartbeatRunId, issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId,
|
||||
provider: parsed.config.provider, pluginId: pluginProvider.resolved.plugin.id, driverKey: parsed.config.provider,
|
||||
resolvedConfig: { ...parsed.config },
|
||||
configSchema: pluginProvider.resolved.driver.configSchema as Record<string, unknown> | undefined,
|
||||
config: sandboxConfigForLeaseMetadata(storedConfig), requestKey: createHash("sha256").update(JSON.stringify(storedConfig)).digest("hex"),
|
||||
metadata: { ...sandboxConfigForLeaseMetadata(storedConfig), driver: input.environment.driver,
|
||||
pluginId: pluginProvider.resolved.plugin.id, pluginKey: pluginProvider.resolved.plugin.pluginKey, sandboxProviderPlugin: true },
|
||||
timeoutMs: resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs),
|
||||
}, async identity => {
|
||||
try {
|
||||
return await pluginWorkerManager.call(
|
||||
pluginProvider.resolved.plugin.id,
|
||||
"environmentAcquireLease",
|
||||
return await pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentAcquireLease",
|
||||
{
|
||||
driverKey: parsed.config.provider,
|
||||
companyId: input.companyId,
|
||||
@@ -2143,7 +1879,8 @@ function createSandboxEnvironmentDriver(
|
||||
// Plugin SDK requires a string; ad-hoc test leases use a fresh
|
||||
// UUID so providers that validate or persist the runId still see
|
||||
// a well-formed identifier.
|
||||
runId: acquisitionRunId,
|
||||
runId: identity.runId,
|
||||
creationOperationId: identity.id,
|
||||
workspaceMode: input.executionWorkspaceMode ?? undefined,
|
||||
agentId: input.agentId ?? undefined,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? undefined,
|
||||
@@ -2162,62 +1899,18 @@ function createSandboxEnvironmentDriver(
|
||||
? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) }
|
||||
: {}),
|
||||
},
|
||||
resolvePluginSandboxRpcTimeoutMs(
|
||||
workerConfig,
|
||||
pluginProvider.resolved.driver.defaultAcquireTimeoutMs,
|
||||
),
|
||||
);
|
||||
resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs));
|
||||
} catch (error) {
|
||||
captureEnvironmentAcquisitionDiagnostic(error, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId,
|
||||
}, null);
|
||||
const cleanup = readEnvironmentCreationCleanupError(error);
|
||||
// The authenticated worker may identify its uncertain allocation,
|
||||
// but cannot redirect cleanup to another company, environment, or run.
|
||||
if (cleanup && cleanup.companyId === input.companyId &&
|
||||
cleanup.environmentId === input.environment.id && cleanup.runId === acquisitionRunId) {
|
||||
if (error instanceof JsonRpcCallError) captureEnvironmentAcquisitionDiagnostic(error, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId,
|
||||
}, readEnvironmentAcquisitionDiagnostic(error));
|
||||
const cleanupMetadata = {
|
||||
...sandboxConfigForLeaseMetadata(storedConfig),
|
||||
failedCreateCleanup: cleanup,
|
||||
driver: input.environment.driver,
|
||||
pluginId: pluginProvider.resolved.plugin.id,
|
||||
pluginKey: pluginProvider.resolved.plugin.pluginKey,
|
||||
sandboxProviderPlugin: true,
|
||||
};
|
||||
// Record before retrying the provider. Existing pending-cleanup
|
||||
// recovery (including its durable spool) survives controller loss.
|
||||
const cleanupConfirmed = await cleanUpRejectedOrphanSandbox({
|
||||
record: {
|
||||
companyId: input.companyId, environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
issueId: input.issueId ?? null, heartbeatRunId: input.heartbeatRunId ?? null,
|
||||
provider: parsed.config.provider, providerLeaseId: cleanup.providerLeaseId,
|
||||
metadata: cleanupMetadata,
|
||||
},
|
||||
cause: error,
|
||||
canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id),
|
||||
teardown: (pendingLeaseId) => destroyWithCreationObservation({
|
||||
leaseId: pendingLeaseId, metadata: cleanupMetadata,
|
||||
teardown: (metadata) => pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentDestroyLease", {
|
||||
driverKey: parsed.config.provider, companyId: input.companyId,
|
||||
environmentId: input.environment.id, issueId: input.issueId,
|
||||
config: workerConfig, providerLeaseId: cleanup.providerLeaseId,
|
||||
leaseMetadata: metadata,
|
||||
}, resolvePluginSandboxRpcTimeoutMs(workerConfig)),
|
||||
}),
|
||||
});
|
||||
if (cleanupConfirmed) {
|
||||
throw new Error("Sandbox creation failed; allocated sandbox cleanup was confirmed.", { cause: error });
|
||||
}
|
||||
}
|
||||
// A cleanup record is not a usable lease. Keep acquisition failed,
|
||||
// even when its compensating deletion succeeds on this attempt.
|
||||
const diagnostic = error instanceof JsonRpcCallError && cleanup?.companyId === input.companyId &&
|
||||
cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId
|
||||
? readEnvironmentAcquisitionDiagnostic(error) : null;
|
||||
captureEnvironmentAcquisitionDiagnostic(error, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, runId: identity.runId,
|
||||
}, diagnostic);
|
||||
throw error;
|
||||
}
|
||||
})();
|
||||
});
|
||||
|
||||
// Ad-hoc test leases are never publishable for reuse: storing them
|
||||
// as `reuse_by_environment` would let a concurrent heartbeat resume
|
||||
@@ -2275,82 +1968,34 @@ function createSandboxEnvironmentDriver(
|
||||
: {}),
|
||||
...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
|
||||
};
|
||||
try {
|
||||
return await environmentsSvc.acquireLease({
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
issueId: input.issueId,
|
||||
heartbeatRunId: input.heartbeatRunId,
|
||||
assertCompanyBinding: input.assertCompanyBinding,
|
||||
leasePolicy: resolvedLeasePolicy,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: acquiredLease.providerLeaseId,
|
||||
expiresAt: providerAttestedLeaseExpiry(
|
||||
input.requestedExpiresAt,
|
||||
acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined,
|
||||
),
|
||||
metadata: pluginLeaseMetadata,
|
||||
reusesReusableLeaseId:
|
||||
providerLease &&
|
||||
reusableLease?.heartbeatRunId === input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
replacesReusableLeaseId:
|
||||
providerLease &&
|
||||
reusableLease?.heartbeatRunId !== input.heartbeatRunId
|
||||
? reusableLease?.id
|
||||
: null,
|
||||
});
|
||||
} catch (error) {
|
||||
// The conditional lease insert rejected, so no lease row exists. A
|
||||
// managed reconciliation can bind the environment to another company
|
||||
// between the route guard and this insert, so the insert fails closed
|
||||
// with `environment_company_mismatch`. This call already provisioned the
|
||||
// remote plugin sandbox above, so tear it down now. Without this step
|
||||
// the rejected insert leaks a live sandbox that no lease row tracks. The
|
||||
// Claude login runs on a plugin-backed sandbox, so this path is the one
|
||||
// the login uses. Do not tear down a reused sandbox that an earlier
|
||||
// lease still owns.
|
||||
if (!reusableLease || acquiredLease.providerLeaseId !== reusableLease.providerLeaseId) {
|
||||
// Record the durable pending-cleanup row before the teardown, then
|
||||
// tear the sandbox down. On a successful teardown the handler releases
|
||||
// the row. If the durable write and the teardown both fail, the
|
||||
// handler throws a `SandboxOrphanCleanupWriteError` that carries the
|
||||
// original rejection as its cause.
|
||||
await cleanUpRejectedOrphanSandbox({
|
||||
record: {
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
heartbeatRunId: input.heartbeatRunId ?? null,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: acquiredLease.providerLeaseId,
|
||||
metadata: pluginLeaseMetadata,
|
||||
},
|
||||
cause: error,
|
||||
canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id),
|
||||
teardown: async () => {
|
||||
return await pluginWorkerManager.call(
|
||||
pluginProvider.resolved.plugin.id,
|
||||
"environmentDestroyLease",
|
||||
{
|
||||
driverKey: parsed.config.provider,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
issueId: input.issueId,
|
||||
config: workerConfig,
|
||||
providerLeaseId: acquiredLease.providerLeaseId,
|
||||
leaseMetadata: acquiredLease.metadata ?? undefined,
|
||||
},
|
||||
resolvePluginSandboxRpcTimeoutMs(workerConfig),
|
||||
);
|
||||
},
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
return await environmentsSvc.acquireLease({
|
||||
creationOperationId: providerLease ? undefined : acquiredLease.metadata?.creationOperationId as string | undefined,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
issueId: input.issueId,
|
||||
heartbeatRunId: input.heartbeatRunId,
|
||||
assertCompanyBinding: input.assertCompanyBinding,
|
||||
leasePolicy: resolvedLeasePolicy,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: acquiredLease.providerLeaseId,
|
||||
expiresAt: providerAttestedLeaseExpiry(
|
||||
input.requestedExpiresAt,
|
||||
acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined,
|
||||
),
|
||||
metadata: pluginLeaseMetadata,
|
||||
reusesReusableLeaseId:
|
||||
providerLease &&
|
||||
reusableLease?.heartbeatRunId === input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
replacesReusableLeaseId:
|
||||
providerLease &&
|
||||
reusableLease?.heartbeatRunId !== input.heartbeatRunId
|
||||
? reusableLease?.id
|
||||
: null,
|
||||
});
|
||||
|
||||
}
|
||||
|
||||
// Built-in sandbox provider path. Same guard as the plugin-backed path:
|
||||
@@ -2441,10 +2086,16 @@ function createSandboxEnvironmentDriver(
|
||||
|
||||
let providerLease;
|
||||
try {
|
||||
providerLease = await acquireSandboxProviderLease({
|
||||
providerLease = await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId,
|
||||
issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId, provider: parsed.config.provider,
|
||||
driverKey: parsed.config.provider, config: sandboxConfigForLeaseMetadata(parsed.config),
|
||||
requestKey: createHash("sha256").update(JSON.stringify(parsed.config)).digest("hex"),
|
||||
metadata: { ...sandboxConfigForLeaseMetadata(parsed.config), driver: input.environment.driver },
|
||||
}, identity => acquireSandboxProviderLease({
|
||||
config: parsed.config,
|
||||
environmentId: input.environment.id,
|
||||
heartbeatRunId: input.heartbeatRunId ?? randomUUID(),
|
||||
heartbeatRunId: identity.runId,
|
||||
issueId: input.issueId,
|
||||
agentId: input.agentId,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
@@ -2452,7 +2103,7 @@ function createSandboxEnvironmentDriver(
|
||||
// Forward a caller deadline so the provider configures a provider-side
|
||||
// expiry at or before it and returns the real provider expiry.
|
||||
requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt),
|
||||
});
|
||||
}));
|
||||
} catch (error) {
|
||||
if (reusableLease) {
|
||||
throw new ReusableSandboxResumeError({
|
||||
@@ -2521,72 +2172,35 @@ function createSandboxEnvironmentDriver(
|
||||
: {}),
|
||||
...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
|
||||
};
|
||||
try {
|
||||
return await environmentsSvc.acquireLease({
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
issueId: input.issueId,
|
||||
heartbeatRunId: input.heartbeatRunId,
|
||||
assertCompanyBinding: input.assertCompanyBinding,
|
||||
leasePolicy: resolvedLeasePolicy,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: providerLease.providerLeaseId,
|
||||
expiresAt: providerAttestedLeaseExpiry(
|
||||
input.requestedExpiresAt,
|
||||
providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined,
|
||||
),
|
||||
metadata: builtinLeaseMetadata,
|
||||
reusesReusableLeaseId:
|
||||
reusableLease &&
|
||||
providerLease.providerLeaseId === reusableLease.providerLeaseId &&
|
||||
reusableLease.heartbeatRunId === input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
replacesReusableLeaseId:
|
||||
reusableLease &&
|
||||
providerLease.providerLeaseId === reusableLease.providerLeaseId &&
|
||||
reusableLease.heartbeatRunId !== input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
});
|
||||
} catch (error) {
|
||||
// The conditional lease insert rejected, so no lease row exists. A managed
|
||||
// reconciliation can bind the environment to another company between the
|
||||
// route guard and this insert, so the insert fails closed with
|
||||
// `environment_company_mismatch`. This call already provisioned the remote
|
||||
// sandbox above, so release it now. Without this teardown the rejected
|
||||
// insert leaks a live sandbox that no lease row tracks. Do not tear down a
|
||||
// reused sandbox that an earlier lease still owns.
|
||||
if (!reusableLease || providerLease.providerLeaseId !== reusableLease.providerLeaseId) {
|
||||
// Record the durable pending-cleanup row before the teardown, then
|
||||
// release the remote sandbox. On a successful teardown the handler
|
||||
// releases the row. If the durable write and the teardown both fail,
|
||||
// the handler throws a `SandboxOrphanCleanupWriteError` that carries the
|
||||
// original rejection as its cause.
|
||||
await cleanUpRejectedOrphanSandbox({
|
||||
record: {
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
heartbeatRunId: input.heartbeatRunId ?? null,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: providerLease.providerLeaseId,
|
||||
metadata: builtinLeaseMetadata,
|
||||
},
|
||||
cause: error,
|
||||
canTeardown: true,
|
||||
teardown: async () => {
|
||||
await destroySandboxProviderLease({
|
||||
config: parsed.config,
|
||||
providerLeaseId: providerLease.providerLeaseId,
|
||||
});
|
||||
},
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
return await environmentsSvc.acquireLease({
|
||||
creationOperationId: providerLease.metadata?.creationOperationId as string | undefined,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
issueId: input.issueId,
|
||||
heartbeatRunId: input.heartbeatRunId,
|
||||
assertCompanyBinding: input.assertCompanyBinding,
|
||||
leasePolicy: resolvedLeasePolicy,
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: providerLease.providerLeaseId,
|
||||
expiresAt: providerAttestedLeaseExpiry(
|
||||
input.requestedExpiresAt,
|
||||
providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined,
|
||||
),
|
||||
metadata: builtinLeaseMetadata,
|
||||
reusesReusableLeaseId:
|
||||
reusableLease &&
|
||||
providerLease.providerLeaseId === reusableLease.providerLeaseId &&
|
||||
reusableLease.heartbeatRunId === input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
replacesReusableLeaseId:
|
||||
reusableLease &&
|
||||
providerLease.providerLeaseId === reusableLease.providerLeaseId &&
|
||||
reusableLease.heartbeatRunId !== input.heartbeatRunId
|
||||
? reusableLease.id
|
||||
: null,
|
||||
});
|
||||
},
|
||||
|
||||
async releaseRunLease(input) {
|
||||
@@ -2719,7 +2333,7 @@ function createSandboxEnvironmentDriver(
|
||||
`Sandbox provider "${recordedProvider}" needs a plugin worker manager for cleanup, but none is available.`,
|
||||
);
|
||||
}
|
||||
const pinnedPluginId = exportResume ? readString(input.lease.metadata?.pluginId) : null;
|
||||
const pinnedPluginId = exportResume || input.lease.metadata?.creationOperationId ? readString(input.lease.metadata?.pluginId) : null;
|
||||
if (exportResume && !pinnedPluginId) throw new Error("Workspace export cleanup has no recorded provider plugin.");
|
||||
const pluginProvider = await resolveSandboxProviderPlugin({ provider: recordedProvider, ...(pinnedPluginId ? { pluginId: pinnedPluginId } : {}) });
|
||||
if (pluginProvider.state !== "running") {
|
||||
@@ -3534,6 +3148,7 @@ const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([
|
||||
"remoteCwd",
|
||||
"shellCommand",
|
||||
"sandboxProviderPlugin",
|
||||
"creationOperationId",
|
||||
"sandboxLeaseAcquisition",
|
||||
"nativeHarnessBackup",
|
||||
"nativeWorkspaceSync",
|
||||
@@ -3698,13 +3313,20 @@ function createPluginEnvironmentDriver(
|
||||
resumed = undefined;
|
||||
}
|
||||
}
|
||||
const providerLease = resumed ?? await workerManager.call(plugin.id, "environmentAcquireLease", {
|
||||
const providerLease = resumed ?? await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId,
|
||||
issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId,
|
||||
provider: `plugin:${parsed.config.pluginKey}:${parsed.config.driverKey}`, pluginId: plugin.id, driverKey: parsed.config.driverKey,
|
||||
config: {}, timeoutMs: rpcTimeoutMs, requestKey: fingerprint,
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: parsed.config.pluginKey, driverKey: parsed.config.driverKey },
|
||||
}, identity => workerManager.call(plugin.id, "environmentAcquireLease", {
|
||||
driverKey: parsed.config.driverKey,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
issueId: input.issueId,
|
||||
config: parsed.config.driverConfig,
|
||||
runId: input.heartbeatRunId ?? randomUUID(),
|
||||
runId: identity.runId,
|
||||
creationOperationId: identity.id,
|
||||
workspaceMode: input.executionWorkspaceMode ?? undefined,
|
||||
agentId: input.agentId ?? undefined,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? undefined,
|
||||
@@ -3715,9 +3337,10 @@ function createPluginEnvironmentDriver(
|
||||
...(requestedExpiresAtParam(input.requestedExpiresAt) !== undefined
|
||||
? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) }
|
||||
: {}),
|
||||
} as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride);
|
||||
} as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride));
|
||||
|
||||
return await environmentsSvc.acquireLease({
|
||||
creationOperationId: resumed ? undefined : providerLease.metadata?.creationOperationId as string | undefined,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environment.id,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
@@ -3964,14 +3587,13 @@ export function environmentRuntimeService(
|
||||
pluginWorkerManager?: PluginWorkerManager;
|
||||
pluginWorkerReadyTimeoutMs?: number;
|
||||
pluginWorkerReadyPollMs?: number;
|
||||
pendingCleanupWriteAttempts?: number;
|
||||
pendingCleanupWriteBackoffMs?: number;
|
||||
deferredOrphanCleanupBufferLimit?: number;
|
||||
orphanCleanupSpool?: SandboxOrphanCleanupSpool;
|
||||
orphanCleanupSpoolDir?: string;
|
||||
} = {},
|
||||
) {
|
||||
const environmentsSvc = environmentService(db);
|
||||
let orphanRecoveryFlush: Promise<{ recovered: number; pending: number }> | null = null;
|
||||
const drivers = new Map<string, EnvironmentRuntimeDriver>();
|
||||
|
||||
const defaultDrivers = [
|
||||
@@ -3981,8 +3603,6 @@ export function environmentRuntimeService(
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
pluginWorkerReadyTimeoutMs: options.pluginWorkerReadyTimeoutMs,
|
||||
pluginWorkerReadyPollMs: options.pluginWorkerReadyPollMs,
|
||||
pendingCleanupWriteAttempts: options.pendingCleanupWriteAttempts,
|
||||
pendingCleanupWriteBackoffMs: options.pendingCleanupWriteBackoffMs,
|
||||
deferredOrphanCleanupBufferLimit: options.deferredOrphanCleanupBufferLimit,
|
||||
orphanCleanupSpool: options.orphanCleanupSpool,
|
||||
orphanCleanupSpoolDir: options.orphanCleanupSpoolDir,
|
||||
@@ -4439,14 +4059,53 @@ export function environmentRuntimeService(
|
||||
return driver.isPendingCleanupWorkerReady(input);
|
||||
},
|
||||
|
||||
// Flush the in-process orphan-cleanup buffers of every driver that keeps one.
|
||||
// The cleanup sweep calls this each tick, so a buffered orphan lands a durable
|
||||
// `pending_cleanup` row once the database recovers, and the same sweep tears
|
||||
// it down. The dispatcher sums the recovered and pending counts across all
|
||||
// drivers. A driver with no buffer contributes nothing.
|
||||
// Recover committed creation operations before replaying legacy cleanup inputs.
|
||||
// New creation failures retain their database operation and do not make files.
|
||||
// Legacy buffers and files remain until their cleanup rows are durable.
|
||||
async flushDeferredOrphanCleanups(): Promise<{ recovered: number; pending: number }> {
|
||||
return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values())
|
||||
.flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : []));
|
||||
orphanRecoveryFlush ??= (async () => {
|
||||
await recoverSandboxCreations(db, async operation => {
|
||||
if (!operation.pluginId || operation.metadata.sandboxProviderPlugin !== true) return null;
|
||||
if (!options.pluginWorkerManager) throw new Error("Sandbox creation recovery worker is unavailable");
|
||||
const plugin = await pluginRegistryService(db).getById(operation.pluginId);
|
||||
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === operation.driverKey);
|
||||
if (driver?.creationOperationVersion !== 1) return null;
|
||||
if (!options.pluginWorkerManager.isRunning(operation.pluginId) ||
|
||||
!options.pluginWorkerManager.getWorker(operation.pluginId)?.supportedMethods.includes("environmentInspectLease")) {
|
||||
throw new Error("Sandbox creation recovery worker is unavailable");
|
||||
}
|
||||
let resolved: SandboxEnvironmentConfig;
|
||||
try {
|
||||
resolved = await resolveSandboxCleanupConfigSecrets(db, operation.companyId,
|
||||
operation.config as SandboxEnvironmentConfig, { issueId: operation.issueId });
|
||||
} catch (error) {
|
||||
// Missing or explicitly inactive saved credentials need an operator.
|
||||
// Temporary database/provider failures still retain retry ownership.
|
||||
const code = error instanceof HttpError ? (error.details as { code?: string } | undefined)?.code : undefined;
|
||||
if (error instanceof HttpError && (
|
||||
(error.status === 404 && (code === "secret_deleted" || code === "version_missing" ||
|
||||
(error.details === undefined && error.message === "Secret not found"))) ||
|
||||
(error.status === 422 && (code === "secret_inactive" || code === "version_inactive" ||
|
||||
(error.details === undefined && (error.message === "Provider vault is disabled" ||
|
||||
error.message === "Provider vault is locked while coming soon"))))
|
||||
)) return null;
|
||||
throw error;
|
||||
}
|
||||
const credentialHashes = operation.metadata.creationCredentialHashes as Record<string, string> | undefined;
|
||||
for (const [path, expected] of Object.entries(credentialHashes ?? {})) {
|
||||
const actual = readConfigValueAtPath({ ...resolved }, path);
|
||||
if (typeof actual !== "string" || createHash("sha256").update(actual).digest("hex") !== expected) return null;
|
||||
}
|
||||
return options.pluginWorkerManager.call(operation.pluginId, "environmentInspectLease", {
|
||||
driverKey: operation.driverKey, companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
|
||||
config: stripSandboxProviderEnvelope(resolved), providerLeaseId: operation.id,
|
||||
creationOperation: { id: operation.id, runId: operation.runId },
|
||||
});
|
||||
});
|
||||
return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values())
|
||||
.flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : []));
|
||||
})().finally(() => { orphanRecoveryFlush = null; });
|
||||
return orphanRecoveryFlush;
|
||||
},
|
||||
|
||||
async destroyReusableSandboxLeases(input: {
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { createSandboxOrphanCleanupSpool, type DeferredOrphanCleanupRecord } from "./sandbox-orphan-cleanup-spool.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { attachSandboxCreation, environmentService } from "./environments.js";
|
||||
import { createOwnedSandbox, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
||||
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
|
||||
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
||||
@@ -234,59 +232,38 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({runnerTaskExecutionStopped:true,
|
||||
runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
|
||||
const fresh = await workers.call(plugin.id, "environmentAcquireLease", {driverKey:metadata.driverKey,
|
||||
const replacementDriverKey = metadata.driverKey;
|
||||
const fresh = await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
|
||||
issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
|
||||
pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey, runnerTaskLease: true },
|
||||
}, identity => workers.call(plugin.id, "environmentAcquireLease", {driverKey:replacementDriverKey,
|
||||
companyId:input.companyId,environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,
|
||||
runId:run!.id,agentId:run!.agentId,adapterType:"paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})});
|
||||
runId:identity.runId,creationOperationId:identity.id,agentId:run!.agentId,adapterType:"paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})}));
|
||||
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
||||
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
|
||||
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
|
||||
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
|
||||
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
|
||||
try {
|
||||
{
|
||||
const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
|
||||
const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
|
||||
if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
|
||||
(deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
|
||||
throw new Error("Plugin replacement task expiry is not bounded");
|
||||
const replacement = await db.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
|
||||
metadata:replacementMetadata,
|
||||
await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
|
||||
const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
|
||||
metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
|
||||
}).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
|
||||
eq(environmentLeases.status,"active"))).returning({id:environmentLeases.id});
|
||||
if (replacement.length !== 1) throw new Error("Plugin task replacement ownership changed");
|
||||
} catch (error) {
|
||||
try {
|
||||
await workers.call(plugin.id,"environmentReleaseLease",{driverKey:metadata.driverKey,companyId:input.companyId,
|
||||
environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,providerLeaseId:fresh.providerLeaseId,
|
||||
leaseMetadata:replacementMetadata});
|
||||
} catch {
|
||||
const orphan:DeferredOrphanCleanupRecord={companyId:input.companyId,environmentId:environment!.id,
|
||||
executionWorkspaceId:lease.executionWorkspaceId,issueId:lease.issueId,heartbeatRunId:lease.heartbeatRunId,
|
||||
provider:lease.provider!,providerLeaseId:fresh.providerLeaseId,metadata:replacementMetadata};
|
||||
try {
|
||||
await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"});
|
||||
} catch {
|
||||
// The existing orphan spool recovers cleanup even when the database is unavailable.
|
||||
const spool = createSandboxOrphanCleanupSpool();
|
||||
const persisted = await spool.append(orphan);
|
||||
const finish = beginIdleTrackedWork();
|
||||
let flushing: Promise<{ recovered: number; pending: number }> | null = null;
|
||||
const unregister = registerDeferredEnvironmentCleanup(db, () => {
|
||||
flushing ??= (async () => {
|
||||
await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"});
|
||||
await spool.remove(orphan);
|
||||
unregister();
|
||||
finish();
|
||||
return { recovered: 1, pending: 0 };
|
||||
})().catch(error => { flushing = null; throw error; });
|
||||
// Keep successful completion for a concurrent sweep that already
|
||||
// captured this callback before it was unregistered.
|
||||
return flushing;
|
||||
});
|
||||
logger.error({providerLeaseId:fresh.providerLeaseId,persisted},"Plugin replacement cleanup requires recovery");
|
||||
}
|
||||
}
|
||||
throw error;
|
||||
if (!updated) throw new Error("Plugin task replacement ownership changed");
|
||||
return (await environmentService(tx).getLeaseById(updated.id))!;
|
||||
});
|
||||
|
||||
}
|
||||
return executeEnvironmentTask(db, workers, input);
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
companySecretBindings,
|
||||
environmentCustomImageSetupSessions,
|
||||
environmentLeases,
|
||||
sandboxCreationOperations,
|
||||
environments,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
@@ -48,6 +49,43 @@ import {
|
||||
type ManagedResourceStockStatus,
|
||||
} from "./managed-resource-drift.js";
|
||||
|
||||
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
|
||||
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
|
||||
return db.transaction(async tx => {
|
||||
await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
|
||||
const [operation] = await tx.select().from(sandboxCreationOperations)
|
||||
.where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update");
|
||||
if (scope.heartbeatRunId) {
|
||||
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"),
|
||||
)).for("update");
|
||||
if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed");
|
||||
}
|
||||
if (operation?.state === "attached" && operation.leaseId) {
|
||||
const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId);
|
||||
if (existing?.status === "active") return existing;
|
||||
}
|
||||
if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred");
|
||||
const lease = await attach(tx as unknown as Db);
|
||||
await notifyEnvironmentCleanup(tx, { companyId: scope.companyId });
|
||||
await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
|
||||
.where(eq(sandboxCreationOperations.id, id));
|
||||
return lease;
|
||||
});
|
||||
}
|
||||
|
||||
export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {
|
||||
const [environment] = await db.select({ id: environments.id }).from(environments)
|
||||
.where(eq(environments.id, environmentId)).for("update");
|
||||
if (!environment) throw new Error("Sandbox creation environment no longer exists");
|
||||
const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources)
|
||||
.where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId)));
|
||||
if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) {
|
||||
throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" });
|
||||
}
|
||||
}
|
||||
|
||||
type EnvironmentRow = typeof environments.$inferSelect;
|
||||
type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
|
||||
const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
|
||||
@@ -1428,6 +1466,7 @@ export function environmentService(db: Db) {
|
||||
},
|
||||
|
||||
acquireLease: async (input: {
|
||||
creationOperationId?: string;
|
||||
companyId: string;
|
||||
environmentId: string;
|
||||
executionWorkspaceId?: string | null;
|
||||
@@ -1465,6 +1504,11 @@ export function environmentService(db: Db) {
|
||||
*/
|
||||
assertCompanyBinding?: boolean;
|
||||
}): Promise<EnvironmentLease> => {
|
||||
if (input.creationOperationId) {
|
||||
const { creationOperationId, ...leaseInput } = input;
|
||||
return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId },
|
||||
tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true }));
|
||||
}
|
||||
const now = new Date();
|
||||
const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null;
|
||||
const values = {
|
||||
@@ -1515,48 +1559,7 @@ export function environmentService(db: Db) {
|
||||
companyId: input.companyId, runId: input.heartbeatRunId,
|
||||
});
|
||||
if (input.assertCompanyBinding) {
|
||||
// Lock the environment row first. Managed reconciliation locks the
|
||||
// same sandbox environment rows with `for update` before it writes a
|
||||
// company binding, so this lock serializes the two transactions on
|
||||
// this row and closes the time-of-check to time-of-use window.
|
||||
await tx
|
||||
.select({ id: environments.id })
|
||||
.from(environments)
|
||||
.where(eq(environments.id, input.environmentId))
|
||||
.for("update");
|
||||
// Re-read the company binding inside the locked transaction. A
|
||||
// binding a reconciliation committed after the route guard now
|
||||
// appears here. Reject a foreign-company environment before the
|
||||
// insert, so the login holds no lease.
|
||||
const boundRows = await tx
|
||||
.select({ companyId: builtInManagedResources.companyId })
|
||||
.from(builtInManagedResources)
|
||||
.where(
|
||||
and(
|
||||
eq(
|
||||
builtInManagedResources.resourceKind,
|
||||
MANAGED_ENVIRONMENT_RESOURCE_KIND,
|
||||
),
|
||||
eq(
|
||||
builtInManagedResources.resourceId,
|
||||
input.environmentId,
|
||||
),
|
||||
),
|
||||
);
|
||||
const boundCompanyIds = Array.from(
|
||||
new Set(boundRows.map((boundRow) => boundRow.companyId)),
|
||||
);
|
||||
if (
|
||||
boundCompanyIds.length > 0 &&
|
||||
!boundCompanyIds.includes(input.companyId)
|
||||
) {
|
||||
throw forbidden(
|
||||
"The selected environment belongs to another company.",
|
||||
{
|
||||
code: "environment_company_mismatch",
|
||||
},
|
||||
);
|
||||
}
|
||||
await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId);
|
||||
}
|
||||
if (input.replacesReusableLeaseId) {
|
||||
await notifyChatInboundWakeNotices(tx, { companyId: input.companyId });
|
||||
@@ -1699,6 +1702,7 @@ export function environmentService(db: Db) {
|
||||
const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId,
|
||||
heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status,
|
||||
cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt,
|
||||
creationOperationId: sql<string | null>`${environmentLeases.metadata}->>'creationOperationId'`,
|
||||
}).from(environmentLeases).where(condition).for("update");
|
||||
if (!previous) return null;
|
||||
await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined });
|
||||
@@ -1715,6 +1719,11 @@ export function environmentService(db: Db) {
|
||||
companyId: previous.companyId, runId: previous.heartbeatRunId,
|
||||
});
|
||||
}
|
||||
if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
|
||||
await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
|
||||
.where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId),
|
||||
eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending")));
|
||||
}
|
||||
return tx.update(environmentLeases)
|
||||
.set({
|
||||
status,
|
||||
|
||||
@@ -54,6 +54,7 @@ const WORK_CHECKS = [
|
||||
state NOT IN ('saved', 'unchanged', 'resolved', 'unavailable', 'conflict')
|
||||
OR process_stopped_at IS NULL OR next_attempt_at IS NOT NULL
|
||||
OR receipt ? 'baseline' OR receipt->>'cleanupPending' = 'true'`,
|
||||
`SELECT 1 FROM sandbox_creation_operations WHERE state = 'pending'`,
|
||||
`SELECT 1 FROM budget_reservations WHERE state NOT IN ('settled', 'released')`,
|
||||
`SELECT 1 FROM budget_policies WHERE enforcement_version > enforcement_delivered_version`,
|
||||
`SELECT 1 FROM agents WHERE lifecycle_state IN
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
import { collectSecretRefPaths, isUuidSecretRef, parseSecretRefBindingObject, readConfigValueAtPath } from "./json-schema-secret-refs.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { and, eq, lte, or, sql } from "drizzle-orm";
|
||||
import { companies, environments, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk";
|
||||
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
|
||||
import { ENVIRONMENT_CLEANUP_RETRY_MS } from "./environment-cleanup-state.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
|
||||
|
||||
type CreationOwner = {
|
||||
companyId: string;
|
||||
environmentId: string;
|
||||
executionWorkspaceId?: string | null;
|
||||
issueId?: string | null;
|
||||
heartbeatRunId?: string | null;
|
||||
provider: string;
|
||||
pluginId?: string;
|
||||
driverKey: string;
|
||||
config: Record<string, unknown>;
|
||||
metadata: Record<string, unknown>;
|
||||
timeoutMs?: number;
|
||||
requestKey?: string;
|
||||
configSchema?: Record<string, unknown> | null;
|
||||
resolvedConfig?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
export async function createOwnedSandbox(
|
||||
db: Db, owner: CreationOwner,
|
||||
create: (identity: { id: string; runId: string }) => Promise<PluginEnvironmentLease>,
|
||||
): Promise<PluginEnvironmentLease> {
|
||||
const credentialHashes: Record<string, string> = {};
|
||||
for (const path of collectSecretRefPaths(owner.configSchema)) {
|
||||
const value = readConfigValueAtPath(owner.config, path);
|
||||
if (value != null && value !== "" && !(typeof value === "string" && isUuidSecretRef(value)) && !parseSecretRefBindingObject(value)) {
|
||||
throw new Error("Sandbox creation requires saved secret references");
|
||||
}
|
||||
if (value != null && value !== "") {
|
||||
const resolved = readConfigValueAtPath(owner.resolvedConfig ?? {}, path);
|
||||
if (typeof resolved !== "string") throw new Error("Sandbox creation credential identity is unavailable");
|
||||
credentialHashes[path] = createHash("sha256").update(resolved).digest("hex");
|
||||
}
|
||||
}
|
||||
const id = randomUUID();
|
||||
const runId = owner.heartbeatRunId ?? randomUUID();
|
||||
const operation = await db.transaction(async tx => {
|
||||
await notifyEnvironmentCleanup(tx, { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined });
|
||||
await assertEnvironmentCompanyBinding(tx as unknown as Db, owner.environmentId, owner.companyId);
|
||||
const [inserted] = await tx.insert(sandboxCreationOperations).values({
|
||||
id, companyId: owner.companyId, environmentId: owner.environmentId, originalEnvironmentId: owner.environmentId,
|
||||
runId, requestKey: owner.requestKey ?? "run", issueId: owner.issueId, executionWorkspaceId: owner.executionWorkspaceId,
|
||||
provider: owner.provider, pluginId: owner.pluginId, driverKey: owner.driverKey,
|
||||
config: owner.config, metadata: { ...owner.metadata, creationCredentialHashes: credentialHashes },
|
||||
nextAttemptAt: new Date(Date.now() + Math.max(30 * 60_000, (owner.timeoutMs ?? 0) + 60_000)),
|
||||
}).onConflictDoNothing().returning();
|
||||
if (inserted) return inserted;
|
||||
const [existing] = await tx.select().from(sandboxCreationOperations).where(and(
|
||||
eq(sandboxCreationOperations.companyId, owner.companyId), eq(sandboxCreationOperations.originalEnvironmentId, owner.environmentId),
|
||||
eq(sandboxCreationOperations.runId, runId), eq(sandboxCreationOperations.requestKey, owner.requestKey ?? "run"),
|
||||
));
|
||||
if (!existing) throw new Error("Sandbox creation ownership was not recorded");
|
||||
return existing;
|
||||
});
|
||||
if (operation.id !== id) {
|
||||
if (operation.state !== "attached" || !operation.leaseId) throw new Error("Sandbox creation requires reconciliation before another attempt");
|
||||
const existing = await environmentService(db).getLeaseById(operation.leaseId);
|
||||
if (!existing || existing.status !== "active" || !existing.providerLeaseId) throw new Error("Sandbox creation has already transferred ownership");
|
||||
return { providerLeaseId: existing.providerLeaseId, expiresAt: existing.expiresAt?.toISOString(),
|
||||
metadata: { ...existing.metadata, creationOperationId: operation.id } };
|
||||
}
|
||||
const lease = await create({ id, runId });
|
||||
return { ...lease, metadata: { ...lease.metadata, creationOperationId: id } };
|
||||
}
|
||||
|
||||
/** Keep an already returned resource identity even when task attachment fails.
|
||||
* This enriches the existing owner; it never transfers or reopens ownership. */
|
||||
export async function recordSandboxCreationResource(db: Db, input: {
|
||||
id: string; companyId: string; environmentId: string; providerLeaseId: string;
|
||||
}): Promise<void> {
|
||||
const [recorded] = await db.update(sandboxCreationOperations).set({
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
|
||||
or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
|
||||
)).returning({ id: sandboxCreationOperations.id });
|
||||
if (!recorded) throw new Error("Sandbox creation resource identity changed");
|
||||
}
|
||||
|
||||
export async function recoverSandboxCreations(db: Db,
|
||||
inspect: (operation: SandboxCreationOperation) => Promise<PluginEnvironmentLeaseObservation | null>,
|
||||
): Promise<void> {
|
||||
const due = await db.select().from(sandboxCreationOperations).where(and(
|
||||
eq(sandboxCreationOperations.state, "pending"), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
|
||||
)).limit(100);
|
||||
for (const candidate of due) {
|
||||
try {
|
||||
const [existingCompany] = await db.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).limit(1);
|
||||
const observation = existingCompany ? await inspect(candidate) : null;
|
||||
const cleanup = observation?.creationCleanup ? readEnvironmentCreationCleanupError({ data: {
|
||||
schema: "paperclip/environment-creation-cleanup/v1", cleanup: observation.creationCleanup,
|
||||
} }) : null;
|
||||
if (cleanup && (cleanup.companyId !== candidate.companyId || cleanup.environmentId !== candidate.originalEnvironmentId ||
|
||||
cleanup.runId !== candidate.runId || cleanup.attemptId !== candidate.id || cleanup.providerLeaseId !== observation!.providerLeaseId)) {
|
||||
throw new Error("Sandbox creation recovery returned foreign ownership");
|
||||
}
|
||||
await db.transaction(async tx => {
|
||||
const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share");
|
||||
await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, candidate.originalEnvironmentId)).for("update");
|
||||
const [operation] = await tx.select().from(sandboxCreationOperations)
|
||||
.where(eq(sandboxCreationOperations.id, candidate.id)).for("update");
|
||||
if (!operation || operation.state !== "pending") return;
|
||||
await notifyEnvironmentCleanup(tx, { companyId: operation.companyId });
|
||||
if (!company || !cleanup) {
|
||||
logger.warn({ errorKind: "sandbox_creation_needs_reconciliation", operationId: operation.id, companyId: operation.companyId },
|
||||
"Sandbox creation could not be identified; operator reconciliation is required");
|
||||
await tx.update(sandboxCreationOperations).set({ state: "needs_reconciliation", updatedAt: new Date(),
|
||||
metadata: { ...operation.metadata, creationReconciliationReason: company ? "creation_identity_unavailable" : "company_deleted" },
|
||||
})
|
||||
.where(eq(sandboxCreationOperations.id, operation.id));
|
||||
return;
|
||||
}
|
||||
const lease = await environmentService(tx as unknown as Db).insertPendingCleanupLease({
|
||||
companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
|
||||
provider: operation.provider, providerLeaseId: cleanup.providerLeaseId,
|
||||
metadata: { ...operation.metadata, creationOperationId: operation.id, failedCreateCleanup: cleanup },
|
||||
failureReason: "interrupted_sandbox_creation",
|
||||
});
|
||||
await tx.update(sandboxCreationOperations).set({ state: "cleanup_pending", leaseId: lease.id, updatedAt: new Date() })
|
||||
.where(eq(sandboxCreationOperations.id, operation.id));
|
||||
});
|
||||
} catch {
|
||||
const retry = await db.transaction(async tx => {
|
||||
const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share");
|
||||
await notifyEnvironmentCleanup(tx, { companyId: candidate.companyId });
|
||||
await tx.update(sandboxCreationOperations).set(company
|
||||
? { nextAttemptAt: new Date(Date.now() + ENVIRONMENT_CLEANUP_RETRY_MS), updatedAt: new Date() }
|
||||
: { state: "needs_reconciliation", updatedAt: new Date(),
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || '{"creationReconciliationReason":"company_deleted"}'::jsonb` })
|
||||
.where(and(eq(sandboxCreationOperations.id, candidate.id), eq(sandboxCreationOperations.state, "pending")));
|
||||
return Boolean(company);
|
||||
});
|
||||
logger.warn({ errorKind: retry ? "sandbox_creation_recovery_failed" : "sandbox_creation_needs_reconciliation", operationId: candidate.id, companyId: candidate.companyId },
|
||||
retry ? "Sandbox creation recovery will retry at its saved deadline" : "Sandbox creation company was deleted; operator reconciliation is required");
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user