Merge master and preserve controlled shared storage fixture

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/DATABASE.md

@@ -1545,6 +1545,23 @@ named `review_wait_benchmark...` can be supplied through
 `PAPERCLIP_TEST_DATABASE_URL` to compare another PostgreSQL major version.
 DataRow bytes exclude response metadata and transport overhead.
 
+
+### Sandbox creation ownership
+
+Migration 0418 adds `sandbox_creation_operations`. Admission commits company,
+original environment, run, request key and provider identity before external
+creation. Configuration contains nonsecret values and existing secret references.
+The original identity survives environment, run, task and company deletion; only
+the optional live environment FK uses SET NULL. Company/run deletion must not
+erase an outstanding remote-resource obligation.
+
+The unique company/environment/run/request index fences duplicate creation and
+ambiguous successful lease commits. Normal lease attachment and operation transfer
+share a transaction. Recovery transfers one claimed cleanup lease after provider
+ownership lookup. The pending deadline index drives the existing cleanup worker.
+`needs_reconciliation` is a durable manual outcome, not an automatic retry or an
+idle blocker. See [cleanup ownership](idle-environment-cleanup.md).
+
 ## Object-backed run evidence
 
 Optional segmented run logs and raw provider traces store their payloads and

doc/DEVELOPING.md

@@ -903,6 +903,32 @@ npx paperclipai allowed-hostname dotta-macbook-pro
 
 ## Agent status recovery
 
+Lifecycle verification stops automatic attempts after missing authentication,
+an unavailable model or command, an invalid working directory, or unexpected
+probe output. A generic failed test gets up to three failures before automatic
+checks stop. The saved `verificationFailures` count only advances when the host
+verification test fails; claims deferred before the test do not spend that budget.
+An explicit retry or execution configuration change resets that count.
+The agent stays in `verifying`
+with execution paused and a safe error message. Its saved lifecycle operation
+records `retryRequired`; process restarts and queue scans retain that wait.
+Changing execution configuration or using **Retry now** clears it and runs the
+normal verification again. Name and title edits do not restart verification.
+Timeouts, unavailable test environments, plugin outages and unclassified runtime
+failures retain automatic retry. Pause, termination, resource cleanup and idle
+sleep checks keep their existing safety gates.
+
+Remote directory and command checks preserve timeout, interruption and transport
+failures as typed execution-target failures. Adapters carry that signal in the
+optional `AdapterEnvironmentCheck.retryable` field. A retryable error becomes
+`test_environment_unavailable` before configuration classification, so it neither
+parks verification nor consumes the generic failed-test budget. A completed check
+that confirms a missing directory or command still waits for configuration repair.
+
+The retry persistence tests can use a dedicated local PostgreSQL database ending
+in `_test` through `PAPERCLIP_LIFECYCLE_VERIFICATION_RETRY_TEST_DATABASE_URL` when
+embedded PostgreSQL is unavailable. The test validates that URL before use.
+
 Heartbeat completion locks the agent row before it reads remaining running work
 and writes the agent status. A concurrent execution start cannot be overwritten
 by a completion that read older run state. Run terminalization and agent status

doc/idle-environment-cleanup.md

@@ -11,7 +11,7 @@ or an orphaned active lease. Explicit user retry keeps its existing direct path.
 | Duty | Truth | Trigger and next deadline | Quiet condition |
 | --- | --- | --- | --- |
 | Failed teardown | Existing `environment_leases` rows in `pending_cleanup` | Commit signal; later of row backoff and saved retry/ownership deadline | No pending lease or deferred orphan |
-| Failed acquisition before a DB row exists | Existing local orphan spool and the producing runtime's buffer | Local buffer registration; five-minute retry while buffered; startup spool load | Durable row owns recovery and spool copy is removed |
+| Interrupted acquisition | `sandbox_creation_operations` committed before provider creation | Transaction signal; saved acquisition deadline; five-minute backoff after recovery failure | Normal lease or one claimed cleanup lease owns the resource; unsupported uncertainty requires operator reconciliation |
 | Historical provider verification | Existing failed ephemeral leases and finished custom-image setups without matching cleanup proof | Commit signal; five-minute retry while an unverified candidate remains | No unverified candidates, including retained verified history |
 
 `environment-cleanup-state.ts` owns the deadline query and the shared safe reader
@@ -47,8 +47,9 @@ settlement and uncertainty handling through the shared DB implementation.
 - `heartbeat/recovery.ts` and `heartbeat/run-control.ts`: orphan-active recovery,
   retry settlement, and explicit recovery of historical leases.
 - `environment-task-runtime.ts`: unconfirmed plugin task termination and plugin
-  replacement orphans, including a registered deferred flusher when the row
-  cannot be written. The flusher retains an idle hold until the row is durable.
+  replacement creations. Replacement intent commits before the provider call.
+- `sandbox-creation-operations.ts`: admission, atomic lease transfer, and bounded
+  recovery. An unavailable database stops admission before external creation.
 - `sandbox-stop-and-retain.ts`: persisted Stop intent and its settlement.
 - `native-workspace-export-resume.ts` and `native-workspace-export-retry.ts`:
   terminal-export cleanup, resumed-export cleanup claims, and settlement.
@@ -64,7 +65,49 @@ already pending claim remain covered by its existing deadline. Deletion removes
 work; an already scheduled reconciliation observes that removal. Independent SQL
 writers and another server process are not covered by these process-local signals.
 
-## Deferred orphan buffers
+## Creation ownership
+
+A creation operation retains company, original environment, run, request identity,
+provider, and nonsecret configuration references. Its environment FK uses SET NULL;
+company/run/task identifiers remain after those records are removed. One operation
+owns each company/environment/run/request. A retry of an attached operation returns
+its active lease. Pending or unresolved ownership never authorizes another create.
+Normal success transfers operation ownership and the lease in one transaction.
+Current company binding and run authority are checked again at attachment.
+Declared secret-reference fields retain only their references and nonsecret value
+hashes. Recovery refuses lookup under a changed credential value; it records manual
+reconciliation instead of treating a missing resource in a new account as proof.
+A missing, deleted, disabled, or archived saved secret, a missing or inactive saved version,
+or an explicitly disabled/coming-soon provider vault also records manual
+reconciliation before provider lookup. Admin-disabled credentials require operator
+reconciliation; recovery does not reactivate or use them. Temporary secret-provider
+or database failures retain the pending operation and its retry deadline.
+
+Recovery never resumes partially prepared execution. It asks a capable provider to
+inspect the stable creation identity, validates its ownership envelope, and transfers
+one cleanup lease under the operation lock. The existing claimed cleanup worker
+owns deletion and confirms resolution. Failed lookup/transfer records a bounded
+retry and permits other operations to proceed. An attached operation cannot transfer
+to cleanup. Pending creation blocks instance sleep until its obligation transfers.
+
+Daytona version 0.1.10 declares creationOperationVersion 1. It uses the committed
+operation UUID in its existing creation name and labels. Recovery uses an explicit
+creationOperation request on the existing read-only inspect hook and strict failed-creation deletion path. Normal lease
+inspection still uses its recorded resource ID; creationOperationId metadata is
+provenance only. A missing name alone cannot prove a late allocation will never appear. Missing identity or an
+unsupported provider records needs_reconciliation and an operation-scoped warning.
+A deleted company also records manual reconciliation before credential lookup or
+cleanup transfer. A company lock fences deletion during the transfer transaction.
+That manual diagnostic has no automatic timer and does not prevent idle. The row
+must be reconciled before retrying the same request. Other providers do not yet
+claim automatic creation lookup. Generic plugin operations retain identity only, including a returned task resource
+ID before expiry checks or lease attachment. A failed or ambiguous attachment keeps
+that known ID for reconciliation; it does not authorize unconditional release.
+These operations do not copy credentials, resolved configuration, or provider
+payloads into the operation record. A lost provider reply can still leave the
+resource ID unknown.
+
+## Legacy deferred orphan buffers
 
 Runtimes that hold an unwritten orphan register their flusher in a registry scoped
 to the shared database handle. Registration wakes the worker without contacting
@@ -75,8 +118,9 @@ runtime. The registry retains only runtimes with unresolved buffer ownership.
 Unreadable directories, records, and malformed records retain a retry; a missing
 spool directory is an empty spool. A spool read failure still allows other runtimes to flush their buffers and
 persisted leases to clean up, then fails the sweep so recovery stays scheduled.
-Each concrete cleanup pass reloads the spool, including entries appended by
-plugin task replacement after startup and entries beyond buffer capacity.
+Each concrete cleanup pass reloads existing legacy spool entries, including entries
+beyond buffer capacity. Normal acquisition and plugin task replacement do not append
+new fallback files. Retained entries remain available for the separate cutover inventory.
 No empty-spool filesystem or database polling is introduced. Existing limits on a simultaneous database,
 spool, and buffer-capacity failure remain; this is not a new durable queue.
 

doc/skill-ephemeral-web-servers.md

@@ -131,6 +131,13 @@ perform provider migration: `local_storage` and `provider_mismatch` are blocking
 results for cloud qualification, not instructions to switch configuration and retry.
 A directory of extracted caches is never a durable object backend.
 
+The maintenance CLI reads the existing storage configuration directly so it can
+verify legacy local-disk objects before cutover. On Cloud it still requires the
+persisted runtime configuration and never seeds or changes that configuration.
+This maintenance boundary does not start a server or relax Cloud server startup
+requirements for external PostgreSQL, shared object storage and a stable secret
+key. Apply operations retain their existing stopped-writer gates.
+
 Conversion establishes the rollback floor. Afterward only compatible version-reading
 binaries may overlap or be rolled back against the same remote state. Reattaching an
 old volume does not roll back committed content. Filesystem-authoritative binaries

maps/agent-access-map/agents.md

@@ -270,6 +270,10 @@ Policy: Agent creation/configuration uses action grants, resource policies and c
 consent; some controls require a board actor.
 Permanent deletion is board-only. Retained case handoff references return
 `409 agent_case_history_retained` before dependency cleanup; keep the agent terminated.
+Failed configuration verification waits for a changed execution configuration
+or an authorized explicit retry. That wait does not grant an agent permission
+to retry itself, resume execution, or change configuration. Temporary verification
+outages retain automatic recovery.
 
 Evidence: `source-only` — [implementation](../../server/src/routes/agents.ts#L1395) at
 the reviewed commit above.

maps/feature-map/agents.md

@@ -215,7 +215,9 @@ Manual: Save a harmless instruction edit, reload, and verify its revision and ef
 
 For failed setup, use an unavailable model or invalid command on a disposable agent.
 Confirm the profile names the failed verification step and gives a relevant action.
-Check that it explains automatic retry and offers **Retry now**. Fix the configuration,
+Check that a failed configuration test explains that automatic verification is paused
+and offers **Retry now**. Repeated refreshes must not start another test. Timeouts
+and temporary test-environment failures still retry automatically. Fix the configuration,
 retry, and reload; the failure must disappear only after readiness commits.
 Provider output and credentials must not appear in the lifecycle message.
 

packages/adapter-utils/src/execution-target.test.ts

@@ -8,6 +8,9 @@ import {
   prepareGitHubOperationLaunchers,
   adapterExecutionTargetUsesManagedHome,
   ensureAdapterExecutionTargetRuntimeCommandInstalled,
+  ensureAdapterExecutionTargetDirectory,
+  ensureAdapterExecutionTargetCommandResolvable,
+  isAdapterExecutionTargetUnavailableError,
   resolveAdapterExecutionTargetCwd,
   runAdapterExecutionTargetProcess,
   runAdapterExecutionTargetShellCommand,
@@ -32,6 +35,58 @@ function sshExecutionTarget(): AdapterSshExecutionTarget {
   };
 }
 
+describe("execution target validation failures", () => {
+  afterEach(() => vi.restoreAllMocks());
+  const result = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "", pid: null, startedAt: "" };
+
+  it.each(["directory", "command"] as const)("preserves temporary %s probe failures", async check => {
+    for (const failure of ["timeout", "transport", "interrupted"] as const) {
+      const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", runner: {
+        execute: vi.fn(async () => {
+          if (failure === "transport") throw Object.assign(new Error("private transport detail"), { code: "ECONNRESET" });
+          return { ...result, exitCode: null, timedOut: failure === "timeout", signal: failure === "interrupted" ? "SIGTERM" : null };
+        }),
+      } };
+      const work = check === "directory"
+        ? ensureAdapterExecutionTargetDirectory("test", target, "/workspace", { cwd: "/workspace", env: {} })
+        : ensureAdapterExecutionTargetCommandResolvable("opencode", target, "/workspace", {});
+      const error = await work.catch(error => error);
+      expect(isAdapterExecutionTargetUnavailableError(error)).toBe(true);
+    }
+  });
+
+  it("preserves SSH disconnections instead of reporting a missing directory", async () => {
+    vi.spyOn(ssh, "runSshCommand").mockRejectedValue(Object.assign(new Error("private SSH detail"), { code: 255 }));
+    const error = await ensureAdapterExecutionTargetDirectory("test", sshExecutionTarget(), "/workspace", {
+      cwd: "/workspace", env: {},
+    }).catch(error => error);
+    expect(isAdapterExecutionTargetUnavailableError(error)).toBe(true);
+  });
+
+  it.each(["directory", "command"] as const)("retains a confirmed missing %s as a configuration failure", async check => {
+    const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", runner: {
+      execute: vi.fn(async () => ({ ...result, exitCode: 1 })),
+    } };
+    const error = await (check === "directory"
+      ? ensureAdapterExecutionTargetDirectory("test", target, "/workspace", { cwd: "/workspace", env: {} })
+      : ensureAdapterExecutionTargetCommandResolvable("opencode", target, "/workspace", {})).catch(error => error);
+    expect(error).toBeInstanceOf(Error);
+    expect(isAdapterExecutionTargetUnavailableError(error)).toBe(false);
+  });
+
+  it("does not turn an installation timeout into a permanent missing command", async () => {
+    const execute = vi.fn().mockResolvedValueOnce({ ...result, exitCode: 1 })
+      .mockResolvedValueOnce({ ...result, exitCode: null, timedOut: true })
+      .mockResolvedValueOnce({ ...result, exitCode: 1 });
+    const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", runner: { execute } };
+    const error = await ensureAdapterExecutionTargetCommandResolvable("opencode", target, "/workspace", {}, {
+      installCommand: "install-fixture",
+    }).catch(error => error);
+    expect(execute).toHaveBeenCalledTimes(3);
+    expect(isAdapterExecutionTargetUnavailableError(error)).toBe(true);
+  });
+});
+
 describe("runAdapterExecutionTargetShellCommand", () => {
   afterEach(() => {
     vi.restoreAllMocks();

packages/adapter-utils/src/execution-target.ts

@@ -735,23 +735,53 @@ export async function ensureAdapterExecutionTargetCommandResolvable(
   });
 }
 
+function executionTargetCheckUnavailable(message: string, cause?: unknown): Error {
+  return Object.assign(new Error(message, { cause }), { code: "adapter_execution_target_unavailable" });
+}
+
+/** Preserve target failures before an adapter translates them into a failed check. */
+export function isAdapterExecutionTargetUnavailableError(error: unknown): boolean {
+  return typeof error === "object" && error !== null && "code" in error
+    && error.code === "adapter_execution_target_unavailable";
+}
+
+async function runRemoteExecutionTargetCheck(
+  target: AdapterExecutionTarget,
+  description: string,
+  check: () => Promise<RunProcessResult>,
+): Promise<RunProcessResult> {
+  let result: RunProcessResult;
+  try {
+    result = await check();
+  } catch (error) {
+    throw executionTargetCheckUnavailable(`Could not complete ${description}`, error);
+  }
+  if (result.timedOut || result.signal || result.exitCode === null
+    || (target.kind === "remote" && target.transport === "ssh" && result.exitCode === 255)) {
+    throw executionTargetCheckUnavailable(`${result.timedOut ? "Timed out" : "Could not complete"} ${description}`);
+  }
+  return result;
+}
+
 async function probeSandboxCommandResolvable(
   command: string,
   target: AdapterSandboxExecutionTarget,
   env: Record<string, string>,
-): Promise<{ resolved: boolean; timedOut: boolean; stderr: string }> {
+): Promise<{ resolved: boolean; stderr: string }> {
   const runner = requireSandboxRunner(target);
   const probeScript = `command -v ${shellQuote(command)}`;
-  const result = await runner.execute({
-    command: "sh",
-    args: ["-c", probeScript],
-    cwd: target.remoteCwd,
-    env,
-    timeoutMs: target.timeoutMs ?? 15_000,
-  });
+  const result = await runRemoteExecutionTargetCheck(
+    target, `checking command "${command}" on sandbox target.`,
+    () => runner.execute({
+      command: "sh",
+      args: ["-c", probeScript],
+      cwd: target.remoteCwd,
+      env,
+      timeoutMs: target.timeoutMs ?? 15_000,
+    }),
+  );
   return {
-    resolved: !result.timedOut && (result.exitCode ?? 1) === 0,
-    timedOut: result.timedOut,
+    resolved: result.exitCode === 0,
     stderr: result.stderr.trim(),
   };
 }
@@ -772,15 +802,13 @@ async function ensureSandboxCommandResolvable(
   // SSH's buildSshSpawnTarget) so this and the hello probe agree on PATH.
   let probe = await probeSandboxCommandResolvable(command, target, env);
   if (probe.resolved) return;
-  if (probe.timedOut) {
-    throw new Error(`Timed out checking command "${command}" on sandbox target.`);
-  }
 
   // If the caller supplied an install command, attempt the install once via
   // the sandbox runner (which the sandbox provider wraps in a login shell)
   // and re-probe before reporting failure. This lets fresh sandbox leases
   // bring up the CLI before the resolvability gate, mirroring the test path.
   let installFailureDetail: string | null = null;
+  let installUnavailable: Error | null = null;
   if (installCommand) {
     const runner = requireSandboxRunner(target);
     const installTimeoutMs =
@@ -797,6 +825,9 @@ async function ensureSandboxCommandResolvable(
       });
       if (installResult.timedOut) {
         installFailureDetail = `install command timed out: ${installCommand}`;
+        installUnavailable = executionTargetCheckUnavailable("The runtime installation timed out.");
+      } else if (installResult.signal || installResult.exitCode === null) {
+        installUnavailable = executionTargetCheckUnavailable("The runtime installation did not complete.");
       } else if ((installResult.exitCode ?? 0) !== 0) {
         const tail = (text: string) =>
           text.split(/\r?\n/).filter((line) => line.trim().length > 0).slice(-2).join(" | ").slice(0, 240);
@@ -805,12 +836,11 @@ async function ensureSandboxCommandResolvable(
       }
     } catch (err) {
       installFailureDetail = `install command threw: ${err instanceof Error ? err.message : String(err)}`;
+      installUnavailable = executionTargetCheckUnavailable("The runtime installation was unavailable.", err);
     }
     probe = await probeSandboxCommandResolvable(command, target, env);
     if (probe.resolved) return;
-    if (probe.timedOut) {
-      throw new Error(`Timed out checking command "${command}" on sandbox target.`);
-    }
+    if (installUnavailable) throw installUnavailable;
   }
 
   const probeStderr = probe.stderr.length > 0 ? ` probe stderr: ${probe.stderr}` : "";
@@ -1270,7 +1300,8 @@ export async function ensureAdapterExecutionTargetFile(
  * `mkdir -p` (when allowed) followed by a `[ -d ]` check so the result reflects
  * the directory state inside the environment, not on the Paperclip host.
  *
- * Throws an Error with a human-readable message on failure.
+ * Throws an Error with a human-readable message on failure. Temporary target
+ * failures carry a typed code so adapters can retain them as retryable checks.
  */
 export async function ensureAdapterExecutionTargetDirectory(
   runId: string,
@@ -1296,17 +1327,16 @@ export async function ensureAdapterExecutionTargetDirectory(
     ? `mkdir -p ${quoted} && [ -d ${quoted} ]`
     : `[ -d ${quoted} ]`;
 
-  const result = await runAdapterExecutionTargetShellCommand(runId, target, script, {
-    cwd: target.kind === "remote" ? target.remoteCwd : cwd,
-    env: options.env,
-    timeoutSec: options.timeoutSec ?? 15,
-    graceSec: options.graceSec ?? 5,
-    onLog: options.onLog,
-  });
-
-  if (result.timedOut) {
-    throw new Error(`Timed out checking working directory on remote target: "${cwd}"`);
-  }
+  const result = await runRemoteExecutionTargetCheck(
+    target, `checking working directory on remote target: "${cwd}"`,
+    () => runAdapterExecutionTargetShellCommand(runId, target, script, {
+      cwd: target.remoteCwd,
+      env: options.env,
+      timeoutSec: options.timeoutSec ?? 15,
+      graceSec: options.graceSec ?? 5,
+      onLog: options.onLog,
+    }),
+  );
   if ((result.exitCode ?? 1) !== 0) {
     const detail = (result.stderr || result.stdout || "").trim();
     if (createIfMissing) {

packages/adapter-utils/src/types.ts

@@ -308,6 +308,8 @@ export interface AdapterEnvironmentCheck {
   message: string;
   detail?: string | null;
   hint?: string | null;
+  /** The check could not complete because its execution target was unavailable. */
+  retryable?: boolean;
 }
 
 export type AdapterEnvironmentTestStatus = "pass" | "warn" | "fail";

packages/adapters/claude-local/src/server/test.ts

@@ -14,6 +14,7 @@ import {
 import {
   ensureAdapterExecutionTargetCommandResolvable,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   runAdapterExecutionTargetProcess,
   resolveAdapterExecutionTargetCwd,
   resolveAdapterExecutionTargetCommandForLogs,
@@ -114,6 +115,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "claude_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -168,6 +170,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "claude_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/codex-local/src/server/test.ts

@@ -11,6 +11,7 @@ import {
 import {
   ensureAdapterExecutionTargetCommandResolvable,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   maybeRunSandboxInstallCommand,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
@@ -308,6 +309,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "codex_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -339,6 +341,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "codex_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/cursor-local/src/server/test.ts

@@ -13,6 +13,7 @@ import {
 import {
   ensureAdapterExecutionTargetCommandResolvable,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   maybeRunSandboxInstallCommand,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
@@ -119,6 +120,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "cursor_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -173,6 +175,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "cursor_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/gemini-local/src/server/test.ts

@@ -16,6 +16,7 @@ import {
   ensureAdapterExecutionTargetCommandResolvable,
   maybeRunSandboxInstallCommand,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
   resolveAdapterExecutionTargetCwd,
@@ -107,6 +108,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "gemini_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -141,6 +143,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "gemini_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/grok-local/src/server/test.ts

@@ -15,6 +15,7 @@ import {
   prepareAdapterExecutionTargetRuntime,
   ensureAdapterExecutionTargetCommandResolvable,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   resolveAdapterExecutionTargetCwd,
   runAdapterExecutionTargetProcess,
 } from "@paperclipai/adapter-utils/execution-target";
@@ -144,6 +145,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "grok_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -204,6 +206,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "grok_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/kimi-local/src/server/test.ts

@@ -17,6 +17,7 @@ import {
   ensureAdapterExecutionTargetCommandResolvable,
   maybeRunSandboxInstallCommand,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
   resolveAdapterExecutionTargetCwd,
@@ -156,6 +157,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "kimi_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -187,6 +189,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "kimi_command_unresolvable",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Command is not executable",
       detail: command,

packages/adapters/opencode-local/src/server/test.ts

@@ -19,6 +19,7 @@ import {
   ensureAdapterExecutionTargetCommandResolvable,
   maybeRunSandboxInstallCommand,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
   resolveAdapterExecutionTargetCwd,
@@ -102,6 +103,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "opencode_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -219,6 +221,7 @@ export async function testEnvironment(
       } catch (err) {
         checks.push({
           code: "opencode_command_unresolvable",
+          ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
           level: "error",
           message: err instanceof Error ? err.message : "Command is not executable",
           detail: command,

packages/adapters/pi-local/src/server/test.ts

@@ -16,6 +16,7 @@ import {
   ensureAdapterExecutionTargetCommandResolvable,
   maybeRunSandboxInstallCommand,
   ensureAdapterExecutionTargetDirectory,
+  isAdapterExecutionTargetUnavailableError,
   runAdapterExecutionTargetProcess,
   describeAdapterExecutionTarget,
   resolveAdapterExecutionTargetCwd,
@@ -115,6 +116,7 @@ export async function testEnvironment(
   } catch (err) {
     checks.push({
       code: "pi_cwd_invalid",
+      ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
       level: "error",
       message: err instanceof Error ? err.message : "Invalid working directory",
       detail: cwd,
@@ -156,6 +158,7 @@ export async function testEnvironment(
     } catch (err) {
       checks.push({
         code: "pi_command_unresolvable",
+        ...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
         level: "error",
         message: err instanceof Error ? err.message : "Command is not executable",
         detail: command,

packages/db/src/migrations/0418_curved_lady_mastermind.sql

@@ -0,0 +1,33 @@
+CREATE TABLE IF NOT EXISTS "sandbox_creation_operations" (
+	"id" uuid PRIMARY KEY NOT NULL,
+	"company_id" uuid NOT NULL,
+	"environment_id" uuid,
+	"original_environment_id" uuid NOT NULL,
+	"run_id" uuid NOT NULL,
+	"request_key" text DEFAULT 'run' NOT NULL,
+	"issue_id" uuid,
+	"execution_workspace_id" uuid,
+	"provider" text NOT NULL,
+	"plugin_id" uuid,
+	"driver_key" text NOT NULL,
+	"state" text DEFAULT 'pending' NOT NULL,
+	"config" jsonb NOT NULL,
+	"metadata" jsonb NOT NULL,
+	"lease_id" uuid,
+	"next_attempt_at" timestamp with time zone NOT NULL,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+DO $$ BEGIN
+  IF NOT EXISTS (
+    SELECT 1 FROM pg_constraint
+    WHERE conname = 'sandbox_creation_operations_environment_id_environments_id_fk'
+      AND conrelid = 'sandbox_creation_operations'::regclass
+  ) THEN
+    ALTER TABLE "sandbox_creation_operations" ADD CONSTRAINT "sandbox_creation_operations_environment_id_environments_id_fk" FOREIGN KEY ("environment_id") REFERENCES "public"."environments"("id") ON DELETE set null ON UPDATE no action;
+  END IF;
+END $$;
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "sandbox_creation_operations_pending_deadline_idx" ON "sandbox_creation_operations" USING btree ("next_attempt_at") WHERE "sandbox_creation_operations"."state" = 'pending';--> statement-breakpoint
+CREATE UNIQUE INDEX IF NOT EXISTS "sandbox_creation_operations_owner_idx" ON "sandbox_creation_operations" USING btree ("company_id","original_environment_id","run_id","request_key");

packages/db/src/migrations/meta/0418_snapshot.json

@@ -1,6 +1,6 @@
 {
-  "id": "5690a14a-f9cc-41a6-af51-c27f688f4ef6",
-  "prevId": "3b8bb508-028d-4154-abdd-af870f1f3ca6",
+  "id": "c777fa84-85a6-418f-bea1-0d60b56cf071",
+  "prevId": "c2c69a4a-8281-419c-9eeb-a7bcbc2beaae",
   "version": "7",
   "dialect": "postgresql",
   "tables": {
@@ -1351,6 +1351,39 @@
           "method": "btree",
           "with": {}
         },
+        "activity_log_company_entity_created_idx": {
+          "name": "activity_log_company_entity_created_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "entity_type",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "entity_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "created_at",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
         "activity_log_company_user_issue_actor_idx": {
           "name": "activity_log_company_user_issue_actor_idx",
           "columns": [
@@ -19019,6 +19052,98 @@
       "checkConstraints": {},
       "isRLSEnabled": false
     },
+    "public.company_skill_mutation_receipts": {
+      "name": "company_skill_mutation_receipts",
+      "schema": "",
+      "columns": {
+        "company_id": {
+          "name": "company_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "key": {
+          "name": "key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "fingerprint": {
+          "name": "fingerprint",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "skill_id": {
+          "name": "skill_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "version_id": {
+          "name": "version_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "response": {
+          "name": "response",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "created_at": {
+          "name": "created_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "now()"
+        }
+      },
+      "indexes": {
+        "company_skill_mutation_receipts_key_idx": {
+          "name": "company_skill_mutation_receipts_key_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "key",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": true,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        }
+      },
+      "foreignKeys": {
+        "company_skill_mutation_receipts_company_id_companies_id_fk": {
+          "name": "company_skill_mutation_receipts_company_id_companies_id_fk",
+          "tableFrom": "company_skill_mutation_receipts",
+          "tableTo": "companies",
+          "columnsFrom": [
+            "company_id"
+          ],
+          "columnsTo": [
+            "id"
+          ],
+          "onDelete": "cascade",
+          "onUpdate": "no action"
+        }
+      },
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {},
+      "policies": {},
+      "checkConstraints": {},
+      "isRLSEnabled": false
+    },
     "public.company_skill_repository_snapshots": {
       "name": "company_skill_repository_snapshots",
       "schema": "",
@@ -19046,7 +19171,46 @@
           "name": "file_inventory",
           "type": "jsonb",
           "primaryKey": false,
-          "notNull": true
+          "notNull": true,
+          "default": "'[]'::jsonb"
+        },
+        "storage": {
+          "name": "storage",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "upload": {
+          "name": "upload",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "restore": {
+          "name": "restore",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "migration_at": {
+          "name": "migration_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": false,
+          "default": "now()"
+        },
+        "maintenance_at": {
+          "name": "maintenance_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": false,
+          "default": "now()"
+        },
+        "restore_retry_at": {
+          "name": "restore_retry_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": false
         },
         "created_at": {
           "name": "created_at",
@@ -19077,6 +19241,38 @@
           "concurrently": false,
           "method": "btree",
           "with": {}
+        },
+        "company_skill_snapshots_migration_idx": {
+          "name": "company_skill_snapshots_migration_idx",
+          "columns": [
+            {
+              "expression": "migration_at",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null",
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
+        "company_skill_snapshots_maintenance_idx": {
+          "name": "company_skill_snapshots_maintenance_idx",
+          "columns": [
+            {
+              "expression": "maintenance_at",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null",
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
         }
       },
       "foreignKeys": {
@@ -19966,6 +20162,25 @@
           "primaryKey": false,
           "notNull": false
         },
+        "package_snapshot_id": {
+          "name": "package_snapshot_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "package_file_map": {
+          "name": "package_file_map",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "migration_at": {
+          "name": "migration_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": false,
+          "default": "now()"
+        },
         "revision_number": {
           "name": "revision_number",
           "type": "integer",
@@ -20024,6 +20239,52 @@
         }
       },
       "indexes": {
+        "company_skill_versions_migration_idx": {
+          "name": "company_skill_versions_migration_idx",
+          "columns": [
+            {
+              "expression": "migration_at",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"company_skill_versions\".\"migration_at\" is not null",
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
+        "company_skill_versions_package_snapshot_idx": {
+          "name": "company_skill_versions_package_snapshot_idx",
+          "columns": [
+            {
+              "expression": "package_snapshot_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
+        "company_skill_versions_repository_snapshot_idx": {
+          "name": "company_skill_versions_repository_snapshot_idx",
+          "columns": [
+            {
+              "expression": "repository_snapshot_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
         "company_skill_versions_skill_revision_idx": {
           "name": "company_skill_versions_skill_revision_idx",
           "columns": [
@@ -20135,6 +20396,19 @@
           "onDelete": "restrict",
           "onUpdate": "no action"
         },
+        "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": {
+          "name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk",
+          "tableFrom": "company_skill_versions",
+          "tableTo": "company_skill_repository_snapshots",
+          "columnsFrom": [
+            "package_snapshot_id"
+          ],
+          "columnsTo": [
+            "id"
+          ],
+          "onDelete": "restrict",
+          "onUpdate": "no action"
+        },
         "company_skill_versions_author_agent_id_agents_id_fk": {
           "name": "company_skill_versions_author_agent_id_agents_id_fk",
           "tableFrom": "company_skill_versions",
@@ -20580,6 +20854,78 @@
       "checkConstraints": {},
       "isRLSEnabled": false
     },
+    "public.skill_package_deletions": {
+      "name": "skill_package_deletions",
+      "schema": "",
+      "columns": {
+        "id": {
+          "name": "id",
+          "type": "uuid",
+          "primaryKey": true,
+          "notNull": true,
+          "default": "gen_random_uuid()"
+        },
+        "company_id": {
+          "name": "company_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "provider": {
+          "name": "provider",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "archive_key": {
+          "name": "archive_key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "manifest_key": {
+          "name": "manifest_key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "delete_after": {
+          "name": "delete_after",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true
+        }
+      },
+      "indexes": {
+        "skill_package_deletions_due_idx": {
+          "name": "skill_package_deletions_due_idx",
+          "columns": [
+            {
+              "expression": "delete_after",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        }
+      },
+      "foreignKeys": {},
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {},
+      "policies": {},
+      "checkConstraints": {},
+      "isRLSEnabled": false
+    },
     "public.company_transfer_runs": {
       "name": "company_transfer_runs",
       "schema": "",
@@ -33816,7 +34162,7 @@
             }
           ],
           "isUnique": false,
-          "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n        AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n        AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n        AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n          AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE",
+          "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n        AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n        AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n        AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n          AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE",
           "concurrently": false,
           "method": "btree",
           "with": {}
@@ -51228,6 +51574,195 @@
       "checkConstraints": {},
       "isRLSEnabled": false
     },
+    "public.sandbox_creation_operations": {
+      "name": "sandbox_creation_operations",
+      "schema": "",
+      "columns": {
+        "id": {
+          "name": "id",
+          "type": "uuid",
+          "primaryKey": true,
+          "notNull": true
+        },
+        "company_id": {
+          "name": "company_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "environment_id": {
+          "name": "environment_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "original_environment_id": {
+          "name": "original_environment_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "run_id": {
+          "name": "run_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "request_key": {
+          "name": "request_key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "'run'"
+        },
+        "issue_id": {
+          "name": "issue_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "execution_workspace_id": {
+          "name": "execution_workspace_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "provider": {
+          "name": "provider",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "plugin_id": {
+          "name": "plugin_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "driver_key": {
+          "name": "driver_key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "state": {
+          "name": "state",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "'pending'"
+        },
+        "config": {
+          "name": "config",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "metadata": {
+          "name": "metadata",
+          "type": "jsonb",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "lease_id": {
+          "name": "lease_id",
+          "type": "uuid",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "next_attempt_at": {
+          "name": "next_attempt_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "created_at": {
+          "name": "created_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "now()"
+        },
+        "updated_at": {
+          "name": "updated_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "now()"
+        }
+      },
+      "indexes": {
+        "sandbox_creation_operations_pending_deadline_idx": {
+          "name": "sandbox_creation_operations_pending_deadline_idx",
+          "columns": [
+            {
+              "expression": "next_attempt_at",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"sandbox_creation_operations\".\"state\" = 'pending'",
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        },
+        "sandbox_creation_operations_owner_idx": {
+          "name": "sandbox_creation_operations_owner_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "original_environment_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "run_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "request_key",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": true,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        }
+      },
+      "foreignKeys": {
+        "sandbox_creation_operations_environment_id_environments_id_fk": {
+          "name": "sandbox_creation_operations_environment_id_environments_id_fk",
+          "tableFrom": "sandbox_creation_operations",
+          "tableTo": "environments",
+          "columnsFrom": [
+            "environment_id"
+          ],
+          "columnsTo": [
+            "id"
+          ],
+          "onDelete": "set null",
+          "onUpdate": "no action"
+        }
+      },
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {},
+      "policies": {},
+      "checkConstraints": {},
+      "isRLSEnabled": false
+    },
     "public.secret_access_events": {
       "name": "secret_access_events",
       "schema": "",

packages/db/src/migrations/meta/_journal.json

@@ -2913,6 +2913,13 @@
       "when": 1793371684483,
       "tag": "0417_chunky_red_hulk",
       "breakpoints": true
+    },
+    {
+      "idx": 418,
+      "version": "7",
+      "when": 1793395364937,
+      "tag": "0418_curved_lady_mastermind",
+      "breakpoints": true
     }
   ]
 }

packages/db/src/schema/index.ts

@@ -39,6 +39,7 @@ export {
 } from "./execution_workspace_restore_points.js";
 export { executionWorkspaceRuntimeLeases } from "./execution_workspace_runtime_leases.js";
 export { environments } from "./environments.js";
+export { sandboxCreationOperations } from "./sandbox_creation_operations.js";
 export { environmentLeases } from "./environment_leases.js";
 export { environmentCustomImageTemplates } from "./environment_custom_image_templates.js";
 export { environmentCustomImageSetupSessions } from "./environment_custom_image_setup_sessions.js";

packages/db/src/schema/sandbox_creation_operations.ts

@@ -0,0 +1,27 @@
+import { index, jsonb, pgTable, text, timestamp, uuid, uniqueIndex } from "drizzle-orm/pg-core";
+import { sql } from "drizzle-orm";
+import { environments } from "./environments.js";
+
+export const sandboxCreationOperations = pgTable("sandbox_creation_operations", {
+  id: uuid("id").primaryKey(),
+  companyId: uuid("company_id").notNull(),
+  environmentId: uuid("environment_id").references(() => environments.id, { onDelete: "set null" }),
+  originalEnvironmentId: uuid("original_environment_id").notNull(),
+  runId: uuid("run_id").notNull(),
+  requestKey: text("request_key").notNull().default("run"),
+  issueId: uuid("issue_id"),
+  executionWorkspaceId: uuid("execution_workspace_id"),
+  provider: text("provider").notNull(),
+  pluginId: uuid("plugin_id"),
+  driverKey: text("driver_key").notNull(),
+  state: text("state").notNull().default("pending"),
+  config: jsonb("config").$type<Record<string, unknown>>().notNull(),
+  metadata: jsonb("metadata").$type<Record<string, unknown>>().notNull(),
+  leaseId: uuid("lease_id"),
+  nextAttemptAt: timestamp("next_attempt_at", { withTimezone: true }).notNull(),
+  createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
+  updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
+}, table => ({
+  pendingDeadline: index("sandbox_creation_operations_pending_deadline_idx").on(table.nextAttemptAt).where(sql`${table.state} = 'pending'`),
+  owner: uniqueIndex("sandbox_creation_operations_owner_idx").on(table.companyId, table.originalEnvironmentId, table.runId, table.requestKey),
+}));

packages/plugins/sandbox-providers/daytona/README.md

@@ -63,3 +63,22 @@ second export after reconnect. The test creates and deletes one disposable
 - `src/manifest.ts` declares the sandbox-provider driver metadata
 - `src/plugin.ts` implements the environment lifecycle hooks
 - `paperclipPlugin.manifest` and `paperclipPlugin.worker` point the host at the built plugin entrypoints in `dist/`
+
+
+### Interrupted run-lease creation
+
+The host commits a company-scoped creation operation before acquiring a new run
+lease. The operation UUID becomes `paperclip-create-<uuid>` and the
+`paperclip-create-attempt` label. The existing inspection hook can resolve that
+name and validate company, environment and run labels without starting a sandbox.
+Recovery persists the observed provider ID before the existing cleanup worker
+deletes it. A missing creation name is an uncertain outcome, not proof of removal.
+The host retains that operation for manual reconciliation and never repeats the
+creation request blindly. Credential identity must still match admission.
+
+This capability is declared as `creationOperationVersion: 1` in manifest version
+0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement.
+Legacy provider `environmentProbe` and custom-image interactive setup have separate
+lifecycles and are outside this lease-orphan fallback replacement. The normal Test
+path which acquires a runtime lease is covered. Unit fixtures exercise lookup and
+deletion; live provider qualification requires an explicit disposable test account.

packages/plugins/sandbox-providers/daytona/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@paperclipai/plugin-daytona",
-  "version": "0.1.9",
+  "version": "0.1.10",
   "description": "Daytona sandbox provider plugin for Paperclip environments",
   "license": "MIT",
   "homepage": "https://github.com/paperclipai/paperclip",

packages/plugins/sandbox-providers/daytona/src/manifest.ts

@@ -15,7 +15,7 @@ export const DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS = 300_000;
 // 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner.
 // 0.1.7 exposes host-owned warm/cold runner lifecycle controls.
 // 0.1.8 declares the default provider acquisition budget to the host.
-const PLUGIN_VERSION = "0.1.9";
+const PLUGIN_VERSION = "0.1.10";
 
 const manifest: PaperclipPluginManifestV1 = {
   id: PLUGIN_ID,
@@ -39,6 +39,7 @@ const manifest: PaperclipPluginManifestV1 = {
       description:
         "Provisions Daytona sandboxes with configurable image or snapshot selection, startup timeouts, and lease reuse.",
       supportsReusableLeases: true,
+      creationOperationVersion: 1,
       // Daytona keeps a persistent session and tails its callback log form, so it
       // emits incremental session output while the command runs. Declare the
       // opt-in capability so the host selects the session-output streaming path.

packages/plugins/sandbox-providers/daytona/src/plugin.test.ts

@@ -51,6 +51,16 @@ describe("Daytona read-only resource inspection", () => {
     for (const method of [sandbox.start, sandbox.stop, sandbox.delete, sandbox.archive, sandbox.refreshData]) expect(method).not.toHaveBeenCalled();
     expect(mockCreate).not.toHaveBeenCalled();
   });
+  it.each(["active", "destroyed"])("inspects a normal lease with creation provenance by recorded ID: %s", async state => {
+    const provenance = { creationOperationId: "16cd534e-27a1-45de-a4ad-5adbb6481361" };
+    if (state === "active") mockGet.mockResolvedValue(createMockSandbox());
+    else mockGet.mockRejectedValue(new MockDaytonaNotFoundError("gone"));
+    expect(await plugin.definition.onEnvironmentInspectLease!({ ...params, leaseMetadata: provenance }))
+      .toEqual({ providerLeaseId: "sandbox-123", state });
+    expect(mockGet).toHaveBeenCalledExactlyOnceWith("sandbox-123");
+    expect(mockCreate).not.toHaveBeenCalled();
+    if (state === "destroyed") expect(await plugin.definition.onIdleDrain!(new AbortController().signal)).toBe("none");
+  });
   it("accepts only typed provider not-found as deletion proof", async () => {
     __resetDaytonaSandboxHandleCacheForTest();
     mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not found"));
@@ -80,6 +90,46 @@ describe("Daytona read-only resource inspection", () => {
   });
 });
 
+describe("Daytona durable creation identity", () => {
+  const id = "16cd534e-27a1-45de-a4ad-5adbb6481361";
+  const runId = "8cbd0194-e322-4b5b-9800-df4d5a682479";
+  const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
+    providerLeaseId: id, config: { apiKey: "fixture-not-a-key" },
+    creationOperation: { id, runId } };
+  beforeEach(() => { mockCreate.mockReset(); mockGet.mockReset(); __resetDaytonaSandboxHandleCacheForTest(); });
+  afterEach(() => { vi.unstubAllEnvs(); __resetDaytonaSandboxHandleCacheForTest(); });
+  function resource() { return { ...createMockSandbox({ name: `paperclip-create-${id}` }), labels: {
+    "paperclip-provider": "daytona", "paperclip-create-attempt": id,
+    "paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId,
+    "paperclip-run-id": runId,
+  } }; }
+  it("locates the exact resource by stable name without creation or lifecycle calls", async () => {
+    const sandbox = resource(); mockGet.mockResolvedValue(sandbox);
+    const observed = await plugin.definition.onEnvironmentInspectLease!(params);
+    expect(mockGet).toHaveBeenCalledWith(`paperclip-create-${id}`);
+    expect(observed).toMatchObject({ providerLeaseId: `paperclip-create-${id}`, state: "active",
+      creationCleanup: { observedProviderLeaseId: sandbox.id, attemptId: id, runId } });
+    expect(mockCreate).not.toHaveBeenCalled(); expect(sandbox.delete).not.toHaveBeenCalled();
+    await plugin.definition.onEnvironmentDestroyLease!({ ...params, providerLeaseId: observed.providerLeaseId,
+      leaseMetadata: { failedCreateCleanup: observed.creationCleanup } });
+    expect(sandbox.delete).toHaveBeenCalledOnce();
+  });
+  it("retains uncertainty when a creation name is absent", async () => {
+    mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not yet visible"));
+    expect(await plugin.definition.onEnvironmentInspectLease!(params)).toEqual({ providerLeaseId: `paperclip-create-${id}`, state: "unknown" });
+    expect(mockCreate).not.toHaveBeenCalled();
+  });
+  it("rejects a name or label owned by another creation", async () => {
+    mockGet.mockResolvedValue({ ...resource(), labels: { "paperclip-company-id": "foreign" } });
+    await expect(plugin.definition.onEnvironmentInspectLease!(params)).rejects.toThrow("ownership does not match");
+  });
+  it("requires the recorded endpoint when ambient configuration changes", async () => {
+    vi.stubEnv("DAYTONA_API_URL", "https://changed.example.test");
+    expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("unknown");
+    expect(mockGet).not.toHaveBeenCalled();
+  });
+});
+
 describe("Daytona idle eligibility", () => {
   const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
     config: { apiKey: "fixture-not-a-key", reuseLease: true } };
@@ -537,7 +587,7 @@ describe("Daytona sandbox provider plugin", () => {
     // The bundled-plugin boot reconcile refreshes the stored manifest for an
     // existing install only when the version changes. The acquisition budget needs
     // the bump to reach an existing install.
-    expect(manifest.version).toBe("0.1.9");
+    expect(manifest.version).toBe("0.1.10");
   });
 
   it.each([false, true])("closes duplex routes on lease release even when bridge drain hangs: %s", async (hangDrain) => {

packages/plugins/sandbox-providers/daytona/src/plugin.ts

@@ -994,7 +994,8 @@ async function createSandbox(
   }
   // The SDK can create a sandbox and then throw while waiting for it to start.
   // Keep an attempt-specific provider name so that failure cannot lose ownership.
-  const attemptId = randomUUID();
+  const attemptId = "creationOperationId" in params && params.creationOperationId ? params.creationOperationId : randomUUID();
+  if (!/^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(attemptId)) throw new Error("Invalid creation operation identity");
   const name = `paperclip-create-${attemptId}`;
   const labels = { ...buildSandboxLabels({
     companyId: params.companyId,
@@ -2988,6 +2989,32 @@ const plugin = definePlugin(trackDaytonaHooks({
     if (!config.apiUrl && (process.env.DAYTONA_API_URL?.trim() || process.env.DAYTONA_SERVER_URL?.trim())) {
       return { providerLeaseId: params.providerLeaseId, state: "unknown" };
     }
+    if (params.creationOperation !== undefined) {
+      const { id: attemptId, runId } = params.creationOperation;
+      if (![attemptId, runId].every(value => typeof value === "string" && /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i.test(value))) {
+        throw new Error("Invalid creation operation identity");
+      }
+      const name = `paperclip-create-${attemptId}`;
+      try {
+        const sandbox = await withLivenessTimeout("sandbox.creationLookup", 10_000, () => createDaytonaClient(config).get(name));
+        const labels = {
+          "paperclip-provider": "daytona", "paperclip-create-attempt": attemptId,
+          "paperclip-company-id": params.companyId, "paperclip-environment-id": params.environmentId,
+          "paperclip-run-id": runId,
+        };
+        if (sandbox.name !== name || Object.entries(labels).some(([key, value]) => sandbox.labels?.[key] !== value)) {
+          throw new Error("Creation operation ownership does not match");
+        }
+        return { providerLeaseId: name, state: "active", creationCleanup: {
+          providerLeaseId: name, observedProviderLeaseId: sandbox.id, companyId: params.companyId,
+          environmentId: params.environmentId, runId: labels["paperclip-run-id"], attemptId,
+          accountFingerprint: sandboxAccountDiscriminator(config), labels,
+        } };
+      } catch (error) {
+        if (error instanceof DaytonaNotFoundError) return { providerLeaseId: name, state: "unknown" };
+        throw error;
+      }
+    }
     const proof = idleActivity.proof(providerActivityKey(params, config, params.providerLeaseId));
     // Read directly from the provider. The setup-status path and cached lease
     // reconnect path can start a stopped sandbox, so neither is an observation.

packages/plugins/sdk/src/protocol.ts

@@ -641,6 +641,7 @@ export interface PluginEnvironmentRunnerIngressEndpointParams
 
 export interface PluginEnvironmentAcquireLeaseParams extends PluginEnvironmentDriverBaseParams {
   runId: string;
+  creationOperationId?: string;
   /** Explicit platform probes may request a provider-owned immutable snapshot
    * identity. Supporting providers create by that ID and return snapshotIdentity
    * in lease metadata. Older providers may omit it; the host then cannot qualify
@@ -701,11 +702,13 @@ export interface PluginEnvironmentDestroyLeaseParams extends PluginEnvironmentRe
 /** Observe this exact allocation without starting, stopping or deleting it.
  * Missing resources must be distinguished from permission/transport failures. */
 export interface PluginEnvironmentInspectLeaseParams extends PluginEnvironmentDriverBaseParams {
+  creationOperation?: { id: string; runId: string };
   providerLeaseId: string;
   leaseMetadata?: Record<string, unknown>;
 }
 
 export interface PluginEnvironmentLeaseObservation {
+  creationCleanup?: import("./environment-creation-cleanup.js").PluginEnvironmentCreationCleanup;
   providerLeaseId: string;
   state: "active" | "stopped" | "destroyed" | "unknown";
 }

packages/shared/src/types/agent-lifecycle.ts

@@ -13,6 +13,10 @@ export interface AgentLifecycleOperation {
   leaseOwner?: string;
   leaseUntil?: string;
   retryAt?: string;
+  /** A failed configuration check waits for configuration repair or an explicit retry. */
+  retryRequired?: boolean;
+  /** Failed generic host verification tests for this saved configuration. */
+  verificationFailures?: number;
   attempts: number;
   responsibleUserId?: string | null;
 }

packages/shared/src/types/agent.ts

@@ -333,6 +333,8 @@ export interface AdapterEnvironmentCheck {
   message: string;
   detail?: string | null;
   hint?: string | null;
+  /** The check could not complete because its execution target was unavailable. */
+  retryable?: boolean;
 }
 
 export interface AdapterEnvironmentTestResult {

packages/shared/src/types/plugin.ts

@@ -198,6 +198,7 @@ export interface PluginEnvironmentDriverDeclaration {
   /** Provides Runner execution, resource preparation, and PRP connections via environmentTask. */
   supportsTasks?: boolean;
   freshWorkspaceVersion?: 1;
+  creationOperationVersion?: 1;
   /** Codex task retention and authorized run attachment, version 1. */
   warmTaskVersion?: 1;
   /** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */

packages/shared/src/validators/plugin.ts

@@ -181,6 +181,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({
   requiresAgent: z.boolean().optional(),
   supportsTasks: z.boolean().optional(),
   freshWorkspaceVersion: z.literal(1).optional(),
+  creationOperationVersion: z.literal(1).optional(),
   warmTaskVersion: z.literal(1).optional(),
   driverKey: z.string().min(1).regex(
     /^[a-z0-9][a-z0-9._-]*$/,

server/scripts/migrate-skill-packages.ts

@@ -1,5 +1,8 @@
 /** Run from the server package with its configured storage and DATABASE_URL. */
 import { createDb } from "@paperclipai/db";
+import { loadStorageConfig } from "../src/storage/config.js";
+import { createStorageProviderFromConfig } from "../src/storage/provider-registry.js";
+import { createStorageService } from "../src/storage/service.js";
 import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js";
 import { skillInstallationMaintenance } from "../src/services/skill-installation-maintenance.js";
 import { isCloudManagedInstance } from "../src/services/cloud-instance.js";
@@ -19,8 +22,11 @@ try {
     if (!current) throw new Error("Initialize Cloud runtime configuration before running skill package maintenance");
     setCloudRuntimeConfigSnapshot(current);
   }
-  const service = skillPackageMaintenance(db);
-  const installations = skillInstallationMaintenance(db);
+  // Maintenance must inspect the old provider before Cloud cutover. Server
+  // startup prerequisites still apply to the normal global storage service.
+  const storage = createStorageService(createStorageProviderFromConfig(loadStorageConfig()));
+  const service = skillPackageMaintenance(db, storage);
+  const installations = skillInstallationMaintenance(db, storage);
   const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions,
     orphans: service.collectOrphans, deletions: service.deleteObjects,
     installations: installations.installations, eligibility: installations.eligibility, verification: installations.verifyVersions };

server/src/__tests__/agent-harness-verification-remote.test.ts

@@ -0,0 +1,67 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
+
+// Keep the real directory, command and adapter checks. Runtime staging is a
+// separate operation and must not copy files or contact a provider in this test.
+vi.mock("@paperclipai/adapter-utils/execution-target", async importOriginal => ({
+  ...await importOriginal<typeof import("@paperclipai/adapter-utils/execution-target")>(),
+  prepareAdapterExecutionTargetRuntime: async () => ({
+    workspaceRemoteDir: "/workspace", runtimeRootDir: "/workspace/.paperclip-runtime",
+    assetDirs: {}, restoreWorkspace: async () => {},
+  }),
+}));
+
+import { testEnvironment } from "@paperclipai/adapter-opencode-local/server";
+import { assertHarnessTestPassed } from "../services/agent-harness-verification.js";
+import { lifecycleFailureMessage, lifecycleVerificationNeedsRetry } from "../modules/agent-lifecycle/application/failure.js";
+
+describe("remote harness verification recovery", () => {
+  let configHome: string;
+  beforeAll(async () => { configHome = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-verification-")); });
+  afterAll(async () => { if (configHome) await rm(configHome, { recursive: true, force: true }); });
+  const context = { stage: "host" as const, phase: "verifying" as const, verificationFailures: 100 };
+
+  it.each([
+    ["directory", "timeout"], ["directory", "transport"],
+    ["command", "timeout"], ["command", "transport"],
+    ["directory", "missing"], ["command", "missing"],
+  ] as const)("retains %s %s through the real adapter check and recovers when repaired", async (check, fault) => {
+    let failing = true;
+    const target: AdapterExecutionTarget = {
+      kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: {
+        execute: async input => {
+          const script = input.args?.join(" ") ?? "";
+          const selected = check === "directory" ? script.includes("[ -d ") : script.includes("command -v ");
+          if (failing && selected && fault === "transport") {
+            throw Object.assign(new Error("private provider output"), { code: "ECONNRESET" });
+          }
+          return {
+            exitCode: failing && selected ? fault === "timeout" ? null : 1 : 0,
+            timedOut: failing && selected && fault === "timeout", signal: null, stderr: "", pid: null,
+            startedAt: new Date().toISOString(),
+            stdout: input.command === "opencode" ? JSON.stringify({ type: "text", part: { text: "hello" } }) : "",
+          };
+        },
+      },
+    };
+    const run = () => testEnvironment({ companyId: "company", adapterType: "opencode_local", executionTarget: target,
+      config: { command: "opencode", model: "provider/model", env: { XDG_CONFIG_HOME: configHome } } });
+    const result = await run();
+    const failed = result.checks.find(entry => entry.code === (check === "directory" ? "opencode_cwd_invalid" : "opencode_command_unresolvable"));
+    expect(failed?.level).toBe("error");
+    expect(failed?.retryable === true).toBe(fault !== "missing");
+    let error: unknown;
+    try { assertHarnessTestPassed(result); } catch (caught) { error = caught; }
+    expect(error).toMatchObject({ code: fault !== "missing" ? "test_environment_unavailable"
+      : check === "directory" ? "harness_cwd_invalid" : "harness_command_unavailable" });
+    expect(lifecycleVerificationNeedsRetry(context, error)).toBe(fault === "missing");
+    expect(lifecycleFailureMessage(context, error)).not.toContain("private provider output");
+    if (fault !== "missing") expect(lifecycleFailureMessage(context, error)).toContain("retry automatically");
+    failing = false;
+    const recovered = await run();
+    expect(() => assertHarnessTestPassed(recovered)).not.toThrow();
+  });
+});

server/src/__tests__/agent-harness-verification.test.ts

@@ -24,6 +24,13 @@ describe("saved harness verification", () => {
       expect(error).toMatchObject({ code, message: "The harness test failed" });
     }
   });
+  it("preserves a retryable failed check before classifying configuration failures", () => {
+    const temporary = result("fail", ["opencode_cwd_invalid", "opencode_command_unresolvable"]);
+    temporary.checks[0] = { ...temporary.checks[0], level: "error", retryable: true };
+    expect(() => assertHarnessTestPassed(temporary)).toThrow(expect.objectContaining({ code: "test_environment_unavailable" }));
+    temporary.checks[0].level = "info";
+    expect(() => assertHarnessTestPassed(temporary)).toThrow(expect.objectContaining({ code: "harness_command_unavailable" }));
+  });
   it.each([
     ["fail", []], ["warn", ["claude_hello_probe_auth_required"]],
     ["warn", ["gemini_hello_probe_timed_out"]], ["pass", ["claude_hello_probe_skipped_custom_command"]],

server/src/__tests__/environment-cleanup-work.test.ts

@@ -1,3 +1,5 @@
+import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
+import { createOwnedSandbox } from "../services/sandbox-creation-operations.js";
 import { mkdtemp, writeFile, rm } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
@@ -7,7 +9,7 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
 import { randomUUID } from "node:crypto";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
 import { eq, sql } from "drizzle-orm";
-import { companies, createDb, environmentLeases, environments, type Db } from "@paperclipai/db";
+import { companies, createDb, sandboxCreationOperations, environmentLeases, environments, type Db } from "@paperclipai/db";
 import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
 import { registerEnvironmentCleanupWork } from "../services/environment-cleanup-work.js";
@@ -34,7 +36,7 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => {
     await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
   }, 90_000);
   beforeEach(async () => {
-    await db.execute(sql`TRUNCATE companies, environments CASCADE`);
+    await db.execute(sql`TRUNCATE sandbox_creation_operations, companies, environments CASCADE`);
     companyId = randomUUID(); environmentId = randomUUID(); draining = false; failing = false; errors.length = 0;
     await db.insert(companies).values({ id: companyId, name: "Cleanup worker", issuePrefix: "CLN" });
     await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
@@ -99,6 +101,64 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => {
     await settle(180_000); expect(inspections).toHaveBeenCalledTimes(3);
     await quiet();
   });
+  it("reconstructs a creation deadline after restart and leaves manual reconciliation quiet", async () => {
+    await start(); await quiet();
+    await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
+      async () => ({ providerLeaseId: "fixture-unattached" }));
+    await settle();
+    expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now());
+    worker!.stop(); await coordinator!.stop();
+    await start();
+    expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now());
+    await settle(31 * 60_000);
+    await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([
+      expect.objectContaining({ state: "needs_reconciliation" }),
+    ]));
+    await quiet();
+  });
+  it("settles a deleted-company creation as manual history across restart without idle polling", async () => {
+    await start(); await quiet();
+    await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
+      async () => ({ providerLeaseId: "fixture-deleted-company" }));
+    await db.delete(companies).where(eq(companies.id, companyId));
+    await settle(31 * 60_000);
+    await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([
+      expect.objectContaining({ state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }),
+    ]));
+    await quiet();
+    worker!.stop(); await coordinator!.stop();
+    await start(); await quiet();
+    expect(releases).not.toHaveBeenCalled(); expect(inspections).not.toHaveBeenCalled();
+    const ownerId = randomUUID();
+    const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 };
+    expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none"))
+      .toEqual({ version: 1, backgroundWork: "none" });
+    await quiet();
+  });
+  it("verifies ordinary cleanup carrying creation provenance and stops provider polling", async () => {
+    const lease = await pending();
+    await environmentService(db).updateLeaseMetadata(lease.id, { driver: "sandbox", creationOperationId: randomUUID() });
+    await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" });
+    await start(true);
+    expect(inspections).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ providerLeaseId: lease.providerLeaseId,
+      metadata: expect.objectContaining({ creationOperationId: expect.any(String) }) }));
+    await quiet();
+    worker!.stop(); await coordinator!.stop();
+    await start(true); expect(inspections).toHaveBeenCalledTimes(1); await quiet();
+  });
+  it("preserves a first creation signal during drain and disarms after atomic attachment", async () => {
+    await start(); await quiet(); draining = true;
+    const created = await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
+      async () => ({ providerLeaseId: "fixture-attached" }));
+    const count = reads.mock.calls.length;
+    await settle(60_000, false); expect(reads).toHaveBeenCalledTimes(count);
+    draining = false;
+    await environmentService(db).acquireLease({ companyId, environmentId, provider: "fixture", providerLeaseId: created.providerLeaseId,
+      creationOperationId: String(created.metadata!.creationOperationId), metadata: created.metadata });
+    await settle();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "attached" })]);
+    await quiet();
+  });
   it("wakes on a real orphan insert, waits for its backoff, then disarms after deletion", async () => {
     await start(); await quiet();
     const lease = await pending(); await settle();

server/src/__tests__/fixtures/sandbox-creation-receiver.ts

@@ -0,0 +1,34 @@
+import { createDb } from "@paperclipai/db";
+import { environmentRuntimeService } from "../../services/environment-runtime.js";
+import { environmentService } from "../../services/environments.js";
+import type { PluginWorkerManager } from "../../services/plugin-worker-manager.js";
+
+const db = createDb(process.env.CREATION_FIXTURE_DATABASE_URL!);
+const workers = {
+  isRunning: () => true,
+  getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }),
+  call: async (_pluginId: string, method: string, input: unknown) => {
+    const response = await fetch(process.env.CREATION_FIXTURE_PROVIDER_URL!, {
+      method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ method, input }),
+    });
+    if (!response.ok) throw new Error("Controlled provider unavailable");
+    return response.json();
+  },
+} as unknown as PluginWorkerManager;
+try {
+  const runtime = environmentRuntimeService(db, {
+    pluginWorkerManager: workers, orphanCleanupSpoolDir: process.env.CREATION_FIXTURE_EMPTY_DISK,
+  });
+  if (process.env.CREATION_FIXTURE_ACTION === "acquire") {
+    await runtime.acquireRunLease({ companyId: process.env.CREATION_FIXTURE_COMPANY_ID!,
+      environment: (await environmentService(db).getById(process.env.CREATION_FIXTURE_ENVIRONMENT_ID!))!,
+      heartbeatRunId: process.env.CREATION_FIXTURE_RUN_ID!, issueId: null, persistedExecutionWorkspace: null });
+  } else {
+    await runtime.flushDeferredOrphanCleanups();
+  }
+  process.stdout.write(JSON.stringify({ ok: true }));
+} catch {
+  process.stdout.write(JSON.stringify({ ok: false }));
+} finally {
+  await db.$client.end({ timeout: 1 });
+}

server/src/__tests__/heartbeat-plugin-environment.test.ts

@@ -278,6 +278,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
     }, { timeout: 5_000 });
 
     expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", {
+      creationOperationId: expect.any(String),
       driverKey: "sandbox",
       companyId,
       environmentId,
@@ -745,6 +746,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
     }, { timeout: 5_000 });
 
     expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentAcquireLease", {
+      creationOperationId: expect.any(String),
       driverKey: "sandbox",
       companyId,
       environmentId: newEnvironmentId,

server/src/__tests__/managed-skill-content.test.ts

@@ -166,7 +166,12 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
     }
   }, 180_000);
 
-  it("reports a provider mismatch for local versions under persisted Cloud configuration", async () => {
+  it.each([
+    { storage: "legacy local", env: { PAPERCLIP_STORAGE_PROVIDER: "local_disk" }, expectedStatus: "local_storage" },
+    { storage: "shared S3", env: { PAPERCLIP_STORAGE_PROVIDER: "s3",
+      PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture", PAPERCLIP_STORAGE_S3_REGION: "us-east-1",
+      PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" }, expectedStatus: "provider_mismatch" },
+  ])("loads persisted Cloud configuration for read-only $storage maintenance without seeding it", async ({ env, expectedStatus }) => {
     const companyId = await company();
     const skill = await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
     const local = await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 });
@@ -178,8 +183,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
       `--phase=${phase}`, `--company=${companyId}`, "--limit=50",
     ], { env: { ...process.env, DATABASE_URL: database.connectionString,
       PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture",
-      PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture",
-      PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_SECRETS_MASTER_KEY: "a".repeat(64) },
+      PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects"), ...env },
       timeout: 60_000, maxBuffer: 1024 * 1024 });
     await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration");
     expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0);
@@ -188,7 +192,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
       const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!);
       expect(eligibility.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "version_backed" })]);
       const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!);
-      expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "provider_mismatch" })]);
+      expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: expectedStatus })]);
       expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toEqual([persisted]);
     } finally {
       await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key));

server/src/__tests__/plugin-warm-tasks.integration.test.ts

@@ -1,11 +1,13 @@
-import { flushDeferredEnvironmentCleanup, hasDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, subscribeDeferredEnvironmentCleanup } from "../services/environment-cleanup-notifications.js";
 import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
+import { mkdtemp, readdir, rm } from "node:fs/promises";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
 import { randomUUID } from "node:crypto";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { eq } from "drizzle-orm";
-import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects } from "@paperclipai/db";
+import { eq, sql } from "drizzle-orm";
+import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects, sandboxCreationOperations } from "@paperclipai/db";
 import { environmentRuntimeService } from "../services/environment-runtime.js";
-import * as orphanCleanup from "../services/sandbox-orphan-cleanup-spool.js";
+import { recordSandboxCreationResource } from "../services/sandbox-creation-operations.js";
 import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
 import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
 import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
@@ -51,7 +53,7 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
     if(p.operation.kind==="describe") return {kind:"description",taskId:p.taskId,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`},warmTaskVersion:1};
     return {kind:"accepted",taskId:p.taskId};
    }} as unknown as PluginWorkerManager;
-  const runtime=()=>environmentRuntimeService(db,{pluginWorkerManager:workers});
+  const runtime=(orphanCleanupSpoolDir?:string)=>environmentRuntimeService(db,{pluginWorkerManager:workers,orphanCleanupSpoolDir});
   const acquire=()=>runtime().acquireRunLease({companyId,environment:environment as never,issueId,agentId,heartbeatRunId:runId,persistedExecutionWorkspace:null,adapterType:"paperclip_runner",runnerTaskProjectIds:projectIds,requestedExpiresAt:deadline});
   const first=await acquire();
   const identity={runnerInstanceId:"runner",environmentLeaseId:first.lease.id,runId,normalizedSessionId:"session",turnId:"turn",itemId:"item"};
@@ -126,58 +128,64 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
   await expect(resolvePluginTaskResume(db,{companyId:f.companyId,leaseId:next.lease.id,priorIdentity:f.identity})).resolves.toBe(f.first.lease.providerLeaseId);
  });
 
- it.each(["ownership-lost","database-error","cleanup-error","spool"] as const)("compensates an unowned replacement task: %s",async mode=>{
+ it.each(["ownership-lost","database-error"] as const)("retains a durable operation for an unattached replacement task: %s",async mode=>{
   const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped();
-  if(mode==="cleanup-error" || mode==="spool") f.failRelease();
-  const append=vi.fn(async()=>true);
-  let delayFlush=false,flushCalls=0,releaseFlush!:()=>void;
-  const flushGate=new Promise<void>(resolve=>{releaseFlush=resolve;});
-  const stopGate=mode==="spool" ? registerDeferredEnvironmentCleanup(db,async()=>{
-   if(delayFlush && ++flushCalls===1) await flushGate;
-   return {recovered:0,pending:0};
-  }) : undefined;
-  const wake=vi.fn();
-  const unsubscribe=subscribeDeferredEnvironmentCleanup(db,wake);
-  const spool=mode==="spool" ? vi.spyOn(orphanCleanup,"createSandboxOrphanCleanupSpool").mockReturnValue({append,load:async()=>[],remove:async()=>{}}) : undefined;
-  let replacementFailed=false;
-  const transact=db.transaction.bind(db);
-  const transaction=mode==="spool" ? vi.spyOn(db,"transaction").mockImplementation((...args)=>replacementFailed
-   ? Promise.reject(Error("database unavailable")) : transact(...args)) : undefined;
-  const original=db.update.bind(db);
-  const update=vi.spyOn(db,"update").mockImplementation(((table:any)=>{
-   const query=original(table),set=query.set.bind(query);
-   query.set=((values:any)=>values.providerLeaseId === "task-2"
-    ? {where:()=>({returning:async()=>{replacementFailed=true;if(mode!=="ownership-lost") throw Error("replacement write failed");return [];}})}
-    : set(values)) as never;
-   return query;
-  }) as never);
+  const disk=await mkdtemp(join(tmpdir(),"replacement-intent-"));
+  const runtime=()=>f.runtime(disk);
+  await db.execute(sql.raw(`create or replace function reject_task_replacement() returns trigger language plpgsql as $$ begin
+    if new.provider_lease_id = 'task-2' then ${mode === "ownership-lost" ? "return null;" : "raise exception 'replacement write failed';"} end if; return new; end $$`));
+  await db.execute(sql`create trigger reject_task_replacement before update on environment_leases for each row execute function reject_task_replacement()`);
   try {
-   await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}}))
-    .rejects.toThrow("reconcile the same task");
-   if(mode==="spool") {
-    expect(wake).toHaveBeenCalledOnce();
-    expect(hasDeferredEnvironmentCleanup(db)).toBe(true);
-    await expect(flushDeferredEnvironmentCleanup(db)).rejects.toThrow("Deferred orphan cleanup failed");
-    expect(hasDeferredEnvironmentCleanup(db)).toBe(true);
-   }
-  } finally {update.mockRestore();transaction?.mockRestore();spool?.mockRestore();unsubscribe();}
-  expect(f.calls).toContain("environmentReleaseLease");
-  if(mode==="spool") expect(append).toHaveBeenCalledWith(expect.objectContaining({providerLeaseId:"task-2",metadata:expect.objectContaining({driver:"plugin",runnerTaskDispatched:false})}));
-  const rows=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId));
-  expect(rows.filter(row=>row.providerLeaseId === "task-2")).toEqual(mode==="cleanup-error"
-   ? [expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned",metadata:expect.objectContaining({runnerTaskDispatched:false})})] : []);
-  if(mode==="spool") {
-   delayFlush=true;
-   const delayed=flushDeferredEnvironmentCleanup(db);
-   try { expect(await flushDeferredEnvironmentCleanup(db)).toEqual({recovered:1,pending:0}); }
-   finally { releaseFlush(); stopGate!(); }
-   expect(await delayed).toEqual({recovered:1,pending:0});
-   expect(hasDeferredEnvironmentCleanup(db)).toBe(false);
-   const recovered=await db.select().from(environmentLeases).where(eq(environmentLeases.companyId,f.companyId));
-   expect(recovered.filter(row=>row.providerLeaseId==="task-2")).toEqual([
-    expect.objectContaining({status:"pending_cleanup",failureReason:"plugin_task_replacement_unowned"}),
-   ]);
-  }
+   await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
+  } finally { await db.execute(sql`drop trigger reject_task_replacement on environment_leases`); }
+  expect(f.calls).not.toContain("environmentReleaseLease");
+  const [lease]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));
+  expect(lease.providerLeaseId).toBe(next.lease.providerLeaseId);
+  const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
+  const pending=operations.filter(row=>row.state === "pending");
+  expect(pending).toHaveLength(1);expect(pending[0].requestKey).toBe(next.lease.providerLeaseId);
+  expect(pending[0].metadata.creationProviderLeaseId).toBe("task-2");expect(pending[0].config).toEqual({});
+  expect(f.acquireRequests.at(-1)?.creationOperationId).toBe(pending[0].id);
+  const creates=f.acquireRequests.length;
+  await expect(runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
+  expect(f.acquireRequests).toHaveLength(creates);
+  await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,pending[0].id));
+  await runtime().flushDeferredOrphanCleanups();
+  const [manual]=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,pending[0].id));
+  expect(manual.state).toBe("needs_reconciliation");expect(manual.metadata.creationProviderLeaseId).toBe("task-2");
+  await runtime().flushDeferredOrphanCleanups();expect(f.acquireRequests).toHaveLength(creates);
+  expect(f.calls).not.toContain("environmentReleaseLease");
+  expect(await readdir(disk)).toEqual([]);
+  await rm(disk,{recursive:true,force:true});
+ });
+
+ it("retains the known replacement identity after a committed attachment response is lost",async()=>{
+  const f=await fixture();await f.nextRun();const next=await f.acquire();f.stopped();
+  const transaction=db.transaction.bind(db);let lost=false;
+  const response=vi.spyOn(db,"transaction").mockImplementation((async(...args:any[])=>{
+   const result=await (transaction as any)(...args);
+   if(!lost && result?.providerLeaseId === "task-2") {lost=true;throw new Error("Attachment commit response lost");}
+   return result;
+  }) as typeof db.transaction);
+  try {
+   await expect(f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).rejects.toThrow("reconcile the same task");
+  } finally {response.mockRestore();}
+  expect(lost).toBe(true);
+  const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
+  const replacement=operations.find(operation=>operation.requestKey === next.lease.providerLeaseId)!;
+  expect(replacement).toMatchObject({state:"attached",leaseId:next.lease.id,metadata:{creationProviderLeaseId:"task-2"}});
+  const identity={id:replacement.id,companyId:f.companyId,environmentId:replacement.originalEnvironmentId,providerLeaseId:"task-2"};
+  await recordSandboxCreationResource(db,identity);
+  await expect(recordSandboxCreationResource(db,{...identity,providerLeaseId:"different-resource"})).rejects.toThrow("identity changed");
+  await expect(recordSandboxCreationResource(db,{...identity,companyId:randomUUID()})).rejects.toThrow("identity changed");
+  expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id,replacement.id)))[0])
+   .toMatchObject({state:"attached",metadata:{creationProviderLeaseId:"task-2"}});
+  const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));
+  expect(saved).toMatchObject({providerLeaseId:"task-2",status:"active"});
+  await db.update(sandboxCreationOperations).set({nextAttemptAt:new Date(0)}).where(eq(sandboxCreationOperations.id,replacement.id));
+  await f.runtime().flushDeferredOrphanCleanups();
+  expect((await f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}})).taskId).toBe("task-2");
+  expect(f.acquireRequests).toHaveLength(2);expect(f.calls).not.toContain("environmentReleaseLease");
  });
 
  it.each(["added","removed"] as const)("retires retained mounts when an authorized project is %s after controller restart",async mode=>{
@@ -203,7 +211,9 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
   const replacing=f.runtime().task({companyId:f.companyId,leaseId:next.lease.id,operation:{kind:"describe"}});
   if(["violating","missing-bounded","invalid","expired"].includes(mode)) {
    await expect(replacing).rejects.toThrow("reconcile the same task");
-   expect(f.calls).toContain("environmentReleaseLease");
+   expect(f.calls).not.toContain("environmentReleaseLease");
+   const operations=await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.companyId,f.companyId));
+   expect(operations.filter(row=>row.state === "pending")).toHaveLength(1);
   } else {
    const description=await replacing;
    const [saved]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,next.lease.id));

server/src/__tests__/sandbox-creation-operations.test.ts

@@ -0,0 +1,333 @@
+import { execFile } from "node:child_process";
+import { createServer } from "node:http";
+import { promisify } from "node:util";
+import { fileURLToPath } from "node:url";
+import { randomUUID } from "node:crypto";
+import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { eq, sql } from "drizzle-orm";
+import { agents, builtInManagedResources, companies, companySecretVersions, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
+import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+import { environmentRuntimeService } from "../services/environment-runtime.js";
+import { environmentService } from "../services/environments.js";
+import { HttpError } from "../errors.js";
+import * as secretOwner from "../services/secrets.js";
+import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
+import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
+import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
+
+describe("durable sandbox creation through the runtime entry point", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: ReturnType<typeof createDb>;
+  let companyId: string, environmentId: string, pluginId: string, runId: string, spool: string;
+  let creates: number, destroys: number, loseReply: boolean, failDestroy: boolean;
+  let beforeReply: (() => Promise<void>) | undefined;
+  const resources = new Map<string, string>();
+  let workers: PluginWorkerManager;
+  beforeAll(async () => {
+    database = await startEmbeddedPostgresTestDatabase("sandbox-creation-operations-");
+    db = createDb(database.connectionString);
+  }, 90_000);
+  beforeEach(async () => {
+    await db.execute(sql`truncate sandbox_creation_operations, companies, environments, plugins cascade`);
+    companyId = randomUUID(); environmentId = randomUUID(); pluginId = randomUUID(); runId = randomUUID();
+    const agentId = randomUUID(); creates = 0; destroys = 0; loseReply = false; failDestroy = false; beforeReply = undefined; resources.clear();
+    spool = await mkdtemp(join(tmpdir(), "empty-sandbox-web-"));
+    await db.insert(companies).values({ id: companyId, name: "Creation fixture", issuePrefix: "CRT" });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "codex_local" });
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", invocationSource: "manual" });
+    await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox", config: { provider: "creation-fixture", image: "fixture", reuseLease: false } });
+    await db.insert(plugins).values({ id: pluginId, pluginKey: "paperclip.creation-fixture", packageName: "creation-fixture", version: "1.0.0", apiVersion: 1,
+      status: "ready", categories: ["automation"], manifestJson: { id: "paperclip.creation-fixture", version: "1.0.0", apiVersion: 1,
+      displayName: "Creation fixture", description: "Controlled remote provider", author: "Paperclip", categories: ["automation"],
+      capabilities: ["environment.drivers.register"], entrypoints: { worker: "fixture.js" }, environmentDrivers: [{
+        driverKey: "creation-fixture", kind: "sandbox_provider", displayName: "Fixture", creationOperationVersion: 1, configSchema: { type: "object" },
+      }] } });
+    workers = {
+      isRunning: () => true,
+      getWorker: () => ({ supportedMethods: ["environmentAcquireLease", "environmentInspectLease", "environmentDestroyLease"] }),
+      call: vi.fn(async (_pluginId: string, method: string, input: any) => {
+        if (method === "environmentAcquireLease") {
+          expect(await db.select().from(sandboxCreationOperations)).toContainEqual(expect.objectContaining({ id: input.creationOperationId, state: "pending" }));
+          creates++;
+          const resource = `provider-${input.creationOperationId}`;
+          resources.set(input.creationOperationId, resource);
+          await beforeReply?.();
+          if (loseReply) throw new Error("Lost provider reply");
+          return { providerLeaseId: resource, metadata: { remoteCwd: "/workspace" } };
+        }
+        if (method === "environmentInspectLease") {
+          const id = input.creationOperation.id;
+          const resource = resources.get(id);
+          if (!resource) return { providerLeaseId: id, state: "unknown" };
+          return { providerLeaseId: resource, state: "active", creationCleanup: { providerLeaseId: resource,
+            observedProviderLeaseId: resource, companyId, environmentId, runId: input.creationOperation.runId, attemptId: id,
+            accountFingerprint: "a".repeat(64), labels: { "paperclip-create-attempt": id } } };
+        }
+        if (method === "environmentDestroyLease") {
+          destroys++;
+          if (failDestroy) throw new Error("Fixture teardown failed");
+          resources.delete(input.leaseMetadata.creationOperationId);
+          return { providerLeaseId: input.providerLeaseId, state: "destroyed" };
+        }
+        throw new Error(`Unexpected fixture method ${method}`);
+      }),
+    } as unknown as PluginWorkerManager;
+  });
+  afterEach(async () => {
+    await db.execute(sql`drop trigger if exists reject_fixture_lease on environment_leases`);
+    await db.execute(sql`drop trigger if exists reject_fixture_creation on sandbox_creation_operations`);
+    await rm(spool, { recursive: true, force: true });
+  });
+  afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
+  function runtime() { return environmentRuntimeService(db, { pluginWorkerManager: workers, orphanCleanupSpoolDir: spool }); }
+  async function acquire() {
+    return runtime().acquireRunLease({ companyId, environment: (await environmentService(db).getById(environmentId))!,
+      heartbeatRunId: runId, issueId: null, persistedExecutionWorkspace: null, assertCompanyBinding: true });
+  }
+  async function due() { await db.update(sandboxCreationOperations).set({ nextAttemptAt: new Date(0) }); }
+  async function rejectWrites(table: "environment_leases" | "sandbox_creation_operations", trigger: string) {
+    await db.execute(sql`create or replace function reject_fixture_write() returns trigger language plpgsql as $$ begin raise exception 'Fixture database write unavailable'; end $$`);
+    await db.execute(sql.raw(`create trigger ${trigger} before insert on ${table} for each row execute function reject_fixture_write()`));
+  }
+  it("can reapply the creation migration without losing a committed intent", async () => {
+    loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
+    const [before] = await db.select().from(sandboxCreationOperations);
+    const migration = await readFile(new URL("../../../packages/db/src/migrations/0418_curved_lady_mastermind.sql", import.meta.url), "utf8");
+    for (let attempt = 0; attempt < 2; attempt++) {
+      for (const statement of migration.split("--> statement-breakpoint").map(value => value.trim()).filter(Boolean)) {
+        await db.execute(sql.raw(statement));
+      }
+    }
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([before]);
+    expect(creates).toBe(1);
+  });
+  it("fails before the external action when the durable admission write fails", async () => {
+    await rejectWrites("sandbox_creation_operations", "reject_fixture_creation");
+    await expect(acquire()).rejects.toThrow();
+    expect(creates).toBe(0); expect(resources.size).toBe(0); expect(await readdir(spool)).toEqual([]);
+  });
+  it("attaches once and reuses the committed transfer after a lost response", async () => {
+    const first = await acquire();
+    const retry = await acquire();
+    expect(retry.lease.id).toBe(first.lease.id); expect(creates).toBe(1); expect(destroys).toBe(0);
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(environmentLeases)).toHaveLength(1);
+    expect(await nextEnvironmentCleanupAt(db)).toBeNull();
+  });
+  it("recovers a failed lease write with a fresh receiver and empty web disk", async () => {
+    await rejectWrites("environment_leases", "reject_fixture_lease");
+    await expect(acquire()).rejects.toThrow();
+    expect(creates).toBe(1); expect(await readdir(spool)).toEqual([]);
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "pending" })]);
+    await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`);
+    await due();
+    await Promise.all([runtime().flushDeferredOrphanCleanups(), runtime().flushDeferredOrphanCleanups()]);
+    const leases = await db.select().from(environmentLeases);
+    expect(leases).toEqual([expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] })]);
+    expect(creates).toBe(1);
+    failDestroy = true;
+    await expect(runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! })).rejects.toThrow("Fixture teardown failed");
+    expect(resources.size).toBe(1);
+    failDestroy = false;
+    await runtime().retryPendingSandboxTeardown({ environment: null, lease: (await environmentService(db).getLeaseById(leases[0]!.id))! });
+    expect(resources.size).toBe(0); expect(destroys).toBe(2);
+  });
+  it("reconstructs ownership in a replacement receiver process with a different empty web disk", async () => {
+    const provider = createServer(async (request, response) => {
+      try {
+        let body = ""; for await (const chunk of request) body += chunk.toString();
+        const { method, input } = JSON.parse(body);
+        response.setHeader("content-type", "application/json");
+        response.end(JSON.stringify(await (workers.call as any)(pluginId, method, input)));
+      } catch { response.statusCode = 500; response.end("{}"); }
+    });
+    await new Promise<void>(resolve => provider.listen(0, "127.0.0.1", resolve));
+    const address = provider.address() as { port: number };
+    const replacementDisk = await mkdtemp(join(tmpdir(), "replacement-sandbox-web-"));
+    const fixture = fileURLToPath(new URL("./fixtures/sandbox-creation-receiver.ts", import.meta.url));
+    async function receiver(action: string, disk: string) {
+      const result = await promisify(execFile)(process.execPath, ["--import", fileURLToPath(import.meta.resolve("tsx")), fixture], {
+        cwd: fileURLToPath(new URL("../../..", import.meta.url)), timeout: 60_000,
+        env: { ...process.env, CREATION_FIXTURE_ACTION: action, CREATION_FIXTURE_EMPTY_DISK: disk,
+          CREATION_FIXTURE_DATABASE_URL: database.connectionString, CREATION_FIXTURE_PROVIDER_URL: `http://127.0.0.1:${address.port}`,
+          CREATION_FIXTURE_COMPANY_ID: companyId, CREATION_FIXTURE_ENVIRONMENT_ID: environmentId, CREATION_FIXTURE_RUN_ID: runId },
+      });
+      return result.stdout;
+    }
+    try {
+      await rejectWrites("environment_leases", "reject_fixture_lease");
+      expect(await receiver("acquire", spool)).toContain('"ok":false');
+      expect(creates).toBe(1); expect(resources.size).toBe(1); expect(await readdir(spool)).toEqual([]);
+      await db.execute(sql`drop trigger reject_fixture_lease on environment_leases`);
+      await due();
+      expect(await receiver("recover", replacementDisk)).toContain('"ok":true');
+      expect(await receiver("recover", replacementDisk)).toContain('"ok":true');
+      expect(await db.select().from(environmentLeases)).toEqual([
+        expect.objectContaining({ status: "pending_cleanup", providerLeaseId: [...resources.values()][0] }),
+      ]);
+      expect(creates).toBe(1); expect(await readdir(replacementDisk)).toEqual([]);
+    } finally {
+      await new Promise<void>((resolve, reject) => provider.close(error => error ? reject(error) : resolve()));
+      await rm(replacementDisk, { recursive: true, force: true });
+    }
+  }, 180_000);
+  it("never repeats creation after a lost provider reply", async () => {
+    loseReply = true;
+    await expect(acquire()).rejects.toThrow("Lost provider reply");
+    await expect(acquire()).rejects.toThrow("requires reconciliation");
+    expect(creates).toBe(1);
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]);
+  });
+  it("retains cleanup ownership when the environment is deleted during creation", async () => {
+    beforeReply = async () => { await environmentService(db).remove(environmentId); };
+    await expect(acquire()).rejects.toThrow();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ environmentId: null, originalEnvironmentId: environmentId, state: "pending" })]);
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ environmentId: null, status: "pending_cleanup" })]);
+  });
+  it("rejects attachment after a company rebind and retains cleanup", async () => {
+    beforeReply = async () => {
+      const foreign = randomUUID(); await db.insert(companies).values({ id: foreign, name: "Foreign", issuePrefix: "FOR" });
+      await db.insert(builtInManagedResources).values({ companyId: foreign, bundleKey: "managed-environment", resourceKind: "environment",
+        resourceKey: "sandbox", resourceId: environmentId, stockVersion: "1", stockHash: "fixture" });
+    };
+    await expect(acquire()).rejects.toMatchObject({ status: 403 });
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ companyId, status: "pending_cleanup" })]);
+  });
+  it("backs off a failed lookup and recovers unrelated creations in the same pass", async () => {
+    loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
+    const [first] = await db.select().from(sandboxCreationOperations);
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
+    runId = randomUUID(); await db.insert(heartbeatRuns).values({ ...run!, id: runId });
+    await expect(acquire()).rejects.toThrow("Lost provider reply");
+    const originalCall = workers.call;
+    (workers as any).call = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => {
+      if (method === "environmentInspectLease" && params.creationOperation.id === first!.id) throw new Error("Transient fixture lookup");
+      return originalCall(plugin, method as any, params, timeout);
+    });
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    const rows = await db.select().from(sandboxCreationOperations);
+    expect(rows.find(value => value.id === first!.id)).toMatchObject({ state: "pending" });
+    expect(rows.find(value => value.id === first!.id)!.nextAttemptAt.getTime()).toBeGreaterThan(Date.now());
+    expect(rows.filter(value => value.state === "cleanup_pending")).toHaveLength(1);
+  });
+  it("does not attach a sandbox to a run stopped during creation", async () => {
+    beforeReply = async () => { await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); };
+    await expect(acquire()).rejects.toThrow("run authority changed");
+    expect(await db.select().from(environmentLeases)).toHaveLength(0);
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(environmentLeases)).toEqual([expect.objectContaining({ status: "pending_cleanup" })]);
+  });
+  it.each(["before lookup", "during lookup", "lookup fails after deletion"])("records manual reconciliation when the company is deleted %s", async (deletion) => {
+    loseReply = true; await expect(acquire()).rejects.toThrow("Lost provider reply");
+    async function deleteCompany() {
+      await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
+      await db.delete(agents).where(eq(agents.companyId, companyId));
+      await db.delete(companies).where(eq(companies.id, companyId));
+    }
+    if (deletion === "before lookup") await deleteCompany();
+    const originalCall = workers.call;
+    const calls = vi.fn(async (plugin: string, method: string, params: any, timeout: any) => {
+      if (method === "environmentInspectLease" && deletion !== "before lookup") {
+        await deleteCompany();
+        if (deletion === "lookup fails after deletion") throw new Error("Company credential no longer exists");
+      }
+      return originalCall(plugin, method as any, params, timeout);
+    });
+    (workers as any).call = calls;
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([
+      expect.objectContaining({ companyId, state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }),
+    ]);
+    expect(await db.select().from(environmentLeases)).toEqual([]);
+    expect(calls.mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(deletion === "before lookup" ? 0 : 1);
+    const count = calls.mock.calls.length;
+    await runtime().flushDeferredOrphanCleanups(); await runtime().flushDeferredOrphanCleanups();
+    expect(calls).toHaveBeenCalledTimes(count);
+    expect(await nextEnvironmentCleanupAt(db)).toBeNull(); expect(creates).toBe(1);
+    expect(await readdir(spool)).toEqual([]);
+  });
+  async function savedCreationCredential(providerConfigId?: string) {
+    const secrets = secretOwner.secretService(db);
+    const secret = await secrets.create(companyId, {
+      name: `creation-key-${randomUUID()}`, provider: "local_encrypted", providerConfigId, value: "fixture-private-key",
+    });
+    await secrets.createBinding({ companyId, secretId: secret.id, targetType: "environment", targetId: environmentId, configPath: "apiKey" });
+    await db.update(plugins).set({ manifestJson: sql`jsonb_set(${plugins.manifestJson}, '{environmentDrivers,0,configSchema}',
+      '{"type":"object","properties":{"apiKey":{"type":"string","format":"secret-ref"}}}'::jsonb)` }).where(eq(plugins.id, pluginId));
+    await db.update(environments).set({ config: { provider: "creation-fixture", image: "fixture", reuseLease: false, apiKey: secret.id } })
+      .where(eq(environments.id, environmentId));
+    loseReply = true;
+    await expect(acquire()).rejects.toThrow("Lost provider reply");
+    return secret;
+  }
+  it.each(["deleted", "disabled", "archived", "version-disabled", "vault-disabled", "vault-coming-soon"] as const)(
+    "retains a %s creation credential as quiet manual reconciliation across restart", async state => {
+    const secrets = secretOwner.secretService(db);
+    const vault = state.startsWith("vault-") ? await secrets.createProviderConfig(companyId, {
+      provider: "local_encrypted", displayName: "Creation vault", status: "ready", config: {},
+    }) : undefined;
+    const secret = await savedCreationCredential(vault?.id);
+    if (state === "deleted") await secrets.remove(secret.id);
+    else if (state === "disabled" || state === "archived") await secrets.update(secret.id, { status: state });
+    else if (state === "version-disabled") await db.update(companySecretVersions).set({ status: "disabled" })
+      .where(eq(companySecretVersions.secretId, secret.id));
+    else await secrets.updateProviderConfig(vault!.id, { status: state === "vault-disabled" ? "disabled" : "coming_soon" });
+    const inspectCalls = () => vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease").length;
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([
+      expect.objectContaining({ state: "needs_reconciliation" }),
+    ]);
+    expect(await db.select().from(environmentLeases)).toEqual([]);
+    expect(inspectCalls()).toBe(0);
+    expect(await nextEnvironmentCleanupAt(db)).toBeNull();
+    const restarted = runtime();
+    await restarted.flushDeferredOrphanCleanups(); await restarted.flushDeferredOrphanCleanups();
+    expect(inspectCalls()).toBe(0); expect(creates).toBe(1); expect(destroys).toBe(0);
+    expect(await nextEnvironmentCleanupAt(db)).toBeNull();
+    await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
+    const ownerId = randomUUID();
+    const hold = { ownerId, draining: true, startedAt: new Date(Date.now() - 1000), expiresAt: new Date(Date.now() + 60_000), activeRuns: 0, pendingWakes: 0 };
+    // An installed plugin is a separate blocker until its worker proves quiet.
+    expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none"))
+      .toEqual({ version: 1, backgroundWork: "present" });
+    expect(await readIdleSleepSafety(db, () => hold, Date.now, ownerId, async () => "none",
+      async () => ({ backgroundWork: "none", pluginIds: [pluginId] })))
+      .toEqual({ version: 1, backgroundWork: "none" });
+    expect(await readdir(spool)).toEqual([]);
+  });
+  it("retries a temporary creation credential resolution failure without losing ownership", async () => {
+    await savedCreationCredential();
+    const original = secretOwner.secretService;
+    const resolution = vi.fn(async () => { throw new HttpError(503, "Temporary credential provider failure"); });
+    const owner = vi.spyOn(secretOwner, "secretService").mockImplementation(client => ({
+      ...original(client), resolveSecretValueForSandboxCleanup: resolution,
+    }));
+    try {
+      await due(); await runtime().flushDeferredOrphanCleanups();
+      expect(resolution).toHaveBeenCalledTimes(1);
+      expect(await db.select().from(sandboxCreationOperations)).toEqual([
+        expect.objectContaining({ state: "pending", nextAttemptAt: expect.any(Date) }),
+      ]);
+      expect(await nextEnvironmentCleanupAt(db)).toBeGreaterThan(Date.now());
+      expect(vi.mocked(workers.call).mock.calls.filter(value => value[1] === "environmentInspectLease")).toHaveLength(0);
+    } finally { owner.mockRestore(); }
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "cleanup_pending" })]);
+    expect(await db.select().from(environmentLeases)).toHaveLength(1);
+    expect(creates).toBe(1); expect(destroys).toBe(0); expect(await readdir(spool)).toEqual([]);
+  });
+  it("makes unsupported uncertain outcomes explicit without another creation", async () => {
+    loseReply = true; await expect(acquire()).rejects.toThrow();
+    await db.update(plugins).set({ manifestJson: sql`${plugins.manifestJson} #- '{environmentDrivers,0,creationOperationVersion}'` }).where(eq(plugins.id, pluginId));
+    await due(); await runtime().flushDeferredOrphanCleanups();
+    expect(await db.select().from(sandboxCreationOperations)).toEqual([expect.objectContaining({ state: "needs_reconciliation" })]);
+    expect(await nextEnvironmentCleanupAt(db)).toBeNull();
+    await expect(acquire()).rejects.toThrow("requires reconciliation"); expect(creates).toBe(1);
+  });
+});

server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts

@@ -5,7 +5,7 @@ import type { LifecycleAgent, LifecycleStore, LifecycleDriver } from "../applica
 import { startAgentLifecycle } from "../index.js";
 import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
 import { logger } from "../../../middleware/logger.js";
-import { lifecycleFailureMessage } from "../application/failure.js";
+import { lifecycleFailureMessage, lifecycleVerificationNeedsRetry } from "../application/failure.js";
 
 const agent: LifecycleAgent = { id: "agent", companyId: "company", lifecycleState: "verifying", lifecycleVersion: 3,
   lifecycleRequiredPluginIds: null, lifecycleError: null,
@@ -67,6 +67,20 @@ describe("lifecycle readiness handoff", () => {
 });
 
 describe("lifecycle failure diagnostics", () => {
+  it.each(["harness_auth_failed", "harness_model_unavailable", "harness_command_unavailable", "harness_cwd_invalid", "harness_probe_unexpected_output"])("waits for repair only for a host verification failure: %s", code => {
+    const error = { code };
+    expect(lifecycleVerificationNeedsRetry({ stage: "host", phase: "verifying" }, error)).toBe(true);
+    expect(lifecycleVerificationNeedsRetry({ stage: "plugin", phase: "verifying" }, error)).toBe(false);
+    expect(lifecycleVerificationNeedsRetry({ stage: "host", phase: "cleaning_up" }, error)).toBe(false);
+  });
+
+  it("allows two automatic retries of a generic failed test before waiting for repair", () => {
+    for (const verificationFailures of [1, 2, 3, 8_000]) {
+      const context = { stage: "host" as const, phase: "verifying" as const, attempts: 8_000, verificationFailures };
+      expect(lifecycleVerificationNeedsRetry(context, { code: "harness_test_failed" })).toBe(verificationFailures >= 3);
+    }
+  });
+
   it.each([
     ["host", "harness_model_unavailable", "Execution configuration check failed. The selected model is unavailable."],
     ["plugin", "required_plugin_unavailable", "Execution configuration check: required plugin step failed. A required plugin is unavailable."],
@@ -80,7 +94,8 @@ describe("lifecycle failure diagnostics", () => {
     if (stage === "select_plugins") driver.requiredPluginIds.mockRejectedValue(error);
     if (stage === "transition") store.change.mockRejectedValue(error);
     await worker.process(agent.id);
-    expect(store.defer).toHaveBeenCalledWith(agent, expect.any(String), expect.stringContaining(expected), expect.any(Date));
+    expect(store.defer).toHaveBeenCalledWith(agent, expect.any(String), expect.stringContaining(expected), expect.any(Date),
+      ...(stage === "host" ? [{ retryRequired: true }] : []));
     expect(JSON.stringify(store.defer.mock.calls)).not.toContain("secret provider output");
     expect(JSON.stringify(store.defer.mock.calls)).not.toContain("unknown private code");
     await worker.stop();

server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts

@@ -0,0 +1,182 @@
+import { randomUUID } from "node:crypto";
+import { agents, companies, createDb, applyPendingMigrations, closeRegisteredClients, type Db } from "@paperclipai/db";
+import { eq, sql } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
+import { createAgentLifecycle, configureAgentLifecycle } from "../../../services/agent-lifecycle.js";
+import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
+import { startAgentLifecycle } from "../index.js";
+import { createLifecycleStore } from "../adapters/postgres.js";
+import { createLifecycleWorker } from "../application/worker.js";
+import type { LifecycleAgent } from "../application/ports.js";
+import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js";
+
+const external = resolveExternalTestDatabaseUrl("PAPERCLIP_LIFECYCLE_VERIFICATION_RETRY_TEST_DATABASE_URL");
+const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
+(support.supported ? describe : describe.skip)("failed lifecycle verification retries", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: Db;
+  let companyId: string;
+  const workers: Array<{ stop(): Promise<void> }> = [];
+  beforeAll(async () => {
+    if (external) { await applyPendingMigrations(external); db = createDb(external); }
+    else { database = await startEmbeddedPostgresTestDatabase("paperclip-verification-retry-"); db = createDb(database.connectionString); }
+  }, 90_000);
+  afterAll(async () => { if (external) await closeRegisteredClients(external); else await database?.cleanup(); });
+  beforeEach(async () => {
+    companyId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Verification retry", issuePrefix: `V${companyId.slice(0, 7)}` });
+  });
+  afterEach(async () => {
+    for (const worker of workers.splice(0)) await worker.stop();
+    // Retire only this test's synthetic records so later coordinator reads stay isolated.
+    await db.update(agents).set({ lifecycleState: "terminated", status: "terminated" }).where(eq(agents.companyId, companyId));
+  });
+  const current = async (id: string) => (await db.select().from(agents).where(eq(agents.id, id)))[0]!;
+  async function failed(code = "harness_test_failed") {
+    const runHost = vi.fn(async (agent: LifecycleAgent) => {
+      if (agent.lifecycleState === "verifying") throw Object.assign(new Error("private provider detail"), { code });
+      return "complete" as const;
+    });
+    const driver = { requiredPluginIds: async () => [], runHost, runPlugin: async () => "complete" as const };
+    const work = configureAgentLifecycle(db, driver); workers.push(work);
+    const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "Retry fixture", adapterType: "process", adapterConfig: { command: "missing-command" } });
+    await work.process(agent.id);
+    if (code === "harness_test_failed") {
+      for (let retry = 0; retry < 2; retry++) {
+        expect((await current(agent.id)).lifecycleOperation?.retryRequired).toBeUndefined();
+        await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
+        await work.process(agent.id);
+      }
+    }
+    return { agent: await current(agent.id), work, driver, runHost };
+  }
+
+  it("persists a failed test across worker restarts without scanning, claiming, or probing it again", async () => {
+    const { agent, work, driver, runHost } = await failed();
+    expect(agent).toMatchObject({ lifecycleState: "verifying", status: "paused", lifecycleOperation: { retryRequired: true } });
+    expect(agent.lifecycleOperation).not.toHaveProperty("retryAt");
+    expect(agent.lifecycleOperation).not.toHaveProperty("leaseOwner");
+    expect(agent.lifecycleError).toContain("Automatic verification is paused");
+    expect(agent.lifecycleError).not.toContain("private provider detail");
+    await work.stop();
+    const restarted = startAgentLifecycle(db, createAgentLifecycleEffects(), driver, () => true); workers.push(restarted);
+    const calls = runHost.mock.calls.length;
+    expect(await restarted.hasPending()).toBe(false);
+    expect(restarted.nextRunAt()).toBeNull();
+    await restarted.sweep();
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    const tomorrow = new Date(Date.now() + 86_400_000);
+    expect(await store.pending(100, tomorrow)).not.toContain(agent.id);
+    expect(await store.claim(agent.id, "new-owner", tomorrow)).toBeNull();
+    await work.process(agent.id);
+    expect(runHost).toHaveBeenCalledTimes(calls);
+    expect((await current(agent.id)).lifecycleOperation).toEqual(agent.lifecycleOperation);
+  });
+
+  it("permits an explicit retry of the same operation and completes verification before readiness", async () => {
+    const { agent, work, runHost } = await failed();
+    runHost.mockResolvedValue("complete");
+    await createAgentLifecycle(db).retry(agent.id);
+    expect((await current(agent.id)).lifecycleOperation).not.toHaveProperty("retryRequired");
+    expect((await current(agent.id)).lifecycleOperation).not.toHaveProperty("verificationFailures");
+    expect((await current(agent.id)).lifecycleOperation!.id).toBe(agent.lifecycleOperation!.id);
+    await work.process(agent.id);
+    expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null });
+  });
+
+  it("does not spend verification retries on claims deferred before the host test", async () => {
+    const [agent] = await db.insert(agents).values({ companyId, name: "Deferred verification", status: "paused",
+      lifecycleState: "verifying", lifecycleRequiredPluginIds: [],
+      lifecycleOperation: { id: randomUUID(), hostComplete: false, completedPluginIds: [], attempts: 0 } }).returning();
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    let allowed = true;
+    let deferBeforeHost = true;
+    const runHost = vi.fn(async () => { throw Object.assign(new Error("test failed"), { code: "harness_test_failed" }); });
+    const work = createLifecycleWorker({ ...store, async claim(...args) {
+      const claimed = await store.claim(...args);
+      if (claimed && deferBeforeHost) allowed = false;
+      return claimed;
+    } }, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost }, () => {}, () => allowed);
+    workers.push(work);
+    const due = async () => {
+      allowed = true;
+      await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
+      await work.process(agent.id);
+    };
+    await due(); await due();
+    expect(runHost).not.toHaveBeenCalled();
+    expect((await current(agent.id)).lifecycleOperation).toMatchObject({ attempts: 2 });
+    expect((await current(agent.id)).lifecycleOperation).not.toHaveProperty("verificationFailures");
+    deferBeforeHost = false;
+    await due();
+    expect((await current(agent.id)).lifecycleOperation).toMatchObject({ attempts: 3, verificationFailures: 1 });
+    expect((await current(agent.id)).lifecycleOperation?.retryRequired).toBeUndefined();
+    // Another deferred claim preserves the already recorded failed test.
+    deferBeforeHost = true; await due();
+    expect((await current(agent.id)).lifecycleOperation?.verificationFailures).toBe(1);
+    deferBeforeHost = false;
+    await due();
+    expect((await current(agent.id)).lifecycleOperation?.retryRequired).toBeUndefined();
+    await due();
+    expect(runHost).toHaveBeenCalledTimes(3);
+    expect((await current(agent.id)).lifecycleOperation).toMatchObject({ verificationFailures: 3, retryRequired: true });
+  });
+
+  it("recovers from a generic failed test before the automatic retry budget is exhausted", async () => {
+    const { agent, work, runHost } = await failed("harness_probe_timed_out");
+    runHost.mockRejectedValueOnce(Object.assign(new Error("temporary test failure"), { code: "harness_test_failed" }));
+    await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
+    await work.process(agent.id);
+    expect((await current(agent.id)).lifecycleOperation?.retryRequired).toBeUndefined();
+    runHost.mockResolvedValue("complete");
+    await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
+    await work.process(agent.id);
+    expect((await current(agent.id)).lifecycleState).toBe("ready");
+  });
+
+  it("rechecks changed execution configuration but leaves a name edit waiting", async () => {
+    const { agent, work, runHost } = await failed();
+    const lifecycle = createAgentLifecycle(db);
+    await lifecycle.updateConfiguration(agent.id, { name: "Renamed fixture" });
+    expect((await current(agent.id)).lifecycleOperation!.retryRequired).toBe(true);
+    runHost.mockResolvedValue("complete");
+    await lifecycle.updateConfiguration(agent.id, { adapterConfig: { command: process.execPath } });
+    const changed = await current(agent.id);
+    expect(changed.lifecycleOperation).not.toHaveProperty("retryRequired");
+    expect(changed.lifecycleOperation).not.toHaveProperty("verificationFailures");
+    expect(changed.lifecycleOperation!.id).not.toBe(agent.lifecycleOperation!.id);
+    await work.process(agent.id);
+    expect((await current(agent.id)).lifecycleState).toBe("ready");
+  });
+
+  it("does not let a stale verification failure park newly saved configuration", async () => {
+    const { agent, work } = await failed("harness_auth_failed");
+    await work.stop();
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    await store.change(agent.id, "retry");
+    const claimed = await store.claim(agent.id, "old-verification", new Date());
+    expect(claimed).not.toBeNull();
+    await createAgentLifecycle(db).updateConfiguration(agent.id, { adapterConfig: { command: process.execPath } });
+    expect(await store.defer(claimed!, "old-verification", "Old failure", new Date(), { retryRequired: true })).toBe(false);
+    expect((await current(agent.id)).lifecycleOperation?.retryRequired).toBeUndefined();
+  });
+
+  it.each(["harness_probe_timed_out", "test_environment_unavailable", "ECONNRESET", "unclassified"])("keeps automatic recovery for %s", async code => {
+    const { agent } = await failed(code);
+    expect(agent.lifecycleOperation).not.toHaveProperty("retryRequired");
+    expect(Date.parse(agent.lifecycleOperation!.retryAt!)).toBeGreaterThan(Date.now());
+    const store = createLifecycleStore(db, createAgentLifecycleEffects());
+    expect(await store.pending(100, new Date(Date.now() + 120_000))).toContain(agent.id);
+    expect(await store.claim(agent.id, "automatic-retry", new Date(Date.now() + 120_000))).not.toBeNull();
+  });
+
+  it("still completes termination and cleanup when verification needs repair", async () => {
+    const { agent, work, runHost } = await failed();
+    runHost.mockResolvedValue("complete");
+    await createAgentLifecycle(db).terminateAgent(agent.id);
+    await work.process(agent.id);
+    expect((await current(agent.id)).lifecycleState).toBe("terminated");
+    expect(runHost.mock.calls.map(([a]) => a.lifecycleState)).toContain("cleaning_up");
+  });
+});

server/src/modules/agent-lifecycle/adapters/configuration.ts

@@ -37,7 +37,8 @@ export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects
     ...(!prepareSetup && current.lifecycleHolds.includes("setup") && ["preparing", "verifying"].includes(current.lifecycleState)
       ? { lifecycleState: "pausing" as const } : {}),
     lifecycleOperation: { ...current.lifecycleOperation!, id: randomUUID(), hostComplete: false,
-      completedPluginIds: [], leaseOwner: undefined, leaseUntil: undefined, retryAt: undefined },
+      completedPluginIds: [], attempts: 0, leaseOwner: undefined, leaseUntil: undefined,
+      retryAt: undefined, retryRequired: undefined, verificationFailures: undefined },
   } : {}) }).where(eq(agents.id, id)).returning();
   await effects.completeConfiguration(tx, current, updated, patch, options, publications);
   return agentRecordQueries(tx).getById(id);

server/src/modules/agent-lifecycle/adapters/postgres.ts

@@ -17,6 +17,7 @@ type ChangeOptions = { version?: number; owner?: string; reason?: string };
 export const pendingLifecycleWork = and(
   inArray(agents.lifecycleState, ["preparing", "verifying", "pausing", "resuming", "terminating", "cleaning_up"]),
   sql`not (${agents.lifecycleState} = 'verifying' and ${agents.lifecycleHolds} @> '["setup"]'::jsonb)`,
+  sql`not (${agents.lifecycleState} = 'verifying' and coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true')`,
 );
 
 /** The caller owns the accounting transaction and has locked this agent. */
@@ -33,7 +34,7 @@ async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: Lifec
     if (agent.lifecycleOperation?.leaseUntil && Date.parse(agent.lifecycleOperation.leaseUntil) > Date.now()) return agent;
     await effects.notifyWork(tx);
     const [updated] = await tx.update(agents).set({ lifecycleError: null,
-      lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'leaseOwner' - 'leaseUntil'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
+      lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
     return updated;
   }
   if (command === "resume" && !["ready", "paused", "pausing", "resuming"].includes(agent.lifecycleState)) transition(agent.lifecycleState, command);
@@ -177,11 +178,17 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
         sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
       return rows.length > 0;
     },
-    async defer(agent, owner, error, now) {
+    async defer(agent, owner, error, now, options) {
+      const retry: Partial<AgentLifecycleOperation> = options?.retryRequired && agent.lifecycleState === "verifying"
+        ? { retryRequired: true }
+        : { retryAt: new Date(now.getTime() + (error ? 60_000 : 2_000)).toISOString() };
+      if (options?.verificationFailures !== undefined && agent.lifecycleState === "verifying") {
+        retry.verificationFailures = options.verificationFailures;
+      }
       const rows = await db.update(agents).set({
         lifecycleError: error,
-        lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil')
-          || jsonb_build_object('retryAt', ${new Date(now.getTime() + (error ? 60_000 : 2_000)).toISOString()}::text)`,
+        lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'retryRequired')
+          || ${JSON.stringify(retry)}::jsonb`,
         updatedAt: now,
       }).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
         sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });

server/src/modules/agent-lifecycle/application/failure.ts

@@ -15,6 +15,16 @@ const reasons = {
 } as const;
 
 const transientCodes = new Set(["ECONNREFUSED", "ECONNRESET", "ETIMEDOUT", "40001", "40P01", "53300", "57P01"]);
+const configurationFailures = new Set([
+  "harness_auth_failed", "harness_model_unavailable",
+  "harness_command_unavailable", "harness_cwd_invalid", "harness_probe_unexpected_output",
+]);
+
+export function lifecycleVerificationNeedsRetry(context: LifecycleFailure, error: unknown): boolean {
+  const code = lifecycleFailureCode(error) ?? "";
+  return context.phase === "verifying" && context.stage === "host"
+    && (configurationFailures.has(code) || (code === "harness_test_failed" && (context.verificationFailures ?? 0) >= 3));
+}
 
 export function lifecycleFailureCode(error: unknown): string | undefined {
   const candidate = error as { code?: unknown; cause?: { code?: unknown } } | null;
@@ -36,5 +46,6 @@ export function lifecycleFailureMessage(context: LifecycleFailure, error: unknow
   const code = lifecycleFailureCode(error);
   const reason = code && Object.hasOwn(reasons, code) ? reasons[code as keyof typeof reasons]
     : "Paperclip could not complete this step. It will retry automatically. If this continues, contact your instance administrator.";
-  return `${step} failed. ${reason}`;
+  return `${step} failed. ${reason}${lifecycleVerificationNeedsRetry(context, error)
+    ? " Automatic verification is paused. Update Configuration or choose Retry now." : ""}`;
 }

server/src/modules/agent-lifecycle/application/ports.ts

@@ -20,7 +20,7 @@ export interface LifecycleStore {
   setRequiredPlugins(agent: LifecycleAgent, owner: string, pluginIds: string[]): Promise<boolean>;
   completeHost(agent: LifecycleAgent, owner: string): Promise<boolean>;
   completePlugin(agent: LifecycleAgent, owner: string, pluginId: string): Promise<boolean>;
-  defer(agent: LifecycleAgent, owner: string, error: string | null, now: Date): Promise<boolean>;
+  defer(agent: LifecycleAgent, owner: string, error: string | null, now: Date, options?: { retryRequired: boolean; verificationFailures?: number }): Promise<boolean>;
   pending(limit: number, now: Date): Promise<string[]>;
 }
 
@@ -40,5 +40,7 @@ export interface LifecycleFailure {
   phase?: AgentLifecycleState;
   operationId?: string;
   version?: number;
+  attempts?: number;
+  verificationFailures?: number;
   pluginId?: string;
 }

server/src/modules/agent-lifecycle/application/worker.ts

@@ -1,7 +1,7 @@
 import { randomUUID } from "node:crypto";
 import type { LifecycleDriver, LifecycleStore, LifecycleFailure } from "./ports.js";
 import { isLifecycleWorkPending } from "../domain/policy.js";
-import { lifecycleFailureMessage } from "./failure.js";
+import { lifecycleFailureCode, lifecycleFailureMessage, lifecycleVerificationNeedsRetry } from "./failure.js";
 
 /** Database leases protect each attempt across server processes. */
 export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDriver, reportFailure: (context: LifecycleFailure, error: unknown) => void, canRun = () => true) {
@@ -20,7 +20,8 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr
       const agent = await attempt({ stage: "claim", agentId: id }, () => store.claim(id, owner, new Date()));
       if (!agent || !isLifecycleWorkPending(agent.lifecycleState)) return;
       const context = { agentId: agent.id, companyId: agent.companyId, phase: agent.lifecycleState,
-        operationId: agent.lifecycleOperation?.id, version: agent.lifecycleVersion };
+        operationId: agent.lifecycleOperation?.id, version: agent.lifecycleVersion,
+        attempts: agent.lifecycleOperation?.attempts };
       let failureContext: LifecycleFailure = { ...context, stage: "host" };
       const runStep = <T>(stage: LifecycleFailure["stage"], work: () => Promise<T>, pluginId?: string) => {
         const stepContext = { ...context, stage, ...(pluginId ? { pluginId } : {}) };
@@ -70,8 +71,16 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr
         }
         if (changed && !isLifecycleWorkPending(changed.lifecycleState)) return;
       } catch (error) {
+        // Claims can defer before runHost. Only a failed host test spends this budget.
+        const verificationFailures = failureContext.stage === "host" && agent.lifecycleState === "verifying"
+          && lifecycleFailureCode(error) === "harness_test_failed"
+          ? (agent.lifecycleOperation?.verificationFailures ?? 0) + 1 : undefined;
+        failureContext = { ...failureContext, verificationFailures };
         const message = lifecycleFailureMessage(failureContext, error);
-        if (!await runStep("defer", () => store.defer(agent, owner, message, new Date()))) continue;
+        const retryRequired = lifecycleVerificationNeedsRetry(failureContext, error);
+        if (!await runStep("defer", () => retryRequired || verificationFailures !== undefined
+          ? store.defer(agent, owner, message, new Date(), { retryRequired, ...(verificationFailures !== undefined ? { verificationFailures } : {}) })
+          : store.defer(agent, owner, message, new Date()))) continue;
         return;
       } finally {
         clearInterval(renewal);

server/src/services/agent-harness-verification.ts

@@ -23,7 +23,8 @@ export function assertHarnessTestPassed(result: AdapterEnvironmentTestResult) {
   const incomplete = !verified && (result.status === "warn" || result.checks.some(check => check.code.includes("hello_probe")));
   if (result.status === "fail" || authenticationFailed || incomplete) {
     const has = (suffix: string) => result.checks.some(check => check.code.endsWith(suffix));
-    const code = authenticationFailed ? "harness_auth_failed"
+    const code = result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable"
+      : authenticationFailed ? "harness_auth_failed"
       : has("_hello_probe_model_unavailable") || has("_model_invalid") ? "harness_model_unavailable"
       : has("_command_unresolvable") ? "harness_command_unavailable"
       : has("_cwd_invalid") ? "harness_cwd_invalid"

server/src/services/environment-cleanup-state.ts

@@ -1,5 +1,5 @@
 import { eq, sql } from "drizzle-orm";
-import { environmentLeases, type Db } from "@paperclipai/db";
+import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
 import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
 
 export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
@@ -20,7 +20,10 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
   const [row] = await db.select({ next: sql<string | null>`min(greatest(
     extract(epoch from ${environmentLeases.updatedAt}) * 1000 + ${backoffMs},
     ${pendingCleanupDeadlineSql()}))` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
-  const saved = row?.next == null ? null : Number(row.next);
+  const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
+    .from(sandboxCreationOperations).where(eq(sandboxCreationOperations.state, "pending"));
+  const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
+  const saved = deadlines.length ? Math.min(...deadlines) : null;
   const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
   return saved === null ? buffered : buffered === null ? saved : Math.min(saved, buffered);
 }

server/src/services/environment-runtime.ts

@@ -1,3 +1,6 @@
+import { HttpError } from "../errors.js";
+import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
+import { createOwnedSandbox, recoverSandboxCreations } from "./sandbox-creation-operations.js";
 import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js";
 import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
 import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
@@ -11,7 +14,6 @@ import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/nati
 import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
 import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
 import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
-import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
 import { remoteTerminationReceipt } from "./remote-execution-termination.js";
 import { hasNativeWorkspaceExportResume, releaseCompletedNativeWorkspaceExportRetention } from "./native-runtime/native-workspace-export-resume.js";
 import { createHash, randomUUID } from "node:crypto";
@@ -93,7 +95,7 @@ import {
   resolvePluginExecuteRpcTimeoutMs,
   resumePluginEnvironmentLease,
 } from "./plugin-environment-driver.js";
-import { collectSecretRefPaths } from "./json-schema-secret-refs.js";
+import { collectSecretRefPaths, readConfigValueAtPath } from "./json-schema-secret-refs.js";
 import { buildWorkspaceRealizationRecordFromDriverInput } from "./workspace-realization.js";
 import {
   createSandboxOrphanCleanupSpool,
@@ -698,39 +700,6 @@ export interface EnvironmentRuntimeDriver {
   flushDeferredOrphanCleanups?(): Promise<{ recovered: number; pending: number }>;
 }
 
-/**
- * The acquire provisioned a remote sandbox, the lease insert rejected it, the
- * compensating teardown failed, and every retry of the durable pending-cleanup
- * write also failed. A live sandbox now has no lease row and no cleanup record,
- * so no sweep can find it. The acquire throws this error so the failure stays visible
- * and never resolves to a clean rejection. The `cause` field holds the original
- * lease insert rejection. The `cleanupWriteError` field holds the write failure.
- */
-export class SandboxOrphanCleanupWriteError extends Error {
-  readonly provider: string;
-  readonly providerLeaseId: string | null;
-  readonly cleanupWriteError: unknown;
-
-  constructor(input: {
-    provider: string;
-    providerLeaseId: string | null;
-    cause: unknown;
-    cleanupWriteError: unknown;
-  }) {
-    super(
-      `Sandbox provider "${input.provider}" leaked lease ` +
-        `"${input.providerLeaseId ?? "unknown"}": the acquire could not tear it down ` +
-        "and could not record a durable pending-cleanup row. The live sandbox has no " +
-        "durable cleanup state and needs a manual teardown.",
-      { cause: input.cause },
-    );
-    this.name = "SandboxOrphanCleanupWriteError";
-    this.provider = input.provider;
-    this.providerLeaseId = input.providerLeaseId;
-    this.cleanupWriteError = input.cleanupWriteError;
-  }
-}
-
 /** A reusable sandbox could not be resumed, but has not been proven lost. */
 export class ReusableSandboxResumeError extends Error {
   readonly provider: string;
@@ -785,17 +754,6 @@ export type ProviderResourceDisposition =
 const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000;
 const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100;
 
-// The durable pending-cleanup write is the only automated way to find a leaked
-// sandbox. A single insert can lose to a transient database fault (a dropped
-// connection, a lock timeout, a serialization conflict). So the acquire retries
-// the write a few times with a short backoff before it gives up. A later
-// attempt can still land the durable row that a sweep finds.
-const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS = 3;
-const DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS = 100;
-
-// A prolonged database outage after a failed teardown can buffer many orphan
-// records in-process. The bound stops that buffer from growing without limit. A
-// full buffer keeps the error log as the last durable handle for a new orphan.
 const DEFAULT_DEFERRED_ORPHAN_CLEANUP_BUFFER_LIMIT = 256;
 
 function delay(ms: number): Promise<void> {
@@ -1311,8 +1269,6 @@ function createSandboxEnvironmentDriver(
     pluginWorkerManager?: PluginWorkerManager;
     pluginWorkerReadyTimeoutMs?: number;
     pluginWorkerReadyPollMs?: number;
-    pendingCleanupWriteAttempts?: number;
-    pendingCleanupWriteBackoffMs?: number;
     deferredOrphanCleanupBufferLimit?: number;
     orphanCleanupSpool?: SandboxOrphanCleanupSpool;
     orphanCleanupSpoolDir?: string;
@@ -1321,16 +1277,6 @@ function createSandboxEnvironmentDriver(
   const pluginWorkerManager = options.pluginWorkerManager;
   const pluginWorkerReadyTimeoutMs = options.pluginWorkerReadyTimeoutMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS;
   const pluginWorkerReadyPollMs = options.pluginWorkerReadyPollMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS;
-  // The retry count is at least one attempt, so a zero or negative override
-  // never skips the durable write. The backoff is at least zero milliseconds.
-  const pendingCleanupWriteAttempts = Math.max(
-    1,
-    options.pendingCleanupWriteAttempts ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_ATTEMPTS,
-  );
-  const pendingCleanupWriteBackoffMs = Math.max(
-    0,
-    options.pendingCleanupWriteBackoffMs ?? DEFAULT_SANDBOX_ORPHAN_CLEANUP_WRITE_BACKOFF_MS,
-  );
   // The buffer holds at least one orphan, so a zero or negative override never
   // drops every buffered record.
   const deferredOrphanCleanupBufferLimit = Math.max(
@@ -1339,15 +1285,6 @@ function createSandboxEnvironmentDriver(
   );
   const environmentsSvc = environmentService(db);
 
-  // A live sandbox whose teardown failed needs a durable `pending_cleanup` row,
-  // so a sweep can find and release it. When every synchronous write attempt
-  // fails, the database is down but the process still runs. So the driver keeps
-  // the orphan record in this in-process buffer, and a later cleanup sweep
-  // flushes it back to the database. The buffer closes the common window where
-  // the database recovers after the synchronous attempts but while the process
-  // still runs. The `pending_cleanup` failure reason matches the synchronous
-  // write, so a flushed row and a synchronous row are indistinguishable to the
-  // sweep.
   const deferredOrphanCleanups: DeferredOrphanCleanupRecord[] = [];
   // Keep the token through queue splices, failed flushes and buffer overflow.
   // Release only when the database durably owns cleanup of this exact orphan.
@@ -1363,16 +1300,6 @@ function createSandboxEnvironmentDriver(
   };
   const DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON = "acquire_rejected_teardown_failed";
 
-  // The in-process buffer alone loses an orphan on a restart. So the driver also
-  // persists each buffered orphan to this durable local spool. The spool keeps
-  // the record on the local disk, next to the other durable server state, so a
-  // restart still finds it. The first flush after a restart loads the spool back
-  // into the buffer, so the sweep re-inserts the durable row. The spool holds no
-  // secret value: it persists the same sanitized metadata the database row
-  // holds, so a restart-recovered record still authorizes the teardown but never
-  // exposes a credential. Only a crash during the outage before the durable
-  // spool write lands loses the record; the error log stays the last handle for
-  // that narrow case.
   const orphanCleanupSpool =
     options.orphanCleanupSpool ?? createSandboxOrphanCleanupSpool(options.orphanCleanupSpoolDir);
 
@@ -1460,143 +1387,6 @@ function createSandboxEnvironmentDriver(
     return orphanFlush;
   };
 
-  // Build the safe diagnostic fields for an orphan record. The provider
-  // identifiers are the only fields any diagnostic log emits, and they carry no
-  // secret. A caught write exception never enters a log: it can carry a
-  // credential in its name, code, message, cause, or stack.
-  const orphanDiagnosticFields = (record: DeferredOrphanCleanupRecord) => ({
-    errorKind: SANDBOX_ORPHAN_CLEANUP_WRITE_ERROR_KIND,
-    provider: record.provider,
-    providerLeaseId: record.providerLeaseId,
-    companyId: record.companyId,
-    environmentId: record.environmentId,
-    executionWorkspaceId: record.executionWorkspaceId,
-    issueId: record.issueId,
-    heartbeatRunId: record.heartbeatRunId,
-  });
-
-  // Try to write the durable `pending_cleanup` row for an orphan sandbox, with a
-  // few retries. Report the new lease id on success, or a null lease id and the
-  // last write error on total failure. This function never buffers, never logs
-  // at error level, and never throws. The caller decides whether a total failure
-  // is fatal, because a later successful teardown removes the orphan and needs
-  // no durable row.
-  //
-  // The row carries the provider lease id and the same resolution metadata a
-  // normal lease holds, so a later cleanup sweep finds and releases the orphan.
-  // One atomic insert writes the row directly in the terminal `pending_cleanup`
-  // state, and the insert skips the company-binding assertion, so it records the
-  // orphan even for a foreign-bound environment. A transient database fault (a
-  // dropped connection, a lock timeout, a serialization conflict) can reject one
-  // insert but clear on the next, so the write retries a few times with a short
-  // backoff.
-  const tryWriteDurablePendingCleanup = async (
-    record: DeferredOrphanCleanupRecord,
-  ): Promise<{ leaseId: string | null; lastError: unknown }> => {
-    const diagnosticFields = orphanDiagnosticFields(record);
-    let lastCleanupWriteError: unknown;
-    for (let attempt = 1; attempt <= pendingCleanupWriteAttempts; attempt += 1) {
-      try {
-        const lease = await environmentsSvc.insertPendingCleanupLease({
-          companyId: record.companyId,
-          environmentId: record.environmentId,
-          executionWorkspaceId: record.executionWorkspaceId,
-          issueId: record.issueId,
-          heartbeatRunId: record.heartbeatRunId,
-          provider: record.provider,
-          providerLeaseId: record.providerLeaseId,
-          metadata: record.metadata,
-          failureReason: DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON,
-        });
-        // The durable row exists now, so a sweep can find and release the orphan.
-        return { leaseId: lease.id, lastError: undefined };
-      } catch (cleanupWriteError) {
-        lastCleanupWriteError = cleanupWriteError;
-        if (attempt < pendingCleanupWriteAttempts) {
-          // The write failed, but attempts remain. Log the retry at warn level
-          // and back off, then try again. A later attempt can still land the
-          // durable row.
-          logger.warn(
-            { ...diagnosticFields, attempt, maxAttempts: pendingCleanupWriteAttempts },
-            "sandbox orphan cleanup write failed; retrying the durable pending-cleanup write",
-          );
-          await delay(pendingCleanupWriteBackoffMs * attempt);
-        }
-      }
-    }
-    return { leaseId: null, lastError: lastCleanupWriteError };
-  };
-
-  // Release the durable `pending_cleanup` row after a successful inline teardown.
-  // The teardown removed the orphan sandbox, so the row is no longer needed. The
-  // release moves the row to the terminal `expired` state with a `success`
-  // cleanup status, the same state the sweep records for a successful teardown.
-  //
-  // The release is best-effort. If it fails, the row stays `pending_cleanup`, so
-  // a later sweep runs the idempotent teardown on the already-gone sandbox and
-  // releases the row itself. A failed release never loses or leaks the orphan.
-  const releaseCleanedUpOrphanRow = async (
-    leaseId: string,
-    diagnosticFields: Record<string, unknown>,
-    receipt: unknown,
-  ): Promise<void> => {
-    try {
-      const lease = await environmentsSvc.getLeaseById(leaseId);
-      await environmentsSvc.releaseLease(leaseId, "expired", {
-        ...(lease ? { remoteExecutionTermination: remoteTerminationReceipt(lease, receipt) } : {}),
-        cleanupStatus: "success",
-        failureReason: "acquire_rejected_teardown_succeeded",
-      });
-    } catch {
-      // The caught release exception never enters a log: it can carry a
-      // credential in its name, code, message, cause, or stack.
-      logger.warn(
-        diagnosticFields,
-        "could not release the pending-cleanup row after a successful orphan teardown; a later sweep reconciles the released row",
-      );
-    }
-  };
-
-  // Neither the durable write nor the inline teardown worked, so the orphan
-  // sandbox is live with no durable row. Persist the orphan to the durable spool
-  // and buffer it in-process, so a later cleanup-sweep flush lands the durable
-  // row. The spool survives a restart, so the flush recovers the orphan even
-  // after a crash. Then log the plain provider identifiers at error level, so an
-  // operator keeps a handle to tear the sandbox down by hand. Only a crash before
-  // the spool write lands loses the record, so the error log is the last handle
-  // for that narrow case. Throw `SandboxOrphanCleanupWriteError`, which the
-  // caller propagates. The error keeps the original insert rejection as its
-  // `cause`, so the real reason the acquire failed stays visible in the error
-  // chain.
-  const escalateUnwrittenOrphan = async (
-    record: DeferredOrphanCleanupRecord,
-    cause: unknown,
-    cleanupWriteError: unknown,
-  ): Promise<never> => {
-    trackOrphanIdleWork(record);
-    const persisted = await orphanCleanupSpool.append(record);
-    const buffered = enqueueDeferredOrphanCleanup(record);
-    logger.error(
-      {
-        ...orphanDiagnosticFields(record),
-        persisted,
-        buffered,
-        deferredBufferSize: deferredOrphanCleanups.length,
-      },
-      persisted
-        ? "sandbox orphan cleanup write failed; persisted the orphan to the durable spool for a later cleanup-sweep flush"
-        : buffered
-          ? "sandbox orphan cleanup write failed and the durable spool write failed; buffered the orphan in-process only for a later cleanup-sweep flush"
-          : "sandbox orphan cleanup write failed and neither the durable spool nor the in-process buffer kept the orphan; live sandbox needs a manual teardown",
-    );
-    throw new SandboxOrphanCleanupWriteError({
-      provider: record.provider,
-      providerLeaseId: record.providerLeaseId,
-      cause,
-      cleanupWriteError,
-    });
-  };
-
   // Resolve an uncertain creation to a verified provider ID, journal it, then
   // authorize deletion. This makes a later missing-ID response conclusive even
   // when the destructive RPC reply or the final lease update was lost.
@@ -1628,68 +1418,7 @@ function createSandboxEnvironmentDriver(
     }
   };
 
-  // Clean up an orphan after a rejected insert or uncertain provider creation.
-  // The acquire provisioned a remote sandbox, then the insert rejected (a
-  // foreign-company binding), so no lease row tracks the live sandbox. This
-  // handler records the durable `pending_cleanup` row FIRST, before the inline
-  // teardown, then tears the sandbox down.
-  //
-  // The write goes first on purpose. The conditional insert just returned a
-  // definitive rejection, so the database is reachable at this instant. An
-  // inline teardown can run long, and the database can drop during it. So a
-  // write after the teardown can fail and leave the orphan only in memory, which
-  // a restart loses. With the write first, the durable row lands while the
-  // database is proven up, so a crash after it still leaves a row that a sweep
-  // finds and releases.
-  //
-  // The teardown then runs. On success the orphan is gone, so the handler
-  // releases the durable row. On a failed teardown the row stays for the sweep.
-  // If the durable write itself failed and the teardown also failed, the handler
-  // buffers the orphan and throws `SandboxOrphanCleanupWriteError`. A successful
-  // teardown after a failed write leaves no orphan, so it needs no error.
-  const cleanUpRejectedOrphanSandbox = async (input: {
-    record: DeferredOrphanCleanupRecord;
-    cause: unknown;
-    canTeardown: boolean;
-    teardown: (pendingLeaseId: string | null) => Promise<unknown>;
-  }): Promise<boolean> => {
-    const durable = await tryWriteDurablePendingCleanup(input.record);
-    let teardownFailed = !input.canTeardown;
-    let receipt: unknown;
-    if (!teardownFailed) {
-      try {
-        receipt = await input.teardown(durable.leaseId);
-      } catch {
-        teardownFailed = true;
-      }
-    }
-    if (!teardownFailed) {
-      // The teardown removed the orphan, so drop the durable row if we wrote one.
-      if (durable.leaseId !== null) {
-        await releaseCleanedUpOrphanRow(durable.leaseId, orphanDiagnosticFields(input.record), receipt);
-      }
-      return true;
-    }
-    // The teardown failed, so the orphan sandbox is still live.
-    if (durable.leaseId !== null) {
-      // The durable row already tracks the orphan, so a sweep finds and releases
-      // it. The caller rethrows the original insert rejection.
-      return false;
-    }
-    return await escalateUnwrittenOrphan(input.record, input.cause, durable.lastError);
-  };
 
-  // The run-time exec parent context, held per lease id. A plugin sandbox
-  // provider can open a persistent session on the first command and delete it
-  // on lease release. The session open runs inside `execute`, under the run
-  // parent (the same context the `sandbox.exec` span reads). The session delete
-  // runs inside the lease-release RPC, which the run orchestrator calls after
-  // the run, outside that scope. Without a parent the host mints no
-  // `traceparent` and drops the provider `session.teardown` span. So `execute`
-  // records the exec parent here, and the release paths replay it around the
-  // release RPC. The host still mints and validates the `traceparent` itself;
-  // the value only widens which host calls carry a run parent. An entry is
-  // removed on release, so the map holds at most one context per live lease.
   const runExecParentByLeaseId = new Map<string, StartupSpanContext>();
 
   // Run a lease-release RPC under the lease's recorded exec parent context, and
@@ -2128,12 +1857,19 @@ function createSandboxEnvironmentDriver(
             });
           }
         }
-        const acquisitionRunId = input.heartbeatRunId ?? randomUUID();
-        const acquiredLease = providerLease ?? await (async () => {
+        const acquiredLease = providerLease ?? await createOwnedSandbox(db, {
+          companyId: input.companyId, environmentId: input.environment.id,
+          heartbeatRunId: input.heartbeatRunId, issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId,
+          provider: parsed.config.provider, pluginId: pluginProvider.resolved.plugin.id, driverKey: parsed.config.provider,
+          resolvedConfig: { ...parsed.config },
+          configSchema: pluginProvider.resolved.driver.configSchema as Record<string, unknown> | undefined,
+          config: sandboxConfigForLeaseMetadata(storedConfig), requestKey: createHash("sha256").update(JSON.stringify(storedConfig)).digest("hex"),
+          metadata: { ...sandboxConfigForLeaseMetadata(storedConfig), driver: input.environment.driver,
+            pluginId: pluginProvider.resolved.plugin.id, pluginKey: pluginProvider.resolved.plugin.pluginKey, sandboxProviderPlugin: true },
+          timeoutMs: resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs),
+        }, async identity => {
           try {
-            return await pluginWorkerManager.call(
-              pluginProvider.resolved.plugin.id,
-              "environmentAcquireLease",
+            return await pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentAcquireLease",
               {
                 driverKey: parsed.config.provider,
                 companyId: input.companyId,
@@ -2143,7 +1879,8 @@ function createSandboxEnvironmentDriver(
                 // Plugin SDK requires a string; ad-hoc test leases use a fresh
                 // UUID so providers that validate or persist the runId still see
                 // a well-formed identifier.
-                runId: acquisitionRunId,
+                runId: identity.runId,
+                creationOperationId: identity.id,
                 workspaceMode: input.executionWorkspaceMode ?? undefined,
                 agentId: input.agentId ?? undefined,
                 executionWorkspaceId: input.executionWorkspaceId ?? undefined,
@@ -2162,62 +1899,18 @@ function createSandboxEnvironmentDriver(
                   ? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) }
                   : {}),
               },
-              resolvePluginSandboxRpcTimeoutMs(
-                workerConfig,
-                pluginProvider.resolved.driver.defaultAcquireTimeoutMs,
-              ),
-            );
+          resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs));
           } catch (error) {
-            captureEnvironmentAcquisitionDiagnostic(error, {
-              companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId,
-            }, null);
             const cleanup = readEnvironmentCreationCleanupError(error);
-            // The authenticated worker may identify its uncertain allocation,
-            // but cannot redirect cleanup to another company, environment, or run.
-            if (cleanup && cleanup.companyId === input.companyId &&
-                cleanup.environmentId === input.environment.id && cleanup.runId === acquisitionRunId) {
-              if (error instanceof JsonRpcCallError) captureEnvironmentAcquisitionDiagnostic(error, {
-                companyId: input.companyId, environmentId: input.environment.id, runId: acquisitionRunId,
-              }, readEnvironmentAcquisitionDiagnostic(error));
-              const cleanupMetadata = {
-                ...sandboxConfigForLeaseMetadata(storedConfig),
-                failedCreateCleanup: cleanup,
-                driver: input.environment.driver,
-                pluginId: pluginProvider.resolved.plugin.id,
-                pluginKey: pluginProvider.resolved.plugin.pluginKey,
-                sandboxProviderPlugin: true,
-              };
-              // Record before retrying the provider. Existing pending-cleanup
-              // recovery (including its durable spool) survives controller loss.
-              const cleanupConfirmed = await cleanUpRejectedOrphanSandbox({
-                record: {
-                  companyId: input.companyId, environmentId: input.environment.id,
-                  executionWorkspaceId: input.executionWorkspaceId ?? null,
-                  issueId: input.issueId ?? null, heartbeatRunId: input.heartbeatRunId ?? null,
-                  provider: parsed.config.provider, providerLeaseId: cleanup.providerLeaseId,
-                  metadata: cleanupMetadata,
-                },
-                cause: error,
-                canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id),
-                teardown: (pendingLeaseId) => destroyWithCreationObservation({
-                  leaseId: pendingLeaseId, metadata: cleanupMetadata,
-                  teardown: (metadata) => pluginWorkerManager.call(pluginProvider.resolved.plugin.id, "environmentDestroyLease", {
-                    driverKey: parsed.config.provider, companyId: input.companyId,
-                    environmentId: input.environment.id, issueId: input.issueId,
-                    config: workerConfig, providerLeaseId: cleanup.providerLeaseId,
-                    leaseMetadata: metadata,
-                  }, resolvePluginSandboxRpcTimeoutMs(workerConfig)),
-                }),
-              });
-              if (cleanupConfirmed) {
-                throw new Error("Sandbox creation failed; allocated sandbox cleanup was confirmed.", { cause: error });
-              }
-            }
-            // A cleanup record is not a usable lease. Keep acquisition failed,
-            // even when its compensating deletion succeeds on this attempt.
+            const diagnostic = error instanceof JsonRpcCallError && cleanup?.companyId === input.companyId &&
+              cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId
+              ? readEnvironmentAcquisitionDiagnostic(error) : null;
+            captureEnvironmentAcquisitionDiagnostic(error, {
+              companyId: input.companyId, environmentId: input.environment.id, runId: identity.runId,
+            }, diagnostic);
             throw error;
           }
-        })();
+        });
 
         // Ad-hoc test leases are never publishable for reuse: storing them
         // as `reuse_by_environment` would let a concurrent heartbeat resume
@@ -2275,82 +1968,34 @@ function createSandboxEnvironmentDriver(
             : {}),
           ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
         };
-        try {
-          return await environmentsSvc.acquireLease({
-            companyId: input.companyId,
-            environmentId: input.environment.id,
-            executionWorkspaceId: input.executionWorkspaceId,
-            issueId: input.issueId,
-            heartbeatRunId: input.heartbeatRunId,
-            assertCompanyBinding: input.assertCompanyBinding,
-            leasePolicy: resolvedLeasePolicy,
-            provider: parsed.config.provider,
-            providerLeaseId: acquiredLease.providerLeaseId,
-            expiresAt: providerAttestedLeaseExpiry(
-              input.requestedExpiresAt,
-              acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined,
-            ),
-            metadata: pluginLeaseMetadata,
-            reusesReusableLeaseId:
-              providerLease &&
-              reusableLease?.heartbeatRunId === input.heartbeatRunId
-                ? reusableLease.id
-                : null,
-            replacesReusableLeaseId:
-              providerLease &&
-              reusableLease?.heartbeatRunId !== input.heartbeatRunId
-                ? reusableLease?.id
-                : null,
-          });
-        } catch (error) {
-          // The conditional lease insert rejected, so no lease row exists. A
-          // managed reconciliation can bind the environment to another company
-          // between the route guard and this insert, so the insert fails closed
-          // with `environment_company_mismatch`. This call already provisioned the
-          // remote plugin sandbox above, so tear it down now. Without this step
-          // the rejected insert leaks a live sandbox that no lease row tracks. The
-          // Claude login runs on a plugin-backed sandbox, so this path is the one
-          // the login uses. Do not tear down a reused sandbox that an earlier
-          // lease still owns.
-          if (!reusableLease || acquiredLease.providerLeaseId !== reusableLease.providerLeaseId) {
-            // Record the durable pending-cleanup row before the teardown, then
-            // tear the sandbox down. On a successful teardown the handler releases
-            // the row. If the durable write and the teardown both fail, the
-            // handler throws a `SandboxOrphanCleanupWriteError` that carries the
-            // original rejection as its cause.
-            await cleanUpRejectedOrphanSandbox({
-              record: {
-                companyId: input.companyId,
-                environmentId: input.environment.id,
-                executionWorkspaceId: input.executionWorkspaceId ?? null,
-                issueId: input.issueId ?? null,
-                heartbeatRunId: input.heartbeatRunId ?? null,
-                provider: parsed.config.provider,
-                providerLeaseId: acquiredLease.providerLeaseId,
-                metadata: pluginLeaseMetadata,
-              },
-              cause: error,
-              canTeardown: pluginWorkerManager.isRunning(pluginProvider.resolved.plugin.id),
-              teardown: async () => {
-                return await pluginWorkerManager.call(
-                  pluginProvider.resolved.plugin.id,
-                  "environmentDestroyLease",
-                  {
-                    driverKey: parsed.config.provider,
-                    companyId: input.companyId,
-                    environmentId: input.environment.id,
-                    issueId: input.issueId,
-                    config: workerConfig,
-                    providerLeaseId: acquiredLease.providerLeaseId,
-                    leaseMetadata: acquiredLease.metadata ?? undefined,
-                  },
-                  resolvePluginSandboxRpcTimeoutMs(workerConfig),
-                );
-              },
-            });
-          }
-          throw error;
-        }
+        return await environmentsSvc.acquireLease({
+          creationOperationId: providerLease ? undefined : acquiredLease.metadata?.creationOperationId as string | undefined,
+          companyId: input.companyId,
+          environmentId: input.environment.id,
+          executionWorkspaceId: input.executionWorkspaceId,
+          issueId: input.issueId,
+          heartbeatRunId: input.heartbeatRunId,
+          assertCompanyBinding: input.assertCompanyBinding,
+          leasePolicy: resolvedLeasePolicy,
+          provider: parsed.config.provider,
+          providerLeaseId: acquiredLease.providerLeaseId,
+          expiresAt: providerAttestedLeaseExpiry(
+            input.requestedExpiresAt,
+            acquiredLease.expiresAt ? new Date(acquiredLease.expiresAt) : undefined,
+          ),
+          metadata: pluginLeaseMetadata,
+          reusesReusableLeaseId:
+            providerLease &&
+            reusableLease?.heartbeatRunId === input.heartbeatRunId
+              ? reusableLease.id
+              : null,
+          replacesReusableLeaseId:
+            providerLease &&
+            reusableLease?.heartbeatRunId !== input.heartbeatRunId
+              ? reusableLease?.id
+              : null,
+        });
+
       }
 
       // Built-in sandbox provider path. Same guard as the plugin-backed path:
@@ -2441,10 +2086,16 @@ function createSandboxEnvironmentDriver(
 
       let providerLease;
       try {
-        providerLease = await acquireSandboxProviderLease({
+        providerLease = await createOwnedSandbox(db, {
+          companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId,
+          issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId, provider: parsed.config.provider,
+          driverKey: parsed.config.provider, config: sandboxConfigForLeaseMetadata(parsed.config),
+          requestKey: createHash("sha256").update(JSON.stringify(parsed.config)).digest("hex"),
+          metadata: { ...sandboxConfigForLeaseMetadata(parsed.config), driver: input.environment.driver },
+        }, identity => acquireSandboxProviderLease({
           config: parsed.config,
           environmentId: input.environment.id,
-          heartbeatRunId: input.heartbeatRunId ?? randomUUID(),
+          heartbeatRunId: identity.runId,
           issueId: input.issueId,
           agentId: input.agentId,
           executionWorkspaceId: input.executionWorkspaceId,
@@ -2452,7 +2103,7 @@ function createSandboxEnvironmentDriver(
           // Forward a caller deadline so the provider configures a provider-side
           // expiry at or before it and returns the real provider expiry.
           requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt),
-        });
+        }));
       } catch (error) {
         if (reusableLease) {
           throw new ReusableSandboxResumeError({
@@ -2521,72 +2172,35 @@ function createSandboxEnvironmentDriver(
           : {}),
         ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
       };
-      try {
-        return await environmentsSvc.acquireLease({
-          companyId: input.companyId,
-          environmentId: input.environment.id,
-          executionWorkspaceId: input.executionWorkspaceId,
-          issueId: input.issueId,
-          heartbeatRunId: input.heartbeatRunId,
-          assertCompanyBinding: input.assertCompanyBinding,
-          leasePolicy: resolvedLeasePolicy,
-          provider: parsed.config.provider,
-          providerLeaseId: providerLease.providerLeaseId,
-          expiresAt: providerAttestedLeaseExpiry(
-            input.requestedExpiresAt,
-            providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined,
-          ),
-          metadata: builtinLeaseMetadata,
-          reusesReusableLeaseId:
-            reusableLease &&
-            providerLease.providerLeaseId === reusableLease.providerLeaseId &&
-            reusableLease.heartbeatRunId === input.heartbeatRunId
-              ? reusableLease.id
-              : null,
-          replacesReusableLeaseId:
-            reusableLease &&
-            providerLease.providerLeaseId === reusableLease.providerLeaseId &&
-            reusableLease.heartbeatRunId !== input.heartbeatRunId
-              ? reusableLease.id
-              : null,
-        });
-      } catch (error) {
-        // The conditional lease insert rejected, so no lease row exists. A managed
-        // reconciliation can bind the environment to another company between the
-        // route guard and this insert, so the insert fails closed with
-        // `environment_company_mismatch`. This call already provisioned the remote
-        // sandbox above, so release it now. Without this teardown the rejected
-        // insert leaks a live sandbox that no lease row tracks. Do not tear down a
-        // reused sandbox that an earlier lease still owns.
-        if (!reusableLease || providerLease.providerLeaseId !== reusableLease.providerLeaseId) {
-          // Record the durable pending-cleanup row before the teardown, then
-          // release the remote sandbox. On a successful teardown the handler
-          // releases the row. If the durable write and the teardown both fail,
-          // the handler throws a `SandboxOrphanCleanupWriteError` that carries the
-          // original rejection as its cause.
-          await cleanUpRejectedOrphanSandbox({
-            record: {
-              companyId: input.companyId,
-              environmentId: input.environment.id,
-              executionWorkspaceId: input.executionWorkspaceId ?? null,
-              issueId: input.issueId ?? null,
-              heartbeatRunId: input.heartbeatRunId ?? null,
-              provider: parsed.config.provider,
-              providerLeaseId: providerLease.providerLeaseId,
-              metadata: builtinLeaseMetadata,
-            },
-            cause: error,
-            canTeardown: true,
-            teardown: async () => {
-              await destroySandboxProviderLease({
-                config: parsed.config,
-                providerLeaseId: providerLease.providerLeaseId,
-              });
-            },
-          });
-        }
-        throw error;
-      }
+      return await environmentsSvc.acquireLease({
+        creationOperationId: providerLease.metadata?.creationOperationId as string | undefined,
+        companyId: input.companyId,
+        environmentId: input.environment.id,
+        executionWorkspaceId: input.executionWorkspaceId,
+        issueId: input.issueId,
+        heartbeatRunId: input.heartbeatRunId,
+        assertCompanyBinding: input.assertCompanyBinding,
+        leasePolicy: resolvedLeasePolicy,
+        provider: parsed.config.provider,
+        providerLeaseId: providerLease.providerLeaseId,
+        expiresAt: providerAttestedLeaseExpiry(
+          input.requestedExpiresAt,
+          providerLease.expiresAt ? new Date(providerLease.expiresAt) : undefined,
+        ),
+        metadata: builtinLeaseMetadata,
+        reusesReusableLeaseId:
+          reusableLease &&
+          providerLease.providerLeaseId === reusableLease.providerLeaseId &&
+          reusableLease.heartbeatRunId === input.heartbeatRunId
+            ? reusableLease.id
+            : null,
+        replacesReusableLeaseId:
+          reusableLease &&
+          providerLease.providerLeaseId === reusableLease.providerLeaseId &&
+          reusableLease.heartbeatRunId !== input.heartbeatRunId
+            ? reusableLease.id
+            : null,
+      });
     },
 
     async releaseRunLease(input) {
@@ -2719,7 +2333,7 @@ function createSandboxEnvironmentDriver(
             `Sandbox provider "${recordedProvider}" needs a plugin worker manager for cleanup, but none is available.`,
           );
         }
-        const pinnedPluginId = exportResume ? readString(input.lease.metadata?.pluginId) : null;
+        const pinnedPluginId = exportResume || input.lease.metadata?.creationOperationId ? readString(input.lease.metadata?.pluginId) : null;
         if (exportResume && !pinnedPluginId) throw new Error("Workspace export cleanup has no recorded provider plugin.");
         const pluginProvider = await resolveSandboxProviderPlugin({ provider: recordedProvider, ...(pinnedPluginId ? { pluginId: pinnedPluginId } : {}) });
         if (pluginProvider.state !== "running") {
@@ -3534,6 +3148,7 @@ const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([
   "remoteCwd",
   "shellCommand",
   "sandboxProviderPlugin",
+  "creationOperationId",
   "sandboxLeaseAcquisition",
   "nativeHarnessBackup",
   "nativeWorkspaceSync",
@@ -3698,13 +3313,20 @@ function createPluginEnvironmentDriver(
           resumed = undefined;
         }
       }
-      const providerLease = resumed ?? await workerManager.call(plugin.id, "environmentAcquireLease", {
+      const providerLease = resumed ?? await createOwnedSandbox(db, {
+        companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId,
+        issueId: input.issueId, executionWorkspaceId: input.executionWorkspaceId,
+        provider: `plugin:${parsed.config.pluginKey}:${parsed.config.driverKey}`, pluginId: plugin.id, driverKey: parsed.config.driverKey,
+        config: {}, timeoutMs: rpcTimeoutMs, requestKey: fingerprint,
+        metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: parsed.config.pluginKey, driverKey: parsed.config.driverKey },
+      }, identity => workerManager.call(plugin.id, "environmentAcquireLease", {
         driverKey: parsed.config.driverKey,
         companyId: input.companyId,
         environmentId: input.environment.id,
         issueId: input.issueId,
         config: parsed.config.driverConfig,
-        runId: input.heartbeatRunId ?? randomUUID(),
+        runId: identity.runId,
+        creationOperationId: identity.id,
         workspaceMode: input.executionWorkspaceMode ?? undefined,
         agentId: input.agentId ?? undefined,
         executionWorkspaceId: input.executionWorkspaceId ?? undefined,
@@ -3715,9 +3337,10 @@ function createPluginEnvironmentDriver(
         ...(requestedExpiresAtParam(input.requestedExpiresAt) !== undefined
           ? { requestedExpiresAt: requestedExpiresAtParam(input.requestedExpiresAt) }
           : {}),
-      } as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride);
+      } as PluginEnvironmentAcquireLeaseParams, ...timeoutOverride));
 
       return await environmentsSvc.acquireLease({
+        creationOperationId: resumed ? undefined : providerLease.metadata?.creationOperationId as string | undefined,
         companyId: input.companyId,
         environmentId: input.environment.id,
         executionWorkspaceId: input.executionWorkspaceId,
@@ -3964,14 +3587,13 @@ export function environmentRuntimeService(
     pluginWorkerManager?: PluginWorkerManager;
     pluginWorkerReadyTimeoutMs?: number;
     pluginWorkerReadyPollMs?: number;
-    pendingCleanupWriteAttempts?: number;
-    pendingCleanupWriteBackoffMs?: number;
     deferredOrphanCleanupBufferLimit?: number;
     orphanCleanupSpool?: SandboxOrphanCleanupSpool;
     orphanCleanupSpoolDir?: string;
   } = {},
 ) {
   const environmentsSvc = environmentService(db);
+  let orphanRecoveryFlush: Promise<{ recovered: number; pending: number }> | null = null;
   const drivers = new Map<string, EnvironmentRuntimeDriver>();
 
   const defaultDrivers = [
@@ -3981,8 +3603,6 @@ export function environmentRuntimeService(
       pluginWorkerManager: options.pluginWorkerManager,
       pluginWorkerReadyTimeoutMs: options.pluginWorkerReadyTimeoutMs,
       pluginWorkerReadyPollMs: options.pluginWorkerReadyPollMs,
-      pendingCleanupWriteAttempts: options.pendingCleanupWriteAttempts,
-      pendingCleanupWriteBackoffMs: options.pendingCleanupWriteBackoffMs,
       deferredOrphanCleanupBufferLimit: options.deferredOrphanCleanupBufferLimit,
       orphanCleanupSpool: options.orphanCleanupSpool,
       orphanCleanupSpoolDir: options.orphanCleanupSpoolDir,
@@ -4439,14 +4059,53 @@ export function environmentRuntimeService(
       return driver.isPendingCleanupWorkerReady(input);
     },
 
-    // Flush the in-process orphan-cleanup buffers of every driver that keeps one.
-    // The cleanup sweep calls this each tick, so a buffered orphan lands a durable
-    // `pending_cleanup` row once the database recovers, and the same sweep tears
-    // it down. The dispatcher sums the recovered and pending counts across all
-    // drivers. A driver with no buffer contributes nothing.
+    // Recover committed creation operations before replaying legacy cleanup inputs.
+    // New creation failures retain their database operation and do not make files.
+    // Legacy buffers and files remain until their cleanup rows are durable.
     async flushDeferredOrphanCleanups(): Promise<{ recovered: number; pending: number }> {
-      return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values())
-        .flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : []));
+      orphanRecoveryFlush ??= (async () => {
+        await recoverSandboxCreations(db, async operation => {
+          if (!operation.pluginId || operation.metadata.sandboxProviderPlugin !== true) return null;
+          if (!options.pluginWorkerManager) throw new Error("Sandbox creation recovery worker is unavailable");
+          const plugin = await pluginRegistryService(db).getById(operation.pluginId);
+          const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === operation.driverKey);
+          if (driver?.creationOperationVersion !== 1) return null;
+          if (!options.pluginWorkerManager.isRunning(operation.pluginId) ||
+            !options.pluginWorkerManager.getWorker(operation.pluginId)?.supportedMethods.includes("environmentInspectLease")) {
+            throw new Error("Sandbox creation recovery worker is unavailable");
+          }
+          let resolved: SandboxEnvironmentConfig;
+          try {
+            resolved = await resolveSandboxCleanupConfigSecrets(db, operation.companyId,
+              operation.config as SandboxEnvironmentConfig, { issueId: operation.issueId });
+          } catch (error) {
+            // Missing or explicitly inactive saved credentials need an operator.
+            // Temporary database/provider failures still retain retry ownership.
+            const code = error instanceof HttpError ? (error.details as { code?: string } | undefined)?.code : undefined;
+            if (error instanceof HttpError && (
+              (error.status === 404 && (code === "secret_deleted" || code === "version_missing" ||
+                (error.details === undefined && error.message === "Secret not found"))) ||
+              (error.status === 422 && (code === "secret_inactive" || code === "version_inactive" ||
+                (error.details === undefined && (error.message === "Provider vault is disabled" ||
+                  error.message === "Provider vault is locked while coming soon"))))
+            )) return null;
+            throw error;
+          }
+          const credentialHashes = operation.metadata.creationCredentialHashes as Record<string, string> | undefined;
+          for (const [path, expected] of Object.entries(credentialHashes ?? {})) {
+            const actual = readConfigValueAtPath({ ...resolved }, path);
+            if (typeof actual !== "string" || createHash("sha256").update(actual).digest("hex") !== expected) return null;
+          }
+          return options.pluginWorkerManager.call(operation.pluginId, "environmentInspectLease", {
+            driverKey: operation.driverKey, companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
+            config: stripSandboxProviderEnvelope(resolved), providerLeaseId: operation.id,
+            creationOperation: { id: operation.id, runId: operation.runId },
+          });
+        });
+        return flushDeferredEnvironmentCleanup(db, Array.from(drivers.values())
+          .flatMap(driver => driver.flushDeferredOrphanCleanups ? [driver.flushDeferredOrphanCleanups] : []));
+      })().finally(() => { orphanRecoveryFlush = null; });
+      return orphanRecoveryFlush;
     },
 
     async destroyReusableSandboxLeases(input: {

server/src/services/environment-task-runtime.ts

@@ -1,8 +1,6 @@
-import { beginIdleTrackedWork } from "./task-admission.js";
-import { registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
-import { environmentService } from "./environments.js";
-import { createSandboxOrphanCleanupSpool, type DeferredOrphanCleanupRecord } from "./sandbox-orphan-cleanup-spool.js";
-import { logger } from "../middleware/logger.js";
+import { attachSandboxCreation, environmentService } from "./environments.js";
+import { createOwnedSandbox, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
+import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
 import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
 import { readWorkspaceProviderBinding } from "@paperclipai/shared";
 import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
@@ -234,59 +232,38 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
           metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({runnerTaskExecutionStopped:true,
             runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
         }).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
-        const fresh = await workers.call(plugin.id, "environmentAcquireLease", {driverKey:metadata.driverKey,
+        const replacementDriverKey = metadata.driverKey;
+        const fresh = await createOwnedSandbox(db, {
+          companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
+          issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
+          pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
+          metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey, runnerTaskLease: true },
+        }, identity => workers.call(plugin.id, "environmentAcquireLease", {driverKey:replacementDriverKey,
           companyId:input.companyId,environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,
-          runId:run!.id,agentId:run!.agentId,adapterType:"paperclip_runner",
-          ...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})});
+          runId:identity.runId,creationOperationId:identity.id,agentId:run!.agentId,adapterType:"paperclip_runner",
+          ...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})}));
         if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
+        await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
+          environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
         const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
           runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
         const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
           runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
-        try {
+        {
           const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
           const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
           if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
               (deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
             throw new Error("Plugin replacement task expiry is not bounded");
-          const replacement = await db.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
-            metadata:replacementMetadata,
+          await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
+            const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
+            metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
           }).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
             eq(environmentLeases.status,"active"))).returning({id:environmentLeases.id});
-          if (replacement.length !== 1) throw new Error("Plugin task replacement ownership changed");
-        } catch (error) {
-          try {
-            await workers.call(plugin.id,"environmentReleaseLease",{driverKey:metadata.driverKey,companyId:input.companyId,
-              environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,providerLeaseId:fresh.providerLeaseId,
-              leaseMetadata:replacementMetadata});
-          } catch {
-            const orphan:DeferredOrphanCleanupRecord={companyId:input.companyId,environmentId:environment!.id,
-              executionWorkspaceId:lease.executionWorkspaceId,issueId:lease.issueId,heartbeatRunId:lease.heartbeatRunId,
-              provider:lease.provider!,providerLeaseId:fresh.providerLeaseId,metadata:replacementMetadata};
-            try {
-              await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"});
-            } catch {
-              // The existing orphan spool recovers cleanup even when the database is unavailable.
-              const spool = createSandboxOrphanCleanupSpool();
-              const persisted = await spool.append(orphan);
-              const finish = beginIdleTrackedWork();
-              let flushing: Promise<{ recovered: number; pending: number }> | null = null;
-              const unregister = registerDeferredEnvironmentCleanup(db, () => {
-                flushing ??= (async () => {
-                  await environmentService(db).insertPendingCleanupLease({...orphan,failureReason:"plugin_task_replacement_unowned"});
-                  await spool.remove(orphan);
-                  unregister();
-                  finish();
-                  return { recovered: 1, pending: 0 };
-                })().catch(error => { flushing = null; throw error; });
-                // Keep successful completion for a concurrent sweep that already
-                // captured this callback before it was unregistered.
-                return flushing;
-              });
-              logger.error({providerLeaseId:fresh.providerLeaseId,persisted},"Plugin replacement cleanup requires recovery");
-            }
-          }
-          throw error;
+            if (!updated) throw new Error("Plugin task replacement ownership changed");
+            return (await environmentService(tx).getLeaseById(updated.id))!;
+          });
+
         }
         return executeEnvironmentTask(db, workers, input);
       }

server/src/services/environments.ts

@@ -16,6 +16,7 @@ import {
   companySecretBindings,
   environmentCustomImageSetupSessions,
   environmentLeases,
+  sandboxCreationOperations,
   environments,
   executionWorkspaces,
   heartbeatRuns,
@@ -48,6 +49,43 @@ import {
   type ManagedResourceStockStatus,
 } from "./managed-resource-drift.js";
 
+export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
+  attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
+  return db.transaction(async tx => {
+    await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
+    const [operation] = await tx.select().from(sandboxCreationOperations)
+      .where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
+        eq(sandboxCreationOperations.originalEnvironmentId, scope.environmentId))).for("update");
+    if (scope.heartbeatRunId) {
+      const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
+        eq(heartbeatRuns.id, scope.heartbeatRunId), eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.status, "running"),
+      )).for("update");
+      if (!run || operation?.runId !== scope.heartbeatRunId) throw new Error("Sandbox creation run authority changed");
+    }
+    if (operation?.state === "attached" && operation.leaseId) {
+      const existing = await environmentService(tx as unknown as Db).getLeaseById(operation.leaseId);
+      if (existing?.status === "active") return existing;
+    }
+    if (!operation || operation.state !== "pending") throw new Error("Sandbox creation ownership has already transferred");
+    const lease = await attach(tx as unknown as Db);
+    await notifyEnvironmentCleanup(tx, { companyId: scope.companyId });
+    await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
+      .where(eq(sandboxCreationOperations.id, id));
+    return lease;
+  });
+}
+
+export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {
+  const [environment] = await db.select({ id: environments.id }).from(environments)
+    .where(eq(environments.id, environmentId)).for("update");
+  if (!environment) throw new Error("Sandbox creation environment no longer exists");
+  const bindings = await db.select({ companyId: builtInManagedResources.companyId }).from(builtInManagedResources)
+    .where(and(eq(builtInManagedResources.resourceKind, "environment"), eq(builtInManagedResources.resourceId, environmentId)));
+  if (bindings.length && !bindings.some(binding => binding.companyId === companyId)) {
+    throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" });
+  }
+}
+
 type EnvironmentRow = typeof environments.$inferSelect;
 type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect;
 const DEFAULT_LOCAL_ENVIRONMENT_NAME = "Local";
@@ -1428,6 +1466,7 @@ export function environmentService(db: Db) {
     },
 
     acquireLease: async (input: {
+      creationOperationId?: string;
       companyId: string;
       environmentId: string;
       executionWorkspaceId?: string | null;
@@ -1465,6 +1504,11 @@ export function environmentService(db: Db) {
        */
       assertCompanyBinding?: boolean;
     }): Promise<EnvironmentLease> => {
+      if (input.creationOperationId) {
+        const { creationOperationId, ...leaseInput } = input;
+        return attachSandboxCreation(db, creationOperationId, { companyId: input.companyId, environmentId: input.environmentId, heartbeatRunId: input.heartbeatRunId },
+          tx => environmentService(tx).acquireLease({ ...leaseInput, assertCompanyBinding: true }));
+      }
       const now = new Date();
       const replacementLeaseId = input.replacesReusableLeaseId ? randomUUID() : null;
       const values = {
@@ -1515,48 +1559,7 @@ export function environmentService(db: Db) {
                 companyId: input.companyId, runId: input.heartbeatRunId,
               });
               if (input.assertCompanyBinding) {
-                // Lock the environment row first. Managed reconciliation locks the
-                // same sandbox environment rows with `for update` before it writes a
-                // company binding, so this lock serializes the two transactions on
-                // this row and closes the time-of-check to time-of-use window.
-                await tx
-                  .select({ id: environments.id })
-                  .from(environments)
-                  .where(eq(environments.id, input.environmentId))
-                  .for("update");
-                // Re-read the company binding inside the locked transaction. A
-                // binding a reconciliation committed after the route guard now
-                // appears here. Reject a foreign-company environment before the
-                // insert, so the login holds no lease.
-                const boundRows = await tx
-                  .select({ companyId: builtInManagedResources.companyId })
-                  .from(builtInManagedResources)
-                  .where(
-                    and(
-                      eq(
-                        builtInManagedResources.resourceKind,
-                        MANAGED_ENVIRONMENT_RESOURCE_KIND,
-                      ),
-                      eq(
-                        builtInManagedResources.resourceId,
-                        input.environmentId,
-                      ),
-                    ),
-                  );
-                const boundCompanyIds = Array.from(
-                  new Set(boundRows.map((boundRow) => boundRow.companyId)),
-                );
-                if (
-                  boundCompanyIds.length > 0 &&
-                  !boundCompanyIds.includes(input.companyId)
-                ) {
-                  throw forbidden(
-                    "The selected environment belongs to another company.",
-                    {
-                      code: "environment_company_mismatch",
-                    },
-                  );
-                }
+                await assertEnvironmentCompanyBinding(tx as unknown as Db, input.environmentId, input.companyId);
               }
               if (input.replacesReusableLeaseId) {
                 await notifyChatInboundWakeNotices(tx, { companyId: input.companyId });
@@ -1699,6 +1702,7 @@ export function environmentService(db: Db) {
         const [previous] = await tx.select({ companyId: environmentLeases.companyId, issueId: environmentLeases.issueId,
           heartbeatRunId: environmentLeases.heartbeatRunId, status: environmentLeases.status,
           cleanupStatus: environmentLeases.cleanupStatus, releasedAt: environmentLeases.releasedAt,
+          creationOperationId: sql<string | null>`${environmentLeases.metadata}->>'creationOperationId'`,
         }).from(environmentLeases).where(condition).for("update");
         if (!previous) return null;
         await notifyEnvironmentCleanup(tx, { companyId: previous.companyId, runId: previous.heartbeatRunId ?? undefined, leaseId: id });
@@ -1715,6 +1719,11 @@ export function environmentService(db: Db) {
             companyId: previous.companyId, runId: previous.heartbeatRunId,
           });
         }
+        if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
+          await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
+            .where(and(eq(sandboxCreationOperations.id, previous.creationOperationId), eq(sandboxCreationOperations.companyId, previous.companyId),
+              eq(sandboxCreationOperations.leaseId, id), eq(sandboxCreationOperations.state, "cleanup_pending")));
+        }
         return tx.update(environmentLeases)
         .set({
           status,

server/src/services/idle-sleep-safety.ts

@@ -54,6 +54,7 @@ const WORK_CHECKS = [
     state NOT IN ('saved', 'unchanged', 'resolved', 'unavailable', 'conflict')
     OR process_stopped_at IS NULL OR next_attempt_at IS NOT NULL
     OR receipt ? 'baseline' OR receipt->>'cleanupPending' = 'true'`,
+  `SELECT 1 FROM sandbox_creation_operations WHERE state = 'pending'`,
   `SELECT 1 FROM budget_reservations WHERE state NOT IN ('settled', 'released')`,
   `SELECT 1 FROM budget_policies WHERE enforcement_version > enforcement_delivered_version`,
   `SELECT 1 FROM agents WHERE lifecycle_state IN

server/src/services/sandbox-creation-operations.ts

@@ -0,0 +1,150 @@
+import { collectSecretRefPaths, isUuidSecretRef, parseSecretRefBindingObject, readConfigValueAtPath } from "./json-schema-secret-refs.js";
+import { createHash, randomUUID } from "node:crypto";
+import { and, eq, lte, or, sql } from "drizzle-orm";
+import { companies, environments, sandboxCreationOperations, type Db } from "@paperclipai/db";
+import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk";
+import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
+import { ENVIRONMENT_CLEANUP_RETRY_MS } from "./environment-cleanup-state.js";
+import { logger } from "../middleware/logger.js";
+import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
+
+export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
+
+type CreationOwner = {
+  companyId: string;
+  environmentId: string;
+  executionWorkspaceId?: string | null;
+  issueId?: string | null;
+  heartbeatRunId?: string | null;
+  provider: string;
+  pluginId?: string;
+  driverKey: string;
+  config: Record<string, unknown>;
+  metadata: Record<string, unknown>;
+  timeoutMs?: number;
+  requestKey?: string;
+  configSchema?: Record<string, unknown> | null;
+  resolvedConfig?: Record<string, unknown>;
+};
+
+export async function createOwnedSandbox(
+  db: Db, owner: CreationOwner,
+  create: (identity: { id: string; runId: string }) => Promise<PluginEnvironmentLease>,
+): Promise<PluginEnvironmentLease> {
+  const credentialHashes: Record<string, string> = {};
+  for (const path of collectSecretRefPaths(owner.configSchema)) {
+    const value = readConfigValueAtPath(owner.config, path);
+    if (value != null && value !== "" && !(typeof value === "string" && isUuidSecretRef(value)) && !parseSecretRefBindingObject(value)) {
+      throw new Error("Sandbox creation requires saved secret references");
+    }
+    if (value != null && value !== "") {
+      const resolved = readConfigValueAtPath(owner.resolvedConfig ?? {}, path);
+      if (typeof resolved !== "string") throw new Error("Sandbox creation credential identity is unavailable");
+      credentialHashes[path] = createHash("sha256").update(resolved).digest("hex");
+    }
+  }
+  const id = randomUUID();
+  const runId = owner.heartbeatRunId ?? randomUUID();
+  const operation = await db.transaction(async tx => {
+    await notifyEnvironmentCleanup(tx, { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined });
+    await assertEnvironmentCompanyBinding(tx as unknown as Db, owner.environmentId, owner.companyId);
+    const [inserted] = await tx.insert(sandboxCreationOperations).values({
+      id, companyId: owner.companyId, environmentId: owner.environmentId, originalEnvironmentId: owner.environmentId,
+      runId, requestKey: owner.requestKey ?? "run", issueId: owner.issueId, executionWorkspaceId: owner.executionWorkspaceId,
+      provider: owner.provider, pluginId: owner.pluginId, driverKey: owner.driverKey,
+      config: owner.config, metadata: { ...owner.metadata, creationCredentialHashes: credentialHashes },
+      nextAttemptAt: new Date(Date.now() + Math.max(30 * 60_000, (owner.timeoutMs ?? 0) + 60_000)),
+    }).onConflictDoNothing().returning();
+    if (inserted) return inserted;
+    const [existing] = await tx.select().from(sandboxCreationOperations).where(and(
+      eq(sandboxCreationOperations.companyId, owner.companyId), eq(sandboxCreationOperations.originalEnvironmentId, owner.environmentId),
+      eq(sandboxCreationOperations.runId, runId), eq(sandboxCreationOperations.requestKey, owner.requestKey ?? "run"),
+    ));
+    if (!existing) throw new Error("Sandbox creation ownership was not recorded");
+    return existing;
+  });
+  if (operation.id !== id) {
+    if (operation.state !== "attached" || !operation.leaseId) throw new Error("Sandbox creation requires reconciliation before another attempt");
+    const existing = await environmentService(db).getLeaseById(operation.leaseId);
+    if (!existing || existing.status !== "active" || !existing.providerLeaseId) throw new Error("Sandbox creation has already transferred ownership");
+    return { providerLeaseId: existing.providerLeaseId, expiresAt: existing.expiresAt?.toISOString(),
+      metadata: { ...existing.metadata, creationOperationId: operation.id } };
+  }
+  const lease = await create({ id, runId });
+  return { ...lease, metadata: { ...lease.metadata, creationOperationId: id } };
+}
+
+/** Keep an already returned resource identity even when task attachment fails.
+ * This enriches the existing owner; it never transfers or reopens ownership. */
+export async function recordSandboxCreationResource(db: Db, input: {
+  id: string; companyId: string; environmentId: string; providerLeaseId: string;
+}): Promise<void> {
+  const [recorded] = await db.update(sandboxCreationOperations).set({
+    metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId })}::jsonb`,
+    updatedAt: new Date(),
+  }).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
+    eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
+    or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
+      sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
+  )).returning({ id: sandboxCreationOperations.id });
+  if (!recorded) throw new Error("Sandbox creation resource identity changed");
+}
+
+export async function recoverSandboxCreations(db: Db,
+  inspect: (operation: SandboxCreationOperation) => Promise<PluginEnvironmentLeaseObservation | null>,
+): Promise<void> {
+  const due = await db.select().from(sandboxCreationOperations).where(and(
+    eq(sandboxCreationOperations.state, "pending"), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
+  )).limit(100);
+  for (const candidate of due) {
+    try {
+      const [existingCompany] = await db.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).limit(1);
+      const observation = existingCompany ? await inspect(candidate) : null;
+      const cleanup = observation?.creationCleanup ? readEnvironmentCreationCleanupError({ data: {
+        schema: "paperclip/environment-creation-cleanup/v1", cleanup: observation.creationCleanup,
+      } }) : null;
+      if (cleanup && (cleanup.companyId !== candidate.companyId || cleanup.environmentId !== candidate.originalEnvironmentId ||
+        cleanup.runId !== candidate.runId || cleanup.attemptId !== candidate.id || cleanup.providerLeaseId !== observation!.providerLeaseId)) {
+        throw new Error("Sandbox creation recovery returned foreign ownership");
+      }
+      await db.transaction(async tx => {
+        const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share");
+        await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, candidate.originalEnvironmentId)).for("update");
+        const [operation] = await tx.select().from(sandboxCreationOperations)
+          .where(eq(sandboxCreationOperations.id, candidate.id)).for("update");
+        if (!operation || operation.state !== "pending") return;
+        await notifyEnvironmentCleanup(tx, { companyId: operation.companyId });
+        if (!company || !cleanup) {
+          logger.warn({ errorKind: "sandbox_creation_needs_reconciliation", operationId: operation.id, companyId: operation.companyId },
+            "Sandbox creation could not be identified; operator reconciliation is required");
+          await tx.update(sandboxCreationOperations).set({ state: "needs_reconciliation", updatedAt: new Date(),
+            metadata: { ...operation.metadata, creationReconciliationReason: company ? "creation_identity_unavailable" : "company_deleted" },
+          })
+            .where(eq(sandboxCreationOperations.id, operation.id));
+          return;
+        }
+        const lease = await environmentService(tx as unknown as Db).insertPendingCleanupLease({
+          companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
+          provider: operation.provider, providerLeaseId: cleanup.providerLeaseId,
+          metadata: { ...operation.metadata, creationOperationId: operation.id, failedCreateCleanup: cleanup },
+          failureReason: "interrupted_sandbox_creation",
+        });
+        await tx.update(sandboxCreationOperations).set({ state: "cleanup_pending", leaseId: lease.id, updatedAt: new Date() })
+          .where(eq(sandboxCreationOperations.id, operation.id));
+      });
+    } catch {
+      const retry = await db.transaction(async tx => {
+        const [company] = await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).for("key share");
+        await notifyEnvironmentCleanup(tx, { companyId: candidate.companyId });
+        await tx.update(sandboxCreationOperations).set(company
+          ? { nextAttemptAt: new Date(Date.now() + ENVIRONMENT_CLEANUP_RETRY_MS), updatedAt: new Date() }
+          : { state: "needs_reconciliation", updatedAt: new Date(),
+            metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || '{"creationReconciliationReason":"company_deleted"}'::jsonb` })
+          .where(and(eq(sandboxCreationOperations.id, candidate.id), eq(sandboxCreationOperations.state, "pending")));
+        return Boolean(company);
+      });
+      logger.warn({ errorKind: retry ? "sandbox_creation_recovery_failed" : "sandbox_creation_needs_reconciliation", operationId: candidate.id, companyId: candidate.companyId },
+        retry ? "Sandbox creation recovery will retry at its saved deadline" : "Sandbox creation company was deleted; operator reconciliation is required");
+    }
+  }
+}

ui/src/components/AgentLifecycleStatus.test.tsx

@@ -26,6 +26,12 @@ it.each([undefined, "ready", "paused", "pending_approval", "terminated", "reject
   expect(agentLifecycleRefetchInterval(lifecycleState)).toBe(30_000);
 });
 
+it("does not promise automatic retry of a failed configuration check", () => {
+  const html = renderToStaticMarkup(<AgentLifecycleStatus agent={{ lifecycleState: "verifying", lifecycleError: "Update Configuration or choose Retry now." }} onRetry={() => {}} retryPending={false} />);
+  expect(html).toContain("Retry now");
+  expect(html).not.toContain("Paperclip retries automatically");
+});
+
 it("retries a failed step once while pending and removes the failure after recovery", async () => {
   const container = document.createElement("div");
   const root = createRoot(container);

ui/src/components/AgentLifecycleStatus.tsx

@@ -64,6 +64,8 @@ export function AgentLifecycleStatus({ agent, onRetry, retryPending, refreshErro
         {retryPending ? "Retrying…" : "Retry now"}
       </Button>
     </div>}
-    {phase && agent.lifecycleError && <p className="text-muted-foreground">Paperclip retries automatically; Retry now checks again immediately.</p>}
+    {phase && agent.lifecycleError && <p className="text-muted-foreground">{agent.lifecycleState === "verifying"
+      ? "Retry now checks the saved configuration again immediately."
+      : "Paperclip retries automatically; Retry now checks again immediately."}</p>}
   </div>;
 }