fix(daytona): recover an errored sandbox before stop-only cleanup (#19178)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**

A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.

A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.

**Expected behavior**

Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.

**Steps to reproduce**

1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.

**Paperclip version or commit**

Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.

**Deployment mode**

Managed Daytona sandbox execution with plugin-backed stop-only cleanup.

Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.

## What Changed

- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.

Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.

## Verification

All commands ran in a clean worktree from `master`. Vitest ran from the
real path.

- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.

## Risks

- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/native-workspace-finalization-recovery.md

@@ -407,6 +407,10 @@ and startup cancellation. Before dispatch, the controller stores a separate
 and original plugin. The initial cleanup and restart sweep both require that
 plugin to advertise `environmentStopLease`. Missing capability or failed stop
 leaves pending cleanup; ordinary release and destroy are never fallback methods.
+A typed terminal stop failure (`PluginEnvironmentStopError` with reason
+`resource_unrecoverable` or `recovery_failed`) escalates the lease for operator
+attention on its first attempt and uses the long retry interval. The lease stays
+in pending cleanup.
 
 A matching stopped receipt clears the pending intent, leaves the lease released
 and resumable, and saves `sandboxStopAndRetainReceipt`. Its request identity,

doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md

@@ -213,3 +213,5 @@ reintroduce a concurrency capability when a runtime path enforces it.
 Native export recovery requires the separately discovered `environmentStopLease` RPC (`onEnvironmentStopLease`). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback.
 
 Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
+
+A provider can refuse Stop while the allocation is in a provider error state. A stop-only hook may run one bounded provider recovery and then stop. When it cannot stop the allocation, it throws `PluginEnvironmentStopError` with a closed reason. The host treats `resource_unrecoverable` and `recovery_failed` as terminal: it escalates the pending cleanup at once and retries at its slowest interval. Deletion is never a fallback.

packages/plugins/sandbox-providers/daytona/README.md

@@ -36,6 +36,7 @@ Notes:
 - A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
 - The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
 - A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.
+- Daytona rejects Stop for a sandbox in its `error` state, for example after the sandbox disk fills. The stop-only hook confirms the retention policy first. If Daytona marks the error recoverable, the hook calls Recover once and waits at most 20 seconds for the sandbox to start. Then it stops the sandbox normally. The hook never calls Recover a second time in the same call. An unrecoverable error, a sandbox that is back in `error` after recovery, or an unfinished recovery returns a typed `PluginEnvironmentStopError` instead of the generic stop error. The sandbox is never deleted.
 - Directory export normally builds a sandbox archive and downloads it. If archive creation exhausts disk space, quota, or the file-size limit, export streams the archive through a raw terminal instead. Each bounded chunk waits for a host write acknowledgement. Import still requires successful tar completion and the existing archive/path validation. The total archive cannot exceed the provider-reported sandbox disk allocation, and each host write must leave a 1 GiB free-space reserve. This fallback writes no sandbox archive or command log, preserves source files, and retains the configured transfer timeout. Terminal cleanup has a separate five-second limit and cannot hold the host archive open. It requires Bash and GNU `tar`, `dd`, and `base64` in the sandbox. It does not repair a provider session that already failed to save its state or increase the sandbox's disk allocation.
 - Single-file export normally downloads a guarded sandbox copy. If that copy exhausts disk space, quota, or the file-size limit, the provider removes partial copies and streams the source through the same bounded sandbox-user channel. The stream rechecks canonical confinement and regular-file identity, refuses a symlink at open, and publishes host files only after a successful transfer receipt. It never hands a mutable source to the privileged download API. Permission, confinement, and transport failures still fail; a failed stream leaves prior host files intact. This removes the second archive-sized copy needed to download an already captured workspace archive. Creating the workspace archive itself still needs sandbox space.
 

packages/plugins/sandbox-providers/daytona/src/plugin.test.ts

@@ -37,7 +37,7 @@ import plugin, {
   __setDaytonaPluginContextForTest,
 } from "./plugin.js";
 import type { PluginContext } from "@paperclipai/plugin-sdk";
-import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
+import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
 import manifest from "./manifest.js";
 import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js";
 import * as duplexCommands from "./duplex-command-stream.js";
@@ -2590,6 +2590,95 @@ describe("Daytona sandbox provider plugin", () => {
     expect(sandbox.delete).not.toHaveBeenCalled();
   });
 
+  describe("explicit stop of a sandbox in the provider error state", () => {
+    const stopParams = (sandboxId: string) => ({ driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
+      providerLeaseId: sandboxId, config: { reuseLease: false } });
+    function erroredSandbox(recoverable: boolean) {
+      const sandbox = { ...createMockSandbox({ id: "sandbox-disk-full", state: "error", recoverable }),
+        errorReason: "Error response from daemon: mkdir /var/lib/docker/overlay2/layer: no space left on device" };
+      sandbox.stop.mockImplementation(async () => {
+        if (sandbox.state === "error") throw new Error("Sandbox failed to stop with status: error");
+        sandbox.state = "stopped";
+      });
+      mockGet.mockResolvedValue(sandbox);
+      return sandbox;
+    }
+    async function stopFailure(sandboxId: string) {
+      const error = await plugin.definition.onEnvironmentStopLease!(stopParams(sandboxId)).then(
+        () => { throw new Error("stop unexpectedly succeeded"); }, (caught: unknown) => caught);
+      expect(error).toBeInstanceOf(PluginEnvironmentStopError);
+      return readEnvironmentStopFailure({ data: environmentStopErrorData(error) });
+    }
+    beforeEach(() => { process.env.DAYTONA_API_KEY = "host-key"; });
+
+    it("recovers a recoverable sandbox, then stops and retains it", async () => {
+      const sandbox = erroredSandbox(true);
+      sandbox.recover.mockImplementation(async () => { sandbox.state = "started"; });
+      await expect(plugin.definition.onEnvironmentStopLease!(stopParams(sandbox.id)))
+        .resolves.toEqual({ providerLeaseId: sandbox.id, state: "stopped" });
+      expect(sandbox.recover).toHaveBeenCalledExactlyOnceWith(20);
+      expect(sandbox.setAutoDeleteInterval.mock.invocationCallOrder[0]).toBeLessThan(sandbox.recover.mock.invocationCallOrder[0]);
+      expect(sandbox.recover.mock.invocationCallOrder[0]).toBeLessThan(sandbox.stop.mock.invocationCallOrder[0]);
+      expect(sandbox.stop).toHaveBeenCalledOnce();
+      expect(sandbox.state).toBe("stopped");
+      for (const method of [sandbox.start, sandbox.delete, sandbox.archive]) expect(method).not.toHaveBeenCalled();
+    });
+
+    it("returns a typed terminal failure for an unrecoverable sandbox without provider lifecycle calls", async () => {
+      const sandbox = erroredSandbox(false);
+      expect(await stopFailure(sandbox.id)).toEqual({ reason: "resource_unrecoverable", terminal: true });
+      expect(sandbox.setAutoDeleteInterval).toHaveBeenCalledWith(-1);
+      for (const method of [sandbox.recover, sandbox.stop, sandbox.start, sandbox.delete, sandbox.archive]) {
+        expect(method).not.toHaveBeenCalled();
+      }
+    });
+
+    it.each([
+      { name: "recovery fails back to the error state", state: "error", recoverable: true,
+        expected: { reason: "recovery_failed", terminal: true } },
+      { name: "the provider marks the sandbox unrecoverable during recovery", state: "error", recoverable: false,
+        expected: { reason: "resource_unrecoverable", terminal: true } },
+      { name: "recovery is still in progress at the deadline", state: "starting", recoverable: true,
+        expected: { reason: "recovery_incomplete", terminal: false } },
+    ])("makes one recovery attempt when $name", async ({ state, recoverable, expected }) => {
+      const sandbox = erroredSandbox(true);
+      sandbox.recover.mockRejectedValue(new Error("Sandbox failed to start with status: error"));
+      sandbox.refreshData.mockImplementation(async () => {
+        if (sandbox.recover.mock.calls.length > 0) Object.assign(sandbox, { state, recoverable });
+      });
+      expect(await stopFailure(sandbox.id)).toEqual(expected);
+      expect(sandbox.recover).toHaveBeenCalledOnce();
+      expect(sandbox.stop).not.toHaveBeenCalled();
+      expect(sandbox.delete).not.toHaveBeenCalled();
+    });
+
+    it("bounds a recovery call that never settles", async () => {
+      const sandbox = erroredSandbox(true);
+      sandbox.recover.mockImplementation(() => new Promise<void>(() => {}));
+      vi.useFakeTimers();
+      try {
+        const failure = stopFailure(sandbox.id);
+        await vi.advanceTimersByTimeAsync(25_000);
+        expect(await failure).toEqual({ reason: "recovery_failed", terminal: true });
+      } finally { vi.useRealTimers(); }
+      expect(sandbox.recover).toHaveBeenCalledOnce();
+      expect(sandbox.stop).not.toHaveBeenCalled();
+    });
+
+    it("does not recover again when the recovered sandbox fails again before it stops", async () => {
+      const sandbox = erroredSandbox(true);
+      sandbox.recover.mockImplementation(async () => { sandbox.state = "started"; });
+      sandbox.stop.mockImplementation(async () => {
+        sandbox.state = "error";
+        throw new Error("Sandbox failed to stop with status: error");
+      });
+      expect(await stopFailure(sandbox.id)).toEqual({ reason: "recovery_failed", terminal: true });
+      expect(sandbox.recover).toHaveBeenCalledOnce();
+      expect(sandbox.stop).toHaveBeenCalledOnce();
+      expect(sandbox.delete).not.toHaveBeenCalled();
+    });
+  });
+
   it.each([true, false])("preserves an archived source only after fresh retention confirmation: %s", async confirmed => {
     process.env.DAYTONA_API_KEY = "host-key";
     const sandbox = createMockSandbox({ id: "sandbox-already-archived", state: "archived" });

packages/plugins/sandbox-providers/daytona/src/plugin.ts

@@ -9,7 +9,7 @@ import type {
   Resources,
   Sandbox,
 } from "@daytonaio/sdk";
-import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentCreationCleanupError, PluginEnvironmentRunnerIngressError, readEnvironmentCreationCleanupError, withEnvironmentSyncTransferStep, preserveEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk";
+import { decodeChannelBytes, definePlugin, NOOP_PLUGIN_TRACER, PluginEnvironmentCreationCleanupError, PluginEnvironmentRunnerIngressError, PluginEnvironmentStopError, readEnvironmentCreationCleanupError, withEnvironmentSyncTransferStep, preserveEnvironmentSyncErrorDiagnostic } from "@paperclipai/plugin-sdk";
 import type {
   PluginContext,
   PluginEnvironmentCreationCleanup,
@@ -2520,6 +2520,34 @@ async function closeDaytonaDuplexChannelsForLease(providerLeaseId: string): Prom
   await Promise.allSettled(matches.map(closeDaytonaDuplexChannel));
 }
 
+const STOP_RECOVERY_TIMEOUT_SECONDS = 20;
+const STOPPABLE_AFTER_RECOVERY_STATES: ReadonlySet<string> = new Set(["started", "stopped", "archived"]);
+
+function erroredAfterRecovery(sandbox: Sandbox): PluginEnvironmentStopError {
+  return new PluginEnvironmentStopError(sandbox.recoverable === false ? "resource_unrecoverable" : "recovery_failed");
+}
+
+async function refreshedSandbox(sandbox: Sandbox, timeoutMs: number): Promise<boolean> {
+  try {
+    await withLivenessTimeout("sandbox.refreshData", timeoutMs, () => sandbox.refreshData());
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+async function recoverSandboxForStop(sandbox: Sandbox, timeoutSeconds: number, readTimeoutMs: number): Promise<void> {
+  if (sandbox.recoverable !== true) throw new PluginEnvironmentStopError("resource_unrecoverable");
+  try {
+    await withLivenessTimeout("sandbox.recover", timeoutSeconds * 1_000 + LIVENESS_START_TIMEOUT_MARGIN_MS,
+      () => sandbox.recover(timeoutSeconds));
+  } catch {
+    if (!await refreshedSandbox(sandbox, readTimeoutMs)) throw new PluginEnvironmentStopError("recovery_incomplete");
+  }
+  if (sandbox.state === "error") throw erroredAfterRecovery(sandbox);
+  if (!STOPPABLE_AFTER_RECOVERY_STATES.has(sandbox.state ?? "")) throw new PluginEnvironmentStopError("recovery_incomplete");
+}
+
 async function stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise<PluginEnvironmentTerminationReceipt> {
   if (!params.providerLeaseId) throw new Error("Daytona stop requires an exact sandbox identity");
   const config = parseDriverConfig(params.config);
@@ -2539,10 +2567,20 @@ async function stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise<P
     await withLivenessTimeout("sandbox.setAutoDeleteInterval", deadline, () => sandbox.setAutoDeleteInterval(-1));
     await withLivenessTimeout("sandbox.refreshData", deadline, () => sandbox.refreshData());
     if (sandbox.autoDeleteInterval !== -1) throw new Error("Daytona retention policy was not confirmed");
+    const stopSeconds = Math.min(toTimeoutSeconds(config.timeoutMs), 30);
+    const recovering = sandbox.state === "error";
+    if (recovering) await recoverSandboxForStop(sandbox, Math.min(stopSeconds, STOP_RECOVERY_TIMEOUT_SECONDS), deadline);
     // An archive already has no live process and retains its filesystem.
     // Daytona rejects Stop in this state; leave the confirmed archive cold.
     if (sandbox.state !== "stopped" && sandbox.state !== "archived") {
-      await terminateAtProvider(scope, "sandbox.stop", () => sandbox.stop(Math.min(toTimeoutSeconds(config.timeoutMs), 30)));
+      try {
+        await terminateAtProvider(scope, "sandbox.stop", () => sandbox.stop(stopSeconds));
+      } catch (error) {
+        if (recovering && await refreshedSandbox(sandbox, deadline) && sandbox.state === "error") {
+          throw erroredAfterRecovery(sandbox);
+        }
+        throw error;
+      }
     }
     sandboxHandleSessionStore.clear(scope);
     await closeDaytonaDuplexChannelsForLease(params.providerLeaseId);

packages/plugins/sdk/README.md

@@ -64,6 +64,24 @@ from error text. Unknown and configuration failures keep their normal errors.
 The envelope alone grants no retry, accounting or cleanup authority; native
 execution also requires its complete, current controller checkpoint proof.
 
+## Stop failures
+
+Environment providers can throw `PluginEnvironmentStopError` from
+`onEnvironmentStopLease` when they cannot stop a resource that is in a provider
+error state. The error takes one closed reason:
+
+- `resource_unrecoverable`: the provider reports that the resource cannot recover.
+- `recovery_failed`: the resource is in an error state again after one recovery attempt.
+- `recovery_incomplete`: recovery did not finish before the stop deadline.
+
+The worker forwards only the reason. It omits messages, provider error text and
+causes. The host reads it with `readEnvironmentStopFailure`, which derives the
+`terminal` flag from the reason. The first two reasons are terminal: another
+automatic attempt cannot succeed without provider or operator action. The host
+escalates a terminal failure at once and retries it at its slowest cleanup
+interval. It never abandons the pending cleanup. Other stop errors keep their
+normal retry behavior.
+
 ## Worker quick start
 
 ```ts

packages/plugins/sdk/src/environment-stop-error.ts

@@ -0,0 +1,50 @@
+const SCHEMA = "paperclip/environment-stop-error/v1";
+const REASONS = ["resource_unrecoverable", "recovery_failed", "recovery_incomplete"] as const;
+const TERMINAL_REASONS: ReadonlySet<string> = new Set(["resource_unrecoverable", "recovery_failed"]);
+
+export type PluginEnvironmentStopFailureReason = typeof REASONS[number];
+
+export interface PluginEnvironmentStopFailure {
+  reason: PluginEnvironmentStopFailureReason;
+  terminal: boolean;
+}
+
+const MESSAGES: Record<PluginEnvironmentStopFailureReason, string> = {
+  resource_unrecoverable: "The provider reports that the environment resource cannot recover from its error state, so it cannot be stopped.",
+  recovery_failed: "The environment resource returned to an error state after a recovery attempt, so it cannot be stopped.",
+  recovery_incomplete: "The environment resource did not finish recovery from its error state before the stop deadline.",
+};
+
+function field(value: unknown, name: string): unknown {
+  if (!value || typeof value !== "object") return undefined;
+  try { return Object.getOwnPropertyDescriptor(value, name)?.value; }
+  catch { return undefined; }
+}
+
+function validate(value: unknown): PluginEnvironmentStopFailure | null {
+  const reason = field(value, "reason");
+  if (!REASONS.includes(reason as PluginEnvironmentStopFailureReason)) return null;
+  return { reason: reason as PluginEnvironmentStopFailureReason, terminal: TERMINAL_REASONS.has(reason as string) };
+}
+
+export class PluginEnvironmentStopError extends Error {
+  readonly failure: PluginEnvironmentStopFailure | null;
+
+  constructor(reason: PluginEnvironmentStopFailureReason) {
+    const failure = validate({ reason });
+    super(failure ? MESSAGES[failure.reason] : "The environment resource could not be stopped.");
+    this.name = "PluginEnvironmentStopError";
+    this.failure = failure;
+  }
+}
+
+export function environmentStopErrorData(error: unknown): unknown {
+  if (!(error instanceof PluginEnvironmentStopError)) return undefined;
+  const failure = validate(field(error, "failure"));
+  return failure ? { schema: SCHEMA, failure: { reason: failure.reason } } : undefined;
+}
+
+export function readEnvironmentStopFailure(error: unknown): PluginEnvironmentStopFailure | null {
+  const data = field(error, "data");
+  return field(data, "schema") === SCHEMA ? validate(field(data, "failure")) : null;
+}

packages/plugins/sdk/src/index.ts

@@ -464,6 +464,8 @@ export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readE
 export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js";
 export { PluginEnvironmentRunnerIngressError, environmentRunnerIngressErrorData, readEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
 export type { PluginEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
+export { PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "./environment-stop-error.js";
+export type { PluginEnvironmentStopFailure, PluginEnvironmentStopFailureReason } from "./environment-stop-error.js";
 
 export type { AiConnectionPool, AiConnectionPoolConfig, AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, AiConnectionRouterSelection } from "@paperclipai/shared";
 

packages/plugins/sdk/src/worker-rpc-host.ts

@@ -7,6 +7,7 @@ import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paper
 import { environmentCreationCleanupErrorData } from "./environment-creation-cleanup.js";
 import { environmentSyncErrorData, withEnvironmentSyncErrorCapture } from "./environment-sync-error.js";
 import { environmentRunnerIngressErrorData } from "./environment-runner-ingress-error.js";
+import { environmentStopErrorData } from "./environment-stop-error.js";
 import { createInvocationTracer } from "./invocation-tracer.js";
 import { createPluginIdleDrain } from "./idle-drain.js";
 /**
@@ -1646,7 +1647,8 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
         method === "environmentAcquireLease" || method === "environmentDestroyLease"
           ? environmentCreationCleanupErrorData(err, method === "environmentAcquireLease")
           : method === "environmentSyncOut" ? environmentSyncErrorData(err)
-          : method === "environmentRunnerIngressEndpoint" ? environmentRunnerIngressErrorData(err) : undefined));
+          : method === "environmentRunnerIngressEndpoint" ? environmentRunnerIngressErrorData(err)
+          : method === "environmentStopLease" ? environmentStopErrorData(err) : undefined));
     } finally {
       done?.();
     }

packages/plugins/sdk/tests/environment-stop-error.test.ts

@@ -0,0 +1,39 @@
+import { describe, expect, it } from "vitest";
+import { environmentStopErrorData, PluginEnvironmentStopError, readEnvironmentStopFailure } from "../src/environment-stop-error.js";
+
+describe("environment stop failure contract", () => {
+  it.each([
+    ["resource_unrecoverable", true],
+    ["recovery_failed", true],
+    ["recovery_incomplete", false],
+  ] as const)("serializes %s with its fixed terminal class", (reason, terminal) => {
+    const error = new PluginEnvironmentStopError(reason);
+    const data = environmentStopErrorData(error);
+    expect(data).toEqual({ schema: "paperclip/environment-stop-error/v1", failure: { reason } });
+    expect(readEnvironmentStopFailure({ data })).toEqual({ reason, terminal });
+    expect(error.cause).toBeUndefined();
+    expect(error.message).not.toContain(reason);
+  });
+  it("forwards only the SDK error class", () => {
+    const forged = Object.assign(new Error("private-detail"), { failure: { reason: "recovery_failed", terminal: true } });
+    expect(environmentStopErrorData(forged)).toBeUndefined();
+    expect(environmentStopErrorData(new Error("private-detail"))).toBeUndefined();
+  });
+  it.each([{}, null, { reason: "unknown" }, { reason: 1 }, { reason: "recovery_incomplete ", terminal: true }])("rejects unsupported evidence %j", value => {
+    expect(readEnvironmentStopFailure({ data: { schema: "paperclip/environment-stop-error/v1", failure: value } })).toBeNull();
+  });
+  it("derives the terminal class from the reason instead of trusting the sender", () => {
+    expect(readEnvironmentStopFailure({ data: { schema: "paperclip/environment-stop-error/v1",
+      failure: { reason: "recovery_incomplete", terminal: true } } })).toEqual({ reason: "recovery_incomplete", terminal: false });
+    expect(readEnvironmentStopFailure({ data: { schema: "paperclip/environment-runner-ingress-error/v1",
+      failure: { reason: "recovery_failed" } } })).toBeNull();
+  });
+  it("does not invoke inherited or accessor fields", () => {
+    let reads = 0;
+    const getter = Object.defineProperty({}, "data", { get() { reads++; throw new Error("private"); } });
+    expect(readEnvironmentStopFailure(getter)).toBeNull();
+    expect(readEnvironmentStopFailure(Object.create({ data: environmentStopErrorData(
+      new PluginEnvironmentStopError("recovery_failed")) }))).toBeNull();
+    expect(reads).toBe(0);
+  });
+});

packages/plugins/sdk/tests/worker-rpc-host.test.ts

@@ -23,6 +23,7 @@ import {
   type PluginInvocationContext,
 } from "../src/protocol.js";
 import { isWorkerEntrypoint, startWorkerRpcHost } from "../src/worker-rpc-host.js";
+import { PluginEnvironmentStopError, readEnvironmentStopFailure } from "../src/environment-stop-error.js";
 
 describe("isWorkerEntrypoint", () => {
   const tempRoots: string[] = [];
@@ -355,6 +356,7 @@ describe("worker configChanged cross-tenant guard", () => {
             reject(
               Object.assign(new Error(response.error.message), {
                 code: response.error.code,
+                data: response.error.data,
               }),
             );
             return;
@@ -418,6 +420,21 @@ describe("worker configChanged cross-tenant guard", () => {
     } finally { worker.stop(); }
   });
 
+  it.each([true, false])("forwards only a typed stop failure to the host: %s", async typed => {
+    const worker = makeWorker(definePlugin({ async setup() {},
+      async onEnvironmentStopLease() {
+        throw typed ? new PluginEnvironmentStopError("resource_unrecoverable") : new Error("private provider detail");
+      },
+    }));
+    try {
+      await worker.initialize();
+      const error = await worker.callWorker("environmentStopLease", { driverKey: "fixture", companyId: "company",
+        environmentId: "environment", providerLeaseId: "allocation", config: {} }).catch((caught: unknown) => caught);
+      expect(readEnvironmentStopFailure(error)).toEqual(typed ? { reason: "resource_unrecoverable", terminal: true } : null);
+      expect(JSON.stringify((error as { data?: unknown }).data ?? null)).not.toContain("private");
+    } finally { worker.stop(); }
+  });
+
   it.each([false, true])("advertises read-only inspection only when implemented: %s", async supported => {
     let inspections = 0, mutations = 0;
     const worker = makeWorker(definePlugin({ async setup() {},

server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts

@@ -38,6 +38,7 @@ vi.mock("../middleware/logger.js", () => ({
 import { logger } from "../middleware/logger.ts";
 import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/heartbeat.ts";
 import { hasPendingRunCleanup } from "../services/execution-blocker.js";
+import { environmentStopErrorData, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, PluginEnvironmentStopError } from "@paperclipai/plugin-sdk";
 
 const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
 const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
@@ -355,6 +356,42 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
     expect(Number(saved.metadata?.pendingCleanupRetryAfterMs)).toBeGreaterThan(Date.now() + 29 * 60_000);
   });
 
+  it.each([
+    { reason: "resource_unrecoverable", escalated: true },
+    { reason: "recovery_failed", escalated: true },
+    { reason: "recovery_incomplete", escalated: false },
+  ] as const)("escalates only a terminal provider stop failure on its first attempt: $reason", async ({ reason, escalated }) => {
+    const { companyId, environmentId } = await seedCompanyAndEnvironment();
+    const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(Date.now() - 60 * 60_000) });
+    const failure = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "private provider detail",
+      data: environmentStopErrorData(new PluginEnvironmentStopError(reason)) });
+    const teardown = vi.fn(async () => { throw failure; });
+    const service = heartbeatService(db, { environmentRuntime: {
+      retryPendingSandboxTeardown: teardown,
+    } as unknown as HeartbeatEnvironmentRuntime });
+
+    expect(await service.sweepPendingCleanupLeases()).toEqual({ swept: 1, destroyed: 0, capped: escalated ? 1 : 0 });
+    expect(teardown).toHaveBeenCalledOnce();
+    const [saved] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
+    expect(saved.status).toBe("pending_cleanup");
+    expect(saved.cleanupStatus).toBe("failed");
+    expect(saved.metadata?.[ATTEMPTS_KEY]).toBe(1);
+    expect(saved.metadata?.[CAP_WARNED_KEY] === true).toBe(escalated);
+    const retryAfterMs = Number(saved.metadata?.pendingCleanupRetryAfterMs);
+    if (escalated) expect(retryAfterMs).toBeGreaterThan(Date.now() + 29 * 60_000);
+    else expect(retryAfterMs).toBeLessThan(Date.now() + 60_000);
+
+    const warnings = vi.mocked(logger.warn).mock.calls;
+    expect(warnings.find(call => call[1] === "pending_cleanup lease retry failed")?.[0]).toMatchObject({
+      leaseId, attempts: 1, providerStopFailure: reason,
+      errorKind: escalated ? "provider_stop_terminal" : "destroy_failed",
+    });
+    const attention = warnings.filter(call => String(call[1]).startsWith("environment lease needs operator attention"));
+    expect(attention).toHaveLength(escalated ? 1 : 0);
+    if (escalated) expect(attention[0]?.[0]).toMatchObject({ leaseId, providerStopFailure: reason });
+    expect(JSON.stringify(warnings)).not.toContain("private provider detail");
+  });
+
   it("retries after a restart and cooldown when the provider recovers", async () => {
     const { companyId, environmentId } = await seedCompanyAndEnvironment();
     const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId,

server/src/services/heartbeat/recovery.ts

@@ -32,6 +32,7 @@ import {
   revokeExpiredLegacyControllerWithState,
 } from "../legacy-controller-lease.js";
 import { remoteTerminationReceipt } from "../remote-execution-termination.js";
+import { readEnvironmentStopFailure, type PluginEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
 import { runUsedConversationAdapter } from "../conversation-continuation.js";
 import { legacyExecutionNeedsReconciliationWithEvidence, terminalizeLegacyExecution } from "../legacy-execution-recovery.js";
 import { reconcileRetainedWorkspaces } from "../workspace-recovery.js";
@@ -198,10 +199,13 @@ const ORPHANED_ACTIVE_LEASE_SWEEP_PAGE_SIZE = 20;
 // A provider or plugin destroy rejection can carry a bearer credential, a
 // signed URL, or provider response detail in its name, code, message, cause, or
 // stack. The exception fields cross the server boundary, so they are not a
-// trusted enum. The pending_cleanup sweep logs never read the exception. Each
-// catch site logs a constant, locally generated `errorKind` instead.
+// trusted enum. The pending_cleanup sweep logs never read these fields. Each
+// catch site logs a constant, locally generated `errorKind` instead, plus at most
+// the closed stop-failure reason that `readEnvironmentStopFailure` validates.
 const PENDING_CLEANUP_RETRY_ERROR_KIND = "destroy_failed";
 
+const PENDING_CLEANUP_TERMINAL_STOP_ERROR_KIND = "provider_stop_terminal";
+
 const PENDING_CLEANUP_SWEEP_ERROR_KIND = "sweep_failed";
 
 const ORPHANED_ACTIVE_LEASE_SWEEP_ERROR_KIND = "orphaned_active_lease_sweep_failed";
@@ -1438,10 +1442,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
 
   // Atomically claim the one-time cap warning for a lease. The update only
   // matches when the lease is still pending_cleanup, its stored attempt count is
-  // at or above the cap, and it has not yet carried the warned flag. Two
-  // concurrent sweeps that both reach the cap race here, but only one update
-  // sets the flag and returns a row. The loser skips the warning. This keeps the
-  // warning to one log line per lease.
+  // at or above the cap or the provider reported a terminal stop failure, and it
+  // has not yet carried the warned flag. Two concurrent sweeps that both reach
+  // the cap race here, but only one update sets the flag and returns a row. The
+  // loser skips the warning. This keeps the warning to one log line per lease.
   //
   // The status and cap predicates are the last line of defense. They stop a warn
   // flag write to a lease that left pending_cleanup or dropped below the cap
@@ -1449,6 +1453,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
   // `jsonb_set`, so a concurrent write to an unrelated metadata key survives.
   async function claimPendingCleanupCapWarning(
     leaseId: string,
+    terminalProviderFailure = false,
   ): Promise<boolean> {
     const now = new Date();
     const claimed = await db
@@ -1461,7 +1466,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
         and(
           eq(environmentLeases.id, leaseId),
           eq(environmentLeases.status, "pending_cleanup"),
-          sql`${pendingCleanupAttemptsSql()} >= ${PENDING_CLEANUP_SWEEP_ATTEMPT_CAP}`,
+          terminalProviderFailure ? undefined : sql`${pendingCleanupAttemptsSql()} >= ${PENDING_CLEANUP_SWEEP_ATTEMPT_CAP}`,
           sql`${pendingCleanupCapWarnedSql()} = false`,
         ),
       )
@@ -1757,6 +1762,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
       pendingCleanupAttemptsInFlight.add(row.id);
       lease.metadata = { ...lease.metadata, pendingCleanupAttemptId: claimed };
       let activeRenewal: Promise<void> | null = null;
+      let providerStopFailure: PluginEnvironmentStopFailure | null = null;
       const renewal = setInterval(() => {
         if (activeRenewal) return;
         activeRenewal = db.update(environmentLeases).set({
@@ -1803,7 +1809,8 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
             destroyed += 1;
           }
         }
-      } catch {
+      } catch (error) {
+        providerStopFailure = readEnvironmentStopFailure(error);
         // The recorded-data teardown throws on failure, so revert the lease to
         // pending_cleanup for a later sweep. The claimed attempt still counts
         // for backoff, so requests stay bounded. The `destroyRunLease`
@@ -1817,10 +1824,13 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
           });
         }
         // Log a constant errorKind only. The exception can carry a credential in
-        // its name, code, message, cause, or stack, so the sweep never reads it.
+        // its name, code, message, cause, or stack, so the sweep reads only the
+        // closed, validated provider stop-failure reason from it.
         logger.warn(
           {
-            errorKind: PENDING_CLEANUP_RETRY_ERROR_KIND,
+            errorKind: providerStopFailure?.terminal
+              ? PENDING_CLEANUP_TERMINAL_STOP_ERROR_KIND : PENDING_CLEANUP_RETRY_ERROR_KIND,
+            ...(providerStopFailure ? { providerStopFailure: providerStopFailure.reason } : {}),
             leaseId: row.id,
             environmentId: row.environmentId,
             attempts: attempts + 1,
@@ -1833,15 +1843,17 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
         // Late renewals are attempt-fenced and never write the retry cooldown.
         pendingCleanupAttemptsInFlight.delete(row.id);
       }
-      if (attempts + 1 >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP) {
+      const escalated = attempts + 1 >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP || providerStopFailure?.terminal === true;
+      if (escalated) {
         capped += 1;
         // Warn once, then continue automatic cleanup with backoff. The atomic claim
         // keeps the warning to one log line even when two sweeps overlap.
         if (metadata[PENDING_CLEANUP_CAP_WARNED_METADATA_KEY] !== true) {
-          const warned = await claimPendingCleanupCapWarning(row.id);
+          const warned = await claimPendingCleanupCapWarning(row.id, providerStopFailure?.terminal === true);
           if (warned) {
             logger.warn(
-              { leaseId: row.id, environmentId: row.environmentId, attempts },
+              { leaseId: row.id, environmentId: row.environmentId, attempts,
+                ...(providerStopFailure?.terminal ? { providerStopFailure: providerStopFailure.reason } : {}) },
               "environment lease needs operator attention; automatic cleanup continues with backoff",
             );
           }
@@ -1852,7 +1864,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
       await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
         metadata: sql`${pendingCleanupMetadataObjectSql()} || ${JSON.stringify({
           pendingCleanupInFlight: false,
-          pendingCleanupRetryAfterMs: Date.now() + (attempts + 1 >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP
+          pendingCleanupRetryAfterMs: Date.now() + (escalated
             ? 30 * 60_000 : Math.min(30 * 60_000, Math.max(30_000, backoffMs))),
         })}::jsonb`,
       }).where(and(eq(environmentLeases.id, lease.id),