fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown

## Linked Issues or Issue Description

**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.

**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:

1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.

The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.

**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.

## What Changed

- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.

## Verification

- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.

## Risks

- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/plugins/PLUGIN_RUNNER_EXECUTION.md

@@ -390,6 +390,17 @@ collection, another close error from an owner without a task, or continuation
 of the same session still blocks replacement.
 A controller restart is not required for this stopped-owner reset path.
 
+Deleting an environment (with consent to destroy its reusable sandboxes) closes its
+idle warm owners first. An idle owner whose close cannot settle is retired the same
+way: one confirmed `stop`, the credential snapshot, `complete`, then instruction
+collection. An owner whose sandbox lease already carries a `destroyed` termination
+receipt has no process left to stop, so Core releases it without contacting the
+sandbox. Only then are the environment's sandboxes destroyed, through the usual
+workspace-export and restore-source guards. When an owner still cannot be retired,
+the delete returns 409 and names each run and the close and stop errors. Idle-timeout,
+restart and sleep sweeps do not force a stop. They log the exact close error for each
+owner instead of a count.
+
 When retiring the previous owner stays unproven anyway (any close failure,
 on any of the paths a new run uses to retire it: instruction preparation,
 configuration change, session reset, or a prior-owner fence on a resumed