Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 d14a3b88b1 fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - When an idle close failed, the idle-timer path kept the session
registered and never tried again, unless the sandbox lease was already
inactive.
> - The controller-restart path tried once more, but it accepted no
proof that the session was already gone. It checked a destroyed sandbox
only for a forced environment delete.
> - In a release-swap test, one idle close failed with
`runner_remote_process_identity_unavailable`. The session stayed
registered until shutdown, the shutdown close failed again, and the
sandbox was held until the host stopped it.
> - This pull request treats a failed close as complete when the Runner
is proven gone or the sandbox is stopped or destroyed, retries a failed
idle close a bounded number of times, and logs which Runner check
failed.
> - The benefit is that a dead warm session is released, and the next
failure names its cause.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
In a zero-downtime release-swap test, a warm session's idle close failed
4 s after the server left new run starts. The failure was
`runner_remote_process_identity_unavailable`. The idle-timer path kept
the session registered with no retry. At SIGTERM,
`closeIdleWarmNativeSessionsForRestart` tried again and failed the same
way. The log showed "idle warm native sessions did not close" and the
restart-chaos harness counted a warm-shutdown failure. The log did not
say which Runner check failed, so the cause of the unreadable identity
is not known.

**Expected behavior**
A close whose Runner is proven gone, or whose sandbox is stopped or
destroyed, releases the session. A failed idle close is tried again a
bounded number of times. The failure text names the Runner check stage
and reason.

**Steps to reproduce**
1. Keep a warm remote session idle.
2. Make its close fail with a `RemoteRunnerRecoveryError` (for example,
`process_verification/process_not_running`).
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `failed: 1`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only:
the close error's Runner diagnostic is
`process_verification/process_not_running`, or a lease row for the
sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with
state `destroyed`, or the warm run's own lease row has a receipt with
state `stopped`. `pending_cleanup` and a lease without a receipt prove
nothing. A lease read error counts as no proof.
- `closeIdleWarmNativeSessions` (controller restart, environment delete,
idle sleep) uses this proof for every failed close, not only for a
forced environment delete. A released Runner detach failure is
unchanged.
- The idle-timer path uses the same proof. Its earlier status-only rule
from #21504 (any lease status other than `active` or `retained`) is
removed, because `pending_cleanup` is written before the provider stop.
Without proof, it re-arms the timer and tries again, every 5 s, at most
3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The
log line records the attempt number.
- `describeWarmCloseFailure` appends the Runner check as
`[stage/reason/lifecycle]` when the error carries one.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
three tests.
- "an idle warm session whose close fails with
runner_remote_process_identity_unavailable and whose Runner is proven
gone is released at controller restart"
- "names the Runner check in a warm close failure and keeps an unproven
owner registered at controller restart"
  - "retries a failed idle warm close a bounded number of times"
- The #21504 idle-path test is now "forgets a warm owner whose idle
close failed only once its sandbox has a stop or destroy receipt". A
stop or destroy receipt releases the owner. `pending_cleanup`,
`released` without a receipt, and `retained` keep it.
- "reads the warm owner's own lease rows at controller restart" (a stop
receipt for this run releases the owner; `pending_cleanup`, no receipt,
or another run's receipt do not)

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1519
passed, 7 skipped.
- Red on master (with only the two exported constants added so the file
compiles): all three new tests fail. The first fails with `"failed": 1`.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- A session is released without a checkpoint only when its Runner is
proven gone or its sandbox is stopped or destroyed. The last saved
checkpoint stays the recovery source, which is the same as the forced
environment delete path today.
- Environment delete and idle sleep now also accept the proof. Before,
they accepted only a destroyed receipt during a forced delete.
`pending_cleanup` is written before the provider stop, so it never
counts as proof here.
- The idle retry adds at most 3 more close attempts per failed session,
5 s apart.
- A failed idle close whose sandbox stop is still pending now keeps its
owner until a receipt lands or a successor retires it. Before, the
status alone released it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:09:32 -06:00
..
…
…
…
…
…