mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - When an idle close failed, the idle-timer path kept the session registered and never tried again, unless the sandbox lease was already inactive. > - The controller-restart path tried once more, but it accepted no proof that the session was already gone. It checked a destroyed sandbox only for a forced environment delete. > - In a release-swap test, one idle close failed with `runner_remote_process_identity_unavailable`. The session stayed registered until shutdown, the shutdown close failed again, and the sandbox was held until the host stopped it. > - This pull request treats a failed close as complete when the Runner is proven gone or the sandbox is stopped or destroyed, retries a failed idle close a bounded number of times, and logs which Runner check failed. > - The benefit is that a dead warm session is released, and the next failure names its cause. ## Linked Issues or Issue Description Refs #21627 **What happened?** In a zero-downtime release-swap test, a warm session's idle close failed 4 s after the server left new run starts. The failure was `runner_remote_process_identity_unavailable`. The idle-timer path kept the session registered with no retry. At SIGTERM, `closeIdleWarmNativeSessionsForRestart` tried again and failed the same way. The log showed "idle warm native sessions did not close" and the restart-chaos harness counted a warm-shutdown failure. The log did not say which Runner check failed, so the cause of the unreadable identity is not known. **Expected behavior** A close whose Runner is proven gone, or whose sandbox is stopped or destroyed, releases the session. A failed idle close is tried again a bounded number of times. The failure text names the Runner check stage and reason. **Steps to reproduce** 1. Keep a warm remote session idle. 2. Make its close fail with a `RemoteRunnerRecoveryError` (for example, `process_verification/process_not_running`). 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `failed: 1`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only: the close error's Runner diagnostic is `process_verification/process_not_running`, or a lease row for the sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with state `destroyed`, or the warm run's own lease row has a receipt with state `stopped`. `pending_cleanup` and a lease without a receipt prove nothing. A lease read error counts as no proof. - `closeIdleWarmNativeSessions` (controller restart, environment delete, idle sleep) uses this proof for every failed close, not only for a forced environment delete. A released Runner detach failure is unchanged. - The idle-timer path uses the same proof. Its earlier status-only rule from #21504 (any lease status other than `active` or `retained`) is removed, because `pending_cleanup` is written before the provider stop. Without proof, it re-arms the timer and tries again, every 5 s, at most 3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The log line records the attempt number. - `describeWarmCloseFailure` appends the Runner check as `[stage/reason/lifecycle]` when the error carries one. - `server/src/services/native-runtime/native-session-executor.test.ts`: three tests. - "an idle warm session whose close fails with runner_remote_process_identity_unavailable and whose Runner is proven gone is released at controller restart" - "names the Runner check in a warm close failure and keeps an unproven owner registered at controller restart" - "retries a failed idle warm close a bounded number of times" - The #21504 idle-path test is now "forgets a warm owner whose idle close failed only once its sandbox has a stop or destroy receipt". A stop or destroy receipt releases the owner. `pending_cleanup`, `released` without a receipt, and `retained` keep it. - "reads the warm owner's own lease rows at controller restart" (a stop receipt for this run releases the owner; `pending_cleanup`, no receipt, or another run's receipt do not) ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1519 passed, 7 skipped. - Red on master (with only the two exported constants added so the file compiles): all three new tests fail. The first fails with `"failed": 1`. - `npx tsc --noEmit -p server`: no errors. ## Risks - A session is released without a checkpoint only when its Runner is proven gone or its sandbox is stopped or destroyed. The last saved checkpoint stays the recovery source, which is the same as the forced environment delete path today. - Environment delete and idle sleep now also accept the proof. Before, they accepted only a destroyed receipt during a forced delete. `pending_cleanup` is written before the provider stop, so it never counts as proof here. - The idle retry adds at most 3 more close attempts per failed session, 5 s apart. - A failed idle close whose sandbox stop is still pending now keeps its owner until a receipt lands or a successor retires it. Before, the status alone released it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>