mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes an idle session after 5 minutes, and closes it again for an environment delete, idle sleep, or a restart. > - In a zero-downtime release-swap test, two warm closes failed and did not recover. > - In the first case, the Runner of a finished run refused `runner.suspend`. The server kept the Runner alive and the session registered. An environment delete returned 409 thirty-five times, and only an outside sandbox destroy released the session. A live Runner and a paid sandbox were held, and idle sleep was blocked. > - In the second case, the Runner durably suspended and exited, but its process check returned exit 44 (`process_identity_changed`) on every attempt. Exit 44 also covers "a live process without `--runner-id`", so no proof could release the owner. > - Also, the bounded idle retries from #21782 did not run for the first case. > - This pull request stops the retained Runner of a finished owner with the verified port-holder retirement, releases the owner only when a fresh check reports `process_not_running`, splits exit 44, accepts a durably suspended Runner with no live process that carries its runner id, and keeps the idle retries alive after a failed sweep. > - The benefit is that a finished warm session can no longer hold a live Runner and a sandbox until someone destroys the sandbox. ## Linked Issues or Issue Description Refs #21782 Refs #21835 **What happened?** Release-swap gate wave, two warm-session close failures: 1. A finished run's idle close failed with `native_session_close_unrecoverable`: `runner.suspend` failed, `runnerRetained: true`, Runner lifecycle `ready`. Only close attempt 1 was logged. An environment delete with `forceStopUnsettled` could not discard the session, because `closeWarmNativeSession` discards only with a Launcher task owner or a suspended Runner. The delete got 409 thirty-five times. The session was released only after an outside sandbox destroy. 2. A Runner's `runner-state.json` showed lifecycle `suspended`, with drain and suspend `completed`. The process check exited 44 on every attempt, and the Runner process was gone later while the sandbox stayed started. **Expected behavior** A finished owner whose Runner refused suspend is stopped and released, and a failed stop keeps it held. A durably suspended, settled Runner with no live process that carries its runner id releases its owner. A bare `process_identity_changed` never releases an owner. **Steps to reproduce** 1. Keep a warm sandbox session idle after its run succeeds. 2. Make its close fail with `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`. 3. Call `closeWarmNativeSessionsForEnvironment({ forceStopUnsettled: true })`. On master the result is `failed: 1`, every time. **Why the retries did not run** - The transport memoizes `close()`. A second close returns the same rejected promise and does no sandbox work. The 27 environment-delete failures have identical timings (`totalMs: 6055`), and the other Runner's daemon log shows no sandbox commands during its three retries. So a retry can only succeed through a fresh proof, not through the close. - `closeIdleWarmNativeSessions` (environment delete, idle sleep, restart) clears the entry's idle timer, which is also the pending retry timer, and never re-arms it after a failed close. The test "keeps retrying a failed idle warm close after a failed environment sweep cleared its retry timer" is red on master for this reason. - The log shows one more `close()` request on the same transport in the same second as attempt 1, and no attempt 2. Every path that closes an idle owner from outside the timer clears that timer. The log does not show which caller it was, so this part is not proven. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - A warm entry keeps its sandbox runner and remote directory when the run used a sandbox that the server reaches directly (no Launcher task). - `warmCloseFailureProvesOwnerGone` adds two proofs: - The close failed with `NativeSessionCloseUnrecoverableError`, `runnerRetained: true` and not suspended. The server stops the session's own Runner with `retireSandboxRunnerPortHolders` (owner run must be `succeeded` or `cancelled`, same agent, idle state; TERM, then KILL). It releases the owner without a checkpoint only if a fresh process check then exits 45 (`process_not_running`). A stop that cannot be confirmed keeps the owner. - The close failed with the new `process_runner_id_missing`. A fresh sandbox inspection must show lifecycle `suspended`, settled state (empty outbox, no pending terminal delivery, provider cleanup, or warm transition), matching run, session and runner ids, and no process that carries `--runner-id <id>`. - Both the idle-timer failure path and `closeIdleWarmNativeSessions` (environment delete with `forceStopUnsettled`, idle sleep, restart) use these proofs. - The first Runner-check or refused-suspend close failure for a sandbox owner logs one evidence line: stage, reason, Runner lifecycle, suspend command status, proof window, and an inspection of the sandbox (durable lifecycle, settled, identity match, marker pid, and pid/state of each `paperclip-runnerd` or runner-id process). It logs no nonce and no command line. - A sweep that fails to close an owner whose idle close already failed re-arms the bounded idle retry. - `server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`: `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` exits 48 (`process_runner_id_missing`) for a live process whose command line lacks the runner id. Exit 44 now means only an identity file mismatch. - `server/src/services/native-runtime/sandbox-runner-port-holders.ts`: `retireSandboxRunnerPortHolders` takes `only: { sessionDirectory, runId }` to retire one session's Runner. New `sandboxSessionRunnerExited` and `inspectSandboxSessionRunner` (a read-only `node` script). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/sandbox-runner-port-holders.test.ts src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts src/services/runner-lost-mid-turn-continuation.test.ts src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts` - Red on master (master sources with these tests): "stops the retained Runner of a finished warm owner that refused suspend ..." returns `failed: 1`. "keeps retrying a failed idle warm close after a failed environment sweep ..." sees 2 closes, not 3. "releases at controller restart a warm owner whose Runner durably suspended ..." returns `failed`. - Negative tests: a Runner that survives TERM and KILL stays `failed: 1` with no exit check. A run that is still `running` is never signalled. A live process that still carries the runner id, and a bare `process_identity_changed`, keep the owner. - Script tests: the process check exits 48 for a live process without `--runner-id`, 0 with it, and 44 for a changed marker. The inspection script reports state, marker pid and runner processes, never the nonce or a command line, and fails closed without `/proc` or through a symlink. The port-holder integration test stops only the named session's finished Runner (a real local process) and then proves it exited. - `pnpm verify` exited 2 (opaque setup dependency), so focused checks and required CI give the coverage. `npx tsc --noEmit -p server`: no errors. ## Risks - A finished owner's retained Runner is now stopped when its suspend was refused. The session is released without a new checkpoint. The last saved checkpoint stays the recovery source, which is the same as an environment delete today. - The stop uses the same rules as a successor's port-holder retirement: finished owner run, same agent, idle Runner state, and a verified process identity. Any doubt keeps the owner. - The stop can take up to about 20 s (TERM and KILL waits) inside a sweep, while the entry stays fenced. - The first close failure of a sandbox owner runs one read-only inspection command (5 s limit). - Previous-release story: no persisted state or launch identity changes. The scripts are sent with each command, so a server always runs its own exit-code table. Exit 44 from the previous release maps to `process_identity_changed`, which still never releases an owner. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>