mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - A sandbox has one Runner port. Before a new run launches its Runner on a reused sandbox, it retires every other Runner on that port. > - In a zero-downtime release-swap test, a run finished on the outgoing server and stayed warm. 77 ms later, a follow-up run started on the incoming server. It reused the same sandbox, started a fresh provider session, retired the old Runner, and launched its own Runner on the same port. > - The outgoing server's idle close could not suspend the old Runner. Its checkpoint was correctly refused with `native_runner_checkpoint_owner_lost`. The idle timer retried 3 times and then kept the owner registered. That phantom owner blocked idle sleep for 4.8 minutes, and the shutdown close failed again. > - The existing proofs that a warm owner is gone (#21782) do not count "a later run launched its own Runner on this sandbox". > - This pull request adds that proof. When it holds, the server forgets the owner without a checkpoint. The checkpoint fence does not change. > - The benefit is that a superseded warm owner no longer blocks idle sleep or fails a clean shutdown. ## Linked Issues or Issue Description Refs #21627 Refs #21782 **What happened?** An idle warm owner finished on the outgoing server. A follow-up run on the incoming server reused the same sandbox and launched its own Runner on the sandbox's single Runner port. The owner's idle close failed with `native_runner_checkpoint_owner_lost` 4 times. The owner stayed registered, held the idle-work flag, and blocked idle sleep. At SIGTERM the restart sweep logged "idle warm native sessions did not close" with "runner did not durably suspend before checkpoint". No task was affected. **Expected behavior** A failed idle close releases its owner when a later run proves that the owner's Runner is gone. The checkpoint stays refused while the successor is live. **Steps to reproduce** 1. Finish a warm run on a sandbox and keep it idle. 2. Start a successor run for the same task on another server. Give it a lease on the same sandbox, its own `native.process_identity_recorded` event and its own `session.started` event. 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays `present`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: new `warmRunnerPortTakenBySuccessor`, used by `warmCloseFailureProvesOwnerGone` after the existing sandbox-termination proof. It holds only when another run has all of these: - an `environment_leases` row with the same environment and `providerLeaseId` as the owner; - its own `native.process_identity_recorded` event (`source_event_id is null`, so an inherited identity does not count); - a `session.started` event from its own Runner instance (`source_instance_id` equals the run's `runner_instance_id`); - both events created after the owner run's `finishedAt`. - When the proof holds, the idle-timer path and the controller-restart path log "warm native session's Runner or sandbox is gone; released its owner" and forget the owner without a checkpoint. Both paths already call `warmCloseFailureProvesOwnerGone`. - `validateNativeRunnerCheckpointOwner` does not change. A checkpoint still cannot land under a live successor. - Cost: the proof runs only after a failed close. It does one primary-key read of the owner run and one query on `environment_leases.provider_lease_id` (indexed), joined to `heartbeat_runs` by primary key, with `heartbeat_run_events` read through `(company_id, run_id)`. - New real-database test file `server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`. - `native-session-executor.test.ts`: one environment-delete test counts the extra proof read when the first proof does not hold. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`: 6 passed. - "releases an idle warm owner whose checkpoint is refused because a successor on another server launched its own Runner on the same sandbox: own_runner": the close is refused with `native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed: 0`, the idle-work flag is `none`, the owner has no checkpoint row, and `assertNativeRunnerCheckpointOwner` still throws `native_runner_checkpoint_owner_lost`. - These stay failed and keep the owner registered: the successor is on another sandbox; the successor has only `native.process_start_requested`; the successor has only inherited events (`source_event_id` set); the successor's events come before the owner finished. - "releases a superseded idle warm owner from its idle timer without writing a checkpoint": the idle timer releases the owner after the first refused close. - Red on master (test file only, fix reverted): `own_runner` fails with `closed: 0, failed: 1`; the idle-timer test fails with the idle-work flag still `present`. The 4 negative cases pass on master and with the fix. - `cd server && npx vitest run src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`: 9 passed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: all passed. - `cd server && npx tsc --noEmit -p .`: no errors. - `pnpm verify`: see the comment below for the result. - Not run: a restart-chaos wave. The release-swap timing is reproduced in the database test, not on a live sandbox. ## Risks - A superseded owner is now released without a checkpoint. Its old provider conversation cannot be resumed. The successor had already started a different conversation and holds the task, so no one would resume the old one. - The proof needs the successor's own Runner identity and own session start after the owner finished, on the same sandbox lease. A successor that reused the owner's Runner, only requested a start, or recorded its events earlier does not count. In those cases the close fails as before. - The proof adds two small indexed reads after a failed close only. The normal close path does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>