mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, masteraaec67b891) found two failures with one cause. Task 1: run A finished on the outgoing server at 15:13:56. Its warm session stayed idle in that server. The incoming server claimed the follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked `evaluateNativeControllerTakeover`, which allowed the takeover with `remote_controller_lease_released_or_expired`. That lease covers the finished run, not the warm session. The outgoing server was alive until 15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed with `remote_warm_session_unproven`, the failure was saved as ambiguous and owned by the board, and the task stayed blocked with a Retry. Task 2: the same shape on a later swap started 27 automatic runs in 13 minutes. Fourteen of them said "The next attempt starts on a fresh sandbox", but each one resumed the same sandbox and got the same refusal (`journal_unsettled`). **Expected behavior** A follow-up claimed during an overlap waits for the warm session owner to let go, then continues with no Retry banner and no second controller. If the outgoing server dies with the session open, the takeover continues after its ownership ends. A message that promises a fresh sandbox is true, or the retries stop after a bounded number with a true state. **Steps to reproduce** 1. Run the restart-chaos release-swap preset with a warm (opencode or codex) staging mix. 2. Let a run finish on the outgoing server after the traffic switch, with a follow-up comment already queued. 3. See the follow-up fail with `remote_warm_session_unproven` and the task go blocked. **Paperclip version or commit** masteraaec67b891and45531b719e. **Deployment mode** Hosted deploy overlap (release swap) with Daytona sandboxes. ## What Changed - `remote-warm-session-recovery.ts`: before live adoption, the run proves that the server process that ran the finished turn is gone. It uses the shared owner proof (`proveOwnerGone`): a PID check in the same process space, or the `controller_instances` lease in another process space. While that server is alive, the run defers with the existing `native_controller_recovery_waiting_for_owner` wait and checks again after 5 s. The wait ends when the server exits or is fenced, or when the finished turn's warm window (idle timeout plus two minutes) ends. - `native-session-executor.ts` and `run-start-handover.ts`: a server that leaves new run starts to a replacement closes its idle warm sessions after 1 s, through the existing idle-timeout close. A session that becomes idle after that closes after 1 s too. This is the same close that shutdown already does, done earlier. - `predecessor-retirement-fallback.ts`: when a retry meets an unretired owner on the same sandbox that its source already asked to replace, it does not ask again. The run fails without another automatic retry, and the task shows a Retry. - `predecessor-retirement.ts`: the error no longer says "The next attempt starts on a fresh sandbox", because the replacement is only a request. - `doc/execution-semantics.md`: documents the owner wait, the hand-off at the swap, and the retry stop. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts` - New: "waits while the outgoing server that ran the finished turn still holds its warm session, then takes it over once that server exits" (red on master: master returned a takeover while the owner was alive). - New: "waits for a live outgoing server in another process space until its controller lease ends" (red on master). - New: "stops waiting for a live outgoing server once the finished turn's warm window has closed" (bound check). - `cd server && npx vitest run src/__tests__/heartbeat-native-predispatch-retry.test.ts` - New: "stops retrying with a truthful failure when the next attempt met the same unretired sandbox instead of a fresh one" (red on master: master scheduled another retry on the same sandbox). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "replacement server starts"` - New: "closes a warm session promptly when a replacement server starts this server's runs (before/after the turn ends)" (both red on master). - Focused suites: native-session-executor, remote-warm-session-recovery, native-controller-recovery, native-recovered-owner-followup, native-runner-restart-recovery, heartbeat-native-predispatch-retry, environment-runtime-reusable-resume-replacement, task-interruption-presentation, sandbox-runner-port-holders. - `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits 2 (a doc is a global CI input), so required CI supplies broad coverage. - Not run here: the restart-chaos release-swap wave. The maintainer runs it after merge. ## Risks - A follow-up can now wait up to the outgoing server's remaining overlap when that server cannot close its idle session. Before, it failed and needed a Retry. The wait is bounded by the owner's exit or the warm window. - The outgoing server closes idle warm sessions earlier during an overlap. The next turn then resumes from a suspended Runner instead of a live one, as it already does after a shutdown. - A retry chain that meets the same unretired sandbox twice now stops one attempt earlier than the shared cap. The task shows a Retry instead of looping. - Rows without a recorded process space keep the existing one-release local proof, so they behave as before outside the same host. - Previous-release state: this change adds no persisted field. Finalization rows saved by the previous release already record the controller boot, PID, process space and start time, so they carry forward. A follow-up already deferred by the previous release resumes through the same wait code. - Runtime cost: the owner proof adds one `ps` start-time read (same process space) or one `controller_instances` row read (other process space) per inspection. It runs once per attempt, at most every 5 s while a live owner holds the session. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>