mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - Since #21774, a follow-up that the incoming server claims during the overlap waits (`native_controller_recovery_waiting_for_owner`) and checks again every 5 s. > - About 1 s after the traffic switch (#21627), the outgoing server tries to save and close its idle warm session. The checkpoint fence refuses that save with `native_runner_checkpoint_owner_lost`. The task's current run is the parked follow-up. That run has its own finalization row under another controller boot, and the fence does not accept that kind of successor. > - The close retry from #21782 fails the same way each time. So the follow-up waits until the outgoing server exits, or until the predecessor's finish time plus the idle window plus 2 min. In the CHA-5 case that was about 225 s, and the worst case is about 425 s. Speed is a requirement. > - This pull request records which owner a parked follow-up waits on, and lets the fence accept the outgoing owner's final save when the follow-up is parked on exactly that owner and has not started. > - The benefit is that the outgoing server saves and closes the session on its first attempt, and the follow-up continues at its next 5 s check. ## Linked Issues or Issue Description Refs #21774, #21782, #21697, #21627, #21279 **What happened?** During a release-swap overlap, a warm follow-up was claimed by the incoming server while the outgoing server still owned the idle warm session. The follow-up waited correctly. The outgoing server's idle close failed with `native_runner_checkpoint_owner_lost` on every attempt. The wait ended only when the outgoing server exited, about 225 s after the follow-up was created. **Expected behavior** The outgoing server saves and closes the session while the follow-up waits. The follow-up sees the saved session and continues within one 5 s check. **Steps to reproduce** 1. Run a warm native session on a remote sandbox and let a run finish on server A. 2. Start server B with an overlap and switch traffic to it. 3. Post a follow-up comment so that B claims the follow-up while A is still alive. 4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and the follow-up waits until A exits. **Paperclip version or commit** master `d14a3b88b1`. ## What Changed - `NativeControllerRecoveryWaitingError` carries the owner that the follow-up waits on: the predecessor run and its controller boot. `waitForLiveWarmSessionOwner` and the live-owner wait in `inspectRemoteWarmSessionRecovery` set it. - `deferNativeControllerRecovery` stores that owner in the finalization `failureDetail.waitingForOwner`. The executor passes it through. - `validateNativeRunnerCheckpointOwner` accepts one more successor kind: a parked follow-up waiting on this exact owner. All of these must be true, or the fence still refuses: - The checkpoint holds an accepted result and has no new work (unchanged rule). - The follow-up finalization is `retryable_failure` with `native_controller_recovery_waiting_for_owner`, `attempt = 0`, no result, and no lease owner (no claim is running it). - `waitingForOwner` names this owner's run and controller boot. - The follow-up run is a running native run for the same company and task, on the same native session as the owner's run. - The follow-up has no process fields, no own process-start events, and no checkpoint rows. - The fence locks the follow-up's finalization row (`for share`), so a resumption claim and the save cannot interleave. - Tests in `native-controller-recovery.integration.test.ts` (real Postgres): - The outgoing server saves and closes its idle warm session while the incoming follow-up waits, and the follow-up resumes within one recheck. - Negative cases: the save is still refused when the follow-up has its own process, its own process-start event, its own checkpoint, or a live resumption claim, or when it waits on a different owner boot or run. ## Verification - Red on master: with the `native-runner-checkpoint.ts` change removed, the scenario test fails with `native_runner_checkpoint_owner_lost` at the outgoing server's first close attempt. The follow-up keeps waiting. Green with the change. - The negative cases pass with and without the change. - Measured in the test scenario (warm idle window 600 s): - Before: the follow-up waits until the outgoing server exits, or about 720 s (finish + 600 s idle window + 120 s margin). - After: the outgoing save and close succeed on the first attempt. The follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after it parked. - `vitest run` of `native-controller-recovery.integration.test.ts`, `native-runner-restart-recovery.integration.test.ts` and `native-harness-backup-storage.integration.test.ts`: 349 passed, 14 skipped. - `vitest run` of `native-session-executor.test.ts`, `remote-warm-session-recovery.test.ts`, `native-runner-checkpoint-save.test.ts`, `native-runner-checkpoint-retirement.test.ts` and `native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7 skipped. - `tsc --noEmit` for `server` passes. - `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies the broad coverage. - Not run: a release-swap wave. The fix is not yet measured on a live overlap. ## Risks - The fence is wider. It accepts only a final save with an accepted result and no new work, and only while the current run is parked on this exact owner and has not started. A follow-up that has a claim, a process, process-start events or checkpoint rows still causes a refusal. - A follow-up that a server parked before this change has no `waitingForOwner`. The fence refuses its owner's save as before. At its next recheck, the follow-up parks again with the owner recorded. - While a resumption claim runs the follow-up's recheck (a short window every 5 s), the save is refused. The close retry from #21782 then tries again. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>