mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. The outgoing server still serves while the incoming server starts new runs. > - A warm agent session stays with the server that ran its last turn. That server closes the session about 1 s after it leaves new run starts to the replacement (#21627). > - A follow-up that started within about 1 s of its predecessor's finish read the session as suspended, while the outgoing server was still closing it. The follow-up did not wait. It started a second Runner with the wrong OpenCode launch files, and the Runner refused the launch. > - The same-run retry then went to the outgoing server, which had already left run starts. The run failed, and a blocked parent showed a Retry during its scheduled resume. > - This pull request makes the follow-up wait for the live outgoing server, rebuilds the admitted launch files on the new Runner, keeps same-run retries on the replacement, and shows the scheduled resume for a blocked parent. > - The benefit is that a follow-up during a deploy overlap finishes without a failed run or a Retry banner. ## Linked Issues or Issue Description Refs #21627 **What happened?** A zero-downtime release-swap test (Gate B, wave B1 on master890dcbdd66) failed 2 of 8 cross-server hand-offs. In both, the follow-up started 1.0 s and 0.6 s after its predecessor finished on the outgoing server. The other 6 hand-offs started 34 to 217 s later and passed. - The incoming server took the resume path that skips `inspectRemoteWarmSessionRecovery`, because the local authority state read `suspended`. The live-owner wait from #21627 lives only in that inspection, so it never ran (0 waits recorded). - The new home had no harness backup, so the run had no provider artifact receipt. `prepareRemoteRunner` returned early on `legacyProviderContinuity`, and the launch profile named the provider pack folder. Runner core refused it: "OpenCode run.prepare launch does not match the runner-owned qualified profile". - The same-run retry was claimed by the outgoing server after it left run starts. Its predecessor retirement reported `not_suspended`. Both runs failed with `native_continuation_requires_reconciliation`, and recovery started fresh runs. - For a parent already blocked on its subtasks, the task showed "failed" with Retry for 31 s while the same-run resume was scheduled. **Expected behavior** The follow-up waits while the outgoing server may still hold the session. A new Runner gets the same OpenCode launch files as the session's first launch. Only the replacement claims same-run retries. A blocked parent shows the scheduled resume and no Retry. **Steps to reproduce** 1. Run the restart-chaos release-swap preset with fresh homes and a warm OpenCode staging mix. 2. Let a run finish on the outgoing server after the traffic switch, with a follow-up comment already queued. 3. See the follow-up start on the incoming server within about 1 s and fail on `run.attach` with the qualified-profile mismatch. **Paperclip version or commit** master890dcbdd66. **Deployment mode** Hosted deploy overlap (release swap) with Daytona sandboxes and S3 storage. ## What Changed - `remote-warm-session-recovery.ts`: new `waitForLiveWarmSessionOwner`. It reads the finished run's finalization owner. While that server is alive inside the warm window, the follow-up defers with the existing `native_controller_recovery_waiting_for_owner` wait. A harness backup that holds the suspended state and was completed after the turn proves that the owner finished its close, so the follow-up continues at once. - `native-session-executor.ts`: calls that wait before the `suspended` check, so every warm follow-up on another server's session passes through it. The backup check runs only when the owner is alive. - `native-session-executor.ts`: when a resume has a durable session but no receipt from a harness backup or the admission, it reads the provider artifact receipt recorded on the session's last run. The new Runner then gets the admitted launch root, as a cold continuation already does. - `heartbeat/recovery.ts` and `heartbeat.ts`: a server that yields run starts to a replacement does not claim same-run retries, from the recovery sweep or from its own resume timer. It re-checks every 3 s, so it takes the retries back when the replacement stops. - `task-interruption-presentation.ts`: a blocked task whose remote run waits for its `exact_checkpoint_resume` keeps the resume time, so it shows `retry_scheduled` with no Retry. - `doc/execution-semantics.md`: documents these rules. ## Verification Red on master, green with this change: - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "close window waits for its live lease"`: "a follow-up that starts within the outgoing server's close window waits for its live lease instead of relaunching the Runner". Master proceeds and fails with `runner_state_identity_mismatch`. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote opencode-continuation-prior-receipt"`: a resume relaunch without an adopted process uses the original OpenCode file paths. Master launches from the provider pack folder and does not retain the receipt. - `cd server && npx vitest run src/services/heartbeat/recovery.test.ts -t "leaves run starts"`: "an outgoing server does not claim a same-run retry once it leaves run starts". Master claims it. - `cd server && npx vitest run src/__tests__/task-interruption-query.test.ts -t "blocked parent whose run awaits"`: "a blocked parent whose run awaits its exact same-run resume shows retry_scheduled with no Retry". Master shows `failed` with Retry. Also new: "keeps a follow-up waiting for a live outgoing server until that server has checkpointed the session after the turn" in `native-controller-recovery.integration.test.ts` (the backup proof and the owner exit end the wait). Focused suites pass: native-session-executor, remote-warm-session-recovery, native-controller-recovery (integration), heartbeat recovery, task-interruption-query, task-interruption-presentation, native-recovered-owner-followup, heartbeat-native-predispatch-retry, run-start-handover. `npx tsc --noEmit -p server/tsconfig.json` passes. Not run here: the restart-chaos release-swap wave. The maintainer runs it after merge. ## Risks - A follow-up on a session that a live outgoing server ran can now wait until that server exits or the warm window ends. This applies only when no harness backup proves that the server finished its close after the turn. Before, such a follow-up failed and needed recovery. - The backup proof hashes the restored backup. It runs only when the previous owner is alive in another process (a deploy overlap), not on a normal turn. - A resume without a backup receipt now restores the admitted artifacts from storage, as a cold continuation does. A session whose last run recorded no receipt keeps the legacy pack-folder launch. - A yielding server leaves same-run retries to the replacement. If the replacement stops, the yield ends with its lease, and the older server claims them again within seconds. - Previous-release state: no persisted field is added. Receipts, finalization owners and `failureDetail.recoveryMode` are already saved by the previous release. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>