mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for a few minutes, then the old server gets SIGTERM and hands its native runs to the next controller. > - #21046 taught the legacy reaper to leave a run with its owner while the owner's server process is proven alive. Native runs have a similar gap. > - Runtime selection writes a native run's coordinator with no controller and no lease, and stops the run lease. The executor claims the coordinator about 20 seconds later. In that window the other server's sweep sees no owner and marks the run "ownership could not be verified". > - When the old server then hands the run off before any Runner checkpoint exists, no sweep looks at it again. Only the next server boot does, 15 to 40 minutes later. The task and its queued runs wait. > - This pull request leaves such a run with its live preparing server, and lets the serving server re-adopt a handed-off run through the ordinary restart claim. > - The benefit is that a run started during an upgrade continues within one sweep, with no false "blocked" mark and no wait for another upgrade. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, an upgrade started a new server at 12:19:57Z. The old server started a native run and selected its runtime at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership could not be verified; automatic recovery is blocked" (`observed_owner_unverified`). The old server's executor claimed the run at 12:20:51Z. The old server got SIGTERM and suspended the run at 12:20:54Z (`awaiting_runner_reattach`). After that the run showed "Confirming execution" for 29 minutes, and two queued runs waited behind it. At 12:49:45Z the next upgrade's startup recovery restarted its bootstrap, and the run succeeded at 12:58Z. **Expected behavior** - A server does not mark a run unverified, or recover it, while another live server is preparing it. - A run that a graceful shutdown hands to the next controller is re-adopted by the serving server at once, not at the next boot. **Steps to reproduce** 1. Seed a native run whose coordinator is `observed` with no controller and no lease, and whose `controllerProcess` names another server with a live controller instance lease. 2. Run the recovery sweep on this server. See that the run gets `native_execution_ownership_unverified`. 3. Claim the coordinator for the other server, then write what its graceful shutdown writes (`restart_suspended`, `awaiting_runner_reattach`, lapsed lease) and stop its instance lease. Give the run a predecessor's Runner PID. 4. Run the sweep again. See that nothing claims the run. **Root cause** - `recoverActiveExecutions` (`server/src/services/heartbeat/recovery.ts`) treats an `observed` coordinator without a live controller lease as an owner it cannot verify. Between runtime selection and the claim, the only owner on record is the run's `controllerProcess`, and the sweep did not check it. - `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`) selects runs for the restart claim only when they have a controller checkpoint, or no process identity at all. A run handed off just after its claim has no checkpoint, and a follow-up carries its predecessor's Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so only a server boot looked at them again. **Paperclip version or commit** Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server). The behavior is the same on master `849f109f62`. ## What Changed - `recovery.ts`: before it lists expired native runs, the sweep checks runs in the runtime-selection window (coordinator `observed`, no controller, no lease, not tracked here). When the run's `controllerProcess` names another server process that the shared owner prover proves alive, the sweep leaves the run with that server: it does not recover it and does not mark it unverified. It re-checks within 30 seconds. After 5 minutes without a claim (`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the existing rules apply. - `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that proof. - `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities` also selects a run that a graceful shutdown handed off and whose lease lapsed: recovery state `awaiting_runner_reattach` or `awaiting_evidence`, with a `restart_suspended` history entry that no restart claim followed. It stays selected through ownership waits (the departing server can still be alive at the first look). `recovery.ts` schedules a 30-second re-check for an ownership wait that has no retry of its own. The ordinary restart claim then proves the takeover and picks the disposition, exactly as the startup pass of a new server does. A row that an older release already marked unverified is re-adopted too: the claim clears the mark. - `doc/execution-semantics.md`: document both rules. - No schema change. Nothing changes on the path that starts a run or a turn: the new proof runs only in the recovery sweep, and only for runs in the window. ## Verification - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, block "a run the old server started during an upgrade overlap". Red on master, green on this branch: - "leaves the run with its live preparing server, then re-adopts it after that server hands it off": two concurrent sweeps leave the run `running` with no error and do not touch the coordinator. After the old server claims, hands the run off and stops, one sweep of the new server re-adopts it and reaches provider dispatch. Master: the run is marked `native_execution_ownership_unverified`, and after the handoff nothing claims it. - "keeps a handed-off run eligible while the departing server is still alive, and re-adopts it once it exits": the first sweep waits (`awaiting_evidence`), the sweep after the departing process exits re-adopts the run. Red with a filter that matches only the last history entry. - "re-adopts a handed-off run that an older release already marked ownership-unverified": the stuck staging shape. Master: nothing claims it. - Related suites pass: `heartbeat-native-restart-suspension` (all 109), `services/heartbeat/recovery.test.ts` (all 103). - `tsc --noEmit` for `server`. ## Risks - A run that another live server prepares, but never claims, now stays `running` for up to 5 minutes before the existing rules apply. That is the same bound #21046 uses. - The guard trusts the shared owner prover. It reads "alive" only from a live controller instance lease or a live PID in the same PID space. - More handed-off runs reach the restart claim earlier. The claim keeps all its fencing and evidence checks; this change only makes the serving server ask the same question a new boot asks. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>