mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
**HOLD: do not merge until the coordinator confirms (merging resets the zero-downtime gates).** ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - When that turn finishes during the overlap, the outgoing server saves its idle session and closes it. A checkpoint fence decides whether the save is still allowed. > - The task's follow-up run usually starts on the incoming server at the same moment. It waits for the outgoing owner (#21774), and #21791 lets the outgoing server save while the follow-up is parked on it. > - The fence still refused the save (`native_runner_checkpoint_owner_lost`) in every other state of the same unstarted follow-up. In release-swap waves the close then fails at least once and retries, and the follow-up waits longer. If every attempt hits such a window, the owner stays until the outgoing server stops, up to the 510 s overlap. Speed is a requirement. > - This pull request makes the fence accept the outgoing server's final save in every state in which the follow-up has not started: no process, no own process start, no own checkpoint, `attempt = 0`, same session. It still refuses when the follow-up names another owner or has started. > - The benefit is that the outgoing server closes its session on the first attempt, and the follow-up continues at its next owner check. ## Linked Issues or Issue Description Refs #21791, #21774, #21782 **What happened?** Release-swap waves logged `idle warm native session close failed … native_runner_checkpoint_owner_lost`. The close worked on a later attempt. Two recent waves, traced with the wave's 2-second run samples: | Wave | Outgoing run | Attempt | Follow-up at that moment | Old fence | | --- | --- | --- | --- | --- | | 13:48Z | `480c86e4` (finished 14:29:04.7) | 1, 14:29:07.7 | `5396e5ab`, started 14:29:05.6 on the incoming server: `preparing` (legacy, incoming boot) at 06.2, native `observed` with no owner at 08.2 | refused: a preparing follow-up was accepted only on the outgoing server's own boot, and a native follow-up only when parked or re-claimed with `waitingForOwner` | | 15:58Z | `87de6f54` (finished 16:16:14.1) | 1, 16:16:19 | `2009f6bb`, started 16:16:18.7: `preparing` (legacy, incoming boot) at 19.8 | refused: foreign preparation | | 15:58Z | `87de6f54` | 2, 16:16:27 | `2009f6bb`: native `observed`, no owner from 21.8 to 25.8, owner stamped at 27.8 | refused: selected follow-up before its first owner check, or a park or re-claim without `waitingForOwner` | In each case the follow-up was the task's current run and had no process, no process-start event and no checkpoint rows. No successor claim or stale epoch held the session: the outgoing server's own run had committed and released its lease. The fence refused only because it did not recognize these states of the same unstarted follow-up. **Expected behavior** The outgoing server saves and closes the session on the first attempt while the follow-up is in any unstarted state. The follow-up then sees the saved session at its owner check and continues without parking (or within one recheck if it already parked). **Steps to reproduce** 1. Run a warm native session on a remote sandbox, and let a run finish on server A during a swap to server B. 2. Let B claim the follow-up right away. 3. While B prepares it, selects native execution, parks it, or re-claims it, A saves and closes its idle session. The fence refuses the save. **Paperclip version or commit** master `f61b919701` (merged in). The waves ran on master builds without this pull request. ## What Changed - `native-runner-checkpoint.ts`: the parked-follow-up check is now `unstartedSuccessorOfOwner`. It takes `FOR SHARE` on the follow-up's finalization row (if any), then on its run row, in that order, inside a savepoint. The claim and defer paths use the same order. If native selection created the finalization row while the fence waited for the run lock, the fence rolls the savepoint back, which releases the run lock, and takes both locks again in the same order (at most twice). A last statement then reads the current state again and accepts one of these states: - **Preparing on any server** (new): the run is still `legacy`, `preparing`, on this task, has a live controller lease and no process, and has no finalization row. Before, only the outgoing server's own boot was accepted. - **Selected, before its first owner check** (new): native, finalization `observed`, `attempt = 0`, no lease, no failure, empty recovery history. - **Parked**: as before, but `waitingForOwner` may be absent. - **Re-claimed**: as before (live `same_run_resume` claim by the current controller), but `waitingForOwner` may be absent, as in claims from a previous release. - These conditions apply to every state and are unchanged: running; no own process-start event; no own checkpoint rows. For the native states, also: same company, task and native session; `attempt = 0`; no result; no process, unless it is the identity inherited from the outgoing run (`inheritedProcessIdentityRunId`). The fence still requires an accepted result and no new work. - A `waitingForOwner` that names another run or boot is still refused. - `SAME_RUN_RESUME_DISPOSITION` moves to the leaf module `same-run-resume-disposition.ts`. The fence no longer imports `scheduled-resume-continuation.ts`, so the runner E2E server does not load the executor and `@paperclipai/plugin-sdk` through it (`server-startup-import-graph.test.ts`). - The run-row share lock makes native selection (`runtime_selection` locks the run `FOR UPDATE`) and process start wait for the save to commit. A save that waited reads the committed state in its last statement and is refused if the follow-up has started. - Tests, real Postgres: - `native-controller-recovery.integration.test.ts`, new: - The outgoing server saves and closes while the incoming server prepares the follow-up. - Native selection of that follow-up waits for the save. - A save that waited while native selection held the run lock and created the coordinator is accepted (Greptile review). - It saves and closes while the follow-up is selected but before its owner check, also with an inherited process identity, and the follow-up's check resolves at once. - It saves and closes while the follow-up is parked or re-claimed without `waitingForOwner`, including a claim entry in the previous release's shape. - Still refused, preparing: expired controller lease, `dispatching`, own process, own process-start event, other task. - Still refused, selected: own process, a process inherited from another run, own process-start event, own checkpoint, other session, admitted (`attempt = 1` with a lease). - The outgoing-server fixtures share one helper. - `native-harness-backup-storage.integration.test.ts`: the harness backup fence shares this check. A released owner whose follow-up is still preparing on another server now replaces its selected backup. `expired_preparation`, `dispatching_successor` and `foreign_native` still keep the old backup. - `native-runner-restart-recovery.integration.test.ts`: the `foreign_preparation` early-retirement case now expects the save to succeed. `expired_preparation`, `dispatching_successor`, `wrong_issue`, `unclaimed_successor` (another session) and `foreign_native_preparation` still expect a refusal. - From the earlier commits of this pull request: the re-claimed state, `sameRunResumeClaimEntry` recording `waitingForOwner`, the run-row lock and the post-lock recheck. ## Verification - Red/green for this round (only the fence source changed between runs): - With the fence at `ed7227eeda` (this pull request before this round), 9 tests fail with `native_runner_checkpoint_owner_lost`: preparing close (+ `foreign_preparation` ×2), native-selection ordering, selected-before-check ×2, waiting-without-owner ×3. The 11 new refusal cases pass both before and after. - With the fence change: `native-controller-recovery.integration.test.ts` and `native-runner-restart-recovery.integration.test.ts` 306 passed, 14 skipped. `native-harness-backup-storage.integration.test.ts` 73 passed. `scheduled-resume-continuation.test.ts`, `native-runner-checkpoint-save.test.ts` and `native-controller-recovery.integration.test.ts` 110 passed. - Savepoint restart: the new waited-on-native-selection test fails with `native_runner_checkpoint_owner_lost` without it, and passes with it. - `tests/runner-e2e/server-startup-import-graph.test.ts` failed in CI at `cc513d9434` (import path through `scheduled-resume-continuation.ts`). It passes locally after the move. - Earlier rounds (unchanged): red on master for the re-claimed close, the process-start ordering and the claim ordering; red without the run-row lock; red without the recheck. - `tsc --noEmit` for `server` passes. - `pnpm verify` (base `f61b919701`) selected 2842 of 3683 files, because the checkpoint module has a wide import graph. The reusable job passed: 2 files, 12 tests. The fresh job was cut off locally (exit 144, no test failure printed). Required CI supplies that coverage. - `node scripts/agent-preflight.mjs` stops at `pnpm install --frozen-lockfile`: on master, `server/package.json` lists `posthog-node` (#22006) and the lockfile does not have it yet. CI resolves the lockfile inline. Locally, dependencies were installed without the frozen lockfile, and `pnpm-lock.yaml` was not committed. - Not run: a release-swap wave. The coordinator runs the gates. ## Risks - The fence is wider. It now accepts the outgoing server's final save while the follow-up is in any state before it starts. The save needs an accepted result and no new work. The follow-up must have no process (except the identity inherited from this run), no process-start event, no checkpoint of its own, and `attempt = 0`. A follow-up that names another owner, holds an expired claim, has been admitted, or is in another session is still refused. - A follow-up that another server is preparing is now accepted. It has not touched the session: it runs the owner check after native selection, and native selection waits for the save to commit. The earlier same-boot-only rule was stricter than needed. - The fence takes share locks on the follow-up's rows whenever it reaches this check, also when it then refuses. The save transaction is short. The lock order is the finalization row, then the run row, as in the claim and defer paths. - Previous-release state: a claim or park written by a release without `waitingForOwner` is now accepted, not refused. No stored shape changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>