Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 46e38e1387 fix(native): let the outgoing server checkpoint its idle warm session while its follow-up has not started (#21887)
**HOLD: do not merge until the coordinator confirms (merging resets the
zero-downtime gates).**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - When that turn finishes during the overlap, the outgoing server
saves its idle session and closes it. A checkpoint fence decides whether
the save is still allowed.
> - The task's follow-up run usually starts on the incoming server at
the same moment. It waits for the outgoing owner (#21774), and #21791
lets the outgoing server save while the follow-up is parked on it.
> - The fence still refused the save
(`native_runner_checkpoint_owner_lost`) in every other state of the same
unstarted follow-up. In release-swap waves the close then fails at least
once and retries, and the follow-up waits longer. If every attempt hits
such a window, the owner stays until the outgoing server stops, up to
the 510 s overlap. Speed is a requirement.
> - This pull request makes the fence accept the outgoing server's final
save in every state in which the follow-up has not started: no process,
no own process start, no own checkpoint, `attempt = 0`, same session. It
still refuses when the follow-up names another owner or has started.
> - The benefit is that the outgoing server closes its session on the
first attempt, and the follow-up continues at its next owner check.

## Linked Issues or Issue Description

Refs #21791, #21774, #21782

**What happened?**

Release-swap waves logged `idle warm native session close failed …
native_runner_checkpoint_owner_lost`. The close worked on a later
attempt. Two recent waves, traced with the wave's 2-second run samples:

| Wave | Outgoing run | Attempt | Follow-up at that moment | Old fence |
| --- | --- | --- | --- | --- |
| 13:48Z | `480c86e4` (finished 14:29:04.7) | 1, 14:29:07.7 |
`5396e5ab`, started 14:29:05.6 on the incoming server: `preparing`
(legacy, incoming boot) at 06.2, native `observed` with no owner at 08.2
| refused: a preparing follow-up was accepted only on the outgoing
server's own boot, and a native follow-up only when parked or re-claimed
with `waitingForOwner` |
| 15:58Z | `87de6f54` (finished 16:16:14.1) | 1, 16:16:19 | `2009f6bb`,
started 16:16:18.7: `preparing` (legacy, incoming boot) at 19.8 |
refused: foreign preparation |
| 15:58Z | `87de6f54` | 2, 16:16:27 | `2009f6bb`: native `observed`, no
owner from 21.8 to 25.8, owner stamped at 27.8 | refused: selected
follow-up before its first owner check, or a park or re-claim without
`waitingForOwner` |

In each case the follow-up was the task's current run and had no
process, no process-start event and no checkpoint rows. No successor
claim or stale epoch held the session: the outgoing server's own run had
committed and released its lease. The fence refused only because it did
not recognize these states of the same unstarted follow-up.

**Expected behavior**

The outgoing server saves and closes the session on the first attempt
while the follow-up is in any unstarted state. The follow-up then sees
the saved session at its owner check and continues without parking (or
within one recheck if it already parked).

**Steps to reproduce**

1. Run a warm native session on a remote sandbox, and let a run finish
on server A during a swap to server B.
2. Let B claim the follow-up right away.
3. While B prepares it, selects native execution, parks it, or re-claims
it, A saves and closes its idle session. The fence refuses the save.

**Paperclip version or commit**

master `f61b919701` (merged in). The waves ran on master builds without
this pull request.

## What Changed

- `native-runner-checkpoint.ts`: the parked-follow-up check is now
`unstartedSuccessorOfOwner`. It takes `FOR SHARE` on the follow-up's
finalization row (if any), then on its run row, in that order, inside a
savepoint. The claim and defer paths use the same order. If native
selection created the finalization row while the fence waited for the
run lock, the fence rolls the savepoint back, which releases the run
lock, and takes both locks again in the same order (at most twice). A
last statement then reads the current state again and accepts one of
these states:
- **Preparing on any server** (new): the run is still `legacy`,
`preparing`, on this task, has a live controller lease and no process,
and has no finalization row. Before, only the outgoing server's own boot
was accepted.
- **Selected, before its first owner check** (new): native, finalization
`observed`, `attempt = 0`, no lease, no failure, empty recovery history.
  - **Parked**: as before, but `waitingForOwner` may be absent.
- **Re-claimed**: as before (live `same_run_resume` claim by the current
controller), but `waitingForOwner` may be absent, as in claims from a
previous release.
- These conditions apply to every state and are unchanged: running; no
own process-start event; no own checkpoint rows. For the native states,
also: same company, task and native session; `attempt = 0`; no result;
no process, unless it is the identity inherited from the outgoing run
(`inheritedProcessIdentityRunId`). The fence still requires an accepted
result and no new work.
- A `waitingForOwner` that names another run or boot is still refused.
- `SAME_RUN_RESUME_DISPOSITION` moves to the leaf module
`same-run-resume-disposition.ts`. The fence no longer imports
`scheduled-resume-continuation.ts`, so the runner E2E server does not
load the executor and `@paperclipai/plugin-sdk` through it
(`server-startup-import-graph.test.ts`).
- The run-row share lock makes native selection (`runtime_selection`
locks the run `FOR UPDATE`) and process start wait for the save to
commit. A save that waited reads the committed state in its last
statement and is refused if the follow-up has started.
- Tests, real Postgres:
  - `native-controller-recovery.integration.test.ts`, new:
- The outgoing server saves and closes while the incoming server
prepares the follow-up.
    - Native selection of that follow-up waits for the save.
- A save that waited while native selection held the run lock and
created the coordinator is accepted (Greptile review).
- It saves and closes while the follow-up is selected but before its
owner check, also with an inherited process identity, and the
follow-up's check resolves at once.
- It saves and closes while the follow-up is parked or re-claimed
without `waitingForOwner`, including a claim entry in the previous
release's shape.
- Still refused, preparing: expired controller lease, `dispatching`, own
process, own process-start event, other task.
- Still refused, selected: own process, a process inherited from another
run, own process-start event, own checkpoint, other session, admitted
(`attempt = 1` with a lease).
    - The outgoing-server fixtures share one helper.
- `native-harness-backup-storage.integration.test.ts`: the harness
backup fence shares this check. A released owner whose follow-up is
still preparing on another server now replaces its selected backup.
`expired_preparation`, `dispatching_successor` and `foreign_native`
still keep the old backup.
- `native-runner-restart-recovery.integration.test.ts`: the
`foreign_preparation` early-retirement case now expects the save to
succeed. `expired_preparation`, `dispatching_successor`, `wrong_issue`,
`unclaimed_successor` (another session) and `foreign_native_preparation`
still expect a refusal.
- From the earlier commits of this pull request: the re-claimed state,
`sameRunResumeClaimEntry` recording `waitingForOwner`, the run-row lock
and the post-lock recheck.

## Verification

- Red/green for this round (only the fence source changed between runs):
- With the fence at `ed7227eeda` (this pull request before this round),
9 tests fail with `native_runner_checkpoint_owner_lost`: preparing close
(+ `foreign_preparation` ×2), native-selection ordering,
selected-before-check ×2, waiting-without-owner ×3. The 11 new refusal
cases pass both before and after.
- With the fence change:
`native-controller-recovery.integration.test.ts` and
`native-runner-restart-recovery.integration.test.ts` 306 passed, 14
skipped. `native-harness-backup-storage.integration.test.ts` 73 passed.
`scheduled-resume-continuation.test.ts`,
`native-runner-checkpoint-save.test.ts` and
`native-controller-recovery.integration.test.ts` 110 passed.
- Savepoint restart: the new waited-on-native-selection test fails with
`native_runner_checkpoint_owner_lost` without it, and passes with it.
- `tests/runner-e2e/server-startup-import-graph.test.ts` failed in CI at
`cc513d9434` (import path through `scheduled-resume-continuation.ts`).
It passes locally after the move.
- Earlier rounds (unchanged): red on master for the re-claimed close,
the process-start ordering and the claim ordering; red without the
run-row lock; red without the recheck.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` (base `f61b919701`) selected 2842 of 3683 files, because
the checkpoint module has a wide import graph. The reusable job passed:
2 files, 12 tests. The fresh job was cut off locally (exit 144, no test
failure printed). Required CI supplies that coverage.
- `node scripts/agent-preflight.mjs` stops at `pnpm install
--frozen-lockfile`: on master, `server/package.json` lists
`posthog-node` (#22006) and the lockfile does not have it yet. CI
resolves the lockfile inline. Locally, dependencies were installed
without the frozen lockfile, and `pnpm-lock.yaml` was not committed.
- Not run: a release-swap wave. The coordinator runs the gates.

## Risks

- The fence is wider. It now accepts the outgoing server's final save
while the follow-up is in any state before it starts. The save needs an
accepted result and no new work. The follow-up must have no process
(except the identity inherited from this run), no process-start event,
no checkpoint of its own, and `attempt = 0`. A follow-up that names
another owner, holds an expired claim, has been admitted, or is in
another session is still refused.
- A follow-up that another server is preparing is now accepted. It has
not touched the session: it runs the owner check after native selection,
and native selection waits for the save to commit. The earlier
same-boot-only rule was stricter than needed.
- The fence takes share locks on the follow-up's rows whenever it
reaches this check, also when it then refuses. The save transaction is
short. The lock order is the finalization row, then the run row, as in
the claim and defer paths.
- Previous-release state: a claim or park written by a release without
`waitingForOwner` is now accepted, not refused. No stored shape changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:37:16 -06:00
..
…
…
…
…
…