Files
PaperClipAI/server
DottaandClaude Opus 5.5 b1049b174e fix(native): retire a finished remote owner whose suspend never finished once its sandbox leases ended (#20211)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a per-turn follow-up starts a new runner, it retires the
previous turn's finished owner. Retirement requires the owner's
controller journal to be settled
> - A local runner that is proven stopped can no longer act on its own
lifecycle commands, so an unfinished `runner.suspend` from it does not
hold the follow-up
> - On a remote per-turn sandbox the same thing happens when a restart
or drain lands while the runner is suspending. But the remote path had
no equivalent rule, so the unfinished `runner.suspend` held the
follow-up even after the previous run's sandbox lease had ended
> - This pull request treats a remote owner's lifecycle commands as
settled once every sandbox lease of that run has been released or
expired, and none of them is the sandbox the follow-up now uses
> - The benefit is that a C1 follow-up after a restart on Daytona
continues on its own

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503
(revision 9046884811, image 01246d43): 60 of 65 passed. Three C1
follow-ups failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(runner.suspend.pending)`:
- t04 codex, run `c7446d0c`;
- t16 claude, run `4b2c770f`;
- t39 claude, run `3ef7d934`.

For t04, the previous owner `fc585314` committed its `run.terminal`, and
its journal kept only `runner.suspend` pending. Its per-turn lease
(`5576f27d…`) expired at 07:43:52, and the follow-up ran on a new
sandbox. Each Retry succeeded.

**Expected behavior**
Once the previous run's sandbox leases have all ended, and the follow-up
runs on another sandbox, the previous runner cannot act on its suspend.
The finished owner is retired and the follow-up continues.

The owner stays held while any of these holds:
- a lease is active, retained or pending cleanup;
- the follow-up runs on the same sandbox;
- no lease is recorded.

**Steps to reproduce**
1. Finish a remote per-turn run whose controller journal keeps
`runner.suspend` pending after its committed run terminal, and let its
sandbox lease end.
2. Start a follow-up on a new sandbox. Before this change, retirement
refuses with `journal_commands_unsettled(runner.suspend.pending)`.

**Paperclip version or commit**
9046884811 (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`:
- the new `priorRemoteSandboxLeasesEnded` checks the previous run's
`environment_leases`: at least one lease, every one `released` or
`expired` with `releasedAt`, and none on the current target's provider
lease;
- `inspectFinishedPerTurnRunnerdOwner` passes the result as
`remoteRunnerGone` to `readSettledRunnerdProjection`, for the live root
and for quarantined copies. That function then ignores unfinished runner
lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`), exactly as it already does for a stopped local
runner.
- `native-recovered-owner-followup.test.ts`: a red/green test with lease
variants. A lease that ended or was released on another sandbox is
admitted. A retained lease, an ended lease on the same sandbox, or no
lease record stays held.

## Verification

- Red/green: on master, the two admitted cases fail with exactly the
production error; with the change they are admitted. The three held
cases hold on both.
- `vitest run` of the recovered-owner, remote-warm and restart-recovery
test files and `native-session-executor.test.ts`: 7 files, 1634 tests
passed (14 skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk. Only runner lifecycle commands become settled, and only when
the database proves every sandbox of the previous run has ended and the
follow-up runs elsewhere. Unfinished turn, tool or result commands, a
missing run terminal and unanswered tool input still hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:40:13 -06:00
..