Files
PaperClipAI/doc/plans
DottaandClaude ed5e7493bb fix(native): take back a lapsed lease held under the run's own restart claim (#20625)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The owner renews it every 5 minutes.
After a host sleep (or the restart-chaos time fault TM2), the stored
lease looks lapsed although the owner is alive.
> - #20474 lets the same owner take back a lapsed lease when no other
actor recorded anything on the row. One of the checks is
"`recovery_state` is null".
> - Restart recovery writes `recovery_state` (`resuming_session`,
`awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update
that grants the lease. The executor keeps that value until a result is
accepted.
> - So a run that a restart resumed can never take back its own lapsed
lease. Renewal interrupts the turn, and failure settlement is refused
with `native_session_lease_lost`. The run fails with `adapter_failed`,
recovery holds it with `preserve_without_replay_v1`, and the user must
click Retry.
> - This pull request counts the claim's own record as untouched when
the row still shows exactly that claim and nothing recorded since.
> - The benefit is that a resumed run survives a later host sleep with
no Retry. Every real takeover still fences the late owner out.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2,
TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were
Codex runs. In each run, a TM3 kill made restart recovery resume the run
(`resume_dead_runner`, generation 2). The runner restored the
conversation, reported `provider_turn_lost_on_restore`, and the
same-conversation restart continuation started as designed. Then the TM2
fault aged every database time by one hour. The execution lease lapsed
while its owner was alive. The run then failed with
`native_session_lease_lost` (`adapter_failed`), and recovery stopped
with `native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`. One run had already proposed its result
and completed its turn. The Retry click worked every time.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, continues with no Retry. This must also be true after a
restart resumed that run.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`

## What Changed

- `server/src/services/native-runtime/native-execution-lease.ts`: the
"untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts
the run's own local restart-recovery claim. All of these must be true:
- the last `recovery_history` entry is that claim: same controller boot
and generation as the row;
- its disposition matches the current `recovery_state`
(`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` →
`awaiting_runner_reattach`, `bootstrap_incomplete` →
`bootstrap_incomplete`);
  - `recovery_request_id` equals the request that claim recorded.
Any later actor appends a history entry (shutdown suspension, failure
settlement), moves the state (`awaiting_evidence`, `blocked`), or
replaces the owner, boot or generation. Each of these still fences the
late owner out. All other untouched checks are unchanged.
- `native-execution-lease-reacquire.test.ts`: new cases for each claim
kind, a requested claim, failure settlement under the claim, and eight
fenced variants.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new rule.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`.
Without the source change, 5 new cases fail (the three claim kinds, the
requested claim, and failure settlement). With it, all 37 pass.
- Related suites pass: `native-execution-lease.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`heartbeat-native-orphan-sweep-lease-race.test.ts`,
`native-runner-restart-lease-renewal.integration.test.ts`,
`native-runner-restart-preparation.integration.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: the same L7 wave (seed 9107, local) on this branch. The server
log has 0 `native_session_lease_lost` (the baseline had 3). The two
tasks whose cells failed this way in the baseline (codex C1 TM1, codex
C1 TM3) now pass with no Retry. Two other tasks needed Retry for
different causes, which this pull request does not change: codex B5 TM2
(a `runner.drain` timeout, then a same-run retry that may not continue a
turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on
`turn.start` until the retry budget ran out). The L6 wave (seed 9106)
also ran on this branch: 5 of 6 passed. Its one failure is
`execution_recovery_budget_exhausted` and is not related to leases.

## Risks

- The rule widens when an owner may take back its own lapsed lease. It
applies only to local runs, before a result, with the same owner token,
attempt, boot, pid, process start and generation. A claim that any other
actor changed afterwards does not match.
- Rows saved by earlier releases carry the same history entry shape, so
a run claimed before a deploy is handled the same way. A deploy changes
the boot, so the new process never takes back the old process's lease.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 14:31:15 -06:00
..