mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude
ed5e7493bb
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The owner renews it every 5 minutes. After a host sleep (or the restart-chaos time fault TM2), the stored lease looks lapsed although the owner is alive. > - #20474 lets the same owner take back a lapsed lease when no other actor recorded anything on the row. One of the checks is "`recovery_state` is null". > - Restart recovery writes `recovery_state` (`resuming_session`, `awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update that grants the lease. The executor keeps that value until a result is accepted. > - So a run that a restart resumed can never take back its own lapsed lease. Renewal interrupts the turn, and failure settlement is refused with `native_session_lease_lost`. The run fails with `adapter_failed`, recovery holds it with `preserve_without_replay_v1`, and the user must click Retry. > - This pull request counts the claim's own record as untouched when the row still shows exactly that claim and nothing recorded since. > - The benefit is that a resumed run survives a later host sleep with no Retry. Every real takeover still fences the late owner out. ## Linked Issues or Issue Description Refs #20474 **What happened?** The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2, TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were Codex runs. In each run, a TM3 kill made restart recovery resume the run (`resume_dead_runner`, generation 2). The runner restored the conversation, reported `provider_turn_lost_on_restore`, and the same-conversation restart continuation started as designed. Then the TM2 fault aged every database time by one hour. The execution lease lapsed while its owner was alive. The run then failed with `native_session_lease_lost` (`adapter_failed`), and recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`. One run had already proposed its result and completed its turn. The Retry click worked every time. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, continues with no Retry. This must also be true after a restart resumed that run. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` ## What Changed - `server/src/services/native-runtime/native-execution-lease.ts`: the "untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts the run's own local restart-recovery claim. All of these must be true: - the last `recovery_history` entry is that claim: same controller boot and generation as the row; - its disposition matches the current `recovery_state` (`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` → `awaiting_runner_reattach`, `bootstrap_incomplete` → `bootstrap_incomplete`); - `recovery_request_id` equals the request that claim recorded. Any later actor appends a history entry (shutdown suspension, failure settlement), moves the state (`awaiting_evidence`, `blocked`), or replaces the owner, boot or generation. Each of these still fences the late owner out. All other untouched checks are unchanged. - `native-execution-lease-reacquire.test.ts`: new cases for each claim kind, a requested claim, failure settlement under the claim, and eight fenced variants. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new rule. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts`. Without the source change, 5 new cases fail (the three claim kinds, the requested claim, and failure settlement). With it, all 37 pass. - Related suites pass: `native-execution-lease.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `heartbeat-native-orphan-sweep-lease-race.test.ts`, `native-runner-restart-lease-renewal.integration.test.ts`, `native-runner-restart-preparation.integration.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: the same L7 wave (seed 9107, local) on this branch. The server log has 0 `native_session_lease_lost` (the baseline had 3). The two tasks whose cells failed this way in the baseline (codex C1 TM1, codex C1 TM3) now pass with no Retry. Two other tasks needed Retry for different causes, which this pull request does not change: codex B5 TM2 (a `runner.drain` timeout, then a same-run retry that may not continue a turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on `turn.start` until the retry budget ran out). The L6 wave (seed 9106) also ran on this branch: 5 of 6 passed. Its one failure is `execution_recovery_budget_exhausted` and is not related to leases. ## Risks - The rule widens when an owner may take back its own lapsed lease. It applies only to local runs, before a result, with the same owner token, attempt, boot, pid, process start and generation. A claim that any other actor changed afterwards does not match. - Rows saved by earlier releases carry the same history entry shape, so a run claimed before a deploy is handled the same way. A deploy changes the boot, so the new process never takes back the old process's lease. - No schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>