mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master8e3b64a257), t05 (claude M1 parent): 1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each time it continued, and the third continuation `turn_lab_92b03dc7…` was active. 2. At 00:48:29 the next restart blocked it: `native_restart_recovery_blocked` / `provider_checkpoint_permanently_failed`, with replay blocked. 3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from the lost turn), `activeTurnId` set to the continuation, and a last terminal turn fingerprinted `provider_turn_lost_on_restore`. Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass with the same shape, and its continuation had already proposed a semantic result. Both stayed blocked. The parent's Retry was held on the blocked child. **Expected behavior** A failed run terminal that records a restart loss, followed by an active continuation turn, is not a permanent failure. Recovery claims `resume_dead_runner`. The runtime then reconciles the active turn, and a held result settles a lost turn as completed. A real provider failure followed by a later turn still blocks. **Steps to reproduce** 1. Checkpoint a run whose last terminal turn is a restart loss, whose `terminal` is failed, and whose `activeTurnId` is a later continuation (optionally with a semantic result). 2. Claim restart recovery. Before this change it blocks with `provider_checkpoint_permanently_failed`. **Paperclip version or commit**8e3b64a257(wave). Branch rebased on current master. ## What Changed - `contracts/native-session-backend.ts` (runner): the new `nativeRestartContinuationInFlight` is true when all of these hold: - the checkpoint has an active turn; - its last terminal turn is a different turn; - that turn's fingerprint is a failed, result-less, recoverable `provider_turn_lost_on_restore`. - `native-session-runtime.ts`: recovery clears that stale failed terminal for a continuation in flight, as it already does for a checkpointed interruption. Before, it threw `provider_checkpoint_failed_terminal` before it reconciled the active turn. - `native-restart-recovery.ts` (server): `checkpointFailed` now also requires that no restart continuation be in flight. It uses the same contract function through the vendor boundary. - `native-session-runtime.test.ts`: an end-to-end recovery test. A checkpoint with the stale failed terminal and an active continuation resolves on the continuation's result without starting a new turn. - `native-runner-restart-recovery.integration.test.ts`: - a codex case of the stale-terminal shape as saved in production; - an ACPX case with a held result; - a held case where an active turn follows a real provider failure (`model_not_found`). ## Verification - Red/green: on master, the stale-terminal case is blocked with `provider_checkpoint_permanently_failed`. With the change it claims `resume_dead_runner`. The real-failure case stays blocked on both. - Red/green (runtime): without the runtime change, the end-to-end test throws `provider_checkpoint_failed_terminal`. With it, the run completes. - `vitest run` over the restart-recovery and failed-workspace-recovery test files in `server/src/services/native-runtime`: 5 files, 305 tests passed (14 skipped). - `vitest run` over the runner restart, recovery, continuation and terminal test files: 9 files, 385 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm run typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The exception applies only when the recorded failure is itself a recoverable restart loss and a later turn is in flight. Every other recovery gate is unchanged: the provider-attempt budget, controller takeover and process proofs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>