mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:38:22 +01:00
DottaandClaude Opus 5.5
533f135780
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a warm remote sandbox leaves its Runner `ready` in the sandbox. The next run of the task adopts that Runner live, after it proves that the previous owner finished > - When the server is killed after a run parked on a question, the warm owner lived only in the killed server's memory. The Runner in the sandbox was never suspended > - A re-wake on the replacement server resumed that sandbox and tried live adoption. A proof failed, and the new run failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry, although it did no provider work > - #20863 already sends such a run to a fresh sandbox when the predecessor's Runner cannot be retired, but the live-adoption path of a resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to it, and the failed run's inherited Runner identity blocked the "workspace unchanged" proof that the fallback needs > - This pull request connects the two: a refusal of live adoption after the predecessor is durably terminal with an accepted result goes to the #20863 fresh-sandbox path, and an inherited Runner identity no longer counts as a process of the follow-up > - The benefit is that a re-wake seconds after a crash restart continues without a Retry ## Linked Issues or Issue Description Refs #20863, #20844, #20928, #20979, #20953, #20972 **What happened?** Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`, volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh home and overlap), task t08 (W2Q): - Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm owner stayed in the server's memory. - K1 killed the server at 01:30:06. The replacement was ready at 01:30:15. - A board comment woke the task at 01:30:17. Run `8f812df6` started at 01:30:19 and resumed the same sandbox (`run.startup.reuse` `verified_reuse`). - The first live-adoption inspection passed. The second check (`revalidate`, in backend setup, before the Runner attach) refused at 01:30:21 (stack: `remote-warm-session-recovery.ts:217` from `revalidate`). The run failed `native_session_interrupted` with `INSTRUCTION_COLLECTION_UNAVAILABLE`. - The failed run kept its full save because the unchanged-files proof refused with `process_recorded`: the run carried its predecessor's Runner process identity. Retry (`6968dc0f`) worked. **Expected behavior** The predecessor finished with an accepted result before the kill, and the new run did no provider work. The task must continue without a Retry. **Steps to reproduce** 1. Finish a warm remote run that parks on a question. 2. Kill the server (SIGKILL) and start a replacement. 3. Wake the task within seconds. The new run resumes the warm sandbox. 4. Any live-adoption proof fails, and the run fails terminally with `prior_owner_terminal_state_indeterminate`. **Which proof failed** The old refusal had no reason. The wave evidence proves that it was `revalidate`, after a passed inspection, and not the SIGTERM of the next K15 fault (that server logged "shutdown signal received" only at 01:31:09). `revalidate` checks the predecessor's coordinator row, the current run's coordinator row and lease, and the task. The evidence cannot tell which one changed in the first seconds after the restart. This pull request names the failed proof (`warm_recovery_refused=<reason>`), so the next occurrence identifies it. The fix covers the class: every refusal that concerns the predecessor or the sandbox. ## What Changed - `server/src/services/native-runtime/remote-warm-session-recovery.ts`: - Each refusal names its proof from a closed list: `identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`, `execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`, `controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`, `current_lease_changed`, `current_run_not_admitted`, `current_owner_changed`. - After the predecessor's run is `succeeded` and its coordinator is `committed` with a result and no lease, a refusal about the predecessor, its journal or the sandbox throws `NativePredecessorRetirementUnprovenError` (predecessor run, sandbox). The existing #20863 path then marks the sandbox `replaceBeforeReuse` and re-queues the run in the uncharged `interrupted_preparation` lane. The fresh sandbox is used only when the workspace save is clean and the harness backup is verified (unchanged #20863 safety rules); otherwise the sandbox is resumed as before. - A refusal because the new run is no longer admitted (stopped, reassigned, paused, company inactive) or no longer owned by this controller stays an ordinary failure, so a fallback cannot become a path around Stop or reassignment. - `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose recorded Runner process identity is exactly a settled predecessor's identity of the same session (`carriesInheritedRunnerIdentity`, from #20979), with no `native.process_identity_recorded` event of its own, is no longer refused with `process_recorded`. A provider session of its own (`sessionIdAfter`) and every other check still refuse. - `doc/execution-semantics.md`: document both rules. ## Verification - Red on `origin/master` (`8818402e7a`): - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "crash restart"`: `AssertionError: expected Error: runner_state_identity_mismatch: pr… to be an instance of NativePredecessorRetirementUnprovenError`. - `npx vitest run src/services/remote-workspace-unchanged-release.test.ts -t "follow-up that carries"`: `expected false to be true` for the inherited-identity case; the own-identity and own-record cases passed (still refused). - Green with the fix: `native-controller-recovery.integration`, `remote-workspace-unchanged-release`, `native-recovered-owner-followup` and `heartbeat-native-predispatch-retry` pass (149 tests). Also run: `native-session-executor.test`, `explicit-native-continuation.test`, `task-workspace-binding.test`, `native-runner-restart-recovery.integration.test` (see the PR comment for the result). `tsc --noEmit` for the server passes. - The new integration case is the staging shape: a predecessor that parked on a question (unanswered `semantic_tool.input` before its terminal), a replacement controller, a resumed sandbox. It checks the journal refusal, a predecessor row taken by another worker between inspection and revalidation, a changed current lease, and that a cancelled current run is not sent to the fallback. - Chaos: the before/after rerun of seed 7000 is in a PR comment. ## Risks - The fallback starts the next attempt on a fresh sandbox instead of reusing the warm Runner. That costs one cold start, only after live adoption is refused. - The unchanged-files proof accepts an inherited Runner identity. The identity must match a settled predecessor of the same session exactly (PID, group and start time), and the run must have no own identity record, no execution event, no runner checkpoint, no workspace operation and no result. These are the same facts #20979 uses for restart recovery. - Error text of this refusal gains `; warm_recovery_refused=<reason>`. Callers match the `runner_state_identity_mismatch: ` prefix only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>