mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
e3bc4ca798
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native warm runs on a remote sandbox continue the saved session of the previous run. Admission first proves what happened to the previous Runner. > - A deploy restarted the staging server while warm Runners were idle. Three follow-up runs then failed before provider work, each with a different refusal, and their tasks stayed blocked. > - The refusals asked for proof that a restart had made impossible. A restart is a product failure, so recovery must settle it automatically and keep the strict checks for real uncertainty. > - This pull request accepts the proof that the restart leaves behind, in three places: the warm owner on a replaced sandbox, the suspended Runner with an undelivered suspend receipt, and the restore hold of a run that saved nothing new. > - The benefit is that these follow-up runs start, and state saved by older releases carries forward without a migration. ## Linked Issues or Issue Description Refs #20567, #20346 **What happened?** On staging, a deploy replaced the webserver process at about 18:04 UTC on 2026-11-10 (lease controller boot `348ec0bb` on one deployment, then boot `3e84c195` on a new deployment that started at 18:04:16). Three runs of one Codex agent on Daytona failed between 18:08 and 18:15: 1. A warm follow-up got a newly created sandbox. The previous warm Runner stayed `ready` in its own retained sandbox. Live adoption denied it with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` (`remote-warm-session-recovery.ts:127`), because only a resumed sandbox may be adopted. The per-turn retirement path refused warm owners (`lifecycle_not_per_turn`). 2. A warm follow-up resumed the same sandbox and directory. The previous Runner had been suspended by the shutdown, but the controller stopped before it acknowledged the suspend result. The Runner kept its terminal fence (`pendingTerminalDelivery` for `runner.suspend`). The read refused it with `native_controller_recovery_missing_evidence:runner_not_settled` (`native-session-executor.ts:7032`). 3. A continuation into a fresh workspace failed with `remote_workspace_recovery_receipt_invalid` (`remote-workspace-recovery-attachments.ts:60`). Its predecessor had restored the committed save, failed before provider work, and kept a restore hold with no recovery receipt. Its saved reference named the snapshot it restored. **Expected behavior** After a restart, the next run continues. It retires a finished owner that nobody can adopt, reads a finished suspension, and skips a source that left no unsaved files. Real uncertainty still refuses with a named reason. **Steps to reproduce** 1. Run a warm native Codex turn on a reusable Daytona sandbox and let it commit. 2. Restart the server (deploy) while the Runner is idle. 3. Start the next turn. Get a new sandbox (case 1), or resume the same sandbox after the shutdown suspended the Runner (case 2). 4. For case 3, continue a task whose previous run restored the last save and failed before provider work, into a fresh workspace. **Paperclip version or commit** Staging before `2e3cbe57` (the runs predate that deploy). The gaps remain on `master` at `9b21dcbd`. ## What Changed - `inspectRemoteWarmSessionRecovery` returns `null` when this run's sandbox is `created` or `replacement`. Such a sandbox holds no Runner to adopt. The caller then uses ordinary admission. - `inspectFinishedPerTurnRunnerdOwner` also retires a **warm** owner on a remote sandbox runner when this run's sandbox is `created` or `replacement`. It uses all the existing proofs: terminal run, committed result, released coordinator lease, stopped or fenced controller, settled journal, and quarantine copies. Local warm Runners and resumed sandboxes keep the live path. - `restoreRemoteSuspendedRunnerCheckpoint` accepts a suspended Runner whose only open fence is its own undelivered `runner.suspend` receipt. The fence must name `runner.suspend` with lifecycle `suspended`, it must be the last controller command, and the restored journal must not name that command differently. A provider cleanup, unsent events, a turn's terminal fence or a warm transition still refuse with `runner_not_settled`. A later authorized Runner process reconciles the receipt, as the Runner already does. - `attachPreservedRemoteWorkspaceSnapshots` skips a source with a receipt-less restore hold when its saved reference names the exact committed snapshot that its own succeeded `remote_workspace_restore` loaded at the same epoch. Any other saved snapshot still needs the canonical receipt. - `doc/execution-semantics.md` describes the three rules. ## Verification - Red (unfixed code), each in the exact staging shape: - `native-recovered-owner-followup.test.ts` "retires a finished warm owner left behind by a restart when this run's sandbox is created/replacement" failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. - `native-controller-recovery.integration.test.ts` "defers a remote warm continuation…" failed: live adoption rejected a created/replacement sandbox with the same error. - `native-controller-recovery.integration.test.ts` "reads a resumed sandbox's suspended Runner with an undelivered suspend receipt" failed with `native_controller_recovery_missing_evidence:runner_not_settled`. It reads through a sandbox runner that refuses paths outside the workspace base. - `workspace-recovery-preparation.test.ts` "decides a receipt-less restore hold of a source that restored_unchanged" failed with `remote_workspace_recovery_receipt_invalid`. - Green: the same tests pass. Focused files pass: `native-recovered-owner-followup.test.ts` and `native-controller-recovery.integration.test.ts` (104 tests), `workspace-recovery-preparation.test.ts` (85 tests). - Safety cases (strict checks stay strict): - A warm owner on a resumed sandbox is not retired. - A warm owner with an unsettled journal (pending command, unanswered tool input) stays held and names `retirement_refused=journal_…`. - A warm owner whose finalization lease is still held stays held. - A turn terminal fence, a provider cleanup, unsent events, a fence behind a later command, or a journal that names the command differently still fail with `runner_not_settled` and cache nothing. - A restore-hold source that saved another snapshot, whose restore did not succeed, or whose restore was at another epoch still fails with `remote_workspace_recovery_receipt_invalid`. - Neighbouring suites pass: `native-session-executor.test.ts`, `plugin-runner-release-resume.integration.test.ts`, `explicit-native-continuation.test.ts` and `native-harness-filesystem-independence-continuation.integration.test.ts` (1786 tests). The last one asserted the old warm refusal on a created sandbox. It now shows that the warm follow-up resumes the same provider session for the new run. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. ## Risks - Previous-release state: all three rules read state that earlier releases already save (lease outcome, coordinator rows, Runner state in the sandbox, journal commands, run results and workspace operations). Rows saved by older releases carry forward. No migration. - Case 1 starts a new Runner for a warm session whose old Runner may still exist, idle, in its retained sandbox. Its controller is gone and its result is committed, so it can accept no new commands. The provider session follows the existing resume-or-replace policy. - Case 2 trusts a suspend fence only in the exact shape that the Runner writes before it sends the result. Other fences still refuse. - After case 1, a warm follow-up resumes from its own warm checkpoint. That checkpoint carries no replacement policy, so a failed provider resume is still reported as `native_session_recovery_failed` and is not replaced. This is the existing warm behavior. - Case 3 skips a source only when it provably saved nothing beyond a committed snapshot. That snapshot stays in its own workspace. - The three tasks that already failed stay blocked until their next wake or a user message. This PR does not add an automatic continuation for a completed no-replay hold. A follow-up can add that. - `native-runner-restart-recovery.integration.test.ts` (runs only with a runner container) no longer probes a replacement sandbox, because that probe now retires the owner by design. - Not verified live on Daytona. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only browser access to staging). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>