From 412d49e48d6646ec2a3f17495f96d3c726621dc3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 16:34:23 -0600 Subject: [PATCH] fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - The rule for upgrade-safe saved state: work saved by the old build must resume under the new build. - The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke that rule for Claude. The same heartbeat run resumed after the upgrade restart, and the runner's resume probe (`runner.drain`) failed with `ACPX durable launch profile digest does not match runner startup` (`provider_state_incompatible`). t03 and t04's Claude subtask both needed a board Retry. Codex crossed the upgrade cleanly. - The ACPX launch profile digest covers the provider-pack authority and the sidecar artifact hashes, so every deploy changes it. - #18879 already re-binds a suspended session to the new digest. A deploy that restarts the runner mid-turn leaves the same run with an active turn, or with an open session, which is the state a same-build lost-turn restore leaves behind. Neither was re-bound. - A same-build restart already recovers both states safely: it reopens an open session, and recovers an active turn through `restore_conversation_after_lost_turn`, which is guarded by the provider-lifetime fence. An upgraded build should take the same path. ## Linked Issues or Issue Description Refs #18879 **What happened?** A deploy restarted the runner while a Claude (ACPX) task was mid-run. When the same run resumed under the new build, restore refused the saved state with `provider_state_incompatible`. The task stopped and needed a manual Retry, which started a fresh provider session and dropped the conversation. **Steps to reproduce** 1. Start a Claude task on build A and let it reach an active turn. 2. Upgrade to build B, which ships a new provider pack or sidecar build, with a graceful restart while the turn runs. 3. The same run's resume fails with `ACPX durable launch profile digest does not match runner startup`. **Expected behavior** The resumed run recovers exactly as it would after a same-build restart. The conversation is restored, or the lost turn is closed through the normal recovery path, with no Retry. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`: - `rebinds_settled_session` now also re-binds the saved digest for an **interrupted turn or open session of the same heartbeat run** (`rebinds_interrupted_turn`). - The re-bind requires exactly the same-build recovery preconditions: no accepted semantic result, no pending runtime request, no unconfirmed provider exit, and a dormant goal. It also keeps the existing requirements: no candidate admission, and the same sidecar command and arguments. - Recovery then runs unchanged. The provider-lifetime fence still decides whether the conversation is restored or the lost turn is closed. - **Relocated installs:** an upgrade can install the same launch files under a new directory, as an `npx` versioned cache, a new image layout or a new checkout does. The K14 harness moves between worktrees. The re-bind now accepts a launch of the same shape: the same executable and sidecar file names, and identical non-path arguments. It then records this startup's runner-owned paths in the saved descriptor. A renamed executable or a changed argument stays strict. - An active turn or open session saved by a *different* run stays strict, as before. So do suspended sessions with undelivered events, and every qualified-release binding (ACPX, agent server and command digest). - Tests (Rust, `acpx_provider_backend`): - `interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`: red on master. An old-digest saved state with an active turn restores, the digest is re-bound, and identity and command digest are kept. - `open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`: restore gets past the saved-state check, to provider start. - `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`: red on master and on the first fix, with the production error. It covers an interrupted turn of the same run and a settled session attached by the next run, both moved to a new install directory. - `interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`: each unsafe guard still fails with `provider_state_incompatible` and leaves the saved state untouched. - The existing strict tests for other runs are unchanged and still pass. - `packages/paperclip-runner/docs/durable-recovery.md`: documents the extended re-bind. - `tests/acpx_warm_attachment.rs`: the end-to-end test `restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`, from the Phase 2 session's parallel fix. It fails on master with the production error and passes here. ## Verification - `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44 passed. - The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures are timing-based `task_memory` kill-monitor tests on macOS; that file is untouched (last changed 5 days ago). CI runs them on Linux. - `cargo fmt --check`: clean. - Live: the first K14 rerun, at 070365404e, still failed for 3 Claude runs. The cause was the install path: the saved sidecar path pointed into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun has been requested. ## Risks - **The restored conversation runs on the new provider pack.** That is already the case for re-bound suspended sessions. Session start still verifies every artifact and the restored provider identity against the qualified command digest. A changed upstream release still fails strict. - **Old provider process.** For an active turn, lost-turn recovery waits for the provider-lifetime fence before restoring. For an open session, behavior matches the same-build restart, which reopens without the fence. - **A state that really is incompatible** (an upstream provider release change): - **Between turns**, the common upgrade case: the next run's checkpoint carries `allow_replacement_after_resume_failure`, so the refusal opens a fresh provider session automatically, with no Retry. A new test, `continues in a fresh session when a new run's saved session is incompatible…`, pins this in `native-session-runtime.test.ts`. - **Mid-turn, same run:** this still stops for a board Retry, by design. The effects of the lost turn are unknown, and the automatic replacement sweep (`decideNativeReplacement`) only replaces a session when it has a complete effect inventory. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../paperclip-runner/docs/durable-recovery.md | 19 +- .../runner-core/src/acpx_provider_backend.rs | 275 +++++++++++++++++- .../runner-core/tests/acpx_warm_attachment.rs | 125 ++++++++ .../src/native-session-runtime.test.ts | 30 ++ 4 files changed, 439 insertions(+), 10 deletions(-) diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 8bd198be3d4..3f8938c272a 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -110,10 +110,23 @@ digest, so restore cannot tell which part changed. Restore re-binds a saved session to the current launch profile, and keeps the conversation, only when all of these are true: -- The session is suspended. It has no active turn, pending runtime request, - undelivered provider event, or unconfirmed provider exit. +- The session is in one of these two states: + - **Suspended.** It has no active turn, pending runtime request, undelivered + provider event, or unconfirmed provider exit. + - **An interrupted turn or open session of the same heartbeat run.** The + deploy restarted the runner and the same run resumes. There is no accepted + semantic result, no pending runtime request, no unconfirmed provider exit, + and no active goal. Same-build recovery then runs unchanged. It reopens an + open session. For an active turn it uses lost-turn recovery: the + conversation is restored once the original provider lifetime loses its + fence quorum, and otherwise the lost turn is closed. An active turn or open + session saved by a different run stays strict. - The startup has no candidate admission. -- The saved descriptor names this startup's sidecar command and arguments. +- The saved descriptor names the same launch as this startup: the same + executable and sidecar file names, and identical non-path arguments. The + files may be installed under a different directory, as an `npx` versioned + cache, a new image layout or a new checkout does. The re-bind then records + this startup's runner-owned paths in the saved descriptor. Every other binding stays exact. The saved descriptor must match this build's qualified agent profile (ACPX version, agent server package and version, and diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index a6a13bda61d..e16c4f3e7c2 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -626,6 +626,28 @@ struct AcpxDurableState { retired_semantic_calls: BTreeSet, } +/// The saved launch names the same executable and sidecar files as this +/// startup's qualified profile, possibly installed under a different directory +/// (an `npx` versioned cache, a new image layout or checkout). File names and +/// every non-path argument must match exactly. +fn same_launch_shape(descriptor: &AcpxProviderDescriptor, profile: &AcpxLaunchProfile) -> bool { + fn same_file_name(saved: &Path, current: &Path) -> bool { + saved.is_absolute() + && current.is_absolute() + && saved.file_name().is_some() + && saved.file_name() == current.file_name() + } + same_file_name(&descriptor.sidecar_command, &profile.command) + && descriptor.sidecar_args.len() == profile.args.len() + && descriptor + .sidecar_args + .iter() + .zip(&profile.args) + .all(|(saved, current)| { + saved == current || same_file_name(Path::new(saved), Path::new(current)) + }) +} + impl AcpxDurableState { fn new( descriptor: AcpxProviderDescriptor, @@ -1061,15 +1083,35 @@ impl AcpxCommandExecutor { // still verifies every artifact and the restored provider identity. fn rebinds_settled_session(&self, state: &AcpxDurableState) -> bool { self.launch_profile.as_ref().is_some_and(|profile| { - profile.candidate_admission.is_none() - && state.descriptor.sidecar_command == profile.command - && state.descriptor.sidecar_args == profile.args - }) && state.lifecycle == "suspended" - && state.identity.is_some() - && state.active_turn_id.is_none() + profile.candidate_admission.is_none() && same_launch_shape(&state.descriptor, profile) + }) && state.identity.is_some() && !state.provider_exit_unconfirmed - && state.pending_events.is_empty() && state.pending_runtime_requests.is_empty() + && ((state.lifecycle == "suspended" + && state.active_turn_id.is_none() + && state.pending_events.is_empty()) + || self.rebinds_interrupted_turn(state)) + } + + // A deploy that restarts the runner mid-turn resumes the SAME heartbeat + // run with a saved active turn or open session. A same-build restart + // reopens an open session and recovers an active turn + // through `restore_conversation_after_lost_turn`; an upgraded build must + // take the same path instead of refusing the digest. Re-bind only when + // that recovery's own preconditions hold, so its provider-lifetime fence + // check still decides whether the conversation is restored or the turn is + // closed. A different run's active turn stays strict. + fn rebinds_interrupted_turn(&self, state: &AcpxDurableState) -> bool { + state.descriptor.run_id == self.context.run_id + && match state.lifecycle.as_str() { + "turn_starting" | "turn_active" => state.active_turn_id.is_some(), + // A same-build restart restores an open session between turns + // (also the state a lost-turn restore leaves behind). + "session_open" => state.active_turn_id.is_none(), + _ => false, + } + && state.semantic_result.is_none() + && (state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection)) } fn restore(&mut self) -> Result<(), DurableRunnerError> { @@ -1141,6 +1183,15 @@ impl AcpxCommandExecutor { && self.rebinds_settled_session(&state) { state.launch_profile_digest = launch_profile_digest.clone(); + // An upgrade may install the same launch files at a new path. The + // runner-owned qualified profile is authoritative for where they + // live; the saved descriptor only recorded the old location. + let profile = self + .launch_profile + .as_ref() + .expect("a re-bind requires the runner launch profile"); + state.descriptor.sidecar_command = profile.command.clone(); + state.descriptor.sidecar_args = profile.args.clone(); } state.validate(&self.context, &launch_profile_digest)?; self.state = Some(state); @@ -1584,6 +1635,20 @@ impl AcpxCommandExecutor { .state .as_ref() .ok_or_else(|| DurableRunnerError::invalid("ACPX provider has not been prepared"))?; + // The controller attaches with the payload it saved before an upgrade + // moved the install. A launch of the same shape names the same files; + // the runner-owned qualified profile says where they live now. + let mut descriptor = descriptor; + if let Some(profile) = self + .launch_profile + .as_ref() + .filter(|profile| profile.candidate_admission.is_none()) + { + if same_launch_shape(&descriptor, profile) { + descriptor.sidecar_command = profile.command.clone(); + descriptor.sidecar_args = profile.args.clone(); + } + } let mut durable_descriptor = descriptor.clone(); durable_descriptor.run_id = state.descriptor.run_id.clone(); // session/load reconnects MCP using this attachment's bindings. All @@ -6211,6 +6276,202 @@ mod tests { fs::remove_dir_all(directory).unwrap(); } + /// The same heartbeat run resumes after a deploy restarted its runner + /// mid-turn (K14 upgrade mid-run). The old build saved an interrupted turn + /// under its launch profile digest. + #[cfg(unix)] + fn interrupted_turn_after_runner_upgrade( + label: &str, + ) -> (PathBuf, DurableRunnerConfig, PathBuf) { + let (directory, mut config, _, saved) = settled_session_after_runner_upgrade(label); + config.run_id = "run-1".to_owned(); + config.turn_id = "turn-1".to_owned(); + config.item_id = "item-1".to_owned(); + config.validate().unwrap(); + let mut state: AcpxDurableState = serde_json::from_slice(&saved).unwrap(); + state.lifecycle = "turn_active".into(); + state.active_turn_id = Some("turn-1".into()); + let state_path = directory.join(ACPX_PROVIDER_STATE_FILE); + fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap(); + (directory, config, state_path) + } + + #[cfg(unix)] + #[test] + fn interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() { + let (directory, config, state_path) = + interrupted_turn_after_runner_upgrade("upgrade-lost-turn"); + let prior: AcpxDurableState = + serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + let current_digest = config + .acpx_launch_profile + .as_ref() + .unwrap() + .canonical_digest() + .unwrap(); + assert_ne!(prior.launch_profile_digest, current_digest); + + let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config); + recovered + .restore() + .expect("an interrupted turn saved by the previous build takes the same lost-turn recovery as a same-build restart"); + let restored: AcpxDurableState = + serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + assert_eq!(restored.launch_profile_digest, current_digest); + assert_eq!(restored.identity, prior.identity); + assert_eq!( + restored.descriptor.command_digest, + prior.descriptor.command_digest + ); + // The fixture's provider cannot start, so recovery takes the ordinary + // close path of a lost turn instead of failing the restore. + assert!(restored.active_turn_id.is_none()); + fs::remove_dir_all(directory).unwrap(); + } + + #[cfg(unix)] + #[test] + fn open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() { + let (directory, config, state_path) = + interrupted_turn_after_runner_upgrade("upgrade-open-session"); + let mut state: AcpxDurableState = + serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + // A lost-turn restore leaves the session open with undelivered events. + state.lifecycle = "session_open".into(); + state.active_turn_id = None; + state + .push(NormalizedProviderEvent { + event_type: "session.resumed".into(), + priority: EventPriority::P0, + payload: json!({}), + }) + .unwrap(); + fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap(); + + let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config); + // The fixture's provider cannot start; the restore must get that far + // instead of refusing the saved state. + let error = recovered.restore().unwrap_err(); + assert!( + error.to_string().contains("failed to start ACPX provider"), + "{error}" + ); + assert_ne!( + error.classification(), + Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE) + ); + fs::remove_dir_all(directory).unwrap(); + } + + /// Moves the upgraded build's launch files to a new install directory with + /// the same file names, as an `npx` versioned cache, a new image layout or a + /// new checkout does (K14 upgraded between two worktrees). + #[cfg(unix)] + fn relocate_upgraded_install( + directory: &Path, + config: &mut DurableRunnerConfig, + ) -> (PathBuf, Vec) { + let install = directory.join("build-2"); + fs::create_dir_all(&install).unwrap(); + let profile = config.acpx_launch_profile.as_mut().unwrap(); + let command = install.join(profile.command.file_name().unwrap()); + fs::copy(&profile.command, &command).unwrap(); + let old_sidecar = PathBuf::from(&profile.args[0]); + let sidecar = install.join(old_sidecar.file_name().unwrap()); + fs::copy(&old_sidecar, &sidecar).unwrap(); + profile.command = command.clone(); + profile.args = vec![sidecar.to_string_lossy().into_owned()]; + profile.artifacts = vec![artifact(&command), artifact(&sidecar)]; + let args = profile.args.clone(); + config.validate().unwrap(); + (command, args) + } + + #[cfg(unix)] + #[test] + fn saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path() { + // Interrupted turn of the same run (the K14 failure) and a settled + // session attached by the next run. + let (directory, mut config, state_path) = + interrupted_turn_after_runner_upgrade("upgrade-relocated-turn"); + let (command, args) = relocate_upgraded_install(&directory, &mut config); + let current_digest = config + .acpx_launch_profile + .as_ref() + .unwrap() + .canonical_digest() + .unwrap(); + let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config); + recovered + .restore() + .expect("a relocated install of the upgraded build restores the interrupted turn"); + let restored: AcpxDurableState = + serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + assert_eq!(restored.launch_profile_digest, current_digest); + assert_eq!(restored.descriptor.sidecar_command, command); + assert_eq!(restored.descriptor.sidecar_args, args); + fs::remove_dir_all(directory).unwrap(); + + let (directory, mut config, payload, _) = + settled_session_after_runner_upgrade("upgrade-relocated-settled"); + let (command, args) = relocate_upgraded_install(&directory, &mut config); + // The controller attaches with the payload it saved, which still names + // the old install's paths. + let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config); + recovered + .restore() + .expect("a relocated install of the upgraded build restores a settled session"); + recovered + .attach_run(&payload) + .expect("the next run attaches the same conversation"); + let attached: AcpxDurableState = + serde_json::from_slice(&fs::read(recovered.state_path()).unwrap()).unwrap(); + assert_eq!(attached.descriptor.sidecar_command, command); + assert_eq!(attached.descriptor.sidecar_args, args); + assert_eq!(attached.lifecycle, "suspended"); + fs::remove_dir_all(directory).unwrap(); + } + + #[cfg(unix)] + #[test] + fn interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe() { + let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [ + ("provider_exit_unconfirmed", |state| { + state.provider_exit_unconfirmed = true + }), + ("accepted_semantic_result", |state| { + state.semantic_result = Some(json!({"reportedWorkDisposition": "done"})) + }), + ("unanswered_runtime_request", |state| { + state + .pending_runtime_requests + .insert("input-1".to_owned(), pending_input_request("input-1")); + }), + ("active_goal", |state| { + state.goal_projection = json!({"goal": {"status": "active"}}) + }), + ]; + for (label, seed) in guards { + let (directory, config, state_path) = + interrupted_turn_after_runner_upgrade("upgrade-lost-turn-strict"); + let mut state: AcpxDurableState = + serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + seed(&mut state); + fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap(); + let retained = fs::read(&state_path).unwrap(); + let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config); + let error = recovered.restore().unwrap_err(); + assert_eq!( + error.classification(), + Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE), + "{label}: {error}" + ); + assert!(recovered.session.is_none(), "{label}"); + assert_eq!(fs::read(&state_path).unwrap(), retained, "{label}"); + fs::remove_dir_all(directory).unwrap(); + } + } + #[cfg(unix)] #[test] fn unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class() { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs index e502f1cedab..f710d2f4bfe 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs @@ -448,3 +448,128 @@ fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() { assert_eq!(continued.result["status"], "accepted"); restored.shutdown().unwrap(); } + +#[test] +fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() { + let fixture = + Fixture(std::env::temp_dir().join(format!("acpx-upgrade-{}", uuid::Uuid::new_v4()))); + fs::create_dir_all(&fixture.0).unwrap(); + fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap(); + let sidecar = fixture.0.join("sidecar"); + fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap(); + fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap(); + let ports = unused_fence_ports(); + let args = vec![ + "--mode".to_owned(), + "suspend".to_owned(), + "--profile-digest".to_owned(), + PROFILE_DIGEST.to_owned(), + "--lifetime-fence-ports".to_owned(), + format!("{},{},{}", ports[0], ports[1], ports[2]), + ]; + let config = DurableRunnerConfig { + connect_url: "ws://127.0.0.1/runner".to_owned(), + ca_bundle_path: None, + state_dir: fixture.0.clone(), + runner_instance_id: "runner-1".to_owned(), + environment_lease_id: "lease-1".to_owned(), + run_id: "run-1".to_owned(), + normalized_session_id: "session-1".to_owned(), + turn_id: "turn-1".to_owned(), + item_id: "item-1".to_owned(), + runner_version: "0.0.0".to_owned(), + runner_digest: format!("sha256:{}", "a".repeat(64)), + acpx_launch_profile: Some(AcpxLaunchProfile { + candidate_admission: None, + authority_digest: format!("sha256:{}", "d".repeat(64)), + command: sidecar.clone(), + args: args.clone(), + artifacts: vec![QualifiedLaunchArtifact { + path: sidecar.clone(), + sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())), + }], + }), + opencode_launch_profile: None, + cursor_cloud_launch_profile: None, + max_outbox_bytes: 1024 * 1024, + p0_reserve_bytes: 64 * 1024, + max_frame_bytes: 1024 * 1024, + reconnect_delay: Duration::from_millis(1), + reconnect_grace: None, + max_runtime: Duration::from_secs(30), + }; + let descriptor = json!({ + "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime", + "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol", + "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp", + "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex", + "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST, + "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0, + "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0, + "instructions": "Complete the supplied work.", + "permissionMode": "approve-all", "permissionModePinned": true, + }); + let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + original + .execute(&command(1, "run.prepare", json!({"provider": descriptor}))) + .unwrap(); + let opened = original + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + let provider_session = opened.result["providerSessionId"].clone(); + let started = original + .execute(&command( + 3, + "turn.start", + json!({"turnId": "provider-turn-1", "text": "Do the work."}), + )) + .unwrap(); + assert_eq!(started.result["status"], "accepted"); + drop(original); + + let mut upgraded = config.clone(); + upgraded + .acpx_launch_profile + .as_mut() + .unwrap() + .authority_digest = format!("sha256:{}", "e".repeat(64)); + let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &upgraded); + let snapshot = restored + .execute(&command(4, "session.snapshot", json!({}))) + .expect("the new build restores the conversation of the turn the upgrade lost"); + assert_ne!(snapshot.result["status"], "closed"); + assert_eq!(snapshot.result["providerSessionId"], provider_session); + let events = restored.poll_events().unwrap(); + let failed = events + .iter() + .find(|event| event.event_type == "turn.failed") + .expect("the lost turn is terminated"); + assert_eq!(failed.payload["providerTurnId"], "provider-turn-1"); + assert_eq!( + failed.payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(failed.payload["error"]["recoverable"], true); + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); + let continued = restored + .execute(&command( + 5, + "turn.start", + json!({"turnId": "provider-turn-2", "text": "Continue the work."}), + )) + .unwrap(); + assert_eq!(continued.result["status"], "accepted"); + let saved: Value = + serde_json::from_slice(&fs::read(fixture.0.join("acpx-provider-state.json")).unwrap()) + .unwrap(); + let upgraded_digest = upgraded + .acpx_launch_profile + .as_ref() + .unwrap() + .canonical_digest() + .unwrap(); + assert_eq!(saved["launchProfileDigest"], upgraded_digest); + restored.shutdown().unwrap(); +} diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index 45953672e1d..37f0d4605d1 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -6571,6 +6571,36 @@ describe("executeNativeSession recovery", () => { expect(openReplacementSession).toHaveBeenCalledOnce(); }); + it("continues in a fresh session when a new run's saved session is incompatible and the checkpoint allows replacement", async () => { + // A provider release changed between turns (for example an agent-server + // bump). The next run's checkpoint allows replacement, so the user's task + // continues automatically in a fresh provider session, with no Retry. + const checkpoint: PersistedNativeSession = { + backendKind: "mock", sessionId: "driver-upgraded", identity, + providerSessionId: "provider-upgraded", providerRecoveryPolicy: "allow_replacement_after_resume_failure", + cursor: "0", activeTurnId: null, pendingRuntimeRequests: [], lineage: [], + }; + const replacementAttempted = new Error("replacement attempted"); + const openReplacementSession = vi.fn(async () => { throw replacementAttempted; }); + const backend: NativeSessionBackend = { + descriptor: async () => ({ kind: "mock", name: "upgraded", version: "1", + capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true } }), + openSession: vi.fn(), openReplacementSession, + recoverSession: async () => ({ recovered: false, failureClass: PROVIDER_STATE_INCOMPATIBLE, + reason: "ACPX durable launch profile digest does not match runner startup" }), + }; + const port: ControlPlanePort = { + openRun: vi.fn(async () => undefined), appendEvent: vi.fn(), completeRun: vi.fn(), + loadSessionCheckpoint: async () => structuredClone(checkpoint), + replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), + }; + await expect(executeNativeSession({ + input, backend, controlPlane: port, + runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", + })).rejects.toBe(replacementAttempted); + expect(openReplacementSession).toHaveBeenCalledOnce(); + }); + it("bounds paginated recovery replay with one signal and observes a late rejection", async () => { vi.useFakeTimers(); let rejectStalledReplay = (_error: Error) => {};