fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)

## Thinking Path

- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.

## Linked Issues or Issue Description

Refs #18879

**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.

**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.

**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.

## Verification

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at 070365404e, still failed for 3 Claude
runs. The cause was the install path: the saved sidecar path pointed
into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun
has been requested.

## Risks

- **The restored conversation runs on the new provider pack.** That is
already the case for re-bound suspended sessions. Session start still
verifies every artifact and the restored provider identity against the
qualified command digest. A changed upstream release still fails strict.
- **Old provider process.** For an active turn, lost-turn recovery waits
for the provider-lifetime fence before restoring. For an open session,
behavior matches the same-build restart, which reopens without the
fence.
- **A state that really is incompatible** (an upstream provider release
change):
- **Between turns**, the common upgrade case: the next run's checkpoint
carries `allow_replacement_after_resume_failure`, so the refusal opens a
fresh provider session automatically, with no Retry. A new test,
`continues in a fresh session when a new run's saved session is
incompatible…`, pins this in `native-session-runtime.test.ts`.
- **Mid-turn, same run:** this still stops for a board Retry, by design.
The effects of the lost turn are unknown, and the automatic replacement
sweep (`decideNativeReplacement`) only replaces a session when it has a
complete effect inventory.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 16:34:23 -06:00
1 parent 15b987d9b8
commit 412d49e48d
4 files changed
+439 -10

No files matched your search

@@ -110,10 +110,23 @@ digest, so restore cannot tell which part changed.
Restore re-binds a saved session to the current launch profile, and keeps the
conversation, only when all of these are true:
- The session is suspended. It has no active turn, pending runtime request,
undelivered provider event, or unconfirmed provider exit.
- The session is in one of these two states:
- **Suspended.** It has no active turn, pending runtime request, undelivered
provider event, or unconfirmed provider exit.
- **An interrupted turn or open session of the same heartbeat run.** The
deploy restarted the runner and the same run resumes. There is no accepted
semantic result, no pending runtime request, no unconfirmed provider exit,
and no active goal. Same-build recovery then runs unchanged. It reopens an
open session. For an active turn it uses lost-turn recovery: the
conversation is restored once the original provider lifetime loses its
fence quorum, and otherwise the lost turn is closed. An active turn or open
session saved by a different run stays strict.
- The startup has no candidate admission.
- The saved descriptor names this startup's sidecar command and arguments.
- The saved descriptor names the same launch as this startup: the same
executable and sidecar file names, and identical non-path arguments. The
files may be installed under a different directory, as an `npx` versioned
cache, a new image layout or a new checkout does. The re-bind then records
this startup's runner-owned paths in the saved descriptor.
Every other binding stays exact. The saved descriptor must match this build's
qualified agent profile (ACPX version, agent server package and version, and
@@ -626,6 +626,28 @@ struct AcpxDurableState {
retired_semantic_calls: BTreeSet<String>,
}
/// The saved launch names the same executable and sidecar files as this
/// startup's qualified profile, possibly installed under a different directory
/// (an `npx` versioned cache, a new image layout or checkout). File names and
/// every non-path argument must match exactly.
fn same_launch_shape(descriptor: &AcpxProviderDescriptor, profile: &AcpxLaunchProfile) -> bool {
fn same_file_name(saved: &Path, current: &Path) -> bool {
saved.is_absolute()
&& current.is_absolute()
&& saved.file_name().is_some()
&& saved.file_name() == current.file_name()
}
same_file_name(&descriptor.sidecar_command, &profile.command)
&& descriptor.sidecar_args.len() == profile.args.len()
&& descriptor
.sidecar_args
.iter()
.zip(&profile.args)
.all(|(saved, current)| {
saved == current || same_file_name(Path::new(saved), Path::new(current))
})
}
impl AcpxDurableState {
fn new(
descriptor: AcpxProviderDescriptor,
@@ -1061,15 +1083,35 @@ impl AcpxCommandExecutor {
// still verifies every artifact and the restored provider identity.
fn rebinds_settled_session(&self, state: &AcpxDurableState) -> bool {
self.launch_profile.as_ref().is_some_and(|profile| {
profile.candidate_admission.is_none()
&& state.descriptor.sidecar_command == profile.command
&& state.descriptor.sidecar_args == profile.args
}) && state.lifecycle == "suspended"
&& state.identity.is_some()
&& state.active_turn_id.is_none()
profile.candidate_admission.is_none() && same_launch_shape(&state.descriptor, profile)
}) && state.identity.is_some()
&& !state.provider_exit_unconfirmed
&& state.pending_events.is_empty()
&& state.pending_runtime_requests.is_empty()
&& ((state.lifecycle == "suspended"
&& state.active_turn_id.is_none()
&& state.pending_events.is_empty())
|| self.rebinds_interrupted_turn(state))
}
// A deploy that restarts the runner mid-turn resumes the SAME heartbeat
// run with a saved active turn or open session. A same-build restart
// reopens an open session and recovers an active turn
// through `restore_conversation_after_lost_turn`; an upgraded build must
// take the same path instead of refusing the digest. Re-bind only when
// that recovery's own preconditions hold, so its provider-lifetime fence
// check still decides whether the conversation is restored or the turn is
// closed. A different run's active turn stays strict.
fn rebinds_interrupted_turn(&self, state: &AcpxDurableState) -> bool {
state.descriptor.run_id == self.context.run_id
&& match state.lifecycle.as_str() {
"turn_starting" | "turn_active" => state.active_turn_id.is_some(),
// A same-build restart restores an open session between turns
// (also the state a lost-turn restore leaves behind).
"session_open" => state.active_turn_id.is_none(),
_ => false,
}
&& state.semantic_result.is_none()
&& (state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection))
}
fn restore(&mut self) -> Result<(), DurableRunnerError> {
@@ -1141,6 +1183,15 @@ impl AcpxCommandExecutor {
&& self.rebinds_settled_session(&state)
{
state.launch_profile_digest = launch_profile_digest.clone();
// An upgrade may install the same launch files at a new path. The
// runner-owned qualified profile is authoritative for where they
// live; the saved descriptor only recorded the old location.
let profile = self
.launch_profile
.as_ref()
.expect("a re-bind requires the runner launch profile");
state.descriptor.sidecar_command = profile.command.clone();
state.descriptor.sidecar_args = profile.args.clone();
}
state.validate(&self.context, &launch_profile_digest)?;
self.state = Some(state);
@@ -1584,6 +1635,20 @@ impl AcpxCommandExecutor {
.state
.as_ref()
.ok_or_else(|| DurableRunnerError::invalid("ACPX provider has not been prepared"))?;
// The controller attaches with the payload it saved before an upgrade
// moved the install. A launch of the same shape names the same files;
// the runner-owned qualified profile says where they live now.
let mut descriptor = descriptor;
if let Some(profile) = self
.launch_profile
.as_ref()
.filter(|profile| profile.candidate_admission.is_none())
{
if same_launch_shape(&descriptor, profile) {
descriptor.sidecar_command = profile.command.clone();
descriptor.sidecar_args = profile.args.clone();
}
}
let mut durable_descriptor = descriptor.clone();
durable_descriptor.run_id = state.descriptor.run_id.clone();
// session/load reconnects MCP using this attachment's bindings. All
@@ -6211,6 +6276,202 @@ mod tests {
fs::remove_dir_all(directory).unwrap();
}
/// The same heartbeat run resumes after a deploy restarted its runner
/// mid-turn (K14 upgrade mid-run). The old build saved an interrupted turn
/// under its launch profile digest.
#[cfg(unix)]
fn interrupted_turn_after_runner_upgrade(
label: &str,
) -> (PathBuf, DurableRunnerConfig, PathBuf) {
let (directory, mut config, _, saved) = settled_session_after_runner_upgrade(label);
config.run_id = "run-1".to_owned();
config.turn_id = "turn-1".to_owned();
config.item_id = "item-1".to_owned();
config.validate().unwrap();
let mut state: AcpxDurableState = serde_json::from_slice(&saved).unwrap();
state.lifecycle = "turn_active".into();
state.active_turn_id = Some("turn-1".into());
let state_path = directory.join(ACPX_PROVIDER_STATE_FILE);
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
(directory, config, state_path)
}
#[cfg(unix)]
#[test]
fn interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() {
let (directory, config, state_path) =
interrupted_turn_after_runner_upgrade("upgrade-lost-turn");
let prior: AcpxDurableState =
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
let current_digest = config
.acpx_launch_profile
.as_ref()
.unwrap()
.canonical_digest()
.unwrap();
assert_ne!(prior.launch_profile_digest, current_digest);
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
recovered
.restore()
.expect("an interrupted turn saved by the previous build takes the same lost-turn recovery as a same-build restart");
let restored: AcpxDurableState =
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
assert_eq!(restored.launch_profile_digest, current_digest);
assert_eq!(restored.identity, prior.identity);
assert_eq!(
restored.descriptor.command_digest,
prior.descriptor.command_digest
);
// The fixture's provider cannot start, so recovery takes the ordinary
// close path of a lost turn instead of failing the restore.
assert!(restored.active_turn_id.is_none());
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
fn open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() {
let (directory, config, state_path) =
interrupted_turn_after_runner_upgrade("upgrade-open-session");
let mut state: AcpxDurableState =
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
// A lost-turn restore leaves the session open with undelivered events.
state.lifecycle = "session_open".into();
state.active_turn_id = None;
state
.push(NormalizedProviderEvent {
event_type: "session.resumed".into(),
priority: EventPriority::P0,
payload: json!({}),
})
.unwrap();
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
// The fixture's provider cannot start; the restore must get that far
// instead of refusing the saved state.
let error = recovered.restore().unwrap_err();
assert!(
error.to_string().contains("failed to start ACPX provider"),
"{error}"
);
assert_ne!(
error.classification(),
Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE)
);
fs::remove_dir_all(directory).unwrap();
}
/// Moves the upgraded build's launch files to a new install directory with
/// the same file names, as an `npx` versioned cache, a new image layout or a
/// new checkout does (K14 upgraded between two worktrees).
#[cfg(unix)]
fn relocate_upgraded_install(
directory: &Path,
config: &mut DurableRunnerConfig,
) -> (PathBuf, Vec<String>) {
let install = directory.join("build-2");
fs::create_dir_all(&install).unwrap();
let profile = config.acpx_launch_profile.as_mut().unwrap();
let command = install.join(profile.command.file_name().unwrap());
fs::copy(&profile.command, &command).unwrap();
let old_sidecar = PathBuf::from(&profile.args[0]);
let sidecar = install.join(old_sidecar.file_name().unwrap());
fs::copy(&old_sidecar, &sidecar).unwrap();
profile.command = command.clone();
profile.args = vec![sidecar.to_string_lossy().into_owned()];
profile.artifacts = vec![artifact(&command), artifact(&sidecar)];
let args = profile.args.clone();
config.validate().unwrap();
(command, args)
}
#[cfg(unix)]
#[test]
fn saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path() {
// Interrupted turn of the same run (the K14 failure) and a settled
// session attached by the next run.
let (directory, mut config, state_path) =
interrupted_turn_after_runner_upgrade("upgrade-relocated-turn");
let (command, args) = relocate_upgraded_install(&directory, &mut config);
let current_digest = config
.acpx_launch_profile
.as_ref()
.unwrap()
.canonical_digest()
.unwrap();
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
recovered
.restore()
.expect("a relocated install of the upgraded build restores the interrupted turn");
let restored: AcpxDurableState =
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
assert_eq!(restored.launch_profile_digest, current_digest);
assert_eq!(restored.descriptor.sidecar_command, command);
assert_eq!(restored.descriptor.sidecar_args, args);
fs::remove_dir_all(directory).unwrap();
let (directory, mut config, payload, _) =
settled_session_after_runner_upgrade("upgrade-relocated-settled");
let (command, args) = relocate_upgraded_install(&directory, &mut config);
// The controller attaches with the payload it saved, which still names
// the old install's paths.
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
recovered
.restore()
.expect("a relocated install of the upgraded build restores a settled session");
recovered
.attach_run(&payload)
.expect("the next run attaches the same conversation");
let attached: AcpxDurableState =
serde_json::from_slice(&fs::read(recovered.state_path()).unwrap()).unwrap();
assert_eq!(attached.descriptor.sidecar_command, command);
assert_eq!(attached.descriptor.sidecar_args, args);
assert_eq!(attached.lifecycle, "suspended");
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
fn interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe() {
let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [
("provider_exit_unconfirmed", |state| {
state.provider_exit_unconfirmed = true
}),
("accepted_semantic_result", |state| {
state.semantic_result = Some(json!({"reportedWorkDisposition": "done"}))
}),
("unanswered_runtime_request", |state| {
state
.pending_runtime_requests
.insert("input-1".to_owned(), pending_input_request("input-1"));
}),
("active_goal", |state| {
state.goal_projection = json!({"goal": {"status": "active"}})
}),
];
for (label, seed) in guards {
let (directory, config, state_path) =
interrupted_turn_after_runner_upgrade("upgrade-lost-turn-strict");
let mut state: AcpxDurableState =
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
seed(&mut state);
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
let retained = fs::read(&state_path).unwrap();
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
let error = recovered.restore().unwrap_err();
assert_eq!(
error.classification(),
Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE),
"{label}: {error}"
);
assert!(recovered.session.is_none(), "{label}");
assert_eq!(fs::read(&state_path).unwrap(), retained, "{label}");
fs::remove_dir_all(directory).unwrap();
}
}
#[cfg(unix)]
#[test]
fn unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class() {
@@ -448,3 +448,128 @@ fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
assert_eq!(continued.result["status"], "accepted");
restored.shutdown().unwrap();
}
#[test]
fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() {
let fixture =
Fixture(std::env::temp_dir().join(format!("acpx-upgrade-{}", uuid::Uuid::new_v4())));
fs::create_dir_all(&fixture.0).unwrap();
fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
let sidecar = fixture.0.join("sidecar");
fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
let ports = unused_fence_ports();
let args = vec![
"--mode".to_owned(),
"suspend".to_owned(),
"--profile-digest".to_owned(),
PROFILE_DIGEST.to_owned(),
"--lifetime-fence-ports".to_owned(),
format!("{},{},{}", ports[0], ports[1], ports[2]),
];
let config = DurableRunnerConfig {
connect_url: "ws://127.0.0.1/runner".to_owned(),
ca_bundle_path: None,
state_dir: fixture.0.clone(),
runner_instance_id: "runner-1".to_owned(),
environment_lease_id: "lease-1".to_owned(),
run_id: "run-1".to_owned(),
normalized_session_id: "session-1".to_owned(),
turn_id: "turn-1".to_owned(),
item_id: "item-1".to_owned(),
runner_version: "0.0.0".to_owned(),
runner_digest: format!("sha256:{}", "a".repeat(64)),
acpx_launch_profile: Some(AcpxLaunchProfile {
candidate_admission: None,
authority_digest: format!("sha256:{}", "d".repeat(64)),
command: sidecar.clone(),
args: args.clone(),
artifacts: vec![QualifiedLaunchArtifact {
path: sidecar.clone(),
sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
}],
}),
opencode_launch_profile: None,
cursor_cloud_launch_profile: None,
max_outbox_bytes: 1024 * 1024,
p0_reserve_bytes: 64 * 1024,
max_frame_bytes: 1024 * 1024,
reconnect_delay: Duration::from_millis(1),
reconnect_grace: None,
max_runtime: Duration::from_secs(30),
};
let descriptor = json!({
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
"providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol",
"acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp",
"agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
"agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
"sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
"normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
"instructions": "Complete the supplied work.",
"permissionMode": "approve-all", "permissionModePinned": true,
});
let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config);
original
.execute(&command(1, "run.prepare", json!({"provider": descriptor})))
.unwrap();
let opened = original
.execute(&command(2, "session.open", json!({})))
.unwrap();
let provider_session = opened.result["providerSessionId"].clone();
let started = original
.execute(&command(
3,
"turn.start",
json!({"turnId": "provider-turn-1", "text": "Do the work."}),
))
.unwrap();
assert_eq!(started.result["status"], "accepted");
drop(original);
let mut upgraded = config.clone();
upgraded
.acpx_launch_profile
.as_mut()
.unwrap()
.authority_digest = format!("sha256:{}", "e".repeat(64));
let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &upgraded);
let snapshot = restored
.execute(&command(4, "session.snapshot", json!({})))
.expect("the new build restores the conversation of the turn the upgrade lost");
assert_ne!(snapshot.result["status"], "closed");
assert_eq!(snapshot.result["providerSessionId"], provider_session);
let events = restored.poll_events().unwrap();
let failed = events
.iter()
.find(|event| event.event_type == "turn.failed")
.expect("the lost turn is terminated");
assert_eq!(failed.payload["providerTurnId"], "provider-turn-1");
assert_eq!(
failed.payload["error"]["code"],
"provider_turn_lost_on_restore"
);
assert_eq!(failed.payload["error"]["recoverable"], true);
assert!(!events
.iter()
.any(|event| event.event_type == "run.terminal"));
let continued = restored
.execute(&command(
5,
"turn.start",
json!({"turnId": "provider-turn-2", "text": "Continue the work."}),
))
.unwrap();
assert_eq!(continued.result["status"], "accepted");
let saved: Value =
serde_json::from_slice(&fs::read(fixture.0.join("acpx-provider-state.json")).unwrap())
.unwrap();
let upgraded_digest = upgraded
.acpx_launch_profile
.as_ref()
.unwrap()
.canonical_digest()
.unwrap();
assert_eq!(saved["launchProfileDigest"], upgraded_digest);
restored.shutdown().unwrap();
}
@@ -6571,6 +6571,36 @@ describe("executeNativeSession recovery", () => {
expect(openReplacementSession).toHaveBeenCalledOnce();
});
it("continues in a fresh session when a new run's saved session is incompatible and the checkpoint allows replacement", async () => {
// A provider release changed between turns (for example an agent-server
// bump). The next run's checkpoint allows replacement, so the user's task
// continues automatically in a fresh provider session, with no Retry.
const checkpoint: PersistedNativeSession = {
backendKind: "mock", sessionId: "driver-upgraded", identity,
providerSessionId: "provider-upgraded", providerRecoveryPolicy: "allow_replacement_after_resume_failure",
cursor: "0", activeTurnId: null, pendingRuntimeRequests: [], lineage: [],
};
const replacementAttempted = new Error("replacement attempted");
const openReplacementSession = vi.fn(async () => { throw replacementAttempted; });
const backend: NativeSessionBackend = {
descriptor: async () => ({ kind: "mock", name: "upgraded", version: "1",
capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true } }),
openSession: vi.fn(), openReplacementSession,
recoverSession: async () => ({ recovered: false, failureClass: PROVIDER_STATE_INCOMPATIBLE,
reason: "ACPX durable launch profile digest does not match runner startup" }),
};
const port: ControlPlanePort = {
openRun: vi.fn(async () => undefined), appendEvent: vi.fn(), completeRun: vi.fn(),
loadSessionCheckpoint: async () => structuredClone(checkpoint),
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }),
};
await expect(executeNativeSession({
input, backend, controlPlane: port,
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
})).rejects.toBe(replacementAttempted);
expect(openReplacementSession).toHaveBeenCalledOnce();
});
it("bounds paginated recovery replay with one signal and observes a late rejection", async () => {
vi.useFakeTimers();
let rejectStalledReplay = (_error: Error) => {};