mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at 070365404e, still failed for 3 Claude
runs. The cause was the install path: the saved sidecar path pointed
into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun
has been requested.
## Risks
- **The restored conversation runs on the new provider pack.** That is
already the case for re-bound suspended sessions. Session start still
verifies every artifact and the restored provider identity against the
qualified command digest. A changed upstream release still fails strict.
- **Old provider process.** For an active turn, lost-turn recovery waits
for the provider-lifetime fence before restoring. For an open session,
behavior matches the same-build restart, which reopens without the
fence.
- **A state that really is incompatible** (an upstream provider release
change):
- **Between turns**, the common upgrade case: the next run's checkpoint
carries `allow_replacement_after_resume_failure`, so the refusal opens a
fresh provider session automatically, with no Retry. A new test,
`continues in a fresh session when a new run's saved session is
incompatible…`, pins this in `native-session-runtime.test.ts`.
- **Mid-turn, same run:** this still stops for a board Retry, by design.
The effects of the lost turn are unknown, and the automatic replacement
sweep (`decideNativeReplacement`) only replaces a session when it has a
complete effect inventory.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
15b987d9b8
commit
412d49e48d
4 files changed
+439
-10
No files matched your search
@@ -110,10 +110,23 @@ digest, so restore cannot tell which part changed.
|
||||
Restore re-binds a saved session to the current launch profile, and keeps the
|
||||
conversation, only when all of these are true:
|
||||
|
||||
- The session is suspended. It has no active turn, pending runtime request,
|
||||
undelivered provider event, or unconfirmed provider exit.
|
||||
- The session is in one of these two states:
|
||||
- **Suspended.** It has no active turn, pending runtime request, undelivered
|
||||
provider event, or unconfirmed provider exit.
|
||||
- **An interrupted turn or open session of the same heartbeat run.** The
|
||||
deploy restarted the runner and the same run resumes. There is no accepted
|
||||
semantic result, no pending runtime request, no unconfirmed provider exit,
|
||||
and no active goal. Same-build recovery then runs unchanged. It reopens an
|
||||
open session. For an active turn it uses lost-turn recovery: the
|
||||
conversation is restored once the original provider lifetime loses its
|
||||
fence quorum, and otherwise the lost turn is closed. An active turn or open
|
||||
session saved by a different run stays strict.
|
||||
- The startup has no candidate admission.
|
||||
- The saved descriptor names this startup's sidecar command and arguments.
|
||||
- The saved descriptor names the same launch as this startup: the same
|
||||
executable and sidecar file names, and identical non-path arguments. The
|
||||
files may be installed under a different directory, as an `npx` versioned
|
||||
cache, a new image layout or a new checkout does. The re-bind then records
|
||||
this startup's runner-owned paths in the saved descriptor.
|
||||
|
||||
Every other binding stays exact. The saved descriptor must match this build's
|
||||
qualified agent profile (ACPX version, agent server package and version, and
|
||||
|
||||
@@ -626,6 +626,28 @@ struct AcpxDurableState {
|
||||
retired_semantic_calls: BTreeSet<String>,
|
||||
}
|
||||
|
||||
/// The saved launch names the same executable and sidecar files as this
|
||||
/// startup's qualified profile, possibly installed under a different directory
|
||||
/// (an `npx` versioned cache, a new image layout or checkout). File names and
|
||||
/// every non-path argument must match exactly.
|
||||
fn same_launch_shape(descriptor: &AcpxProviderDescriptor, profile: &AcpxLaunchProfile) -> bool {
|
||||
fn same_file_name(saved: &Path, current: &Path) -> bool {
|
||||
saved.is_absolute()
|
||||
&& current.is_absolute()
|
||||
&& saved.file_name().is_some()
|
||||
&& saved.file_name() == current.file_name()
|
||||
}
|
||||
same_file_name(&descriptor.sidecar_command, &profile.command)
|
||||
&& descriptor.sidecar_args.len() == profile.args.len()
|
||||
&& descriptor
|
||||
.sidecar_args
|
||||
.iter()
|
||||
.zip(&profile.args)
|
||||
.all(|(saved, current)| {
|
||||
saved == current || same_file_name(Path::new(saved), Path::new(current))
|
||||
})
|
||||
}
|
||||
|
||||
impl AcpxDurableState {
|
||||
fn new(
|
||||
descriptor: AcpxProviderDescriptor,
|
||||
@@ -1061,15 +1083,35 @@ impl AcpxCommandExecutor {
|
||||
// still verifies every artifact and the restored provider identity.
|
||||
fn rebinds_settled_session(&self, state: &AcpxDurableState) -> bool {
|
||||
self.launch_profile.as_ref().is_some_and(|profile| {
|
||||
profile.candidate_admission.is_none()
|
||||
&& state.descriptor.sidecar_command == profile.command
|
||||
&& state.descriptor.sidecar_args == profile.args
|
||||
}) && state.lifecycle == "suspended"
|
||||
&& state.identity.is_some()
|
||||
&& state.active_turn_id.is_none()
|
||||
profile.candidate_admission.is_none() && same_launch_shape(&state.descriptor, profile)
|
||||
}) && state.identity.is_some()
|
||||
&& !state.provider_exit_unconfirmed
|
||||
&& state.pending_events.is_empty()
|
||||
&& state.pending_runtime_requests.is_empty()
|
||||
&& ((state.lifecycle == "suspended"
|
||||
&& state.active_turn_id.is_none()
|
||||
&& state.pending_events.is_empty())
|
||||
|| self.rebinds_interrupted_turn(state))
|
||||
}
|
||||
|
||||
// A deploy that restarts the runner mid-turn resumes the SAME heartbeat
|
||||
// run with a saved active turn or open session. A same-build restart
|
||||
// reopens an open session and recovers an active turn
|
||||
// through `restore_conversation_after_lost_turn`; an upgraded build must
|
||||
// take the same path instead of refusing the digest. Re-bind only when
|
||||
// that recovery's own preconditions hold, so its provider-lifetime fence
|
||||
// check still decides whether the conversation is restored or the turn is
|
||||
// closed. A different run's active turn stays strict.
|
||||
fn rebinds_interrupted_turn(&self, state: &AcpxDurableState) -> bool {
|
||||
state.descriptor.run_id == self.context.run_id
|
||||
&& match state.lifecycle.as_str() {
|
||||
"turn_starting" | "turn_active" => state.active_turn_id.is_some(),
|
||||
// A same-build restart restores an open session between turns
|
||||
// (also the state a lost-turn restore leaves behind).
|
||||
"session_open" => state.active_turn_id.is_none(),
|
||||
_ => false,
|
||||
}
|
||||
&& state.semantic_result.is_none()
|
||||
&& (state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection))
|
||||
}
|
||||
|
||||
fn restore(&mut self) -> Result<(), DurableRunnerError> {
|
||||
@@ -1141,6 +1183,15 @@ impl AcpxCommandExecutor {
|
||||
&& self.rebinds_settled_session(&state)
|
||||
{
|
||||
state.launch_profile_digest = launch_profile_digest.clone();
|
||||
// An upgrade may install the same launch files at a new path. The
|
||||
// runner-owned qualified profile is authoritative for where they
|
||||
// live; the saved descriptor only recorded the old location.
|
||||
let profile = self
|
||||
.launch_profile
|
||||
.as_ref()
|
||||
.expect("a re-bind requires the runner launch profile");
|
||||
state.descriptor.sidecar_command = profile.command.clone();
|
||||
state.descriptor.sidecar_args = profile.args.clone();
|
||||
}
|
||||
state.validate(&self.context, &launch_profile_digest)?;
|
||||
self.state = Some(state);
|
||||
@@ -1584,6 +1635,20 @@ impl AcpxCommandExecutor {
|
||||
.state
|
||||
.as_ref()
|
||||
.ok_or_else(|| DurableRunnerError::invalid("ACPX provider has not been prepared"))?;
|
||||
// The controller attaches with the payload it saved before an upgrade
|
||||
// moved the install. A launch of the same shape names the same files;
|
||||
// the runner-owned qualified profile says where they live now.
|
||||
let mut descriptor = descriptor;
|
||||
if let Some(profile) = self
|
||||
.launch_profile
|
||||
.as_ref()
|
||||
.filter(|profile| profile.candidate_admission.is_none())
|
||||
{
|
||||
if same_launch_shape(&descriptor, profile) {
|
||||
descriptor.sidecar_command = profile.command.clone();
|
||||
descriptor.sidecar_args = profile.args.clone();
|
||||
}
|
||||
}
|
||||
let mut durable_descriptor = descriptor.clone();
|
||||
durable_descriptor.run_id = state.descriptor.run_id.clone();
|
||||
// session/load reconnects MCP using this attachment's bindings. All
|
||||
@@ -6211,6 +6276,202 @@ mod tests {
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
|
||||
/// The same heartbeat run resumes after a deploy restarted its runner
|
||||
/// mid-turn (K14 upgrade mid-run). The old build saved an interrupted turn
|
||||
/// under its launch profile digest.
|
||||
#[cfg(unix)]
|
||||
fn interrupted_turn_after_runner_upgrade(
|
||||
label: &str,
|
||||
) -> (PathBuf, DurableRunnerConfig, PathBuf) {
|
||||
let (directory, mut config, _, saved) = settled_session_after_runner_upgrade(label);
|
||||
config.run_id = "run-1".to_owned();
|
||||
config.turn_id = "turn-1".to_owned();
|
||||
config.item_id = "item-1".to_owned();
|
||||
config.validate().unwrap();
|
||||
let mut state: AcpxDurableState = serde_json::from_slice(&saved).unwrap();
|
||||
state.lifecycle = "turn_active".into();
|
||||
state.active_turn_id = Some("turn-1".into());
|
||||
let state_path = directory.join(ACPX_PROVIDER_STATE_FILE);
|
||||
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
|
||||
(directory, config, state_path)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() {
|
||||
let (directory, config, state_path) =
|
||||
interrupted_turn_after_runner_upgrade("upgrade-lost-turn");
|
||||
let prior: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
|
||||
let current_digest = config
|
||||
.acpx_launch_profile
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.canonical_digest()
|
||||
.unwrap();
|
||||
assert_ne!(prior.launch_profile_digest, current_digest);
|
||||
|
||||
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
|
||||
recovered
|
||||
.restore()
|
||||
.expect("an interrupted turn saved by the previous build takes the same lost-turn recovery as a same-build restart");
|
||||
let restored: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
|
||||
assert_eq!(restored.launch_profile_digest, current_digest);
|
||||
assert_eq!(restored.identity, prior.identity);
|
||||
assert_eq!(
|
||||
restored.descriptor.command_digest,
|
||||
prior.descriptor.command_digest
|
||||
);
|
||||
// The fixture's provider cannot start, so recovery takes the ordinary
|
||||
// close path of a lost turn instead of failing the restore.
|
||||
assert!(restored.active_turn_id.is_none());
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile() {
|
||||
let (directory, config, state_path) =
|
||||
interrupted_turn_after_runner_upgrade("upgrade-open-session");
|
||||
let mut state: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
|
||||
// A lost-turn restore leaves the session open with undelivered events.
|
||||
state.lifecycle = "session_open".into();
|
||||
state.active_turn_id = None;
|
||||
state
|
||||
.push(NormalizedProviderEvent {
|
||||
event_type: "session.resumed".into(),
|
||||
priority: EventPriority::P0,
|
||||
payload: json!({}),
|
||||
})
|
||||
.unwrap();
|
||||
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
|
||||
|
||||
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
|
||||
// The fixture's provider cannot start; the restore must get that far
|
||||
// instead of refusing the saved state.
|
||||
let error = recovered.restore().unwrap_err();
|
||||
assert!(
|
||||
error.to_string().contains("failed to start ACPX provider"),
|
||||
"{error}"
|
||||
);
|
||||
assert_ne!(
|
||||
error.classification(),
|
||||
Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE)
|
||||
);
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
|
||||
/// Moves the upgraded build's launch files to a new install directory with
|
||||
/// the same file names, as an `npx` versioned cache, a new image layout or a
|
||||
/// new checkout does (K14 upgraded between two worktrees).
|
||||
#[cfg(unix)]
|
||||
fn relocate_upgraded_install(
|
||||
directory: &Path,
|
||||
config: &mut DurableRunnerConfig,
|
||||
) -> (PathBuf, Vec<String>) {
|
||||
let install = directory.join("build-2");
|
||||
fs::create_dir_all(&install).unwrap();
|
||||
let profile = config.acpx_launch_profile.as_mut().unwrap();
|
||||
let command = install.join(profile.command.file_name().unwrap());
|
||||
fs::copy(&profile.command, &command).unwrap();
|
||||
let old_sidecar = PathBuf::from(&profile.args[0]);
|
||||
let sidecar = install.join(old_sidecar.file_name().unwrap());
|
||||
fs::copy(&old_sidecar, &sidecar).unwrap();
|
||||
profile.command = command.clone();
|
||||
profile.args = vec![sidecar.to_string_lossy().into_owned()];
|
||||
profile.artifacts = vec![artifact(&command), artifact(&sidecar)];
|
||||
let args = profile.args.clone();
|
||||
config.validate().unwrap();
|
||||
(command, args)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path() {
|
||||
// Interrupted turn of the same run (the K14 failure) and a settled
|
||||
// session attached by the next run.
|
||||
let (directory, mut config, state_path) =
|
||||
interrupted_turn_after_runner_upgrade("upgrade-relocated-turn");
|
||||
let (command, args) = relocate_upgraded_install(&directory, &mut config);
|
||||
let current_digest = config
|
||||
.acpx_launch_profile
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.canonical_digest()
|
||||
.unwrap();
|
||||
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
|
||||
recovered
|
||||
.restore()
|
||||
.expect("a relocated install of the upgraded build restores the interrupted turn");
|
||||
let restored: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
|
||||
assert_eq!(restored.launch_profile_digest, current_digest);
|
||||
assert_eq!(restored.descriptor.sidecar_command, command);
|
||||
assert_eq!(restored.descriptor.sidecar_args, args);
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
|
||||
let (directory, mut config, payload, _) =
|
||||
settled_session_after_runner_upgrade("upgrade-relocated-settled");
|
||||
let (command, args) = relocate_upgraded_install(&directory, &mut config);
|
||||
// The controller attaches with the payload it saved, which still names
|
||||
// the old install's paths.
|
||||
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
|
||||
recovered
|
||||
.restore()
|
||||
.expect("a relocated install of the upgraded build restores a settled session");
|
||||
recovered
|
||||
.attach_run(&payload)
|
||||
.expect("the next run attaches the same conversation");
|
||||
let attached: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(recovered.state_path()).unwrap()).unwrap();
|
||||
assert_eq!(attached.descriptor.sidecar_command, command);
|
||||
assert_eq!(attached.descriptor.sidecar_args, args);
|
||||
assert_eq!(attached.lifecycle, "suspended");
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe() {
|
||||
let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [
|
||||
("provider_exit_unconfirmed", |state| {
|
||||
state.provider_exit_unconfirmed = true
|
||||
}),
|
||||
("accepted_semantic_result", |state| {
|
||||
state.semantic_result = Some(json!({"reportedWorkDisposition": "done"}))
|
||||
}),
|
||||
("unanswered_runtime_request", |state| {
|
||||
state
|
||||
.pending_runtime_requests
|
||||
.insert("input-1".to_owned(), pending_input_request("input-1"));
|
||||
}),
|
||||
("active_goal", |state| {
|
||||
state.goal_projection = json!({"goal": {"status": "active"}})
|
||||
}),
|
||||
];
|
||||
for (label, seed) in guards {
|
||||
let (directory, config, state_path) =
|
||||
interrupted_turn_after_runner_upgrade("upgrade-lost-turn-strict");
|
||||
let mut state: AcpxDurableState =
|
||||
serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
|
||||
seed(&mut state);
|
||||
fs::write(&state_path, serde_json::to_vec(&state).unwrap()).unwrap();
|
||||
let retained = fs::read(&state_path).unwrap();
|
||||
let mut recovered = AcpxCommandExecutor::with_runner_config(&directory, &config);
|
||||
let error = recovered.restore().unwrap_err();
|
||||
assert_eq!(
|
||||
error.classification(),
|
||||
Some(crate::durable::PROVIDER_STATE_INCOMPATIBLE),
|
||||
"{label}: {error}"
|
||||
);
|
||||
assert!(recovered.session.is_none(), "{label}");
|
||||
assert_eq!(fs::read(&state_path).unwrap(), retained, "{label}");
|
||||
fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class() {
|
||||
|
||||
@@ -448,3 +448,128 @@ fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
|
||||
assert_eq!(continued.result["status"], "accepted");
|
||||
restored.shutdown().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() {
|
||||
let fixture =
|
||||
Fixture(std::env::temp_dir().join(format!("acpx-upgrade-{}", uuid::Uuid::new_v4())));
|
||||
fs::create_dir_all(&fixture.0).unwrap();
|
||||
fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let sidecar = fixture.0.join("sidecar");
|
||||
fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
|
||||
fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let ports = unused_fence_ports();
|
||||
let args = vec![
|
||||
"--mode".to_owned(),
|
||||
"suspend".to_owned(),
|
||||
"--profile-digest".to_owned(),
|
||||
PROFILE_DIGEST.to_owned(),
|
||||
"--lifetime-fence-ports".to_owned(),
|
||||
format!("{},{},{}", ports[0], ports[1], ports[2]),
|
||||
];
|
||||
let config = DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1/runner".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir: fixture.0.clone(),
|
||||
runner_instance_id: "runner-1".to_owned(),
|
||||
environment_lease_id: "lease-1".to_owned(),
|
||||
run_id: "run-1".to_owned(),
|
||||
normalized_session_id: "session-1".to_owned(),
|
||||
turn_id: "turn-1".to_owned(),
|
||||
item_id: "item-1".to_owned(),
|
||||
runner_version: "0.0.0".to_owned(),
|
||||
runner_digest: format!("sha256:{}", "a".repeat(64)),
|
||||
acpx_launch_profile: Some(AcpxLaunchProfile {
|
||||
candidate_admission: None,
|
||||
authority_digest: format!("sha256:{}", "d".repeat(64)),
|
||||
command: sidecar.clone(),
|
||||
args: args.clone(),
|
||||
artifacts: vec![QualifiedLaunchArtifact {
|
||||
path: sidecar.clone(),
|
||||
sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
|
||||
}],
|
||||
}),
|
||||
opencode_launch_profile: None,
|
||||
cursor_cloud_launch_profile: None,
|
||||
max_outbox_bytes: 1024 * 1024,
|
||||
p0_reserve_bytes: 64 * 1024,
|
||||
max_frame_bytes: 1024 * 1024,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
"providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol",
|
||||
"acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp",
|
||||
"agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
|
||||
"agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
|
||||
"sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
|
||||
"normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
|
||||
"instructions": "Complete the supplied work.",
|
||||
"permissionMode": "approve-all", "permissionModePinned": true,
|
||||
});
|
||||
let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config);
|
||||
original
|
||||
.execute(&command(1, "run.prepare", json!({"provider": descriptor})))
|
||||
.unwrap();
|
||||
let opened = original
|
||||
.execute(&command(2, "session.open", json!({})))
|
||||
.unwrap();
|
||||
let provider_session = opened.result["providerSessionId"].clone();
|
||||
let started = original
|
||||
.execute(&command(
|
||||
3,
|
||||
"turn.start",
|
||||
json!({"turnId": "provider-turn-1", "text": "Do the work."}),
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(started.result["status"], "accepted");
|
||||
drop(original);
|
||||
|
||||
let mut upgraded = config.clone();
|
||||
upgraded
|
||||
.acpx_launch_profile
|
||||
.as_mut()
|
||||
.unwrap()
|
||||
.authority_digest = format!("sha256:{}", "e".repeat(64));
|
||||
let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &upgraded);
|
||||
let snapshot = restored
|
||||
.execute(&command(4, "session.snapshot", json!({})))
|
||||
.expect("the new build restores the conversation of the turn the upgrade lost");
|
||||
assert_ne!(snapshot.result["status"], "closed");
|
||||
assert_eq!(snapshot.result["providerSessionId"], provider_session);
|
||||
let events = restored.poll_events().unwrap();
|
||||
let failed = events
|
||||
.iter()
|
||||
.find(|event| event.event_type == "turn.failed")
|
||||
.expect("the lost turn is terminated");
|
||||
assert_eq!(failed.payload["providerTurnId"], "provider-turn-1");
|
||||
assert_eq!(
|
||||
failed.payload["error"]["code"],
|
||||
"provider_turn_lost_on_restore"
|
||||
);
|
||||
assert_eq!(failed.payload["error"]["recoverable"], true);
|
||||
assert!(!events
|
||||
.iter()
|
||||
.any(|event| event.event_type == "run.terminal"));
|
||||
let continued = restored
|
||||
.execute(&command(
|
||||
5,
|
||||
"turn.start",
|
||||
json!({"turnId": "provider-turn-2", "text": "Continue the work."}),
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(continued.result["status"], "accepted");
|
||||
let saved: Value =
|
||||
serde_json::from_slice(&fs::read(fixture.0.join("acpx-provider-state.json")).unwrap())
|
||||
.unwrap();
|
||||
let upgraded_digest = upgraded
|
||||
.acpx_launch_profile
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.canonical_digest()
|
||||
.unwrap();
|
||||
assert_eq!(saved["launchProfileDigest"], upgraded_digest);
|
||||
restored.shutdown().unwrap();
|
||||
}
|
||||
@@ -6571,6 +6571,36 @@ describe("executeNativeSession recovery", () => {
|
||||
expect(openReplacementSession).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("continues in a fresh session when a new run's saved session is incompatible and the checkpoint allows replacement", async () => {
|
||||
// A provider release changed between turns (for example an agent-server
|
||||
// bump). The next run's checkpoint allows replacement, so the user's task
|
||||
// continues automatically in a fresh provider session, with no Retry.
|
||||
const checkpoint: PersistedNativeSession = {
|
||||
backendKind: "mock", sessionId: "driver-upgraded", identity,
|
||||
providerSessionId: "provider-upgraded", providerRecoveryPolicy: "allow_replacement_after_resume_failure",
|
||||
cursor: "0", activeTurnId: null, pendingRuntimeRequests: [], lineage: [],
|
||||
};
|
||||
const replacementAttempted = new Error("replacement attempted");
|
||||
const openReplacementSession = vi.fn(async () => { throw replacementAttempted; });
|
||||
const backend: NativeSessionBackend = {
|
||||
descriptor: async () => ({ kind: "mock", name: "upgraded", version: "1",
|
||||
capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true } }),
|
||||
openSession: vi.fn(), openReplacementSession,
|
||||
recoverSession: async () => ({ recovered: false, failureClass: PROVIDER_STATE_INCOMPATIBLE,
|
||||
reason: "ACPX durable launch profile digest does not match runner startup" }),
|
||||
};
|
||||
const port: ControlPlanePort = {
|
||||
openRun: vi.fn(async () => undefined), appendEvent: vi.fn(), completeRun: vi.fn(),
|
||||
loadSessionCheckpoint: async () => structuredClone(checkpoint),
|
||||
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }),
|
||||
};
|
||||
await expect(executeNativeSession({
|
||||
input, backend, controlPlane: port,
|
||||
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
|
||||
})).rejects.toBe(replacementAttempted);
|
||||
expect(openReplacementSession).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("bounds paginated recovery replay with one signal and observes a late rejection", async () => {
|
||||
vi.useFakeTimers();
|
||||
let rejectStalledReplay = (_error: Error) => {};
|
||||
|
||||
Reference in new issue
Block a user