fix(runner): settle ACPX turns when the cancel reply is lost (#20498)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the native Runner and the ACPX sidecar > - A saved question or a board Stop cancels the active turn with `turn.cancel` > - The Runner waited 30 s for the sidecar's reply to that cancel, and the Runner serves controller commands one at a time > - When that reply was lost, the controller's shutdown commands timed out first, so the run failed and the task needed a human Retry > - This pull request bounds that wait, keeps the transport open, and lets the provider's own turn terminal settle the turn > - The benefit is that a lost cancel reply on Claude settles the same way it already does on Codex (#20336) ## Linked Issues or Issue Description Refs #20336 (Codex equivalent: settle a saved handoff after verified shutdown). Related, no overlap: #20477, #20482, #20483. **What happened?** The restart-chaos harness dropped the ACPX sidecar's reply to the Runner's `turn.cancel` (fault PV2, target `interrupt_ack`). A Claude task that saved a question (Q1) failed with "The waiting state is saved; confirming shutdown of the previous execution is still pending". Its next attempt failed `native_session_cleanup_quarantined`. A Claude task stopped by the board (S1) ended `blocked`. Retry was accepted but started no run. Cause, in `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs` (`interrupt_turn`, master line ~800): the cancel used the ordinary sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`. While it waited: - `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s governed drain plus its 10 s close grace ran out first (`runner did not durably suspend before checkpoint`, suspend command `pending`). - At 30 s, `fail_closed` killed the sidecar and dropped the provider's buffered turn terminal. The poll path then recorded `turn.failed` (`acpx_provider_transport_failed`) and `provider_exit_unconfirmed` (`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned `already_settled`, and `runner.suspend` refused with `ACPX provider lifetime cleanup is not yet proven` (`acpx_provider_backend.rs:2613-2621`). Codex passes the same fault because its Runner does not block on the interrupt reply. **Expected behavior** A lost cancel reply must not fail the run. The saved question must settle as `succeeded`. A board Stop must end the run as cancelled with a proven suspension, so the next turn can start. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4` Failing wave on master: `w-20261110T060120-s207` (t04, t05). **Deployment mode** Native Runner with the Claude ACPX profile, local. Hosted sandboxes run the same Runner code. ## What Changed - `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and waits only to a caller deadline. On a missing reply it keeps the transport usable and retains the request ID, so a late reply is skipped in both the response and the event-poll paths. A rejected reply and transport errors keep their existing fail-closed result. - `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the shorter configured request bound) for the cancel reply. On a missing reply it retires the runtime from warm reuse and returns. The provider's turn terminal still arrives through polling, and `turn.stop` still reaps a provider that never stops. - `AcpxProviderSession::deliver_tool_result`: a board Stop can land while a semantic tool runs. The session's own `turn.cancel` then retires the call in the sidecar before its result arrives, and the turn terminal may still be buffered. The sidecar rejects the result as stale (`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that turn, the Runner now keeps the durable result, like a result for an already closed turn. Every other rejection still fails closed. (`AcpxSidecarTransport::request_tool_resolve` reports whether a rejection was the stale-call case.) - A repeated interruption while a cancel is unanswered sends nothing and returns at once (review finding). - `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that cancel, `tool.resolve` is rejected as stale. The provider ends the turn, and the reply is dropped or written just before the next request. - Three regression tests (below) and a short section in `packages/paperclip-runner/docs/durable-recovery.md`. No TypeScript, server, schema or protocol change. The normal start path is unchanged; nothing waits before turn start. ## Verification Red/green (red runs used the master versions of `acpx_provider_session.rs` and `acpx_sidecar_transport.rs`): - `cargo test -p paperclip-runner-core --test acpx_warm_attachment a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner` - Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request timed out at turn.cancel` (the live error). - Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted` and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and `runner.suspend` completes. - `cargo test -p paperclip-runner-core --test acpx_provider_turns lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport` - Red: `interrupt_turn` fails at the request bound. - Green, for a lost reply and for a late reply: the terminal is polled, warm reuse is refused, and the next request (`session.close`) succeeds after the late reply arrives. A late reply causes no second settle and no second prompt. - `cargo test -p paperclip-runner-core --test acpx_provider_turns a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport` - Red (guard disabled): `ACPX sidecar command tool.resolve was rejected (retryable=false, classification=provider_tool_call_retired)`, the same error as live wave task t03 below. - Green: the result is kept, the pending tool clears, and the turn terminal is polled after it. - `cargo test -p paperclip-runner-core` (whole crate): all pass. Existing `turns-tool-resolution-rejected` coverage still fails closed when no cancel was sent. `cargo fmt --check` passes. - Pack contract digest: unchanged (`sha256:3164357d…` before and after, from `scripts/contracts/contract-fingerprints.mjs`). - Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first fix only), `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1 tasks pass (the asking run `succeeded`, no failed run, answer in the brief). One S1 passes. S1 task t03 failed: Stop landed during a `write_task_document` call and hit the stale tool-result error above. The second commit fixes that case. - Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault applied (asking run `succeeded`, no failed run). S1 t01 passes with the fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails only `provider_fault_applied` (`applied=0`): the agent finished before the Stop window, so nothing was injected and the task proves nothing either way. - Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault applied; t01's Stop took effect (`status=cancelled`), task `done`, no Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the run finished before the Stop window); all its R1–R6 and U checks pass. - Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault applied pass (4 Q1, 2 S1). No product rule failed. The two misses are harness timing (no injection), not product failures. ## Risks - A healthy cancel reply that takes longer than 5 s now retires the warm runtime. The next turn starts a fresh provider process for that session. The observed healthy reply time is about 1 s. - If the provider never stops and the reply is lost, the turn stays active until `turn.stop` terminates the provider and proves the lifetime fence. If that proof fails, the existing settlement-pending and quarantine states are unchanged; this PR adds no success path without proof. - Runner binary change: hosted Daytona images need a normal image rebuild to pick up the fix. The pack `contractDigest` does not change, so existing image packs stay reusable. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 05:11:49 +01:00 · 2026-11-10 03:17:14 -06:00
1 parent 82abb3ea04
commit 914e3ae14f
@@ -112,6 +112,23 @@ unconfirmed cleanup do not qualify. Error messages cannot authorize recovery.
 An arbitrary `closed` provider state stays closed. This path does not establish
 recovery for a Runner killed during an accepted turn.

+## ACPX cancellation without a reply
+
+The Runner serves controller commands one at a time. It therefore waits at most
+five seconds for the sidecar's reply to `turn.cancel`, not the 30-second request
+bound. The provider's turn terminal settles a cancelled turn; the reply does
+not. When the reply is missing, the Runner keeps the transport, retires the
+runtime from warm reuse, and keeps polling. It ignores the late reply if one
+arrives. A repeated interruption while that cancel is unanswered sends nothing
+and returns at once. A provider that never stops is still reaped by `turn.stop` and the
+lifetime-fence proof. A rejected reply or a transport error still fails closed.
+
+A tool result can reach the sidecar after the session's own `turn.cancel`
+retired that call. The sidecar rejects it as stale (`ACPX_TOOL_CALL_STALE`).
+The Runner then keeps the durable result, as it does for a result that arrives
+after a polled terminal. The same rejection without that cancellation still
+fails closed.
+
 ## Settled ACPX sessions across a Runner upgrade

 Saved ACPX provider state records the digest of the launch profile that wrote
@@ -29,6 +29,10 @@ const MAX_ID_CHARS: usize = 240;
 const MAX_MODEL_CHARS: usize = 240;
 pub(crate) const MAX_SYSTEM_INSTRUCTIONS_BYTES: usize = 1024 * 1024;
 const MAX_JSON_SAFE_INTEGER: u64 = 9_007_199_254_740_991;
+// A healthy sidecar confirms cancellation within about a second. The runner
+// serves controller commands one at a time, so this bound must stay well below
+// the controller's 15 s governed-wait drain and 10 s close grace.
+const ACPX_TURN_CANCEL_ACK_TIMEOUT: Duration = Duration::from_secs(5);

 #[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)]
 #[serde(rename_all = "kebab-case")]
@@ -355,6 +359,7 @@ pub struct AcpxProviderSession {
    transport_terminated: bool,
    rejected_start_resumable: bool,
    runtime_retired: bool,
+    cancelled_turn_id: Option<String>,
    completed_turn_observed: bool,
    generation_has_run_turn: bool,
    restarted_after_turn_from: Option<u32>,
@@ -416,6 +421,7 @@ impl AcpxProviderSession {
            transport_terminated: false,
            rejected_start_resumable: false,
            runtime_retired: false,
+            cancelled_turn_id: None,
            completed_turn_observed: false,
            generation_has_run_turn: false,
            restarted_after_turn_from: None,
@@ -798,11 +804,24 @@ impl AcpxProviderSession {
                "ACPX interruption named a stale or inactive turn",
            ));
        }
-        let response = match self.transport.request(
-            GeneratedAcpxSidecarCommand::TurnCancel,
+        self.cancelled_turn_id = Some(turn_id.to_owned());
+        let response = match self.transport.request_turn_cancel(
            json!({"turnId":turn_id,"reason":bounded_reason(reason)}),
+            Instant::now()
+                + ACPX_TURN_CANCEL_ACK_TIMEOUT.min(self.config.transport.request_timeout),
        ) {
-            Ok(response) => response,
+            Ok(Some(response)) => response,
+            Ok(None) => {
+                // The cancel was delivered but its reply is missing. Its turn
+                // terminal still arrives through polling, and turn.stop can
+                // reap a provider that never stops. Failing closed here would
+                // block every controller command for the whole request bound
+                // and discard that terminal. The sidecar may have retired its
+                // runtime in the lost reply, so never admit another prompt.
+                // The caller receives no confirmation: Null is not a reply.
+                self.runtime_retired = true;
+                return Ok(Value::Null);
+            }
            Err(error) => return Err(self.fail_closed(error)),
        };
        if response.get("cancelled").and_then(Value::as_bool) != Some(true) {
@@ -1100,12 +1119,19 @@ impl AcpxProviderSession {
                "error":Value::Null,
            })
        };
-        let response = match self
-            .transport
-            .request(GeneratedAcpxSidecarCommand::ToolResolve, resolution)
-        {
+        let response = match self.transport.request_tool_resolve(resolution) {
            Ok(response) => response,
-            Err(error) => return Err(self.fail_closed(error)),
+            // This session's own turn.cancel retired the call before its result
+            // arrived, and the turn terminal may still be buffered. The provider
+            // can never receive this result, exactly as after a polled terminal,
+            // so keep the durable result instead of failing the transport.
+            Err((_, true)) if self.cancelled_turn_id.as_deref() == Some(turn_id.as_str()) => {
+                self.state = next_state;
+                self.tool_bridge = next_bridge;
+                self.reserved_tool_bridge = next_reserved_bridge;
+                return Ok(());
+            }
+            Err((error, _)) => return Err(self.fail_closed(error)),
        };
        self.verify_resolution(&response, "tool")?;
        self.state = next_state;
@@ -2258,6 +2284,7 @@ mod mode_tests {
            transport_terminated: false,
            rejected_start_resumable: false,
            runtime_retired: false,
+            cancelled_turn_id: None,
            completed_turn_observed: false,
            generation_has_run_turn: false,
            restarted_after_turn_from: None,
@@ -93,6 +93,7 @@ pub struct AcpxSidecarTransport {
    stderr_categories: BTreeSet<&'static str>,
    poisoned: bool,
    abandoned_observation: Option<u64>,
+    abandoned_cancellation: Option<u64>,
 }

 // A fresh Pi process verifies and copies its native closure before ACP admission.
@@ -407,6 +408,7 @@ impl AcpxSidecarTransport {
            stderr_categories: BTreeSet::new(),
            poisoned: false,
            abandoned_observation: None,
+            abandoned_cancellation: None,
        })
    }

@@ -450,6 +452,7 @@ impl AcpxSidecarTransport {
            Ok(CommandOutcome::Rejected {
                error,
                resume_required,
+                ..
            }) => Err(AcpxTurnStartFailure {
                error,
                resume_required,
@@ -461,6 +464,23 @@ impl AcpxSidecarTransport {
        }
    }

+    /// Returns `Err((error, retired))`, where `retired` is true only when the
+    /// sidecar rejected the result because it no longer tracks that tool call.
+    pub(crate) fn request_tool_resolve(
+        &mut self,
+        params: Value,
+    ) -> Result<Value, (LocalRunnerError, bool)> {
+        match self.request_outcome(GeneratedAcpxSidecarCommand::ToolResolve, params) {
+            Ok(CommandOutcome::Success(value)) => Ok(value),
+            Ok(CommandOutcome::Rejected {
+                error,
+                tool_call_retired,
+                ..
+            }) => Err((error, tool_call_retired)),
+            Err(error) => Err((error, false)),
+        }
+    }
+
    fn request_outcome(
        &mut self,
        command: GeneratedAcpxSidecarCommand,
@@ -576,6 +596,51 @@ impl AcpxSidecarTransport {
        }
    }

+    /// Sends `turn.cancel` and waits for its reply only until `deadline`.
+    ///
+    /// The provider's turn terminal settles a cancelled turn, not this reply,
+    /// and the runner cannot service `turn.stop` while it blocks here. A reply
+    /// lost past the bound therefore leaves the transport usable (polling still
+    /// owns every buffered and later frame) and retains the request ID, so a
+    /// late reply cannot masquerade as the answer to another request.
+    pub(crate) fn request_turn_cancel(
+        &mut self,
+        params: Value,
+        deadline: Instant,
+    ) -> Result<Option<Value>, LocalRunnerError> {
+        if self.poisoned {
+            return Err(LocalRunnerError::invalid(
+                "ACPX sidecar transport is unavailable after a protocol failure",
+            ));
+        }
+        if self.abandoned_cancellation.is_some() {
+            // The earlier cancel is still delivered and unanswered. Its runtime
+            // was retired, so no later turn exists; another cancel adds nothing
+            // and must not block the runner again.
+            return Ok(None);
+        }
+        let command = GeneratedAcpxSidecarCommand::TurnCancel;
+        let deadline = self
+            .request_deadline
+            .map_or(deadline, |bound| bound.min(deadline));
+        let result = self.send_request(command, params).and_then(|request_id| {
+            let result = self.receive_response(command, request_id, deadline)?;
+            if result.is_none() {
+                self.abandoned_cancellation = Some(request_id);
+            }
+            Ok(result)
+        });
+        match result {
+            Ok(Some(CommandOutcome::Success(value))) => Ok(Some(value)),
+            Ok(Some(CommandOutcome::Rejected { error, .. })) => Err(error),
+            Ok(None) => Ok(None),
+            Err(error) => {
+                self.poison();
+                Err(error)
+            }
+        }
+    }
+
    pub(crate) fn observed_event_sequence(&self) -> u64 {
        self.last_event_sequence
    }
@@ -660,6 +725,10 @@ impl AcpxSidecarTransport {
                        self.abandoned_observation = None;
                        continue;
                    }
+                    if self.abandoned_cancellation == Some(response.id) {
+                        self.abandoned_cancellation = None;
+                        continue;
+                    }
                    if response.id != request_id {
                        return Err(LocalRunnerError::invalid(format!(
                            "ACPX sidecar response id mismatch: expected {request_id}, received {}",
@@ -676,7 +745,9 @@ impl AcpxSidecarTransport {
                    let resume_required = command == GeneratedAcpxSidecarCommand::TurnStart
                        && error.retryable
                        && error.code == "SESSION_RESUME_REQUIRED";
-                    return Ok(Some(CommandOutcome::Rejected { error: LocalRunnerError::invalid(
+                    let tool_call_retired = command == GeneratedAcpxSidecarCommand::ToolResolve
+                        && error.code == "ACPX_TOOL_CALL_STALE";
+                    return Ok(Some(CommandOutcome::Rejected { tool_call_retired, error: LocalRunnerError::invalid(
                        format!(
                            "ACPX sidecar command {} was rejected (retryable={}, classification={})",
                            command.as_str(),
@@ -712,6 +783,11 @@ impl AcpxSidecarTransport {
                {
                    self.abandoned_observation = None;
                }
+                ParsedFrame::Response(response)
+                    if self.abandoned_cancellation == Some(response.id) =>
+                {
+                    self.abandoned_cancellation = None;
+                }
                ParsedFrame::Response(response) => {
                    return Err(LocalRunnerError::invalid(format!(
                        "ACPX sidecar emitted response {} without a pending request",
@@ -879,6 +955,7 @@ enum CommandOutcome {
    Rejected {
        error: LocalRunnerError,
        resume_required: bool,
+        tool_call_retired: bool,
    },
 }

@@ -69,6 +69,15 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
    let mut source_run = String::new();
    let mut source_pending_completion: Option<String> = None;
    let mut source_goal_revision = 0_u64;
+    // `--cancel-reply lost|late`: the provider stops its prompt, but the reply
+    // to turn.cancel is dropped, or written only before the next request.
+    let cancel_reply = args
+        .windows(2)
+        .find(|pair| pair[0] == "--cancel-reply")
+        .map(|pair| pair[1].clone());
+    let mut late_cancel_reply: Option<Value> = None;
+    // With --cancel-reply, a cancelled turn forgets its pending tool calls.
+    let mut cancel_retired_tools = false;
    let mut source_journal = if mode == "source-reconcile" {
        let path = args
            .windows(2)
@@ -115,6 +124,9 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
            .get("command")
            .and_then(Value::as_str)
            .ok_or("request command is missing")?;
+        if let Some(reply) = late_cancel_reply.take() {
+            write_json(&mut stdout, &reply)?;
+        }
        if command == "turn.start" {
            if let Some(fault) = args
                .windows(2)
@@ -627,7 +639,9 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
            | "suspend-wrong-ack"
            | "suspend-wrong-identity"
            | "suspend-missing-identity" => {
-                if command == "tool.resolve" && mode == "turns-tool-resolution-rejected" {
+                if command == "tool.resolve"
+                    && (mode == "turns-tool-resolution-rejected" || cancel_retired_tools)
+                {
                    write_json(
                        &mut stdout,
                        &json!({
@@ -666,6 +680,27 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
                if command == "session.open" {
                    opened_identity = response.pointer("/result/identity").cloned();
                }
+                if command == "turn.cancel" {
+                    if let Some(fault) = cancel_reply.as_deref() {
+                        write_turn_event(
+                            &mut stdout,
+                            next_sequence,
+                            "runtime.turn_terminal",
+                            "run-1",
+                            request
+                                .pointer("/params/turnId")
+                                .and_then(Value::as_str)
+                                .unwrap(),
+                            json!({"status":"interrupted"}),
+                        )?;
+                        next_sequence += 1;
+                        cancel_retired_tools = true;
+                        if fault == "late" {
+                            late_cancel_reply = Some(response);
+                        }
+                        continue;
+                    }
+                }
                if command == "session.suspend" {
                    if let Some(delay_ms) = suspend_delay_ms {
                        std::thread::sleep(Duration::from_millis(delay_ms));
@@ -792,6 +792,84 @@ fn forced_cancellation_retirement_preserves_terminal_polling_but_rejects_live_re
    }
 }

+#[test]
+fn lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport() {
+    // Restart-chaos PV2 interrupt_ack: the provider stops its prompt, but the
+    // sidecar's reply to turn.cancel is lost or late. The terminal must still
+    // settle the turn, and a late reply must not break the next request.
+    for mode in ["lost", "late"] {
+        let mut config = config("turns");
+        config
+            .transport
+            .args
+            .extend(["--cancel-reply".to_owned(), mode.to_owned()]);
+        let mut session = AcpxProviderSession::start(&config).unwrap();
+        session
+            .start_turn("turn-1", "Please help", &std::env::temp_dir())
+            .unwrap();
+        let started = Instant::now();
+        session
+            .interrupt_turn("turn-1", "Paperclip interruption")
+            .unwrap_or_else(|error| panic!("{mode}: {error}"));
+        assert!(started.elapsed() < Duration::from_secs(3), "{mode}");
+        assert!(session.runtime_retired(), "{mode}");
+        // A repeated Stop while the first cancel is unanswered must not block.
+        let repeated = Instant::now();
+        session
+            .interrupt_turn("turn-1", "Paperclip interruption again")
+            .unwrap_or_else(|error| panic!("{mode} repeat: {error}"));
+        assert!(repeated.elapsed() < Duration::from_millis(200), "{mode}");
+        let mut terminal = None;
+        while terminal.is_none() {
+            let events = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();
+            terminal = events.into_iter().find(|event| {
+                matches!(event, AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1")
+            });
+        }
+        assert!(session.state().active_turn_id().is_none(), "{mode}");
+        assert!(session
+            .start_turn("turn-2", "Must not reuse", &std::env::temp_dir())
+            .unwrap_err()
+            .to_string()
+            .contains("runtime was retired"));
+        session
+            .shutdown("test complete")
+            .unwrap_or_else(|error| panic!("{mode}: {error}"));
+    }
+}
+
+#[test]
+fn a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport() {
+    // Restart-chaos S1 PV2: Stop lands while a semantic tool runs. The cancel
+    // retires the call in the sidecar before its result arrives, and the turn
+    // terminal is still buffered. The result must be kept, not fail the session.
+    let mut config = config("turns-tool");
+    config
+        .transport
+        .args
+        .extend(["--cancel-reply".to_owned(), "lost".to_owned()]);
+    let mut session = AcpxProviderSession::start(&config).unwrap();
+    session
+        .start_turn("turn-1", "Please help", &std::env::temp_dir())
+        .unwrap();
+    let tool = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();
+    assert!(
+        matches!(&tool[0], AcpxProviderStateEvent::ToolCall { call_id, .. } if call_id == "call-1")
+    );
+    session
+        .interrupt_turn("turn-1", "Cancelled by a board operator")
+        .unwrap();
+    session
+        .deliver_tool_result(&late_result("call-1"))
+        .expect("a call retired by our own cancellation keeps its result");
+    assert!(session.state().pending_tool("call-1").is_none());
+    let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();
+    assert!(
+        matches!(&terminal[0], AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1")
+    );
+    session.shutdown("test complete").unwrap();
+}
+
 fn reusable_config(directory: &std::path::Path) -> AcpxProviderSessionConfig {
    let mut config = config("source-reconcile");
    config.transport.args = vec![
@@ -577,3 +577,116 @@ fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() {
    assert_eq!(saved["launchProfileDigest"], upgraded_digest);
    restored.shutdown().unwrap();
 }
+
+#[test]
+fn a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner() {
+    // Restart-chaos PV2 interrupt_ack on Claude: the sidecar's reply to
+    // turn.cancel is lost. The runner must stay responsive, publish the
+    // provider's own terminal, and reach a proven suspension. Before the fix it
+    // blocked for the 30 s request bound, then failed the transport, and
+    // runner.suspend refused because provider exit was unconfirmed.
+    let fixture =
+        Fixture(std::env::temp_dir().join(format!("acpx-cancel-lost-{}", uuid::Uuid::new_v4())));
+    fs::create_dir_all(&fixture.0).unwrap();
+    fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
+    let sidecar = fixture.0.join("sidecar");
+    fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
+    fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
+    let args = vec![
+        "--mode".to_owned(),
+        "turns".to_owned(),
+        "--profile-digest".to_owned(),
+        PROFILE_DIGEST.to_owned(),
+        "--cancel-reply".to_owned(),
+        "lost".to_owned(),
+    ];
+    let config = DurableRunnerConfig {
+        connect_url: "ws://127.0.0.1/runner".to_owned(),
+        ca_bundle_path: None,
+        state_dir: fixture.0.clone(),
+        runner_instance_id: "runner-1".to_owned(),
+        environment_lease_id: "lease-1".to_owned(),
+        run_id: "run-1".to_owned(),
+        normalized_session_id: "session-1".to_owned(),
+        turn_id: "turn-1".to_owned(),
+        item_id: "item-1".to_owned(),
+        runner_version: "0.0.0".to_owned(),
+        runner_digest: format!("sha256:{}", "a".repeat(64)),
+        acpx_launch_profile: Some(AcpxLaunchProfile {
+            candidate_admission: None,
+            authority_digest: format!("sha256:{}", "d".repeat(64)),
+            command: sidecar.clone(),
+            args: args.clone(),
+            artifacts: vec![QualifiedLaunchArtifact {
+                path: sidecar.clone(),
+                sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
+            }],
+        }),
+        opencode_launch_profile: None,
+        cursor_cloud_launch_profile: None,
+        max_outbox_bytes: 1024 * 1024,
+        p0_reserve_bytes: 64 * 1024,
+        max_frame_bytes: 1024 * 1024,
+        reconnect_delay: Duration::from_millis(1),
+        reconnect_grace: None,
+        max_runtime: Duration::from_secs(60),
+        turn_scoped: false,
+    };
+    let descriptor = json!({
+        "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
+        "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol",
+        "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp",
+        "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
+        "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
+        "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
+        "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
+        "instructions": "Complete the supplied work.",
+        "permissionMode": "approve-all", "permissionModePinned": true,
+    });
+    let mut executor = AcpxCommandExecutor::with_runner_config(&fixture.0, &config);
+    executor
+        .execute(&command(1, "run.prepare", json!({"provider": descriptor})))
+        .unwrap();
+    executor
+        .execute(&command(2, "session.open", json!({})))
+        .unwrap();
+    executor
+        .execute(&command(
+            3,
+            "turn.start",
+            json!({"turnId": "provider-turn-1", "text": "Ask one question."}),
+        ))
+        .unwrap();
+    let started = std::time::Instant::now();
+    let interrupted = executor
+        .execute(&command(4, "turn.interrupt", json!({})))
+        .unwrap();
+    assert_eq!(interrupted.result["status"], "interrupt_requested");
+    assert!(started.elapsed() < Duration::from_secs(10));
+    let polled_until = std::time::Instant::now() + Duration::from_secs(5);
+    let mut events = executor.poll_events().unwrap();
+    while !events
+        .iter()
+        .any(|event| event.event_type == "run.terminal")
+    {
+        assert!(std::time::Instant::now() < polled_until, "no turn terminal");
+        events.extend(executor.poll_events().unwrap());
+    }
+    assert!(events
+        .iter()
+        .any(|event| event.event_type == "turn.interrupted"));
+    assert!(!events.iter().any(|event| event.event_type == "turn.failed"));
+    let terminal = events
+        .iter()
+        .find(|event| event.event_type == "run.terminal")
+        .unwrap();
+    assert_eq!(terminal.payload["runTerminalState"], "cancelled");
+    executor
+        .execute(&command(5, "turn.stop", json!({})))
+        .unwrap();
+    let suspended = executor
+        .execute(&command(6, "runner.suspend", json!({})))
+        .unwrap();
+    assert_eq!(suspended.result["status"], "completed");
+    executor.shutdown().unwrap();
+}