From 914e3ae14f1d288e429c39fe4e9f8f36e1d2ef7b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 03:17:14 -0600 Subject: [PATCH] fix(runner): settle ACPX turns when the cancel reply is lost (#20498) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the native Runner and the ACPX sidecar > - A saved question or a board Stop cancels the active turn with `turn.cancel` > - The Runner waited 30 s for the sidecar's reply to that cancel, and the Runner serves controller commands one at a time > - When that reply was lost, the controller's shutdown commands timed out first, so the run failed and the task needed a human Retry > - This pull request bounds that wait, keeps the transport open, and lets the provider's own turn terminal settle the turn > - The benefit is that a lost cancel reply on Claude settles the same way it already does on Codex (#20336) ## Linked Issues or Issue Description Refs #20336 (Codex equivalent: settle a saved handoff after verified shutdown). Related, no overlap: #20477, #20482, #20483. **What happened?** The restart-chaos harness dropped the ACPX sidecar's reply to the Runner's `turn.cancel` (fault PV2, target `interrupt_ack`). A Claude task that saved a question (Q1) failed with "The waiting state is saved; confirming shutdown of the previous execution is still pending". Its next attempt failed `native_session_cleanup_quarantined`. A Claude task stopped by the board (S1) ended `blocked`. Retry was accepted but started no run. Cause, in `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs` (`interrupt_turn`, master line ~800): the cancel used the ordinary sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`. While it waited: - `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s governed drain plus its 10 s close grace ran out first (`runner did not durably suspend before checkpoint`, suspend command `pending`). - At 30 s, `fail_closed` killed the sidecar and dropped the provider's buffered turn terminal. The poll path then recorded `turn.failed` (`acpx_provider_transport_failed`) and `provider_exit_unconfirmed` (`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned `already_settled`, and `runner.suspend` refused with `ACPX provider lifetime cleanup is not yet proven` (`acpx_provider_backend.rs:2613-2621`). Codex passes the same fault because its Runner does not block on the interrupt reply. **Expected behavior** A lost cancel reply must not fail the run. The saved question must settle as `succeeded`. A board Stop must end the run as cancelled with a proven suspension, so the next turn can start. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4` Failing wave on master: `w-20261110T060120-s207` (t04, t05). **Deployment mode** Native Runner with the Claude ACPX profile, local. Hosted sandboxes run the same Runner code. ## What Changed - `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and waits only to a caller deadline. On a missing reply it keeps the transport usable and retains the request ID, so a late reply is skipped in both the response and the event-poll paths. A rejected reply and transport errors keep their existing fail-closed result. - `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the shorter configured request bound) for the cancel reply. On a missing reply it retires the runtime from warm reuse and returns. The provider's turn terminal still arrives through polling, and `turn.stop` still reaps a provider that never stops. - `AcpxProviderSession::deliver_tool_result`: a board Stop can land while a semantic tool runs. The session's own `turn.cancel` then retires the call in the sidecar before its result arrives, and the turn terminal may still be buffered. The sidecar rejects the result as stale (`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that turn, the Runner now keeps the durable result, like a result for an already closed turn. Every other rejection still fails closed. (`AcpxSidecarTransport::request_tool_resolve` reports whether a rejection was the stale-call case.) - A repeated interruption while a cancel is unanswered sends nothing and returns at once (review finding). - `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that cancel, `tool.resolve` is rejected as stale. The provider ends the turn, and the reply is dropped or written just before the next request. - Three regression tests (below) and a short section in `packages/paperclip-runner/docs/durable-recovery.md`. No TypeScript, server, schema or protocol change. The normal start path is unchanged; nothing waits before turn start. ## Verification Red/green (red runs used the master versions of `acpx_provider_session.rs` and `acpx_sidecar_transport.rs`): - `cargo test -p paperclip-runner-core --test acpx_warm_attachment a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner` - Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request timed out at turn.cancel` (the live error). - Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted` and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and `runner.suspend` completes. - `cargo test -p paperclip-runner-core --test acpx_provider_turns lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport` - Red: `interrupt_turn` fails at the request bound. - Green, for a lost reply and for a late reply: the terminal is polled, warm reuse is refused, and the next request (`session.close`) succeeds after the late reply arrives. A late reply causes no second settle and no second prompt. - `cargo test -p paperclip-runner-core --test acpx_provider_turns a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport` - Red (guard disabled): `ACPX sidecar command tool.resolve was rejected (retryable=false, classification=provider_tool_call_retired)`, the same error as live wave task t03 below. - Green: the result is kept, the pending tool clears, and the turn terminal is polled after it. - `cargo test -p paperclip-runner-core` (whole crate): all pass. Existing `turns-tool-resolution-rejected` coverage still fails closed when no cancel was sent. `cargo fmt --check` passes. - Pack contract digest: unchanged (`sha256:3164357d…` before and after, from `scripts/contracts/contract-fingerprints.mjs`). - Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first fix only), `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1 tasks pass (the asking run `succeeded`, no failed run, answer in the brief). One S1 passes. S1 task t03 failed: Stop landed during a `write_task_document` call and hit the stale tool-result error above. The second commit fixes that case. - Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault applied (asking run `succeeded`, no failed run). S1 t01 passes with the fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails only `provider_fault_applied` (`applied=0`): the agent finished before the Stop window, so nothing was injected and the task proves nothing either way. - Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault applied; t01's Stop took effect (`status=cancelled`), task `done`, no Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the run finished before the Stop window); all its R1–R6 and U checks pass. - Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault applied pass (4 Q1, 2 S1). No product rule failed. The two misses are harness timing (no injection), not product failures. ## Risks - A healthy cancel reply that takes longer than 5 s now retires the warm runtime. The next turn starts a fresh provider process for that session. The observed healthy reply time is about 1 s. - If the provider never stops and the reply is lost, the turn stays active until `turn.stop` terminates the provider and proves the lifetime fence. If that proof fails, the existing settlement-pending and quarantine states are unchanged; this PR adds no success path without proof. - Runner binary change: hosted Daytona images need a normal image rebuild to pick up the fix. The pack `contractDigest` does not change, so existing image packs stay reusable. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- .../paperclip-runner/docs/durable-recovery.md | 17 +++ .../runner-core/src/acpx_provider_session.rs | 43 +++++-- .../runner-core/src/acpx_sidecar_transport.rs | 79 +++++++++++- .../runner-core/src/bin/fake-acpx-sidecar.rs | 37 +++++- .../runner-core/tests/acpx_provider_turns.rs | 78 ++++++++++++ .../runner-core/tests/acpx_warm_attachment.rs | 113 ++++++++++++++++++ 6 files changed, 357 insertions(+), 10 deletions(-) diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index ecda817f372..d33a00f7e46 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -112,6 +112,23 @@ unconfirmed cleanup do not qualify. Error messages cannot authorize recovery. An arbitrary `closed` provider state stays closed. This path does not establish recovery for a Runner killed during an accepted turn. +## ACPX cancellation without a reply + +The Runner serves controller commands one at a time. It therefore waits at most +five seconds for the sidecar's reply to `turn.cancel`, not the 30-second request +bound. The provider's turn terminal settles a cancelled turn; the reply does +not. When the reply is missing, the Runner keeps the transport, retires the +runtime from warm reuse, and keeps polling. It ignores the late reply if one +arrives. A repeated interruption while that cancel is unanswered sends nothing +and returns at once. A provider that never stops is still reaped by `turn.stop` and the +lifetime-fence proof. A rejected reply or a transport error still fails closed. + +A tool result can reach the sidecar after the session's own `turn.cancel` +retired that call. The sidecar rejects it as stale (`ACPX_TOOL_CALL_STALE`). +The Runner then keeps the durable result, as it does for a result that arrives +after a polled terminal. The same rejection without that cancellation still +fails closed. + ## Settled ACPX sessions across a Runner upgrade Saved ACPX provider state records the digest of the launch profile that wrote diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs index e588b0a4d68..01d4a5b519b 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs @@ -29,6 +29,10 @@ const MAX_ID_CHARS: usize = 240; const MAX_MODEL_CHARS: usize = 240; pub(crate) const MAX_SYSTEM_INSTRUCTIONS_BYTES: usize = 1024 * 1024; const MAX_JSON_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +// A healthy sidecar confirms cancellation within about a second. The runner +// serves controller commands one at a time, so this bound must stay well below +// the controller's 15 s governed-wait drain and 10 s close grace. +const ACPX_TURN_CANCEL_ACK_TIMEOUT: Duration = Duration::from_secs(5); #[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] @@ -355,6 +359,7 @@ pub struct AcpxProviderSession { transport_terminated: bool, rejected_start_resumable: bool, runtime_retired: bool, + cancelled_turn_id: Option, completed_turn_observed: bool, generation_has_run_turn: bool, restarted_after_turn_from: Option, @@ -416,6 +421,7 @@ impl AcpxProviderSession { transport_terminated: false, rejected_start_resumable: false, runtime_retired: false, + cancelled_turn_id: None, completed_turn_observed: false, generation_has_run_turn: false, restarted_after_turn_from: None, @@ -798,11 +804,24 @@ impl AcpxProviderSession { "ACPX interruption named a stale or inactive turn", )); } - let response = match self.transport.request( - GeneratedAcpxSidecarCommand::TurnCancel, + self.cancelled_turn_id = Some(turn_id.to_owned()); + let response = match self.transport.request_turn_cancel( json!({"turnId":turn_id,"reason":bounded_reason(reason)}), + Instant::now() + + ACPX_TURN_CANCEL_ACK_TIMEOUT.min(self.config.transport.request_timeout), ) { - Ok(response) => response, + Ok(Some(response)) => response, + Ok(None) => { + // The cancel was delivered but its reply is missing. Its turn + // terminal still arrives through polling, and turn.stop can + // reap a provider that never stops. Failing closed here would + // block every controller command for the whole request bound + // and discard that terminal. The sidecar may have retired its + // runtime in the lost reply, so never admit another prompt. + // The caller receives no confirmation: Null is not a reply. + self.runtime_retired = true; + return Ok(Value::Null); + } Err(error) => return Err(self.fail_closed(error)), }; if response.get("cancelled").and_then(Value::as_bool) != Some(true) { @@ -1100,12 +1119,19 @@ impl AcpxProviderSession { "error":Value::Null, }) }; - let response = match self - .transport - .request(GeneratedAcpxSidecarCommand::ToolResolve, resolution) - { + let response = match self.transport.request_tool_resolve(resolution) { Ok(response) => response, - Err(error) => return Err(self.fail_closed(error)), + // This session's own turn.cancel retired the call before its result + // arrived, and the turn terminal may still be buffered. The provider + // can never receive this result, exactly as after a polled terminal, + // so keep the durable result instead of failing the transport. + Err((_, true)) if self.cancelled_turn_id.as_deref() == Some(turn_id.as_str()) => { + self.state = next_state; + self.tool_bridge = next_bridge; + self.reserved_tool_bridge = next_reserved_bridge; + return Ok(()); + } + Err((error, _)) => return Err(self.fail_closed(error)), }; self.verify_resolution(&response, "tool")?; self.state = next_state; @@ -2258,6 +2284,7 @@ mod mode_tests { transport_terminated: false, rejected_start_resumable: false, runtime_retired: false, + cancelled_turn_id: None, completed_turn_observed: false, generation_has_run_turn: false, restarted_after_turn_from: None, diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs index b84526d397c..fc2ff4f575f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs @@ -93,6 +93,7 @@ pub struct AcpxSidecarTransport { stderr_categories: BTreeSet<&'static str>, poisoned: bool, abandoned_observation: Option, + abandoned_cancellation: Option, } // A fresh Pi process verifies and copies its native closure before ACP admission. @@ -407,6 +408,7 @@ impl AcpxSidecarTransport { stderr_categories: BTreeSet::new(), poisoned: false, abandoned_observation: None, + abandoned_cancellation: None, }) } @@ -450,6 +452,7 @@ impl AcpxSidecarTransport { Ok(CommandOutcome::Rejected { error, resume_required, + .. }) => Err(AcpxTurnStartFailure { error, resume_required, @@ -461,6 +464,23 @@ impl AcpxSidecarTransport { } } + /// Returns `Err((error, retired))`, where `retired` is true only when the + /// sidecar rejected the result because it no longer tracks that tool call. + pub(crate) fn request_tool_resolve( + &mut self, + params: Value, + ) -> Result { + match self.request_outcome(GeneratedAcpxSidecarCommand::ToolResolve, params) { + Ok(CommandOutcome::Success(value)) => Ok(value), + Ok(CommandOutcome::Rejected { + error, + tool_call_retired, + .. + }) => Err((error, tool_call_retired)), + Err(error) => Err((error, false)), + } + } + fn request_outcome( &mut self, command: GeneratedAcpxSidecarCommand, @@ -576,6 +596,51 @@ impl AcpxSidecarTransport { } } + /// Sends `turn.cancel` and waits for its reply only until `deadline`. + /// + /// The provider's turn terminal settles a cancelled turn, not this reply, + /// and the runner cannot service `turn.stop` while it blocks here. A reply + /// lost past the bound therefore leaves the transport usable (polling still + /// owns every buffered and later frame) and retains the request ID, so a + /// late reply cannot masquerade as the answer to another request. + pub(crate) fn request_turn_cancel( + &mut self, + params: Value, + deadline: Instant, + ) -> Result, LocalRunnerError> { + if self.poisoned { + return Err(LocalRunnerError::invalid( + "ACPX sidecar transport is unavailable after a protocol failure", + )); + } + if self.abandoned_cancellation.is_some() { + // The earlier cancel is still delivered and unanswered. Its runtime + // was retired, so no later turn exists; another cancel adds nothing + // and must not block the runner again. + return Ok(None); + } + let command = GeneratedAcpxSidecarCommand::TurnCancel; + let deadline = self + .request_deadline + .map_or(deadline, |bound| bound.min(deadline)); + let result = self.send_request(command, params).and_then(|request_id| { + let result = self.receive_response(command, request_id, deadline)?; + if result.is_none() { + self.abandoned_cancellation = Some(request_id); + } + Ok(result) + }); + match result { + Ok(Some(CommandOutcome::Success(value))) => Ok(Some(value)), + Ok(Some(CommandOutcome::Rejected { error, .. })) => Err(error), + Ok(None) => Ok(None), + Err(error) => { + self.poison(); + Err(error) + } + } + } + pub(crate) fn observed_event_sequence(&self) -> u64 { self.last_event_sequence } @@ -660,6 +725,10 @@ impl AcpxSidecarTransport { self.abandoned_observation = None; continue; } + if self.abandoned_cancellation == Some(response.id) { + self.abandoned_cancellation = None; + continue; + } if response.id != request_id { return Err(LocalRunnerError::invalid(format!( "ACPX sidecar response id mismatch: expected {request_id}, received {}", @@ -676,7 +745,9 @@ impl AcpxSidecarTransport { let resume_required = command == GeneratedAcpxSidecarCommand::TurnStart && error.retryable && error.code == "SESSION_RESUME_REQUIRED"; - return Ok(Some(CommandOutcome::Rejected { error: LocalRunnerError::invalid( + let tool_call_retired = command == GeneratedAcpxSidecarCommand::ToolResolve + && error.code == "ACPX_TOOL_CALL_STALE"; + return Ok(Some(CommandOutcome::Rejected { tool_call_retired, error: LocalRunnerError::invalid( format!( "ACPX sidecar command {} was rejected (retryable={}, classification={})", command.as_str(), @@ -712,6 +783,11 @@ impl AcpxSidecarTransport { { self.abandoned_observation = None; } + ParsedFrame::Response(response) + if self.abandoned_cancellation == Some(response.id) => + { + self.abandoned_cancellation = None; + } ParsedFrame::Response(response) => { return Err(LocalRunnerError::invalid(format!( "ACPX sidecar emitted response {} without a pending request", @@ -879,6 +955,7 @@ enum CommandOutcome { Rejected { error: LocalRunnerError, resume_required: bool, + tool_call_retired: bool, }, } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs index cfba5cc3ba9..ff3d63dd1e6 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs @@ -69,6 +69,15 @@ fn run() -> Result<(), Box> { let mut source_run = String::new(); let mut source_pending_completion: Option = None; let mut source_goal_revision = 0_u64; + // `--cancel-reply lost|late`: the provider stops its prompt, but the reply + // to turn.cancel is dropped, or written only before the next request. + let cancel_reply = args + .windows(2) + .find(|pair| pair[0] == "--cancel-reply") + .map(|pair| pair[1].clone()); + let mut late_cancel_reply: Option = None; + // With --cancel-reply, a cancelled turn forgets its pending tool calls. + let mut cancel_retired_tools = false; let mut source_journal = if mode == "source-reconcile" { let path = args .windows(2) @@ -115,6 +124,9 @@ fn run() -> Result<(), Box> { .get("command") .and_then(Value::as_str) .ok_or("request command is missing")?; + if let Some(reply) = late_cancel_reply.take() { + write_json(&mut stdout, &reply)?; + } if command == "turn.start" { if let Some(fault) = args .windows(2) @@ -627,7 +639,9 @@ fn run() -> Result<(), Box> { | "suspend-wrong-ack" | "suspend-wrong-identity" | "suspend-missing-identity" => { - if command == "tool.resolve" && mode == "turns-tool-resolution-rejected" { + if command == "tool.resolve" + && (mode == "turns-tool-resolution-rejected" || cancel_retired_tools) + { write_json( &mut stdout, &json!({ @@ -666,6 +680,27 @@ fn run() -> Result<(), Box> { if command == "session.open" { opened_identity = response.pointer("/result/identity").cloned(); } + if command == "turn.cancel" { + if let Some(fault) = cancel_reply.as_deref() { + write_turn_event( + &mut stdout, + next_sequence, + "runtime.turn_terminal", + "run-1", + request + .pointer("/params/turnId") + .and_then(Value::as_str) + .unwrap(), + json!({"status":"interrupted"}), + )?; + next_sequence += 1; + cancel_retired_tools = true; + if fault == "late" { + late_cancel_reply = Some(response); + } + continue; + } + } if command == "session.suspend" { if let Some(delay_ms) = suspend_delay_ms { std::thread::sleep(Duration::from_millis(delay_ms)); diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs index f31bb5e1da5..814f53e1377 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs @@ -792,6 +792,84 @@ fn forced_cancellation_retirement_preserves_terminal_polling_but_rejects_live_re } } +#[test] +fn lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport() { + // Restart-chaos PV2 interrupt_ack: the provider stops its prompt, but the + // sidecar's reply to turn.cancel is lost or late. The terminal must still + // settle the turn, and a late reply must not break the next request. + for mode in ["lost", "late"] { + let mut config = config("turns"); + config + .transport + .args + .extend(["--cancel-reply".to_owned(), mode.to_owned()]); + let mut session = AcpxProviderSession::start(&config).unwrap(); + session + .start_turn("turn-1", "Please help", &std::env::temp_dir()) + .unwrap(); + let started = Instant::now(); + session + .interrupt_turn("turn-1", "Paperclip interruption") + .unwrap_or_else(|error| panic!("{mode}: {error}")); + assert!(started.elapsed() < Duration::from_secs(3), "{mode}"); + assert!(session.runtime_retired(), "{mode}"); + // A repeated Stop while the first cancel is unanswered must not block. + let repeated = Instant::now(); + session + .interrupt_turn("turn-1", "Paperclip interruption again") + .unwrap_or_else(|error| panic!("{mode} repeat: {error}")); + assert!(repeated.elapsed() < Duration::from_millis(200), "{mode}"); + let mut terminal = None; + while terminal.is_none() { + let events = session.poll_event(Duration::from_secs(1)).unwrap().unwrap(); + terminal = events.into_iter().find(|event| { + matches!(event, AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1") + }); + } + assert!(session.state().active_turn_id().is_none(), "{mode}"); + assert!(session + .start_turn("turn-2", "Must not reuse", &std::env::temp_dir()) + .unwrap_err() + .to_string() + .contains("runtime was retired")); + session + .shutdown("test complete") + .unwrap_or_else(|error| panic!("{mode}: {error}")); + } +} + +#[test] +fn a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport() { + // Restart-chaos S1 PV2: Stop lands while a semantic tool runs. The cancel + // retires the call in the sidecar before its result arrives, and the turn + // terminal is still buffered. The result must be kept, not fail the session. + let mut config = config("turns-tool"); + config + .transport + .args + .extend(["--cancel-reply".to_owned(), "lost".to_owned()]); + let mut session = AcpxProviderSession::start(&config).unwrap(); + session + .start_turn("turn-1", "Please help", &std::env::temp_dir()) + .unwrap(); + let tool = session.poll_event(Duration::from_secs(1)).unwrap().unwrap(); + assert!( + matches!(&tool[0], AcpxProviderStateEvent::ToolCall { call_id, .. } if call_id == "call-1") + ); + session + .interrupt_turn("turn-1", "Cancelled by a board operator") + .unwrap(); + session + .deliver_tool_result(&late_result("call-1")) + .expect("a call retired by our own cancellation keeps its result"); + assert!(session.state().pending_tool("call-1").is_none()); + let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap(); + assert!( + matches!(&terminal[0], AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1") + ); + session.shutdown("test complete").unwrap(); +} + fn reusable_config(directory: &std::path::Path) -> AcpxProviderSessionConfig { let mut config = config("source-reconcile"); config.transport.args = vec![ diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs index bb8cbb6946a..d6163af3de7 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs @@ -577,3 +577,116 @@ fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() { assert_eq!(saved["launchProfileDigest"], upgraded_digest); restored.shutdown().unwrap(); } + +#[test] +fn a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner() { + // Restart-chaos PV2 interrupt_ack on Claude: the sidecar's reply to + // turn.cancel is lost. The runner must stay responsive, publish the + // provider's own terminal, and reach a proven suspension. Before the fix it + // blocked for the 30 s request bound, then failed the transport, and + // runner.suspend refused because provider exit was unconfirmed. + let fixture = + Fixture(std::env::temp_dir().join(format!("acpx-cancel-lost-{}", uuid::Uuid::new_v4()))); + fs::create_dir_all(&fixture.0).unwrap(); + fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap(); + let sidecar = fixture.0.join("sidecar"); + fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap(); + fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap(); + let args = vec![ + "--mode".to_owned(), + "turns".to_owned(), + "--profile-digest".to_owned(), + PROFILE_DIGEST.to_owned(), + "--cancel-reply".to_owned(), + "lost".to_owned(), + ]; + let config = DurableRunnerConfig { + connect_url: "ws://127.0.0.1/runner".to_owned(), + ca_bundle_path: None, + state_dir: fixture.0.clone(), + runner_instance_id: "runner-1".to_owned(), + environment_lease_id: "lease-1".to_owned(), + run_id: "run-1".to_owned(), + normalized_session_id: "session-1".to_owned(), + turn_id: "turn-1".to_owned(), + item_id: "item-1".to_owned(), + runner_version: "0.0.0".to_owned(), + runner_digest: format!("sha256:{}", "a".repeat(64)), + acpx_launch_profile: Some(AcpxLaunchProfile { + candidate_admission: None, + authority_digest: format!("sha256:{}", "d".repeat(64)), + command: sidecar.clone(), + args: args.clone(), + artifacts: vec![QualifiedLaunchArtifact { + path: sidecar.clone(), + sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())), + }], + }), + opencode_launch_profile: None, + cursor_cloud_launch_profile: None, + max_outbox_bytes: 1024 * 1024, + p0_reserve_bytes: 64 * 1024, + max_frame_bytes: 1024 * 1024, + reconnect_delay: Duration::from_millis(1), + reconnect_grace: None, + max_runtime: Duration::from_secs(60), + turn_scoped: false, + }; + let descriptor = json!({ + "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime", + "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol", + "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp", + "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex", + "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST, + "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0, + "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0, + "instructions": "Complete the supplied work.", + "permissionMode": "approve-all", "permissionModePinned": true, + }); + let mut executor = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + executor + .execute(&command(1, "run.prepare", json!({"provider": descriptor}))) + .unwrap(); + executor + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + executor + .execute(&command( + 3, + "turn.start", + json!({"turnId": "provider-turn-1", "text": "Ask one question."}), + )) + .unwrap(); + let started = std::time::Instant::now(); + let interrupted = executor + .execute(&command(4, "turn.interrupt", json!({}))) + .unwrap(); + assert_eq!(interrupted.result["status"], "interrupt_requested"); + assert!(started.elapsed() < Duration::from_secs(10)); + let polled_until = std::time::Instant::now() + Duration::from_secs(5); + let mut events = executor.poll_events().unwrap(); + while !events + .iter() + .any(|event| event.event_type == "run.terminal") + { + assert!(std::time::Instant::now() < polled_until, "no turn terminal"); + events.extend(executor.poll_events().unwrap()); + } + assert!(events + .iter() + .any(|event| event.event_type == "turn.interrupted")); + assert!(!events.iter().any(|event| event.event_type == "turn.failed")); + let terminal = events + .iter() + .find(|event| event.event_type == "run.terminal") + .unwrap(); + assert_eq!(terminal.payload["runTerminalState"], "cancelled"); + executor + .execute(&command(5, "turn.stop", json!({}))) + .unwrap(); + let suspended = executor + .execute(&command(6, "runner.suspend", json!({}))) + .unwrap(); + assert_eq!(suspended.result["status"], "completed"); + executor.shutdown().unwrap(); +}