From d4498ae7ac9d35b5effa7fcc7a9073aabdb30744 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 14:24:21 -0600 Subject: [PATCH] fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call Paperclip tools through the Runner. The Runner records each call and returns the controller's result to the provider (here OpenCode) > - In chat-eval cohort 13, a `list_workflows` result was recorded but never reached OpenCode. OpenCode waited about 290 s, and the call stayed pending > - The next run's `run.attach` then failed with `durable_pending_tool_calls`, so the person's follow-up question never got an answer > - This pull request fixes the result shape that caused the rejection, makes the Runner settle every recorded result, and lets the next run settle a call that a closed turn left pending > - The benefit is that a tool result always reaches the agent, and a stuck call can no longer block every later turn ## Linked Issues or Issue Description **What happened?** Chat-eval cohort 13 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In `cases.case-history` and `cases.create-case`, the CEO called `list_documents` (or `list_projects`) and `list_workflows` (call id 5) together. The server answered `GET /companies/{id}/workflows` with 200 in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome: "unanswered"`, `written: false`, `handlerMs: 289038` (290723 in create-case). The sibling call was answered in about 1 s. The run ended with `runner_settlement_incomplete`: `semanticTools.pending` held call 5 and its `semantic_tool.result` delivery command had status `failed`. The next question started a run whose `run.attach` was refused: `run.attach requires a settled Codex provider session: durable_pending_tool_calls`. The run failed in 17 s, and the question was not answered. Cause: `executeNativeWorkflowTool` returned the Runner API result unchanged: `{ ok: true, status: 200, operationId: "GET /api/companies/{companyId}/workflows", ... }`. Every other API-backed tool renames that field to `apiOperationId`. The Rust tool bridge (`semantic_response_value`) reads an object with a boolean `ok` and an `operationId` as a semantic envelope that must name this tool and call. The REST route did not match, so `apply_result` failed with `semantic result envelope does not match its provider call`. Nothing was sent to the provider, and the call stayed pending forever. The red test reproduces this exact error. The parallel call is not the trigger: the sibling call settled normally, and the OpenCode MCP bridge answered other parallel pairs in the same runs. `list_workflows` first appears in cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools had this shape. **Expected behavior** Every recorded tool result reaches the provider. When a recorded result cannot be delivered, the provider gets an error at once and the call settles. A call that a closed turn left pending does not block the next run. **Steps to reproduce** 1. Run an OpenCode agent with Cases enabled on the durable Runner. 2. Ask a question that makes the agent call `list_workflows`. 3. See the call wait about 290 s. Then ask a follow-up question and see `run.attach` fail with `durable_pending_tool_calls`. **Paperclip version or commit** `master` at `df1b12ad8e`. ## What Changed - `server/src/services/native-runtime/native-workflow-tools.ts`: workflow tool results name the route `apiOperationId`, like case tools and `call_api`. - `provider_bridge.rs`: add `settle_undeliverable_result`. When the bridge rejects a recorded result for a call it still holds, the call settles with a bounded, non-retryable runner error. Conflicting or unknown results still fail as before. - `provider_backend.rs` (`deliver_semantic_result`): use that settlement, deliver the error to the provider, and record a `semantic_tool_result_undeliverable` diagnostic with the call, operation, reason and the rejected result's digest. - `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add `expire_closed_turn_calls`. When nothing is live (no active or ambiguous turn, no provider request waiting), `run.attach` settles calls that a closed turn left pending as an expired failure with an unknown outcome, and records `semantic_tool_result_expired`. The provider stopped waiting when its turn ended, and the new run's controller never delivers the old run's results. A call that a live turn still waits on still blocks attachment. This follows the existing `tool_call_interrupted_by_restart` rule: never execute the operation again, and record an unknown outcome when no result can ever arrive. - Docs: `doc/run-log-events.md` and `packages/paperclip-runner/docs/durable-recovery.md` describe both diagnostics. ## Verification - Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --test codex_provider -- a_rejected_recorded_result run_attach_expires`): - Before: `a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session` fails with `semantic result envelope does not match its provider call` (the production error). `run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with `run.attach requires a settled Codex provider session: durable_pending_tool_calls` (the production refusal). - After: both pass. The fake provider logs `tool-response:failure`, so it got an answer at once. The next `run.attach` succeeds. - Parallel calls (`cargo test -p paperclip-runner-core --lib -- provider_bridge`): `a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends the cohort 13 pair (`list_documents` + `list_workflows`). The rejected call settles, its sibling keeps its own result, and the next turn starts. `only_a_closed_turn_expires_its_pending_calls` checks that an open turn expires nothing. - Upgrade and mid-run cutover: `run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state with a pending call after the turn ends, stops that Runner, and attaches with a new Runner process from the saved state. `durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended` (unchanged) still passes: a late result in the same run still settles as `settled_after_turn`. - Red/green, server (`cd server && npx vitest run src/services/native-runtime/native-case-tools.integration.test.ts -t workflows`): before, `expected { ok: true, status: 201, … } to not have property "operationId"`; after, 2 passed. - `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`) failed once with `EPERM` while signaling a process group under parallel load. It passes alone 3 of 3 and does not touch this change. The integration test binaries pass. - Paid narrow run for `cases.case-history` and `cases.create-case`: see the comment below. ## Risks - Previous-release story: no persisted shape changes. State saved by the previous Runner with a call pending after its turn closed carries forward. The new Runner settles that call at the next `run.attach`, and the session continues (tested from saved state with a new process). A previous-release Runner that receives the corrected `apiOperationId` result delivers it normally, because the result is no longer read as an envelope. - Behavior change: a call left by a closed turn is now settled at the next run's attachment, not kept forever. Its outcome is recorded as unknown, and the operation is never run again. A provider turn that is still live keeps the old refusal (`durable_pending_tool_calls`, `provider_side_effect_pending`), and the controller still never replaces the session. - A recorded result that the bridge rejects now becomes an error for the agent, not an endless wait. The agent is told the action may have completed and must check the current state before repeating it. - Latency: nothing is added before turn start. The attach-time check walks the in-memory pending calls (usually none) and adds no I/O beyond the existing state save. The rejected-delivery path now answers in the same command instead of after the provider's ~290 s client timeout, and the follow-up run no longer fails after 17 s. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21398 (first release where `list_workflows` was called in this eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery fix). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/run-log-events.md | 17 ++ .../paperclip-runner/docs/durable-recovery.md | 9 + .../runner-core/src/provider_backend.rs | 136 +++++++++--- .../crates/runner-core/src/provider_bridge.rs | 201 ++++++++++++++++++ .../runner-core/tests/codex_provider.rs | 170 +++++++++++++++ .../native-case-tools.integration.test.ts | 17 +- .../native-runtime/native-workflow-tools.ts | 13 +- 7 files changed, 534 insertions(+), 29 deletions(-) diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 26b84266d8f..2da5478d001 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -359,6 +359,23 @@ attention request. A true conflict includes the call and operation IDs and both result digests in its error; full arguments belong to the canonical semantic input record, not the error message. +When the tool bridge rejects a recorded result for a call it still holds, for +example a result envelope that names another operation or a value over the +retained limit, the runner still settles that call. It returns a bounded, +non-retryable error to the provider and records +`semantic_tool_result_undeliverable` with warning severity, call ID, operation +ID, the rejection reason, and the rejected result's digest. The provider is not +left waiting for its own client timeout, and the call does not block later +turns or runs. + +`run.attach` settles calls that a closed provider turn left pending. The +provider stopped waiting when that turn ended, and the new run's controller +never delivers the old run's results. Each such call is recorded as a +non-retryable failure with an unknown outcome, and the attachment records one +`semantic_tool_result_expired` diagnostic with the call and operation IDs. A +call that a live provider turn still waits on keeps blocking attachment as +`durable_pending_tool_calls`. + Incomplete close emits the bounded `native_session_settlement_incomplete` runner diagnostic. Its evidence includes runner suspension, provider drain, final provider state, pending call/operation/source-event IDs and input digests, diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 7ea574589c3..befe1482b09 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -700,6 +700,15 @@ result replay as `semantic_tool_result_duplicate` in a diagnostic event, with no task failure. Conflict errors identify the call, operation, and both result digests without including the result bodies. +A recorded result that the bridge rejects for a call it still holds settles +that call with a non-retryable runner error, recorded as +`semantic_tool_result_undeliverable`; the provider receives that error +immediately. `run.attach` settles calls left pending by a closed provider turn +as `semantic_tool_result_expired`, an error result with an unknown outcome: the +provider stopped waiting when its turn ended, and the next run's authority +never delivers the previous run's results. Neither settlement executes the +operation again. + The controller checkpoints a reusable session only after every admitted tool has a matching completed result-delivery command and the provider has drained. A callback rejected before business dispatch can report diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index d364d83b045..7e811ce6c7d 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -27,8 +27,8 @@ use crate::durable::{ }; use crate::provider_bridge::{ authorized_tool_catalog_digest, semantic_value_digest, AuthorizedToolSet, DurableReplayFilter, - PendingToolCall, ProviderBridgeError, ProviderToolBridge, ToolResult, MAX_PENDING_CALLS, - TOOL_SET_SCHEMA, + PendingToolCall, ProviderBridgeError, ProviderToolBridge, ToolResult, EXPIRED_RESULT_CODE, + MAX_PENDING_CALLS, TOOL_SET_SCHEMA, UNDELIVERABLE_RESULT_CODE, }; use crate::provider_events::{ codex_context_usage, normalize_codex_notification, normalize_codex_notification_in, @@ -3443,7 +3443,9 @@ impl CodexCommandExecutor { }) } - fn attach_run(&mut self, payload: &Value) -> Result<(), DurableRunnerError> { + /// Returns the calls a closed turn left pending that this attachment + /// settled as expired. + fn attach_run(&mut self, payload: &Value) -> Result, DurableRunnerError> { self.retain_completed_accounting_tail()?; let mut next_state = self .state @@ -3473,6 +3475,23 @@ impl CodexCommandExecutor { if next_state.ambiguous_turn_start_pending { blockers.push("durable_ambiguous_turn_start"); } + // A call left pending by a closed provider turn can never be answered: + // the provider stopped waiting when its turn ended, and the new run's + // controller never delivers the old run's results. Record it as an + // expired failure with an unknown outcome so the session continues. + // A call the live provider still waits on keeps blocking attachment. + let expired_tool_calls = if blockers.is_empty() { + next_state + .tool_bridge + .expire_closed_turn_calls() + .map_err(|error| { + DurableRunnerError::invalid(format!( + "run.attach could not settle tool calls left by a closed turn: {error}" + )) + })? + } else { + Vec::new() + }; if next_state.tool_bridge.pending_calls().next().is_some() { blockers.push("durable_pending_tool_calls"); } @@ -3669,7 +3688,7 @@ impl CodexCommandExecutor { }; self.persist_state(&next_state)?; self.state = Some(next_state); - Ok(()) + Ok(expired_tool_calls) } fn open_session(&mut self) -> Result { @@ -5002,15 +5021,54 @@ impl CodexCommandExecutor { .pending_calls() .find(|call| call.call_id == result.call_id) .map(|call| (call.operation_id.clone(), call.input.clone())); - next_state - .tool_bridge - .apply_result(result.clone()) - .map_err(|error| { - DurableRunnerError::invalid(format!( + // A recorded result that this bridge rejects must still settle its + // pending call. Otherwise the provider waits until its own client + // times out, and the call blocks every later turn and run. + let mut undeliverable = None; + let result = match next_state.tool_bridge.apply_result(result.clone()) { + Ok(_) => result, + Err(error) => { + let rejected = DurableRunnerError::invalid(format!( "semantic tool result was rejected for call {} operation {}: {error}", result.call_id, result.operation_id - )) - })?; + )); + if was_completed { + // An exact replay of a delivery whose result was already + // rejected and settled (for example after a crash before + // the command finished) is a duplicate, not a conflict. + if next_state + .tool_bridge + .is_settled_undeliverable_replay(&result) + { + return Ok(CommandExecution { + result: json!({ "status": "duplicate", "callId": result.call_id }), + events: vec![( + "harness.diagnostic".to_owned(), + EventPriority::P0, + json!({ + "code": "semantic_tool_result_duplicate", "severity": "warning", + "callId": result.call_id, "operationId": result.operation_id, + "resultDigest": semantic_value_digest(&result.result), + }), + )], + }); + } + return Err(rejected); + } + match next_state.tool_bridge.settle_undeliverable_result(&result) { + Ok(Some(settlement)) => { + undeliverable = Some(json!({ + "code": UNDELIVERABLE_RESULT_CODE, "severity": "warning", + "callId": result.call_id, "operationId": settlement.operation_id, + "reason": error.to_string(), + "resultDigest": semantic_value_digest(&result.result), + })); + settlement + } + _ => return Err(rejected), + } + } + }; if was_completed { return Ok(CommandExecution { result: json!({ "status": "duplicate", "callId": result.call_id }), @@ -5041,11 +5099,24 @@ impl CodexCommandExecutor { next_state.push_event(semantic_result_event(&identity, &result))?; self.persist_state(&next_state)?; self.state = Some(next_state); + let diagnostics = undeliverable + .map(|diagnostic| { + ( + "harness.diagnostic".to_owned(), + EventPriority::P0, + diagnostic, + ) + }) + .into_iter() + .collect::>(); if delivery_detached { - return Ok(CommandExecution::result(json!({ - "status": "settled_after_turn", - "callId": result.call_id, - }))); + return Ok(CommandExecution { + result: json!({ + "status": "settled_after_turn", + "callId": result.call_id, + }), + events: diagnostics, + }); } let provider = self.ensure_provider()?; if terminal_tool_authoritative { @@ -5062,10 +5133,13 @@ impl CodexCommandExecutor { "failed to return semantic tool result to Codex: {error}" )) })?; - Ok(CommandExecution::result(json!({ - "status": "delivered", - "callId": result.call_id, - }))) + Ok(CommandExecution { + result: json!({ + "status": "delivered", + "callId": result.call_id, + }), + events: diagnostics, + }) } fn stop_owned_provider(&mut self) -> Result<(), DurableRunnerError> { @@ -5915,12 +5989,26 @@ impl CommandExecutor for CodexCommandExecutor { match command.command_type.as_str() { "run.prepare" => self.prepare(&command.payload), "run.attach" => { - if self.state.is_none() && command.payload.get("provider").is_some() { - self.prepare(&command.payload)?; - } else { - self.attach_run(&command.payload)?; - } + let expired = + if self.state.is_none() && command.payload.get("provider").is_some() { + self.prepare(&command.payload)?; + Vec::new() + } else { + self.attach_run(&command.payload)? + }; let mut execution = self.open_session()?; + if !expired.is_empty() { + execution.events.push(( + "harness.diagnostic".to_owned(), + EventPriority::P0, + json!({ + "code": EXPIRED_RESULT_CODE, "severity": "warning", + "calls": expired.iter().map(|call| json!({ + "callId": call.call_id, "operationId": call.operation_id, + })).collect::>(), + }), + )); + } let provider = self .state .as_ref() diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs index e9e99c8c220..b1ab2939aad 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs @@ -897,6 +897,72 @@ impl ProviderToolBridge { self.pending.values() } + /// Settle a pending call whose recorded result this bridge rejected, such + /// as a malformed envelope or an oversized value. The provider then gets a + /// bounded error now instead of waiting on a call nothing can answer, and + /// the call stops blocking later turns. Returns None when the call is not + /// pending, so a conflicting or unknown result still fails as before. + pub fn settle_undeliverable_result( + &mut self, + rejected: &ToolResult, + ) -> Result, ProviderBridgeError> { + let Some(pending) = self.pending.get(&rejected.call_id) else { + return Ok(None); + }; + let mut settlement = runner_tool_failure( + pending, + UNDELIVERABLE_RESULT_CODE, + "Paperclip ran this tool, but its result could not be returned to you. The action may already have completed; do not repeat it. Check the current state with a read tool instead.", + ); + // Bind the settlement to the exact rejected result, so a replay of the + // same delivery after a crash is recognized as a duplicate. + settlement.result["error"]["rejectedResultDigest"] = + Value::String(rejected_result_digest(rejected)); + self.apply_result(settlement.clone())?; + Ok(Some(settlement)) + } + + /// True when this exact result was already rejected and its call settled + /// by `settle_undeliverable_result`. + pub fn is_settled_undeliverable_replay(&self, result: &ToolResult) -> bool { + self.completed + .get(&result.call_id) + .or_else(|| self.settled_results.get(&result.call_id)) + .is_some_and(|entry| { + let error = &entry.result.result["error"]; + entry.result.is_error + && entry.result.operation_id == result.operation_id + && error["code"] == UNDELIVERABLE_RESULT_CODE + && error["rejectedResultDigest"] + == Value::String(rejected_result_digest(result)) + }) + } + + /// Settle the calls a closed provider turn left pending. Their provider + /// turn ended without them, and a new run's authority can never deliver + /// the old run's result, so they would block every later run. Each is + /// recorded as a timed-out failure with an unknown outcome. + pub fn expire_closed_turn_calls(&mut self) -> Result, ProviderBridgeError> { + if !self.turn_closed { + return Ok(Vec::new()); + } + let expired = self + .pending + .values() + .map(|pending| { + runner_tool_failure( + pending, + EXPIRED_RESULT_CODE, + "This Paperclip tool call timed out before its result reached you. Its outcome is unknown: check the current state before repeating any change.", + ) + }) + .collect::>(); + for settlement in &expired { + self.apply_result(settlement.clone())?; + } + Ok(expired) + } + /// Close provider admission/delivery, without inventing an operation outcome. /// This marker is durable, including when interruption beats result delivery. pub fn cancel_pending_calls( @@ -1307,6 +1373,26 @@ where deserializer.deserialize_map(RetainedResultsVisitor) } +pub const UNDELIVERABLE_RESULT_CODE: &str = "semantic_tool_result_undeliverable"; +pub const EXPIRED_RESULT_CODE: &str = "semantic_tool_result_expired"; + +fn rejected_result_digest(result: &ToolResult) -> String { + semantic_value_digest(&serde_json::json!({ + "operationId": result.operation_id, "isError": result.is_error, "result": result.result, + })) +} + +fn runner_tool_failure(pending: &PendingToolCall, code: &str, message: &str) -> ToolResult { + ToolResult { + call_id: pending.call_id.clone(), + operation_id: pending.operation_id.clone(), + result: serde_json::json!({ + "error": { "code": code, "retryable": false, "message": message }, + }), + is_error: true, + } +} + fn conflicting_tool_result(existing: &ToolResult, incoming: &ToolResult) -> ProviderBridgeError { // Correlate both receipts without leaking arguments or result content into // a user-visible error or diagnostic stream. @@ -2262,6 +2348,121 @@ mod tests { .expect("a new durable run resets the full legacy ledger"); } + fn parallel_read_bridge() -> ProviderToolBridge { + let operations = ["list_documents", "list_workflows"] + .map(|operation_id| AuthorizedTool { + operation_id: operation_id.to_owned(), + version: 1, + description: "Read company records.".to_owned(), + input_schema: json!({"type": "object"}), + response_schema: json!({"type": "object", "additionalProperties": true}), + }) + .to_vec(); + let mut bridge = ProviderToolBridge::default(); + bridge + .prepare(AuthorizedToolSet { + schema: TOOL_SET_SCHEMA.to_owned(), + schema_version: 1, + catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(), + operations, + }) + .unwrap(); + for (call_id, operation_id) in [("4", "list_documents"), ("5", "list_workflows")] { + bridge + .begin_call(call_id.to_owned(), operation_id.to_owned(), json!({})) + .unwrap(); + } + bridge + } + + #[test] + fn a_rejected_result_in_a_parallel_pair_still_settles_its_own_call() { + // Chat-eval cohort 13 issued list_documents and list_workflows together. + // The workflow result named its REST route as `operationId`, which the + // bridge reads as a mismatched semantic envelope. + let mut bridge = parallel_read_bridge(); + bridge + .apply_result(ToolResult { + call_id: "4".to_owned(), + operation_id: "list_documents".to_owned(), + result: json!({"documents": []}), + is_error: false, + }) + .unwrap(); + let rejected = ToolResult { + call_id: "5".to_owned(), + operation_id: "list_workflows".to_owned(), + result: json!({"ok": true, "status": 200, + "operationId": "GET /api/companies/{companyId}/workflows", "data": []}), + is_error: false, + }; + let error = bridge.apply_result(rejected.clone()).unwrap_err(); + assert!(error.to_string().contains("envelope does not match")); + assert_eq!(bridge.pending_calls().count(), 1); + + let settlement = bridge + .settle_undeliverable_result(&rejected) + .unwrap() + .expect("the pending call settles"); + assert!(settlement.is_error); + assert_eq!(settlement.operation_id, "list_workflows"); + assert_eq!( + settlement.result["error"]["code"], + UNDELIVERABLE_RESULT_CODE + ); + assert_eq!(bridge.pending_calls().count(), 0); + // Its sibling keeps its own result, and a settled call is not settled twice. + assert_eq!( + bridge + .replay_result("4", "list_documents", &json!({})) + .unwrap() + .unwrap() + .result, + json!({"documents": []}) + ); + assert!(bridge + .settle_undeliverable_result(&rejected) + .unwrap() + .is_none()); + assert!(bridge.is_settled_undeliverable_replay(&rejected)); + let mut different = rejected.clone(); + different.result["data"] = json!(["other"]); + assert!(!bridge.is_settled_undeliverable_replay(&different)); + bridge.settle_turn("provider_turn_terminated").unwrap(); + assert!(bridge.is_settled_undeliverable_replay(&rejected)); + bridge.prepare_turn().expect("the next turn starts"); + } + + #[test] + fn only_a_closed_turn_expires_its_pending_calls() { + let mut bridge = parallel_read_bridge(); + assert!(bridge.expire_closed_turn_calls().unwrap().is_empty()); + assert_eq!(bridge.pending_calls().count(), 2); + + bridge.settle_turn("provider_turn_terminated").unwrap(); + let expired = bridge.expire_closed_turn_calls().unwrap(); + assert_eq!( + expired + .iter() + .map(|call| call.call_id.as_str()) + .collect::>(), + ["4", "5"] + ); + assert!(expired + .iter() + .all(|call| call.is_error && call.result["error"]["code"] == EXPIRED_RESULT_CODE)); + assert_eq!(bridge.pending_calls().count(), 0); + let operations = bridge.authorized_tools().cloned().collect::>(); + bridge + .attach_run(AuthorizedToolSet { + schema: TOOL_SET_SCHEMA.to_owned(), + schema_version: 1, + catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(), + operations, + }) + .expect("the next run attaches"); + } + #[test] fn active_receipts_that_block_the_next_turn_are_named() { let mut bridge = completion_bridge(); diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs index 629f117c7d1..225108ceba3 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs @@ -5853,6 +5853,176 @@ fn durable_backend_settles_tools_before_a_natural_terminal_event() { fs::remove_dir_all(directory).unwrap(); } +fn start_turn_with_pending_tool(executor: &mut CodexCommandExecutor, config: CodexProviderConfig) { + executor + .execute(&command( + "prepare", + 1, + "run.prepare", + json!({"provider": config, "authorizedTools": task_context_tool_set()}), + )) + .unwrap(); + executor + .execute(&command("open", 2, "session.open", json!({}))) + .unwrap(); + executor + .execute(&command( + "turn", + 3, + "turn.start", + json!({"text": "Call the tool."}), + )) + .unwrap(); +} + +#[test] +fn a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session() { + // Chat-eval cohort 13: the controller recorded a REST-shaped result whose + // `{ ok, operationId }` the bridge reads as a mismatched semantic envelope. + // The rejected delivery left OpenCode waiting ~290 s and the call pending, + // so the next run's run.attach failed with durable_pending_tool_calls. + let directory = temporary_directory("rejected-recorded-result"); + let config = provider_config(&directory, &["--require-dynamic-tool", "--emit-tool-call"]); + let mut executor = + CodexCommandExecutor::with_runner_config(&directory, &durable_config(&directory)); + start_turn_with_pending_tool(&mut executor, config); + let deadline = event_deadline(); + let mut input_seen = false; + while !input_seen && std::time::Instant::now() < deadline { + input_seen = poll_and_ack(&mut executor) + .unwrap() + .iter() + .any(|event| event.event_type == "semantic_tool.input"); + } + assert!(input_seen, "the provider requests its tool"); + + let rejected = json!({ + "callId": "semantic-call-1", "operationId": "get_task_context", "isError": false, + "result": {"ok": true, "status": 200, "operationId": "GET /api/companies/{companyId}/workflows", + "contentType": "application/json; charset=utf-8", "data": []}, + }); + let delivered = executor + .execute(&command( + "result", + 4, + "semantic_tool.result", + rejected.clone(), + )) + .expect("a recorded result must settle its call even when the bridge rejects it"); + assert_eq!(delivered.result["status"], "delivered"); + let mut observed = Vec::new(); + let deadline = event_deadline(); + while !observed.iter().any(|event: &PolledEvent| { + event.event_type == "turn.failed" || event.event_type == "turn.completed" + }) && std::time::Instant::now() < deadline + { + observed.extend(poll_and_ack(&mut executor).unwrap()); + } + let diagnostic = delivered + .events + .iter() + .find(|(_, _, payload)| payload["code"] == "semantic_tool_result_undeliverable") + .map(|(_, _, payload)| payload) + .expect("the rejected delivery is named"); + assert_eq!(diagnostic["callId"], "semantic-call-1"); + assert!(diagnostic["reason"] + .as_str() + .unwrap() + .contains("envelope does not match")); + let settled = observed + .iter() + .find(|event| event.event_type == "semantic_tool.result") + .expect("the call has a recorded settlement"); + assert_eq!(settled.payload["semantic_tool"]["outcome"], "failed"); + let calls = fs::read_to_string(directory.join("calls.log")).unwrap(); + assert!( + calls.contains("tool-response:failure"), + "the provider must receive an answer, not wait for its client timeout: {calls}" + ); + + // A crash before the command finished replays the same delivery. + let replay = executor + .execute(&command( + "result-replay", + 5, + "semantic_tool.result", + rejected, + )) + .expect("an exact replay of the rejected delivery is a duplicate"); + assert_eq!(replay.result["status"], "duplicate"); + + executor + .execute(&command( + "attach", + 6, + "run.attach", + json!({"authorizedTools": task_context_tool_set()}), + )) + .expect("the next run attaches to the same session"); + executor.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +fn run_attach_expires_a_call_left_pending_by_a_closed_turn() { + // The provider stopped waiting when its turn ended, and the next run's + // controller never delivers the old run's result. Attachment records the + // call as expired instead of refusing every later run. + let directory = temporary_directory("closed-turn-pending-attach"); + let config = provider_config( + &directory, + &[ + "--require-dynamic-tool", + "--emit-tool-call", + "--finish-turn-with-pending-tool", + ], + ); + let runner_config = durable_config(&directory); + let mut first = CodexCommandExecutor::with_runner_config(&directory, &runner_config); + start_turn_with_pending_tool(&mut first, config); + let mut observed = Vec::new(); + let deadline = event_deadline(); + while !observed.iter().any(|event| event == "turn.completed") + && std::time::Instant::now() < deadline + { + observed.extend( + poll_and_ack(&mut first) + .unwrap() + .into_iter() + .map(|event| event.event_type), + ); + } + assert!(observed.iter().any(|event| event == "semantic_tool.input")); + assert!(!observed.iter().any(|event| event == "semantic_tool.result")); + // A deploy lands here: the next Runner restores this saved state. + first.shutdown().unwrap(); + drop(first); + + let mut resumed = CodexCommandExecutor::with_runner_config(&directory, &runner_config); + let attached = resumed + .execute(&command( + "attach", + 4, + "run.attach", + json!({"authorizedTools": task_context_tool_set()}), + )) + .expect("a call left by a closed turn must not block the next run"); + let expired = attached + .events + .iter() + .find(|(event_type, _, payload)| { + event_type == "harness.diagnostic" && payload["code"] == "semantic_tool_result_expired" + }) + .map(|(_, _, payload)| payload.clone()) + .expect("the expired call is named"); + assert_eq!( + expired["calls"], + json!([{"callId": "semantic-call-1", "operationId": "get_task_context"}]) + ); + resumed.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + #[test] fn durable_terminal_delivery_reconciliation_does_not_launch_a_cold_provider() { let directory = temporary_directory("cold-terminal-delivery"); diff --git a/server/src/services/native-runtime/native-case-tools.integration.test.ts b/server/src/services/native-runtime/native-case-tools.integration.test.ts index 16e85ab18a5..ed18e6a9930 100644 --- a/server/src/services/native-runtime/native-case-tools.integration.test.ts +++ b/server/src/services/native-runtime/native-case-tools.integration.test.ts @@ -154,7 +154,16 @@ describe("native case tools through authenticated production routes", () => { expect(created).toMatchObject({ ok: true, status: 201, data: { name: input.name } }); expect(created.data).not.toHaveProperty("prompt"); expect(await f.call("create_workflow", input)).toEqual(created); - expect(await f.call("list_workflows")).toMatchObject({ ok: true, data: [{ id: created.data.id, name: input.name }] }); + const listed = await f.call("list_workflows"); + expect(listed).toMatchObject({ ok: true, data: [{ id: created.data.id, name: input.name }] }); + // The runner reads `{ ok, operationId }` as a semantic envelope for this + // exact tool and call. A REST route there makes the runner reject the + // result, so the agent never receives it (chat-eval cohort 13). + for (const [result, route] of [[created, "POST /api/companies/{companyId}/workflows"], + [listed, "GET /api/companies/{companyId}/workflows"]] as const) { + expect(result).not.toHaveProperty("operationId"); + expect(result).toMatchObject({ apiOperationId: route }); + } let offsetBytes = 0, text = "", snapshot: string | undefined; for (let i = 0; ; i++) { expect(i).toBeLessThan(20); @@ -162,6 +171,8 @@ describe("native case tools through authenticated production routes", () => { ok: boolean; data: string; snapshot: string; responseText: { nextOffsetBytes: number | null }; }; expect(page.ok).toBe(true); + expect(page).toMatchObject({ apiOperationId: "GET /api/workflows/{id}" }); + expect(page).not.toHaveProperty("operationId"); expect(page.snapshot).toMatch(/^[a-f0-9]{64}$/); snapshot ??= page.snapshot; expect(page.snapshot).toBe(snapshot); @@ -175,7 +186,9 @@ describe("native case tools through authenticated production routes", () => { workflowId: created.data.id, kickoff: { toAgentId: f.agentId, phase: "custom phase", instructions: "Read workflow" } }) as { taskId: string; caseId: string }; expect((await documentService(server.db).getIssueDocumentByKey(receipt.taskId, "workflow"))?.body).toBe(prompt); await expect(f.call("read_workflow", { workflowId: created.data.id, responseText: { offsetBytes: 8192 } })).rejects.toThrow("Invalid tool input for read_workflow: A nonzero responseText.offsetBytes continues an earlier read: also send the snapshot it returned"); - expect(await f.call("update_workflow", { workflowId: created.data.id, idempotencyKey: "workflow-edit", prompt: prompt.replaceAll("Reusable", "Adjusted") })).toMatchObject({ ok: true }); + const updated = await f.call("update_workflow", { workflowId: created.data.id, idempotencyKey: "workflow-edit", prompt: prompt.replaceAll("Reusable", "Adjusted") }); + expect(updated).toMatchObject({ ok: true, apiOperationId: "PATCH /api/workflows/{id}" }); + expect(updated).not.toHaveProperty("operationId"); expect(await f.call("read_workflow", { workflowId: created.data.id, snapshot, responseText: { offsetBytes: 8192 } })).toMatchObject({ ok: false, status: 409 }); const restarted = await f.call("read_workflow", { workflowId: created.data.id, responseText: { limitBytes: 4 } }) as { ok: boolean; data: string; snapshot: string }; expect(restarted.ok).toBe(true); diff --git a/server/src/services/native-runtime/native-workflow-tools.ts b/server/src/services/native-runtime/native-workflow-tools.ts index 54cc3fbcffa..59209a1d0d8 100644 --- a/server/src/services/native-runtime/native-workflow-tools.ts +++ b/server/src/services/native-runtime/native-workflow-tools.ts @@ -38,9 +38,16 @@ export async function executeNativeWorkflowTool(input: { ...(input.tool === "read_workflow" ? { responseText: responseText ?? { offsetBytes: 0 } } : {}), }; const key = idempotencyKey ?? createHash("sha256").update(JSON.stringify(request)).digest("hex"); - const result = await input.callApi(`workflow-tool:${input.tool}:${key}`, request); - if (input.tool !== "read_workflow" || !result || typeof result !== "object") return result; - const etag = "etag" in result && typeof result.etag === "string" ? result.etag : ""; + const raw = await input.callApi(`workflow-tool:${input.tool}:${key}`, request); + if (!raw || typeof raw !== "object" || Array.isArray(raw)) return raw; + // Name the REST route apiOperationId, as every other API-backed tool does. + // The runner reads `{ ok, operationId }` as a semantic result envelope that + // must name this tool and call. A REST route there is a mismatch, so the + // result is rejected and never reaches the agent. + const { operationId, ...response } = raw as Record; + const result: Record = { ...response, apiOperationId: operationId }; + if (input.tool !== "read_workflow") return result; + const etag = typeof result.etag === "string" ? result.etag : ""; const snapshot = /^(?:W\/)?"([a-f0-9]{64})"$/.exec(etag)?.[1]; return { ...result, ...(snapshot ? { snapshot } : {}), guidance: "Concatenate data JSON text pages using read_workflow with the same workflowId, returned snapshot, and responseText.offsetBytes=responseText.nextOffsetBytes until null. A 409 conflict requires restarting at offset zero without snapshot. Adoption captures the latest prompt at handoff acceptance." }; }