From 7a0ac92fdf6c49d3a85ff227e97af03fafc3bf2b Mon Sep 17 00:00:00 2001 From: marshall Date: Wed, 18 Nov 2026 16:49:11 -0800 Subject: [PATCH] fix(runner): tell ACPX agents which field a refused semantic call failed (#22037) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call Paperclip semantic tools (for example `request_human_input`) through the runner, and the ACPX runner sends each refusal back to the agent > - Today the ACPX runner replaces every refusal with the fixed text "Paperclip semantic operation failed" > - The real reason (for example `summary: too big, max 1000`) is only in the run events, so the agent cannot see it > - Agents then retry the same bad input. One agent retried a questions card 5 times with the same too-long prompt > - This pull request adds the stable error code and, for a validation refusal, each failing field path with its rule and limit to the message > - The value the agent sent, free-text messages and server errors (5xx) stay hidden, as before > - The benefit is that agents can fix a refused call on the next try instead of retrying blind ## Linked Issues or Issue Description Refs #20134 (fixes one case of this problem: the advertised `request_human_input` bounds) **What happened?** An agent called a semantic tool with input that failed server validation. The tool result said only "Paperclip semantic operation failed". The agent retried the same input 5 times, and each retry failed the same way. The run events showed the real cause: `summary: too big, max 1000`. **What did you expect to happen?** The tool result should say which field failed and which rule it broke, so the agent can correct the input. It should not echo the input value or internal diagnostics. **Steps to reproduce** 1. Run an agent on an ACPX provider. 2. Call `request_human_input` with a `prompt` of 1,100 characters (before #20134), or any semantic tool with input that fails zod validation. 3. See the generic "Paperclip semantic operation failed" in the tool result. ## What Changed - New `acpx_tool_failure_feedback` module in `runner-core`. It reads a failed tool result and builds a bounded message from: - a stable error code (lowercase words joined by `_`, `.` or `-`, for example `paperclip_runner_agent_not_found`); - zod-style issues: field path, rule name, type and numeric limit, for example `payload.prompt: too_big, string max 1000`. - It reads the result as an object, an issue list, or a `{success:false, contentItems:[{inputText}]}` envelope whose text is JSON or starts with `code:`. - It never copies `message`, `input`, free text or the input value. Path keys and rule names that do not look like names become `*` or `invalid`. It shows at most 5 issues and 8 path segments. - Any result with a status of 500 or more, or with no recognizable code or issue, keeps the generic message. - `AcpxProviderSession` uses this message for non-terminal semantic failures. Reserved terminal tools keep their current feedback. - Fake ACPX sidecar mode `resolutions-error-detail` and an integration test that checks the message crosses the sidecar boundary without the value. ## Verification - `cargo test -p runner-core --lib acpx_tool_failure_feedback` (7 unit tests: issue lists, envelopes, codes, server errors, free text, masking, bounds). - `cargo test -p runner-core --test acpx_provider_resolutions` (new `tells_the_agent_which_field_failed_validation_without_its_value`, and the existing redaction test still gets the generic message). - `cargo fmt --check`. ## Risks - Low to medium. The main risk is data in error text. The module copies only values that match strict name and number patterns, and never a `message` or `input` field. Tests check that sample values do not appear in the output. - Agents see a longer error message for validation refusals. The message still starts with "Paperclip semantic operation failed", so text matches on that prefix keep working. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../runner-core/src/acpx_provider_session.rs | 14 +- .../src/acpx_tool_failure_feedback.rs | 353 ++++++++++++++++++ .../runner-core/src/bin/fake-acpx-sidecar.rs | 10 + .../runner/crates/runner-core/src/lib.rs | 1 + .../tests/acpx_provider_resolutions.rs | 25 ++ 5 files changed, 397 insertions(+), 6 deletions(-) create mode 100644 packages/paperclip-runner/runner/crates/runner-core/src/acpx_tool_failure_feedback.rs diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs index e9242bee000..f50b3011c02 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs @@ -10,6 +10,7 @@ use crate::acpx_provider_state::{ PRP_COMPLETION_TOOL_NAME, }; use crate::acpx_sidecar_transport::{AcpxSidecarTransport, AcpxSidecarTransportConfig}; +use crate::acpx_tool_failure_feedback::{semantic_failure_feedback, GENERIC_SEMANTIC_FAILURE}; use crate::generated_acpx_sidecar_contract::{ GeneratedAcpxSidecarCommand, GeneratedAcpxSidecarEventType, GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION, @@ -1197,12 +1198,13 @@ impl AcpxProviderSession { let resolution = if result.is_error { // The durable result remains authoritative for correlation and // retry bookkeeping, but provider-facing failures expose only a - // fixed diagnostic. Internal dispatcher payloads must not cross - // the sidecar boundary on the separate success-result channel. + // stable code and validated field rules. Internal dispatcher + // payloads must not cross the sidecar boundary on the separate + // success-result channel. let message = if is_reserved_terminal_operation(&result.operation_id) { reserved_terminal_feedback(&result.result) } else { - "Paperclip semantic operation failed".to_owned() + semantic_failure_feedback(&result.result) }; json!({ "callId":result.call_id, @@ -1541,7 +1543,7 @@ impl AcpxProviderSession { fn reserved_terminal_feedback(result: &Value) -> String { const MAX_FEEDBACK_CHARS: usize = 2_000; if result.get("success").and_then(Value::as_bool) != Some(false) { - return "Paperclip semantic operation failed".to_owned(); + return GENERIC_SEMANTIC_FAILURE.to_owned(); } let text = result .get("contentItems") @@ -1550,13 +1552,13 @@ fn reserved_terminal_feedback(result: &Value) -> String { .filter(|item| item.get("type").and_then(Value::as_str) == Some("inputText")) .and_then(|item| item.get("text")) .and_then(Value::as_str) - .unwrap_or("Paperclip semantic operation failed"); + .unwrap_or(GENERIC_SEMANTIC_FAILURE); let mut bounded = text.chars().take(MAX_FEEDBACK_CHARS).collect::(); if text.chars().count() > MAX_FEEDBACK_CHARS { bounded.push_str("…"); } if bounded.trim().is_empty() { - "Paperclip semantic operation failed".to_owned() + GENERIC_SEMANTIC_FAILURE.to_owned() } else { bounded } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_tool_failure_feedback.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_tool_failure_feedback.rs new file mode 100644 index 00000000000..08a52223021 --- /dev/null +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_tool_failure_feedback.rs @@ -0,0 +1,353 @@ +//! Provider-facing feedback for a refused ACPX semantic tool call. +//! +//! A failed result can hold dispatcher payloads, request echoes and server +//! diagnostics, so it never crosses the sidecar boundary as is. Only these +//! parts are copied, each checked against a fixed shape: +//! +//! - a stable error code, such as `paperclip_runner_tool_input_invalid`; +//! - for a validation refusal, each failing field path with its rule and +//! numeric limit, such as `payload.prompt: too_big, string max 1000`. +//! +//! Free-text messages, the value the agent sent and every part of a server +//! error (status 5xx) stay behind. Anything else gets the generic message. + +use serde_json::Value; + +pub(crate) const GENERIC_SEMANTIC_FAILURE: &str = "Paperclip semantic operation failed"; + +const MAX_ISSUES: usize = 5; +const MAX_PATH_SEGMENTS: usize = 8; +const MAX_SEGMENT_CHARS: usize = 64; +const MAX_CODE_CHARS: usize = 96; +const MAX_RULE_CHARS: usize = 48; +const MAX_SEARCH_DEPTH: usize = 4; +const MAX_LIMIT: f64 = 1e15; +// Keys that wrap a refusal in the dispatcher, runner API and HTTP error shapes. +const NESTED_KEYS: [&str; 5] = ["error", "data", "details", "issues", "result"]; +const TYPE_NAMES: [&str; 9] = [ + "string", "array", "number", "int", "bigint", "set", "file", "date", "object", +]; + +pub(crate) fn semantic_failure_feedback(result: &Value) -> String { + let result = parsed_text_envelope(result).unwrap_or_else(|| result.clone()); + if reports_server_error(&result, 0) { + return GENERIC_SEMANTIC_FAILURE.to_owned(); + } + let code = find_code(&result, 0); + let issues = find_issues(&result, 0).unwrap_or_default(); + let mut parts = Vec::new(); + if let Some(code) = code { + parts.push(code); + } + if !issues.is_empty() { + let mut rendered = issues + .iter() + .take(MAX_ISSUES) + .map(render_issue) + .collect::>() + .join("; "); + if issues.len() > MAX_ISSUES { + rendered.push_str(&format!("; {} more", issues.len() - MAX_ISSUES)); + } + parts.push(format!("invalid input: {rendered}")); + } + if parts.is_empty() { + GENERIC_SEMANTIC_FAILURE.to_owned() + } else { + format!("{GENERIC_SEMANTIC_FAILURE}: {}", parts.join("; ")) + } +} + +/// A thrown error reaches runnerd as `{success:false, contentItems:[text]}`. +/// Its text is either JSON (a zod issue list) or a message that may start +/// with a stable code (`code: explanation`). +fn parsed_text_envelope(result: &Value) -> Option { + let text = envelope_text(result)?; + if let Ok(parsed) = serde_json::from_str::(text) { + if parsed.is_object() || parsed.is_array() { + return Some(parsed); + } + } + let prefix = text.split(':').next().unwrap_or_default().trim(); + is_code(prefix).then(|| serde_json::json!({ "error": prefix })) +} + +fn envelope_text(result: &Value) -> Option<&str> { + result + .get("contentItems")? + .as_array()? + .first() + .filter(|item| item.get("type").and_then(Value::as_str) == Some("inputText"))? + .get("text")? + .as_str() +} + +fn nested(value: &Value) -> impl Iterator { + NESTED_KEYS.iter().filter_map(|key| value.get(*key)) +} + +fn reports_server_error(value: &Value, depth: usize) -> bool { + if depth > MAX_SEARCH_DEPTH || !value.is_object() { + return false; + } + let status = value.get("status").and_then(Value::as_u64); + status.is_some_and(|status| status >= 500) + || nested(value).any(|child| reports_server_error(child, depth + 1)) +} + +fn find_code(value: &Value, depth: usize) -> Option { + if depth > MAX_SEARCH_DEPTH || !value.is_object() { + return None; + } + for key in ["code", "error"] { + if let Some(code) = value + .get(key) + .and_then(Value::as_str) + .filter(|code| is_code(code)) + { + return Some(code.to_owned()); + } + } + nested(value).find_map(|child| find_code(child, depth + 1)) +} + +fn find_issues(value: &Value, depth: usize) -> Option> { + if depth > MAX_SEARCH_DEPTH { + return None; + } + if let Some(items) = value.as_array() { + let issues = items + .iter() + .filter(|item| is_issue(item)) + .collect::>(); + return (!issues.is_empty() && issues.len() == items.len()).then_some(issues); + } + if !value.is_object() { + return None; + } + nested(value).find_map(|child| find_issues(child, depth + 1)) +} + +fn is_issue(value: &Value) -> bool { + value.get("path").is_some_and(Value::is_array) + && value.get("code").and_then(Value::as_str).is_some() +} + +fn render_issue(issue: &&Value) -> String { + let path = render_path(issue.get("path").and_then(Value::as_array)); + let rule = issue + .get("code") + .and_then(Value::as_str) + .filter(|rule| is_rule(rule)) + .unwrap_or("invalid"); + let origin = ["origin", "type"] + .iter() + .filter_map(|key| issue.get(*key).and_then(Value::as_str)) + .find(|name| TYPE_NAMES.contains(name)); + let limit = [("maximum", "max"), ("minimum", "min")] + .iter() + .find_map(|(key, label)| render_limit(issue.get(*key)?).map(|limit| (*label, limit))); + let mut rendered = format!("{path}: {rule}"); + match (origin, limit) { + (Some(origin), Some((label, limit))) => { + rendered.push_str(&format!(", {origin} {label} {limit}")) + } + (None, Some((label, limit))) => rendered.push_str(&format!(", {label} {limit}")), + _ => {} + } + if rule == "invalid_type" { + if let Some(expected) = issue + .get("expected") + .and_then(Value::as_str) + .filter(|expected| is_rule(expected)) + { + rendered.push_str(&format!(", expected {expected}")); + } + } + rendered +} + +fn render_path(path: Option<&Vec>) -> String { + let Some(path) = path.filter(|path| !path.is_empty()) else { + return "(input)".to_owned(); + }; + let mut rendered = String::new(); + for segment in path.iter().take(MAX_PATH_SEGMENTS) { + match segment { + Value::Number(index) if index.as_u64().is_some() => { + rendered.push_str(&format!("[{}]", index.as_u64().unwrap_or_default())); + } + Value::String(key) if is_path_key(key) => { + if !rendered.is_empty() { + rendered.push('.'); + } + rendered.push_str(key); + } + _ => { + if !rendered.is_empty() { + rendered.push('.'); + } + rendered.push('*'); + } + } + } + if path.len() > MAX_PATH_SEGMENTS { + rendered.push_str(".…"); + } + rendered +} + +fn render_limit(value: &Value) -> Option { + let limit = value + .as_f64() + .filter(|limit| limit.is_finite() && limit.abs() <= MAX_LIMIT)?; + Some(if limit.fract() == 0.0 { + format!("{}", limit as i64) + } else { + format!("{limit}") + }) +} + +fn is_path_key(key: &str) -> bool { + let mut chars = key.chars(); + key.chars().count() <= MAX_SEGMENT_CHARS + && chars + .next() + .is_some_and(|first| first.is_ascii_alphabetic() || first == '_') + && chars.all(|char| char.is_ascii_alphanumeric() || char == '_' || char == '-') +} + +/// A lowercase rule name, such as `too_big` or `invalid_type`. +fn is_rule(rule: &str) -> bool { + let mut chars = rule.chars(); + rule.len() <= MAX_RULE_CHARS + && chars.next().is_some_and(|first| first.is_ascii_lowercase()) + && chars.all(|char| char.is_ascii_lowercase() || char.is_ascii_digit() || char == '_') +} + +/// A stable machine code: lowercase words joined by `_`, `.` or `-`. A single +/// word is refused, so a free-text message such as `Forbidden` never passes. +fn is_code(code: &str) -> bool { + let separated = code.contains(['_', '.', '-']); + let mut chars = code.chars(); + separated + && code.len() <= MAX_CODE_CHARS + && chars.next().is_some_and(|first| first.is_ascii_lowercase()) + && chars.all(|char| { + char.is_ascii_lowercase() || char.is_ascii_digit() || matches!(char, '_' | '.' | '-') + }) + && !code.ends_with(['_', '.', '-']) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn names_each_failing_field_and_rule_from_a_zod_issue_list() { + let feedback = semantic_failure_feedback(&json!([ + {"origin":"string","code":"too_big","maximum":1000,"inclusive":true, + "path":["payload","prompt"],"message":"Too big: secret-value-1"}, + {"code":"invalid_type","expected":"string","path":["title"], + "message":"Invalid input: expected string, received secret-value-2"}, + {"code":"too_small","origin":"array","minimum":1,"path":["payload","questions",0,"options"]}, + ])); + assert_eq!( + feedback, + "Paperclip semantic operation failed: invalid input: \ + payload.prompt: too_big, string max 1000; title: invalid_type, expected string; \ + payload.questions[0].options: too_small, array min 1" + ); + assert!(!feedback.contains("secret-value")); + } + + #[test] + fn reads_a_zod_issue_list_left_as_envelope_text() { + let text = json!([{"code":"too_big","maximum":240,"path":["idempotencyKey"]}]).to_string(); + let feedback = semantic_failure_feedback(&json!({ + "success":false,"contentItems":[{"type":"inputText","text":text}], + })); + assert_eq!( + feedback, + "Paperclip semantic operation failed: invalid input: idempotencyKey: too_big, max 240" + ); + } + + #[test] + fn keeps_only_the_stable_code_of_a_refusal_message() { + let feedback = semantic_failure_feedback(&json!({ + "success":false, + "contentItems":[{"type":"inputText", + "text":"paperclip_runner_agent_not_found: no agent has id \"secret-value\"."}], + })); + assert_eq!( + feedback, + "Paperclip semantic operation failed: paperclip_runner_agent_not_found" + ); + } + + #[test] + fn reports_dispatcher_and_runner_api_codes() { + assert_eq!( + semantic_failure_feedback(&json!({ + "ok":false,"operationId":"get_task_history","callId":"call-1", + "error":{"code":"input_invalid","message":"The requested semantic action was not executed.","retryable":false}, + })), + "Paperclip semantic operation failed: input_invalid" + ); + assert_eq!( + semantic_failure_feedback(&json!({ + "ok":false,"status":422,"error":"api_redirect_not_followed", + "data":{"error":"Validation error","details":[{"code":"too_big","maximum":5,"path":["limit"]}]}, + })), + "Paperclip semantic operation failed: api_redirect_not_followed; invalid input: limit: too_big, max 5" + ); + } + + #[test] + fn hides_server_errors_and_free_text() { + for result in [ + json!({"ok":false,"status":500,"error":"database_unavailable", + "data":{"details":[{"code":"too_big","maximum":5,"path":["limit"]}]}}), + json!({"ok":false,"error":{"status":503,"code":"upstream_failed"}}), + json!({"success":false,"contentItems":[{"type":"inputText","text":"Forbidden for secret-value"}]}), + json!({"diagnostic":"violet-internal-diagnostic-4821","request":{"private":"value"}}), + json!("plain string"), + json!([{"path":["a"],"code":"too_big"}, "secret-value"]), + ] { + assert_eq!( + semantic_failure_feedback(&result), + GENERIC_SEMANTIC_FAILURE, + "{result}" + ); + } + } + + #[test] + fn masks_path_keys_and_rules_that_do_not_look_like_names() { + let long_key = "k".repeat(MAX_SEGMENT_CHARS + 1); + let feedback = semantic_failure_feedback(&json!([ + {"code":"Custom message with secret-value","path":["metadata","has space", long_key, 2]}, + ])); + assert_eq!( + feedback, + "Paperclip semantic operation failed: invalid input: metadata.*.*[2]: invalid" + ); + } + + #[test] + fn bounds_the_issue_count_and_path_depth() { + let issues = (0..7) + .map(|index| json!({"code":"invalid_type","path":[format!("f{index}")]})) + .collect::>(); + let feedback = semantic_failure_feedback(&Value::Array(issues)); + assert!(feedback.ends_with("f4: invalid_type; 2 more"), "{feedback}"); + + let deep = json!([{"code":"invalid_type","path":["a","b","c","d","e","f","g","h","i"]}]); + assert_eq!( + semantic_failure_feedback(&deep), + "Paperclip semantic operation failed: invalid input: a.b.c.d.e.f.g.h.…: invalid_type" + ); + } +} diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs index 370a840ec86..74d777b9c64 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs @@ -935,6 +935,7 @@ fn run() -> Result<(), Box> { | "permissions-wrong-ack" | "resolutions" | "resolutions-error-redaction" + | "resolutions-error-detail" | "resolutions-projected-id" | "resolutions-wrong-ack" | "resolutions-snapshot-wrong-session" @@ -1614,6 +1615,15 @@ fn bootstrap_success( && params.pointer("/error/message").and_then(Value::as_str) == Some("Paperclip semantic operation failed") && !params.to_string().contains("violet-internal-diagnostic-4821") + } else if mode == "resolutions-error-detail" { + params.get("callId").and_then(Value::as_str) == Some("call-1") + && params.get("result").is_none() + && params.pointer("/error/message").and_then(Value::as_str) + == Some( + "Paperclip semantic operation failed: invalid input: \ + payload.prompt: too_big, string max 1000", + ) + && !params.to_string().contains("violet-prompt-value-4821") } else { mode != "resolutions-wrong-ack" } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/lib.rs b/packages/paperclip-runner/runner/crates/runner-core/src/lib.rs index e2e48cfaabb..7e903c75105 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/lib.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/lib.rs @@ -9,6 +9,7 @@ pub mod acpx_provider_checkpoint; pub mod acpx_provider_session; pub mod acpx_provider_state; pub mod acpx_sidecar_transport; +mod acpx_tool_failure_feedback; mod acpx_turn_dispatch; mod attachment_download; pub mod aws_agentcore_provider; diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs index 23105613a6a..d94075d3950 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs @@ -238,6 +238,31 @@ fn redacts_failed_tool_payload_without_losing_correlation_or_retry() { session.shutdown("test complete").unwrap(); } +#[test] +fn tells_the_agent_which_field_failed_validation_without_its_value() { + let mut session = started("resolutions-error-detail"); + // A zod refusal reaches runnerd as its parsed issue list. The issue + // message and the agent's own value must not cross the sidecar boundary. + let failed = ToolResult { + call_id: "call-1".to_owned(), + operation_id: "issues.read".to_owned(), + result: json!([{ + "origin":"string", + "code":"too_big", + "maximum":1000, + "inclusive":true, + "path":["payload","prompt"], + "message":"Too big: violet-prompt-value-4821", + "input":"violet-prompt-value-4821", + }]), + is_error: true, + }; + + session.deliver_tool_result(&failed).unwrap(); + assert!(session.state().pending_tool("call-1").is_none()); + session.shutdown("test complete").unwrap(); +} + #[test] fn rejects_permission_requests_that_bypass_the_pinned_codex_policy() { let mut session = AcpxProviderSession::start(&config("turns-permission")).unwrap();