fix(runner): tell ACPX agents which field a refused semantic call failed (#22037)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip semantic tools (for example
`request_human_input`) through the runner, and the ACPX runner sends
each refusal back to the agent
> - Today the ACPX runner replaces every refusal with the fixed text
"Paperclip semantic operation failed"
> - The real reason (for example `summary: too big, max 1000`) is only
in the run events, so the agent cannot see it
> - Agents then retry the same bad input. One agent retried a questions
card 5 times with the same too-long prompt
> - This pull request adds the stable error code and, for a validation
refusal, each failing field path with its rule and limit to the message
> - The value the agent sent, free-text messages and server errors (5xx)
stay hidden, as before
> - The benefit is that agents can fix a refused call on the next try
instead of retrying blind

## Linked Issues or Issue Description

Refs #20134 (fixes one case of this problem: the advertised
`request_human_input` bounds)

**What happened?**
An agent called a semantic tool with input that failed server
validation. The tool result said only "Paperclip semantic operation
failed". The agent retried the same input 5 times, and each retry failed
the same way. The run events showed the real cause: `summary: too big,
max 1000`.

**What did you expect to happen?**
The tool result should say which field failed and which rule it broke,
so the agent can correct the input. It should not echo the input value
or internal diagnostics.

**Steps to reproduce**
1. Run an agent on an ACPX provider.
2. Call `request_human_input` with a `prompt` of 1,100 characters
(before #20134), or any semantic tool with input that fails zod
validation.
3. See the generic "Paperclip semantic operation failed" in the tool
result.

## What Changed

- New `acpx_tool_failure_feedback` module in `runner-core`. It reads a
failed tool result and builds a bounded message from:
- a stable error code (lowercase words joined by `_`, `.` or `-`, for
example `paperclip_runner_agent_not_found`);
- zod-style issues: field path, rule name, type and numeric limit, for
example `payload.prompt: too_big, string max 1000`.
- It reads the result as an object, an issue list, or a `{success:false,
contentItems:[{inputText}]}` envelope whose text is JSON or starts with
`code:`.
- It never copies `message`, `input`, free text or the input value. Path
keys and rule names that do not look like names become `*` or `invalid`.
It shows at most 5 issues and 8 path segments.
- Any result with a status of 500 or more, or with no recognizable code
or issue, keeps the generic message.
- `AcpxProviderSession` uses this message for non-terminal semantic
failures. Reserved terminal tools keep their current feedback.
- Fake ACPX sidecar mode `resolutions-error-detail` and an integration
test that checks the message crosses the sidecar boundary without the
value.

## Verification

- `cargo test -p runner-core --lib acpx_tool_failure_feedback` (7 unit
tests: issue lists, envelopes, codes, server errors, free text, masking,
bounds).
- `cargo test -p runner-core --test acpx_provider_resolutions` (new
`tells_the_agent_which_field_failed_validation_without_its_value`, and
the existing redaction test still gets the generic message).
- `cargo fmt --check`.

## Risks

- Low to medium. The main risk is data in error text. The module copies
only values that match strict name and number patterns, and never a
`message` or `input` field. Tests check that sample values do not appear
in the output.
- Agents see a longer error message for validation refusals. The message
still starts with "Paperclip semantic operation failed", so text matches
on that prefix keep working.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-18 16:49:11 -08:00
1 parent 92dbf39e6f
commit 7a0ac92fdf
5 files changed
+397 -6

No files matched your search

@@ -10,6 +10,7 @@ use crate::acpx_provider_state::{
PRP_COMPLETION_TOOL_NAME,
};
use crate::acpx_sidecar_transport::{AcpxSidecarTransport, AcpxSidecarTransportConfig};
use crate::acpx_tool_failure_feedback::{semantic_failure_feedback, GENERIC_SEMANTIC_FAILURE};
use crate::generated_acpx_sidecar_contract::{
GeneratedAcpxSidecarCommand, GeneratedAcpxSidecarEventType,
GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
@@ -1197,12 +1198,13 @@ impl AcpxProviderSession {
let resolution = if result.is_error {
// The durable result remains authoritative for correlation and
// retry bookkeeping, but provider-facing failures expose only a
// fixed diagnostic. Internal dispatcher payloads must not cross
// the sidecar boundary on the separate success-result channel.
// stable code and validated field rules. Internal dispatcher
// payloads must not cross the sidecar boundary on the separate
// success-result channel.
let message = if is_reserved_terminal_operation(&result.operation_id) {
reserved_terminal_feedback(&result.result)
} else {
"Paperclip semantic operation failed".to_owned()
semantic_failure_feedback(&result.result)
};
json!({
"callId":result.call_id,
@@ -1541,7 +1543,7 @@ impl AcpxProviderSession {
fn reserved_terminal_feedback(result: &Value) -> String {
const MAX_FEEDBACK_CHARS: usize = 2_000;
if result.get("success").and_then(Value::as_bool) != Some(false) {
return "Paperclip semantic operation failed".to_owned();
return GENERIC_SEMANTIC_FAILURE.to_owned();
}
let text = result
.get("contentItems")
@@ -1550,13 +1552,13 @@ fn reserved_terminal_feedback(result: &Value) -> String {
.filter(|item| item.get("type").and_then(Value::as_str) == Some("inputText"))
.and_then(|item| item.get("text"))
.and_then(Value::as_str)
.unwrap_or("Paperclip semantic operation failed");
.unwrap_or(GENERIC_SEMANTIC_FAILURE);
let mut bounded = text.chars().take(MAX_FEEDBACK_CHARS).collect::<String>();
if text.chars().count() > MAX_FEEDBACK_CHARS {
bounded.push_str("…");
}
if bounded.trim().is_empty() {
"Paperclip semantic operation failed".to_owned()
GENERIC_SEMANTIC_FAILURE.to_owned()
} else {
bounded
}
@@ -0,0 +1,353 @@
//! Provider-facing feedback for a refused ACPX semantic tool call.
//!
//! A failed result can hold dispatcher payloads, request echoes and server
//! diagnostics, so it never crosses the sidecar boundary as is. Only these
//! parts are copied, each checked against a fixed shape:
//!
//! - a stable error code, such as `paperclip_runner_tool_input_invalid`;
//! - for a validation refusal, each failing field path with its rule and
//! numeric limit, such as `payload.prompt: too_big, string max 1000`.
//!
//! Free-text messages, the value the agent sent and every part of a server
//! error (status 5xx) stay behind. Anything else gets the generic message.
use serde_json::Value;
pub(crate) const GENERIC_SEMANTIC_FAILURE: &str = "Paperclip semantic operation failed";
const MAX_ISSUES: usize = 5;
const MAX_PATH_SEGMENTS: usize = 8;
const MAX_SEGMENT_CHARS: usize = 64;
const MAX_CODE_CHARS: usize = 96;
const MAX_RULE_CHARS: usize = 48;
const MAX_SEARCH_DEPTH: usize = 4;
const MAX_LIMIT: f64 = 1e15;
// Keys that wrap a refusal in the dispatcher, runner API and HTTP error shapes.
const NESTED_KEYS: [&str; 5] = ["error", "data", "details", "issues", "result"];
const TYPE_NAMES: [&str; 9] = [
"string", "array", "number", "int", "bigint", "set", "file", "date", "object",
];
pub(crate) fn semantic_failure_feedback(result: &Value) -> String {
let result = parsed_text_envelope(result).unwrap_or_else(|| result.clone());
if reports_server_error(&result, 0) {
return GENERIC_SEMANTIC_FAILURE.to_owned();
}
let code = find_code(&result, 0);
let issues = find_issues(&result, 0).unwrap_or_default();
let mut parts = Vec::new();
if let Some(code) = code {
parts.push(code);
}
if !issues.is_empty() {
let mut rendered = issues
.iter()
.take(MAX_ISSUES)
.map(render_issue)
.collect::<Vec<_>>()
.join("; ");
if issues.len() > MAX_ISSUES {
rendered.push_str(&format!("; {} more", issues.len() - MAX_ISSUES));
}
parts.push(format!("invalid input: {rendered}"));
}
if parts.is_empty() {
GENERIC_SEMANTIC_FAILURE.to_owned()
} else {
format!("{GENERIC_SEMANTIC_FAILURE}: {}", parts.join("; "))
}
}
/// A thrown error reaches runnerd as `{success:false, contentItems:[text]}`.
/// Its text is either JSON (a zod issue list) or a message that may start
/// with a stable code (`code: explanation`).
fn parsed_text_envelope(result: &Value) -> Option<Value> {
let text = envelope_text(result)?;
if let Ok(parsed) = serde_json::from_str::<Value>(text) {
if parsed.is_object() || parsed.is_array() {
return Some(parsed);
}
}
let prefix = text.split(':').next().unwrap_or_default().trim();
is_code(prefix).then(|| serde_json::json!({ "error": prefix }))
}
fn envelope_text(result: &Value) -> Option<&str> {
result
.get("contentItems")?
.as_array()?
.first()
.filter(|item| item.get("type").and_then(Value::as_str) == Some("inputText"))?
.get("text")?
.as_str()
}
fn nested(value: &Value) -> impl Iterator<Item = &Value> {
NESTED_KEYS.iter().filter_map(|key| value.get(*key))
}
fn reports_server_error(value: &Value, depth: usize) -> bool {
if depth > MAX_SEARCH_DEPTH || !value.is_object() {
return false;
}
let status = value.get("status").and_then(Value::as_u64);
status.is_some_and(|status| status >= 500)
|| nested(value).any(|child| reports_server_error(child, depth + 1))
}
fn find_code(value: &Value, depth: usize) -> Option<String> {
if depth > MAX_SEARCH_DEPTH || !value.is_object() {
return None;
}
for key in ["code", "error"] {
if let Some(code) = value
.get(key)
.and_then(Value::as_str)
.filter(|code| is_code(code))
{
return Some(code.to_owned());
}
}
nested(value).find_map(|child| find_code(child, depth + 1))
}
fn find_issues(value: &Value, depth: usize) -> Option<Vec<&Value>> {
if depth > MAX_SEARCH_DEPTH {
return None;
}
if let Some(items) = value.as_array() {
let issues = items
.iter()
.filter(|item| is_issue(item))
.collect::<Vec<_>>();
return (!issues.is_empty() && issues.len() == items.len()).then_some(issues);
}
if !value.is_object() {
return None;
}
nested(value).find_map(|child| find_issues(child, depth + 1))
}
fn is_issue(value: &Value) -> bool {
value.get("path").is_some_and(Value::is_array)
&& value.get("code").and_then(Value::as_str).is_some()
}
fn render_issue(issue: &&Value) -> String {
let path = render_path(issue.get("path").and_then(Value::as_array));
let rule = issue
.get("code")
.and_then(Value::as_str)
.filter(|rule| is_rule(rule))
.unwrap_or("invalid");
let origin = ["origin", "type"]
.iter()
.filter_map(|key| issue.get(*key).and_then(Value::as_str))
.find(|name| TYPE_NAMES.contains(name));
let limit = [("maximum", "max"), ("minimum", "min")]
.iter()
.find_map(|(key, label)| render_limit(issue.get(*key)?).map(|limit| (*label, limit)));
let mut rendered = format!("{path}: {rule}");
match (origin, limit) {
(Some(origin), Some((label, limit))) => {
rendered.push_str(&format!(", {origin} {label} {limit}"))
}
(None, Some((label, limit))) => rendered.push_str(&format!(", {label} {limit}")),
_ => {}
}
if rule == "invalid_type" {
if let Some(expected) = issue
.get("expected")
.and_then(Value::as_str)
.filter(|expected| is_rule(expected))
{
rendered.push_str(&format!(", expected {expected}"));
}
}
rendered
}
fn render_path(path: Option<&Vec<Value>>) -> String {
let Some(path) = path.filter(|path| !path.is_empty()) else {
return "(input)".to_owned();
};
let mut rendered = String::new();
for segment in path.iter().take(MAX_PATH_SEGMENTS) {
match segment {
Value::Number(index) if index.as_u64().is_some() => {
rendered.push_str(&format!("[{}]", index.as_u64().unwrap_or_default()));
}
Value::String(key) if is_path_key(key) => {
if !rendered.is_empty() {
rendered.push('.');
}
rendered.push_str(key);
}
_ => {
if !rendered.is_empty() {
rendered.push('.');
}
rendered.push('*');
}
}
}
if path.len() > MAX_PATH_SEGMENTS {
rendered.push_str(".…");
}
rendered
}
fn render_limit(value: &Value) -> Option<String> {
let limit = value
.as_f64()
.filter(|limit| limit.is_finite() && limit.abs() <= MAX_LIMIT)?;
Some(if limit.fract() == 0.0 {
format!("{}", limit as i64)
} else {
format!("{limit}")
})
}
fn is_path_key(key: &str) -> bool {
let mut chars = key.chars();
key.chars().count() <= MAX_SEGMENT_CHARS
&& chars
.next()
.is_some_and(|first| first.is_ascii_alphabetic() || first == '_')
&& chars.all(|char| char.is_ascii_alphanumeric() || char == '_' || char == '-')
}
/// A lowercase rule name, such as `too_big` or `invalid_type`.
fn is_rule(rule: &str) -> bool {
let mut chars = rule.chars();
rule.len() <= MAX_RULE_CHARS
&& chars.next().is_some_and(|first| first.is_ascii_lowercase())
&& chars.all(|char| char.is_ascii_lowercase() || char.is_ascii_digit() || char == '_')
}
/// A stable machine code: lowercase words joined by `_`, `.` or `-`. A single
/// word is refused, so a free-text message such as `Forbidden` never passes.
fn is_code(code: &str) -> bool {
let separated = code.contains(['_', '.', '-']);
let mut chars = code.chars();
separated
&& code.len() <= MAX_CODE_CHARS
&& chars.next().is_some_and(|first| first.is_ascii_lowercase())
&& chars.all(|char| {
char.is_ascii_lowercase() || char.is_ascii_digit() || matches!(char, '_' | '.' | '-')
})
&& !code.ends_with(['_', '.', '-'])
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn names_each_failing_field_and_rule_from_a_zod_issue_list() {
let feedback = semantic_failure_feedback(&json!([
{"origin":"string","code":"too_big","maximum":1000,"inclusive":true,
"path":["payload","prompt"],"message":"Too big: secret-value-1"},
{"code":"invalid_type","expected":"string","path":["title"],
"message":"Invalid input: expected string, received secret-value-2"},
{"code":"too_small","origin":"array","minimum":1,"path":["payload","questions",0,"options"]},
]));
assert_eq!(
feedback,
"Paperclip semantic operation failed: invalid input: \
payload.prompt: too_big, string max 1000; title: invalid_type, expected string; \
payload.questions[0].options: too_small, array min 1"
);
assert!(!feedback.contains("secret-value"));
}
#[test]
fn reads_a_zod_issue_list_left_as_envelope_text() {
let text = json!([{"code":"too_big","maximum":240,"path":["idempotencyKey"]}]).to_string();
let feedback = semantic_failure_feedback(&json!({
"success":false,"contentItems":[{"type":"inputText","text":text}],
}));
assert_eq!(
feedback,
"Paperclip semantic operation failed: invalid input: idempotencyKey: too_big, max 240"
);
}
#[test]
fn keeps_only_the_stable_code_of_a_refusal_message() {
let feedback = semantic_failure_feedback(&json!({
"success":false,
"contentItems":[{"type":"inputText",
"text":"paperclip_runner_agent_not_found: no agent has id \"secret-value\"."}],
}));
assert_eq!(
feedback,
"Paperclip semantic operation failed: paperclip_runner_agent_not_found"
);
}
#[test]
fn reports_dispatcher_and_runner_api_codes() {
assert_eq!(
semantic_failure_feedback(&json!({
"ok":false,"operationId":"get_task_history","callId":"call-1",
"error":{"code":"input_invalid","message":"The requested semantic action was not executed.","retryable":false},
})),
"Paperclip semantic operation failed: input_invalid"
);
assert_eq!(
semantic_failure_feedback(&json!({
"ok":false,"status":422,"error":"api_redirect_not_followed",
"data":{"error":"Validation error","details":[{"code":"too_big","maximum":5,"path":["limit"]}]},
})),
"Paperclip semantic operation failed: api_redirect_not_followed; invalid input: limit: too_big, max 5"
);
}
#[test]
fn hides_server_errors_and_free_text() {
for result in [
json!({"ok":false,"status":500,"error":"database_unavailable",
"data":{"details":[{"code":"too_big","maximum":5,"path":["limit"]}]}}),
json!({"ok":false,"error":{"status":503,"code":"upstream_failed"}}),
json!({"success":false,"contentItems":[{"type":"inputText","text":"Forbidden for secret-value"}]}),
json!({"diagnostic":"violet-internal-diagnostic-4821","request":{"private":"value"}}),
json!("plain string"),
json!([{"path":["a"],"code":"too_big"}, "secret-value"]),
] {
assert_eq!(
semantic_failure_feedback(&result),
GENERIC_SEMANTIC_FAILURE,
"{result}"
);
}
}
#[test]
fn masks_path_keys_and_rules_that_do_not_look_like_names() {
let long_key = "k".repeat(MAX_SEGMENT_CHARS + 1);
let feedback = semantic_failure_feedback(&json!([
{"code":"Custom message with secret-value","path":["metadata","has space", long_key, 2]},
]));
assert_eq!(
feedback,
"Paperclip semantic operation failed: invalid input: metadata.*.*[2]: invalid"
);
}
#[test]
fn bounds_the_issue_count_and_path_depth() {
let issues = (0..7)
.map(|index| json!({"code":"invalid_type","path":[format!("f{index}")]}))
.collect::<Vec<_>>();
let feedback = semantic_failure_feedback(&Value::Array(issues));
assert!(feedback.ends_with("f4: invalid_type; 2 more"), "{feedback}");
let deep = json!([{"code":"invalid_type","path":["a","b","c","d","e","f","g","h","i"]}]);
assert_eq!(
semantic_failure_feedback(&deep),
"Paperclip semantic operation failed: invalid input: a.b.c.d.e.f.g.h.…: invalid_type"
);
}
}
@@ -935,6 +935,7 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
| "permissions-wrong-ack"
| "resolutions"
| "resolutions-error-redaction"
| "resolutions-error-detail"
| "resolutions-projected-id"
| "resolutions-wrong-ack"
| "resolutions-snapshot-wrong-session"
@@ -1614,6 +1615,15 @@ fn bootstrap_success(
&& params.pointer("/error/message").and_then(Value::as_str)
== Some("Paperclip semantic operation failed")
&& !params.to_string().contains("violet-internal-diagnostic-4821")
} else if mode == "resolutions-error-detail" {
params.get("callId").and_then(Value::as_str) == Some("call-1")
&& params.get("result").is_none()
&& params.pointer("/error/message").and_then(Value::as_str)
== Some(
"Paperclip semantic operation failed: invalid input: \
payload.prompt: too_big, string max 1000",
)
&& !params.to_string().contains("violet-prompt-value-4821")
} else {
mode != "resolutions-wrong-ack"
}
@@ -9,6 +9,7 @@ pub mod acpx_provider_checkpoint;
pub mod acpx_provider_session;
pub mod acpx_provider_state;
pub mod acpx_sidecar_transport;
mod acpx_tool_failure_feedback;
mod acpx_turn_dispatch;
mod attachment_download;
pub mod aws_agentcore_provider;
@@ -238,6 +238,31 @@ fn redacts_failed_tool_payload_without_losing_correlation_or_retry() {
session.shutdown("test complete").unwrap();
}
#[test]
fn tells_the_agent_which_field_failed_validation_without_its_value() {
let mut session = started("resolutions-error-detail");
// A zod refusal reaches runnerd as its parsed issue list. The issue
// message and the agent's own value must not cross the sidecar boundary.
let failed = ToolResult {
call_id: "call-1".to_owned(),
operation_id: "issues.read".to_owned(),
result: json!([{
"origin":"string",
"code":"too_big",
"maximum":1000,
"inclusive":true,
"path":["payload","prompt"],
"message":"Too big: violet-prompt-value-4821",
"input":"violet-prompt-value-4821",
}]),
is_error: true,
};
session.deliver_tool_result(&failed).unwrap();
assert!(session.state().pending_tool("call-1").is_none());
session.shutdown("test complete").unwrap();
}
#[test]
fn rejects_permission_requests_that_bypass_the_pinned_codex_policy() {
let mut session = AcpxProviderSession::start(&config("turns-permission")).unwrap();