mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents change Paperclip records through the `call_api` tool, and the native runtime runs each tool call through a repeated-identical-failure guard (#21360, #21398) > - Some model hosts drop arguments. A model then sends the same failing call again and again, and each attempt costs a model turn > - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input hash > - The guard counted these refusals and stopped running the call after the second failure. But its correction showed only a `"body": {...}` placeholder, so the model had no concrete call to copy. Also, the guard did not count a route's own 400/422 when it came back as an `ok: false` result > - This pull request puts a copyable call in each correction and counts returned 400/422 rejections. It never counts transient failures > - The benefit is that a model gets the exact call to send on the second failure, and the third identical validation failure does not run, whether the rejection is thrown or returned ## Linked Issues or Issue Description Refs #21360, #21398 **What happened?** In chat-eval cohort #14, `questions-and-approvals.review-plan` (run 470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH /api/issues/{id}` with no `body` 7 and 10 times, all with the same input hash. The pre-dispatch check (`assertBodyPresent`) refused each call and returned HTTP 400. The guard corrected the second failure and refused the rest. But the correction said only `Resend with a top-level body argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The model kept sending the identical call, and each refusal cost another model turn: 2 min 18 s in review-plan and 24 s in private-task. Also, when the route itself returns a 400/422 validation response, `call_api` gives back `{ ok: false, status: 422, ... }` as a result and does not throw. The guard did not count that, so an identical invalid request reached the route every time. **Expected behavior** On the second identical deterministic failure, the agent gets a correction with a call it can copy that passes validation. On the third, Paperclip refuses to run the call and tells the agent to change the arguments or to stop and report. Transient failures (5xx, timeouts, rate limits, conflicts) always run again. **Steps to reproduce** 1. Run a native-runtime agent on a host that drops the `body` argument. 2. Have it update a task with `call_api PATCH /api/issues/{id}`. 3. Look at the tool results. They contain only a `{...}` placeholder. A route's 422 returned as `ok: false` is never counted. **Paperclip version or commit** masterd4498ae7ac(cohort #14), reproduced againstecdeed5e7e. ## What Changed - `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a copyable call from the operation's catalog schema: the path parameters the model sent, plus a body with the required fields. If no field is required, the body has one free-text field, for example `{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent id>"},"body":{"title":"<title>"}}`. The "requires body" correction and the "body must be a JSON object, not a JSON-encoded string" correction now include this example in place of the `{...}` placeholder. - `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and HTTP 400/422 as a deterministic failure, in the same way as a thrown 400/422. The first returned rejection passes through unchanged. The second gets a `guidance` note that says the call will not run again. The third and later identical calls are refused without running, and the refusal quotes the response. - Never count transient results: 5xx, 408, 429, 409, 404, and a missing status (timeouts or unknown outcome). - A route can reject a request because of the state it found. So the guard forgets a returned rejection after a different call succeeds. A refusal before sending depends only on the arguments, and the guard keeps it for the run, as before. - Tests: the guard tests cover returned 422 (correct, then refuse), a changed call that still runs, returned 500/503/429/408/409/404/no-status and thrown 429/409/timeout that keep running, and a returned rejection forgotten after a state change. The runner API tests check that the example in each recorded body-less correction is sent when copied as is, that the issue-update example passes `updateIssueSchema`, and that every catalog write with a required body (more than 200) has an example its request schema accepts. ## Verification - `cd server && npx vitest run src/services/native-runtime/repeated-tool-failure-guard.test.ts src/services/native-runtime/runner-api.test.ts`: 2190 passed. - Red/green: with the master guard, the 2 returned-rejection tests fail. With the master client, 6 example tests fail. With this change, all pass. - `node scripts/typecheck-layers.mjs` (server): no errors. - Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a successful call 2.13 µs before, 2.19 µs after. Transient-failure path 2.12 µs. Building an example takes 0.67 µs and happens only on a refused call. Nothing is added before the turn starts. The guard is created with the run's tool authority, as before. - Paid narrow run: see the comment below. ## Risks - Low. The change affects only calls that fail validation twice with identical arguments. - A route that returns 400/422 because of state can succeed later with the same arguments. To cover that, the guard forgets a returned rejection after any different call succeeds. A model that retries with no call in between is refused after two attempts and must change the arguments or report. - The example uses `<field>` placeholders for free text and for ids that the model must look up. If a model copies a placeholder verbatim, the route stores that text. The correction says to send the fields the model means to change. - This does not make a model stop resending a refused call. It only gives better guidance and keeps the refused call from running. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use, code execution, and extended reasoning). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>