Files
PaperClipAI/server/src/services/native-runtime/repeated-tool-failure-guard.test.ts
T
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00

220 lines
14 KiB
TypeScript

import { describe, expect, it, vi } from "vitest";
import { HttpError, badRequest } from "../../errors.js";
import { SessionAuthorityEpoch } from "./native-session-executor.js";
import type { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
import { executeRunnerApi } from "./runner-api-client.js";
import { TOOL_CALL_LOOP_REFUSAL_LIMIT, TOOL_CALL_LOOP_RUN_REFUSAL_LIMIT, ToolCallLoopError } from "./repeated-tool-failure-guard.js";
const context = { companyId: "company-a", issueId: "issue-a", issueIdentifier: "API-1", runId: "run-a", workMode: "standard" };
function epochWith(execute: (call: { tool: string; callId: string; arguments: unknown }) => Promise<unknown>) {
const authority = { execute: vi.fn(execute), close: () => {} };
return { authority, epoch: new SessionAuthorityEpoch("run-a", authority as unknown as PaperclipRunnerToolAuthority) };
}
const failure = (epoch: SessionAuthorityEpoch, call: { tool: string; callId: string; arguments: unknown }) =>
epoch.execute(call).then(() => null, (error: Error) => error);
describe("repeated identical failing tool calls", () => {
// Recorded cohort run: a model host dropped `body`, and the model resent the
// same body-less PATCH twelve times, each refused before sending.
const dropped = { operationId: "PATCH /api/issues/{id}", pathParams: { id: "issue-a" }, query: {}, responseText: { limitBytes: 8192 } };
it("corrects the second identical failure and does not run the third", async () => {
const fetcher = vi.fn<typeof fetch>();
const { authority, epoch } = epochWith(async call => executeRunnerApi(call.arguments as never, context,
{ apiUrl: "http://127.0.0.1:3100", token: "t", fetch: fetcher, readFile: async () => { throw new Error("unused"); }, saveResponse: async () => { throw new Error("unused"); } }));
const call = (n: number) => ({ tool: "call_api", callId: `call-${n}`, arguments: dropped });
const first = await failure(epoch, call(1));
expect(first?.message).toMatch(/^This request was not sent: PATCH \/api\/issues\/\{id\} requires body/);
expect(first?.message).not.toContain("failed 2 times");
const second = await failure(epoch, call(2));
expect(second).toBeInstanceOf(HttpError);
expect((second as HttpError).status).toBe(400);
// The original correction stays visible, followed by what arrived and what to do.
expect(second?.message).toContain(first!.message);
expect(second?.message).toContain("failed 2 times with the same arguments");
expect(second?.message).toContain("Arguments received each time: operationId, pathParams, query, responseText.");
expect(second?.message).toContain("Paperclip will not run this exact call again in this run");
// Both the correction and the refusal name a call the model can copy.
const example = '{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"issue-a"},"body":{"title":"<title>"}}';
expect(first?.message).toContain(`for example: ${example}`);
expect(second?.message).toContain(example);
for (const n of [3, 4, 5]) {
const blocked = await failure(epoch, call(n));
expect(blocked?.message).toMatch(/^This call was not run: call_api already failed 2 times/);
expect(blocked?.message).toContain(first!.message);
expect(blocked?.message).toContain(example);
}
expect(authority.execute).toHaveBeenCalledTimes(2);
expect(fetcher).not.toHaveBeenCalled();
});
it("after N refusals the guard raises a turn-stopping loop error", async () => {
const { authority, epoch } = epochWith(async () => { throw badRequest("body is required"); });
const call = (n: number) => ({ tool: "call_api", callId: `loop-${n}`, arguments: dropped });
await failure(epoch, call(1));
await failure(epoch, call(2));
for (let n = 1; n < TOOL_CALL_LOOP_REFUSAL_LIMIT; n++) {
const refused = await failure(epoch, call(2 + n));
expect(refused).not.toBeInstanceOf(ToolCallLoopError);
expect(refused?.message).toMatch(/^This call was not run/);
}
const stopped = await failure(epoch, call(2 + TOOL_CALL_LOOP_REFUSAL_LIMIT));
expect(stopped).toBeInstanceOf(ToolCallLoopError);
expect(stopped).toMatchObject({ code: "agent_tool_call_loop", tool: "call_api", refusals: TOOL_CALL_LOOP_REFUSAL_LIMIT,
argumentNames: ["operationId", "pathParams", "query", "responseText"] });
expect(authority.execute).toHaveBeenCalledTimes(2);
});
it("stops the turn once refusals across different calls reach the run limit", async () => {
const { epoch } = epochWith(async () => { throw badRequest("body is required"); });
let stopped: Error | null = null;
let refusals = 0;
for (let variant = 0; !stopped && variant < TOOL_CALL_LOOP_RUN_REFUSAL_LIMIT; variant++) {
const args = { ...dropped, pathParams: { id: `issue-${variant}` } };
for (let n = 0; n < 2 + TOOL_CALL_LOOP_REFUSAL_LIMIT - 1 && !stopped; n++) {
const error = await failure(epoch, { tool: "call_api", callId: `v${variant}-${n}`, arguments: args });
if (error instanceof ToolCallLoopError) stopped = error;
else if (error?.message.startsWith("This call was not run")) refusals++;
}
}
expect(stopped).toBeInstanceOf(ToolCallLoopError);
expect(refusals + 1).toBe(TOOL_CALL_LOOP_RUN_REFUSAL_LIMIT);
});
it("runs the call again once its arguments change", async () => {
const { authority, epoch } = epochWith(async call => {
if (!(call.arguments as { body?: unknown }).body) throw badRequest("body is required");
return { ok: true };
});
await failure(epoch, { tool: "call_api", callId: "a", arguments: dropped });
await failure(epoch, { tool: "call_api", callId: "b", arguments: dropped });
await expect(epoch.execute({ tool: "call_api", callId: "c", arguments: { ...dropped, body: { title: "x" } } })).resolves.toEqual({ ok: true });
expect(authority.execute).toHaveBeenCalledTimes(3);
});
it("passes failures that may not repeat through unchanged and keeps running them", async () => {
const outage = new HttpError(503, "Paperclip API unavailable");
const { authority, epoch } = epochWith(async () => { throw outage; });
for (const n of [1, 2, 3]) expect(await failure(epoch, { tool: "call_api", callId: `${n}`, arguments: dropped })).toBe(outage);
expect(authority.execute).toHaveBeenCalledTimes(3);
});
it("passes the first failure through unchanged and counts only matching errors", async () => {
let n = 0;
const { authority, epoch } = epochWith(async () => { throw badRequest(`refusal ${++n}`); });
const errors = [];
for (const id of ["1", "2", "3"]) errors.push(await failure(epoch, { tool: "create_task", callId: id, arguments: { title: "t" } }));
expect(errors.map(error => error?.message)).toEqual(["refusal 1", "refusal 2", "refusal 3"]);
expect(authority.execute).toHaveBeenCalledTimes(3);
});
it("counts identical calls sent in parallel and runs at most two of them", async () => {
const { authority, epoch } = epochWith(async () => { await new Promise(resolve => setTimeout(resolve, 5)); throw badRequest("body is required"); });
const errors = await Promise.all([1, 2, 3, 4].map(n => failure(epoch, { tool: "call_api", callId: `p${n}`, arguments: dropped })));
expect(authority.execute).toHaveBeenCalledTimes(2);
expect(errors[0]?.message).toBe("body is required");
expect(errors[1]?.message).toContain("failed 2 times with the same arguments");
expect(errors[2]?.message).toMatch(/^This call was not run/);
expect(errors[3]?.message).toMatch(/^This call was not run/);
});
it("does not hold back a different call while an identical one runs", async () => {
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const { epoch } = epochWith(async call => { if (call.callId === "slow") await gate; return call.callId; });
const slow = epoch.execute({ tool: "list_agents", callId: "slow", arguments: {} });
await expect(epoch.execute({ tool: "list_projects", callId: "other", arguments: {} })).resolves.toBe("other");
release();
await expect(slow).resolves.toBe("slow");
});
// A route's own validation rejection comes back as data, not as a thrown error.
const rejected = (status: number | null, data: unknown = { error: "Validation error", details: [{ path: ["status"], message: "Invalid enum value" }] }) =>
({ ok: false, status, apiOperationId: "PATCH /api/issues/{id}", contentType: "application/json", data });
const invalid = { operationId: "PATCH /api/issues/{id}", pathParams: { id: "issue-a" }, body: { status: "started" } };
it("counts a route's validation rejection returned as ok:false, corrects the second and does not run the third", async () => {
const { authority, epoch } = epochWith(async () => rejected(422));
const call = (n: number) => ({ tool: "call_api", callId: `r${n}`, arguments: invalid });
expect(await epoch.execute(call(1))).toEqual(rejected(422));
const second = await epoch.execute(call(2)) as Record<string, unknown>;
expect(second).toMatchObject({ ok: false, status: 422, data: rejected(422).data });
expect(second.guidance).toContain("failed 2 times with the same arguments");
expect(second.guidance).toContain("Arguments received each time: body, operationId, pathParams.");
expect(second.guidance).toContain("Paperclip will not run this exact call again in this run");
for (const n of [3, 4]) {
const blocked = await failure(epoch, call(n));
expect(blocked).toBeInstanceOf(HttpError);
expect((blocked as HttpError).status).toBe(422);
expect(blocked?.message).toMatch(/^This call was not run: call_api already failed 2 times/);
expect(blocked?.message).toContain("The response each time was HTTP 422: ");
expect(blocked?.message).toContain("Invalid enum value");
}
expect(authority.execute).toHaveBeenCalledTimes(2);
});
it("still runs a changed call after refusing a returned rejection", async () => {
const { authority, epoch } = epochWith(async call => (call.arguments as typeof invalid).body.status === "started" ? rejected(400) : { ok: true, status: 200 });
for (const n of [1, 2, 3]) await epoch.execute({ tool: "call_api", callId: `r${n}`, arguments: invalid }).catch(() => null);
await expect(epoch.execute({ tool: "call_api", callId: "fixed", arguments: { ...invalid, body: { status: "in_progress" } } }))
.resolves.toEqual({ ok: true, status: 200 });
expect(authority.execute).toHaveBeenCalledTimes(3);
});
it.each([
["server error", 500], ["unavailable", 503], ["rate limit", 429], ["timeout", 408], ["conflict", 409], ["not found", 404], ["no response", null],
] as const)("keeps retrying a returned %s (%s), which may succeed unchanged", async (_label, status) => {
const { authority, epoch } = epochWith(async () => rejected(status, { error: "try again" }));
for (const n of [1, 2, 3, 4]) {
const result = await epoch.execute({ tool: "call_api", callId: `t${n}`, arguments: invalid }) as Record<string, unknown>;
expect(result).toEqual(rejected(status, { error: "try again" }));
}
expect(authority.execute).toHaveBeenCalledTimes(4);
});
it.each([
["rate limit", new HttpError(429, "Too many requests")],
["conflict", new HttpError(409, "Issue was updated by someone else")],
["timeout", Object.assign(new Error("The operation was aborted due to timeout"), { name: "TimeoutError" })],
])("keeps retrying a thrown %s, which may succeed unchanged", async (_label, transient) => {
const { authority, epoch } = epochWith(async () => { throw transient; });
for (const n of [1, 2, 3]) expect(await failure(epoch, { tool: "call_api", callId: `${n}`, arguments: invalid })).toBe(transient);
expect(authority.execute).toHaveBeenCalledTimes(3);
});
it("forgets a returned rejection once a different call succeeds, because the state it checked may have changed", async () => {
let unblocked = false;
const { authority, epoch } = epochWith(async call => call.tool === "call_api"
? (unblocked ? { ok: true, status: 200 } : rejected(422, { error: "Resolve the blockers first" }))
: (unblocked = true, { ok: true }));
const patch = (n: number) => ({ tool: "call_api", callId: `s${n}`, arguments: invalid });
await epoch.execute(patch(1));
await epoch.execute(patch(2));
await expect(epoch.execute({ tool: "update_blockers", callId: "unblock", arguments: { blockedByIssueIds: [] } })).resolves.toEqual({ ok: true });
await expect(epoch.execute(patch(3))).resolves.toEqual({ ok: true, status: 200 });
expect(authority.execute).toHaveBeenCalledTimes(4);
});
it("keeps counting a returned rejection when unrelated calls in between fail transiently", async () => {
const { authority, epoch } = epochWith(async call => call.tool === "call_api"
? rejected(422)
: { ok: false, status: call.callId === "u1" ? 503 : call.callId === "u2" ? 429 : null, error: "try again" });
const patch = (n: number) => ({ tool: "call_api", callId: `i${n}`, arguments: invalid });
await epoch.execute(patch(1));
await epoch.execute({ tool: "list_agents", callId: "u1", arguments: {} });
const second = await epoch.execute(patch(2)) as Record<string, unknown>;
expect(second.guidance).toContain("failed 2 times with the same arguments");
await epoch.execute({ tool: "list_agents", callId: "u2", arguments: {} });
await epoch.execute({ tool: "list_agents", callId: "u3", arguments: {} });
expect((await failure(epoch, patch(3)))?.message).toMatch(/^This call was not run/);
expect(authority.execute.mock.calls.filter(([call]) => call.tool === "call_api")).toHaveLength(2);
});
});