Files
PaperClipAI/tests
DottaandClaude Opus 5.5 64f3abbdef test(evals): write the hard-stop setup note in plain words (#21487)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that the chat agent explains a budget
hard stop to a person
> - The `budgets-costs.explain-hard-stop` setup puts a note on the
conversation. The note said the receipt was "an imported synthetic
accounting fixture, not live model-provider billing"
> - In chat-eval cohort 13, the agent repeated that eval jargon to the
person
> - This pull request rewrites the setup note in plain words, the same
way #21408 and #21429 did for spend-breakdown
> - The benefit is that the eval tests the explanation a real person
would get, without setup jargon to copy

## Linked Issues or Issue Description

Refs #21408
Refs #21429

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
The `budgets-costs.explain-hard-stop` reply gave a correct diagnosis,
but it said "The charge is an imported accounting fixture, not live
model-provider billing". That wording came from the eval's own setup
note.

**Expected behavior**
The setup note uses plain words. It keeps the references the scenario
needs, and it does not give the agent test jargon to repeat.

**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.budgets-costs.explain-hard-stop`.
2. Read the conversation note and the reply.

## What Changed

- `budgetIncidentSourceDescription` in
`tests/runner-e2e/chat-catalog-domains/identity-budget-incidents.ts`
builds the note. The note names the task, the recorded hard-stop
incident and the separately paused agent. It no longer says that the
receipt is an imported synthetic fixture.
- The grader is unchanged. It already proves the receipt's origin from
saved state (`receiptOrigin`), not from the reply. The note's task
identifier, incident reference and paused-agent name are unchanged. The
grader matches no literal from the note.
- TestOps manifest: no change is necessary. The case's oracle version,
source file list, query, context and expected outcome are unchanged.

## Verification

- `npx vitest run -c tests/runner-e2e/vitest.config.ts
chat-catalog-domains/identity-budget-incidents.test.ts` (16 tests pass).
- Red/green: the new test checks that the note keeps the task, incident
and paused-agent references and contains none of the setup jargon. It
fails on master (no plain-words note) and passes with this change.
- A narrow paid run of `budgets-costs.explain-hard-stop` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. Only eval setup text changes. The scenario difficulty is the
same: the note states the same facts about the stop and the manual
pause, without the test-origin remark.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:11 -06:00
..