Files
PaperClipAI/tests
DottaandClaude Opus 5.5 04f12fd2be test(evals): use plain words in the spend fixture and a relative link in the handoff message (#21408)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work for a person
> - Two eval setups put text in front of the agent that a real person
would not write
> - `budgets-costs.spend-breakdown` wrote "imported synthetic
metered-API receipt tasks", "native run" and "native subscription usage"
into the conversation's task description, and the agent repeated these
terms to the person
> - `agent-chat.multiple-handoffs` pasted an absolute
`http://127.0.0.1:<port>/api/attachments/...` link into the person's
message, and the agent echoed it
> - This pull request writes both in user-facing terms, so the eval does
not teach the agent the jargon that it then is marked down for
> - The benefit is that a jargon finding in these cases points at the
product, not at the fixture

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12, the spend-breakdown reply said "completed native
run", "Native subscription usage" and "imported synthetic metered-API
receipt fixtures". All three phrases come from the description that the
eval setup saves on the conversation task. The multiple-handoffs reply
had an absolute `127.0.0.1` attachment link, copied from the setup's own
chat message.

**Expected behavior**
Eval setup text uses words and links that a person would use, so the
reply is graded on product behavior.

**Steps to reproduce**
1. Read `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts`
(spend-breakdown setup description).
2. Read
`tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.ts`
(setup chat message).

## What Changed

- spend-breakdown setup description: "Launch has recorded API charges on
<task> (Morgan) and <task> (Riley). Morgan also finished one work
session on Launch (run <id>) whose usage is saved but not yet priced.
Leave out charges from other projects and earlier weeks. Plan
subscription fees are separate from API charges." The same facts, scope
and exclusions stay. The cost events and the graders do not change.
- multiple-handoffs setup message: the logo link is the relative route
`/api/attachments/<id>/content?download=1`. The grader binds the message
by the attachment id, which stays.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts
tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts
tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.test.ts
tests/runner-e2e/chat-boundary-fingerprint.test.ts`: 85 pass.
- Narrow paid run of `budgets-costs.spend-breakdown` on a branch with
this change.

## Risks

- The setup description no longer says that the charges are imported
test data. The charges are real cost records in the disposable company,
so the agent reports them as recorded charges, which is correct for the
person.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:45:33 -06:00
..