test(evals): use plain words in the spend fixture and a relative link in the handoff message (#21408)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work for a person
> - Two eval setups put text in front of the agent that a real person
would not write
> - `budgets-costs.spend-breakdown` wrote "imported synthetic
metered-API receipt tasks", "native run" and "native subscription usage"
into the conversation's task description, and the agent repeated these
terms to the person
> - `agent-chat.multiple-handoffs` pasted an absolute
`http://127.0.0.1:<port>/api/attachments/...` link into the person's
message, and the agent echoed it
> - This pull request writes both in user-facing terms, so the eval does
not teach the agent the jargon that it then is marked down for
> - The benefit is that a jargon finding in these cases points at the
product, not at the fixture

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12, the spend-breakdown reply said "completed native
run", "Native subscription usage" and "imported synthetic metered-API
receipt fixtures". All three phrases come from the description that the
eval setup saves on the conversation task. The multiple-handoffs reply
had an absolute `127.0.0.1` attachment link, copied from the setup's own
chat message.

**Expected behavior**
Eval setup text uses words and links that a person would use, so the
reply is graded on product behavior.

**Steps to reproduce**
1. Read `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts`
(spend-breakdown setup description).
2. Read
`tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.ts`
(setup chat message).

## What Changed

- spend-breakdown setup description: "Launch has recorded API charges on
<task> (Morgan) and <task> (Riley). Morgan also finished one work
session on Launch (run <id>) whose usage is saved but not yet priced.
Leave out charges from other projects and earlier weeks. Plan
subscription fees are separate from API charges." The same facts, scope
and exclusions stay. The cost events and the graders do not change.
- multiple-handoffs setup message: the logo link is the relative route
`/api/attachments/<id>/content?download=1`. The grader binds the message
by the attachment id, which stays.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts
tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts
tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.test.ts
tests/runner-e2e/chat-boundary-fingerprint.test.ts`: 85 pass.
- Narrow paid run of `budgets-costs.spend-breakdown` on a branch with
this change.

## Risks

- The setup description no longer says that the charges are imported
test data. The charges are real cost records in the disposable company,
so the agent reports them as recorded charges, which is correct for the
person.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 09:45:33 -06:00
1 parent 2659ff1d22
commit 04f12fd2be
2 files changed
+2 -2

No files matched your search

@@ -320,7 +320,7 @@ export const accountingIdentityCases: readonly ChatCatalogCase[] = [
const receipt = await ctx.input.api.post<CostEvent>(`${root}/cost-events`, { agentId, issueId: work.id, projectId: work.projectId, provider: "synthetic-eval-import", biller: "synthetic-eval-import", model: "imported-fixture-api-usage",
billingType: "metered_api", costStatus: "reported", costCents: cents, inputTokens: 0, outputTokens: 0, occurredAt, idempotencyKey: `identity-report:${ctx.input.nonce}:${receipts.length}` }); receipts.push(receipt.id);
}
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: `The requested reporting week is ${weekFrom} through ${weekTo} (UTC). Launch contains imported synthetic metered-API receipt tasks ${morganReceiptTask.identifier} (Morgan) and ${rileyReceiptTask.identifier} (Riley), plus actual completed native run ${state.originalRunId} with unresolved saved usage. Imported fixture receipts are not live provider/campaign billing. Other-project and earlier-week charges must be excluded; native subscription usage/fees are separate.` });
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: `The requested reporting week is ${weekFrom} through ${weekTo} (UTC). Launch has recorded API charges on ${morganReceiptTask.identifier} (Morgan) and ${rileyReceiptTask.identifier} (Riley). Morgan also finished one work session on Launch (run ${state.originalRunId}) whose usage is saved but not yet priced. Leave out charges from other projects and earlier weeks. Plan subscription fees are separate from API charges.` });
return { ...state, riley, morganReceiptTask, rileyReceiptTask, weekFrom, weekTo, importedReceiptIds: receipts, receiptOrigin: "public-imported-synthetic-fixture" };
}, gradeAccounting, { authentication: "human-session", oracleVersion: "paperclip.chat-catalog.spend-breakdown.structural.v5", sourceFiles: [...sources, "chat-catalog-domains/api-read-receipts.ts", "run-observations.ts"], expectedRunCount: 2, attemptTimeoutMs: 600_000, observe: observeAccounting, verifyResult: verifyAccountingLink, verifySetup: verifyUnresolvedSetup,
setupPolicy: { funding: "standard", readiness: "all-ready", invariantCheckIds: ["genuine-unresolved-saved-native-usage"] } }),
@@ -57,7 +57,7 @@ export const runtimeMultipleHandoffCases = [runtimeDefinition("agent-chat.multip
const documents = await ctx.input.api.get<Entity[]>(`/api/issues/${logo.id}/documents`), revisions: Entity[] = [];
for (const doc of documents) revisions.push(...await ctx.input.api.get<Entity[]>(`/api/issues/${logo.id}/documents/${encodeURIComponent(doc.key!)}/revisions`));
const busy = await busyRuntimeWorker(ctx); await openStandingChat(ctx);
const message = `Keep these two existing work references in this conversation: Website ${busy.task.identifier ?? busy.task.id} is assigned to Morgan, who is working on it now; Logo ${logo.identifier ?? logo.id} was completed by Riley and its actual delivered logo is [logo.html](${new URL(`/api/attachments/${artifact.attachmentId}/content?download=1`, ctx.input.api.baseURL).href}). Acknowledge these separate references, without creating, reassigning, editing or starting any task. Keep the Logo deliverable unchanged.`;
const message = `Keep these two existing work references in this conversation: Website ${busy.task.identifier ?? busy.task.id} is assigned to Morgan, who is working on it now; Logo ${logo.identifier ?? logo.id} was completed by Riley and its actual delivered logo is [logo.html](/api/attachments/${artifact.attachmentId}/content?download=1). Acknowledge these separate references, without creating, reassigning, editing or starting any task. Keep the Logo deliverable unchanged.`;
await sendChatMessage(ctx.input.page, message);
const saved = await pollUntil({ label: "Real source history retains separate active Website and completed Logo", deadlineAt: Date.now() + 150_000,
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },