## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat feature evals check that the chat agent helps a person with
real product state
> - The `budgets-costs.repair-cost-recording` case checks that the agent
sends the person to the saved-cost retry control and then reports the
true accounting result
> - The grader checked the agent's replies with keyword regular
expressions, and the scripted person gave the agent the harness's own
accounting readback
> - Keyword checks reject valid wording and accept wrong answers, so the
case did not measure the promise
> - The setup also never formed: the prerequisite run always settled, so
no "Ready to record" saved usage existed
> - This pull request forms that state with a temporary trigger on the
case-owned database (no product change), and replaces the two keyword
checks with checks of saved state, the agent's own product calls, and
rendered links
> - The benefit is a case that can run, and a grader that does not
depend on wording and that still fails the relevant broken behavior
## Linked Issues or Issue Description
**What happened**
The case `budgets-costs.repair-cost-recording` graded the chat replies
with keyword regular expressions (`honest-post-retry-accounting-result`
and `guides-accounting-without-rerun`). The scripted follow-up message
also told the agent the harness's accounting result. Separately, the
setup never formed in paid runs: the prerequisite native run settles as
an unpriced cost event (`missing_price`), not as a `retryable`
saved-usage row. Accounting retries every 5 seconds, so a retryable row
only exists while accounting keeps failing.
**Expected behavior**
The grader accepts any wording. It requires that the guidance before the
retry links the Costs page, that the post-retry run reads saved cost
data before it answers, and that the agent does not write cost, budget
or accounting records or start work.
**Steps to reproduce**
1. Read `gradeAccounting` in
`tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on
master.
2. Reword a valid reply (for example "Head to Costs and press the button
there; the work stays as it is.").
3. See that the keyword checks reject it.
Refs #20600
## What Changed
- New `tests/runner-e2e/chat-accounting-fault.ts`, following
`chat-archive-storage-fault.ts`: a temporary trigger on the case-owned
database refuses cost-event writes for the one prerequisite task. The
product's own accounting transaction rolls back, so the finished run
keeps its saved usage pending and the product's automatic retries keep
failing. Refused writes are counted in a rollback-proof sequence. It is
allowed only when this case is the single selected execution, and it
never writes product rows.
- Setup: the case creates the prerequisite task unassigned, arms the
trigger, assigns the task, and waits until Costs shows the genuine
`retryable` row. The trigger is closed on a setup failure and at
cleanup.
- Act: the trigger is dropped right before the person clicks "Retry cost
recording". Automatic recovery can record it first; the saved accounting
read afterwards classifies either result (an unpriced event still shows
in data quality, so it counts as unresolved).
- `chat-archive-storage-fault.ts`: export its owned-database resolver
for reuse (no behavior change).
- `identity-accounting.ts`: remove the reply keyword checks for the
repair case.
- Add `initial-guidance-links-saved-cost-retry`: a reply from before the
retry has a rendered link to the Costs page (`/<prefix>/activity/costs`
or the `/<prefix>/costs` redirect).
- Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run
made a successful read of a company cost report, or of the cost summary
of the prerequisite task (bound by its redacted input digest).
- Add `no-agent-accounting-fabrication-or-rerun`: no successful
chat-agent call writes cost, finance, budget or accounting records or
starts run or task execution. A denied attempt is not a write.
- `observe` collects the chat agent's redacted `call_api` uses and
results (`leadApiCalls`).
- The scripted follow-up now reports only the toast that the page
showed. That toast says "Costs recorded." even when the usage still has
no price.
- Own `oracleVersion`
(`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and
source files in code and in `chat-catalog-descriptors.json`.
- Calibration tests: reworded valid replies pass; a missing or late
Costs link, a run link only, a foreign link, no fresh read, a read in
the wrong run, a denied read, an unrelated read, another task's cost
summary, and agent cost or execution writes fail.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-accounting-fault.test.ts
tests/runner-e2e/chat-archive-storage-fault.test.ts
tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The
embedded-Postgres test drives the product's `accountRunCost`,
`reconcileRunCosts`, data-quality report and board retry against the
trigger: only the faulted task's run stays `retryable`, other debt in
the company settles, and the retry records it once the trigger is
dropped.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run still needed:
`chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`.
## Risks
- Low risk. The change affects only this eval case.
- The new read check needs the chat agent to have company cost-read
access. Agents cannot read `costs/data-quality` (board only), so the
agent uses the cost reports or the task cost summary.
- The trigger runs DDL on the case-owned database only; the same
isolation checks as the archive storage fault apply, and it is dropped
before the retry and removed at cleanup.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
