Files
PaperClipAI/tests/runner-e2e/chat-accounting-fault.ts
T
DottaandClaude Opus 5.5 afcf34c9da fix(evals): form repair-cost-recording's saved-usage debt and grade it by saved state (#20895)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat feature evals check that the chat agent helps a person with
real product state
> - The `budgets-costs.repair-cost-recording` case checks that the agent
sends the person to the saved-cost retry control and then reports the
true accounting result
> - The grader checked the agent's replies with keyword regular
expressions, and the scripted person gave the agent the harness's own
accounting readback
> - Keyword checks reject valid wording and accept wrong answers, so the
case did not measure the promise
> - The setup also never formed: the prerequisite run always settled, so
no "Ready to record" saved usage existed
> - This pull request forms that state with a temporary trigger on the
case-owned database (no product change), and replaces the two keyword
checks with checks of saved state, the agent's own product calls, and
rendered links
> - The benefit is a case that can run, and a grader that does not
depend on wording and that still fails the relevant broken behavior

## Linked Issues or Issue Description

**What happened**
The case `budgets-costs.repair-cost-recording` graded the chat replies
with keyword regular expressions (`honest-post-retry-accounting-result`
and `guides-accounting-without-rerun`). The scripted follow-up message
also told the agent the harness's accounting result. Separately, the
setup never formed in paid runs: the prerequisite native run settles as
an unpriced cost event (`missing_price`), not as a `retryable`
saved-usage row. Accounting retries every 5 seconds, so a retryable row
only exists while accounting keeps failing.

**Expected behavior**
The grader accepts any wording. It requires that the guidance before the
retry links the Costs page, that the post-retry run reads saved cost
data before it answers, and that the agent does not write cost, budget
or accounting records or start work.

**Steps to reproduce**
1. Read `gradeAccounting` in
`tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on
master.
2. Reword a valid reply (for example "Head to Costs and press the button
there; the work stays as it is.").
3. See that the keyword checks reject it.

Refs #20600

## What Changed

- New `tests/runner-e2e/chat-accounting-fault.ts`, following
`chat-archive-storage-fault.ts`: a temporary trigger on the case-owned
database refuses cost-event writes for the one prerequisite task. The
product's own accounting transaction rolls back, so the finished run
keeps its saved usage pending and the product's automatic retries keep
failing. Refused writes are counted in a rollback-proof sequence. It is
allowed only when this case is the single selected execution, and it
never writes product rows.
- Setup: the case creates the prerequisite task unassigned, arms the
trigger, assigns the task, and waits until Costs shows the genuine
`retryable` row. The trigger is closed on a setup failure and at
cleanup.
- Act: the trigger is dropped right before the person clicks "Retry cost
recording". Automatic recovery can record it first; the saved accounting
read afterwards classifies either result (an unpriced event still shows
in data quality, so it counts as unresolved).
- `chat-archive-storage-fault.ts`: export its owned-database resolver
for reuse (no behavior change).

- `identity-accounting.ts`: remove the reply keyword checks for the
repair case.
- Add `initial-guidance-links-saved-cost-retry`: a reply from before the
retry has a rendered link to the Costs page (`/<prefix>/activity/costs`
or the `/<prefix>/costs` redirect).
- Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run
made a successful read of a company cost report, or of the cost summary
of the prerequisite task (bound by its redacted input digest).
- Add `no-agent-accounting-fabrication-or-rerun`: no successful
chat-agent call writes cost, finance, budget or accounting records or
starts run or task execution. A denied attempt is not a write.
- `observe` collects the chat agent's redacted `call_api` uses and
results (`leadApiCalls`).
- The scripted follow-up now reports only the toast that the page
showed. That toast says "Costs recorded." even when the usage still has
no price.
- Own `oracleVersion`
(`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and
source files in code and in `chat-catalog-descriptors.json`.
- Calibration tests: reworded valid replies pass; a missing or late
Costs link, a run link only, a foreign link, no fresh read, a read in
the wrong run, a denied read, an unrelated read, another task's cost
summary, and agent cost or execution writes fail.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-accounting-fault.test.ts
tests/runner-e2e/chat-archive-storage-fault.test.ts
tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The
embedded-Postgres test drives the product's `accountRunCost`,
`reconcileRunCosts`, data-quality report and board retry against the
trigger: only the faulted task's run stays `retryable`, other debt in
the company settles, and the retry records it once the trigger is
dropped.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run still needed:
`chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`.

## Risks

- Low risk. The change affects only this eval case.
- The new read check needs the chat agent to have company cost-read
access. Agents cannot read `costs/data-quality` (board only), so the
agent uses the cost reports or the task cost summary.
- The trigger runs DDL on the case-owned database only; the same
isolation checks as the archive storage fault apply, and it is dropped
before the retry and removed at cleanup.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:29:54 -06:00

73 lines
4.3 KiB
TypeScript

import { randomBytes } from "node:crypto";
import { reviewedChatExecutionIds } from "./chat-runtime-profile.js";
import { ownedDatabaseUrl } from "./chat-archive-storage-fault.js";
/**
* Test-only accounting fault for `budgets-costs.repair-cost-recording`.
*
* Saved usage is "Ready to record" only while recording a finished run's cost keeps failing; the
* product settles every other run within seconds. A temporary trigger on the case-owned database
* makes the product's own cost-event write for one seeded task fail, so the whole accounting
* transaction rolls back and the run keeps its saved usage pending. Failures are counted in a
* rollback-proof sequence. The trigger is dropped before the person retries; it never writes or
* edits product rows, and all saved state comes from the product.
*/
export const ACCOUNTING_SCENARIO = "budgets-costs.repair-cost-recording";
export interface AccountingFaultBinding { companyId: string; issueId: string }
export interface AccountingFault {
/** Cost-event writes the trigger refused, read from a rollback-proof sequence. */
failures(): Promise<number>;
release(): Promise<void>;
close(): Promise<void>;
}
const uuid = /^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$/i;
export function validAccountingFaultBinding(value: AccountingFaultBinding) {
return !!value && Object.keys(value).sort().join() === "companyId,issueId" && uuid.test(value.companyId) && uuid.test(value.issueId);
}
/** Statements only; ids are validated above and are the only interpolated values. */
export function accountingFaultSql(tag: string, binding: AccountingFaultBinding) {
if (!/^[a-f0-9]{12}$/.test(tag) || !validAccountingFaultBinding(binding)) throw new Error("Accounting fault requires an exact owned binding");
const name = `paperclip_e2e_accounting_fault_${tag}`, hits = `${name}_hits`;
return {
install: [
`create sequence ${hits}`,
`create function ${name}() returns trigger language plpgsql as $fault$
begin
if NEW.company_id = '${binding.companyId}'::uuid and NEW.issue_id = '${binding.issueId}'::uuid then
perform nextval('${hits}'::regclass);
raise exception 'could not write cost event' using errcode = '58030';
end if;
return NEW;
end
$fault$`,
`create trigger ${name} before insert on cost_events for each row execute function ${name}()`,
],
release: [`drop trigger if exists ${name} on cost_events`],
remove: [`drop trigger if exists ${name} on cost_events`, `drop function if exists ${name}()`, `drop sequence if exists ${hits}`],
failures: `select case when is_called then last_value else 0 end as hits from ${hits}`,
};
}
export async function armAccountingFault(input: { executionId: string; binding: AccountingFaultBinding; env?: NodeJS.ProcessEnv }): Promise<AccountingFault> {
const env = input.env ?? process.env;
let selected: unknown; try { selected = JSON.parse(env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "null"); } catch { selected = null; }
if (!reviewedChatExecutionIds(ACCOUNTING_SCENARIO).includes(input.executionId) || !Array.isArray(selected) || selected.length !== 1 || selected[0] !== input.executionId)
throw new Error("Accounting fault is allowed only when its own case is the single selected execution");
const statements = accountingFaultSql(randomBytes(6).toString("hex"), input.binding);
const { createDb } = await import("../../packages/db/src/index.js");
const db = createDb(await ownedDatabaseUrl(env), { maxConnections: 1, connectTimeoutSeconds: 5 }), client = db.$client;
let closed = false, released = false;
const run = async (list: string[]) => { for (const statement of list) await client.unsafe(statement); };
try { await client.begin(async tx => { for (const statement of statements.install) await tx.unsafe(statement); }); }
catch (error) { await client.end({ timeout: 5 }); throw error; }
return {
async failures() {
if (closed) throw new Error("Accounting fault is closed");
const [row] = await client.unsafe(statements.failures); return Number(row?.hits ?? 0);
},
async release() { if (closed || released) throw new Error("Accounting fault was already released"); await run(statements.release); released = true; },
async close() { if (closed) return; closed = true; try { await run(statements.remove); } finally { await client.end({ timeout: 5 }); } },
};
}