mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature evals check that the chat agent helps a person with real product state > - The `budgets-costs.repair-cost-recording` case checks that the agent sends the person to the saved-cost retry control and then reports the true accounting result > - The grader checked the agent's replies with keyword regular expressions, and the scripted person gave the agent the harness's own accounting readback > - Keyword checks reject valid wording and accept wrong answers, so the case did not measure the promise > - The setup also never formed: the prerequisite run always settled, so no "Ready to record" saved usage existed > - This pull request forms that state with a temporary trigger on the case-owned database (no product change), and replaces the two keyword checks with checks of saved state, the agent's own product calls, and rendered links > - The benefit is a case that can run, and a grader that does not depend on wording and that still fails the relevant broken behavior ## Linked Issues or Issue Description **What happened** The case `budgets-costs.repair-cost-recording` graded the chat replies with keyword regular expressions (`honest-post-retry-accounting-result` and `guides-accounting-without-rerun`). The scripted follow-up message also told the agent the harness's accounting result. Separately, the setup never formed in paid runs: the prerequisite native run settles as an unpriced cost event (`missing_price`), not as a `retryable` saved-usage row. Accounting retries every 5 seconds, so a retryable row only exists while accounting keeps failing. **Expected behavior** The grader accepts any wording. It requires that the guidance before the retry links the Costs page, that the post-retry run reads saved cost data before it answers, and that the agent does not write cost, budget or accounting records or start work. **Steps to reproduce** 1. Read `gradeAccounting` in `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on master. 2. Reword a valid reply (for example "Head to Costs and press the button there; the work stays as it is."). 3. See that the keyword checks reject it. Refs #20600 ## What Changed - New `tests/runner-e2e/chat-accounting-fault.ts`, following `chat-archive-storage-fault.ts`: a temporary trigger on the case-owned database refuses cost-event writes for the one prerequisite task. The product's own accounting transaction rolls back, so the finished run keeps its saved usage pending and the product's automatic retries keep failing. Refused writes are counted in a rollback-proof sequence. It is allowed only when this case is the single selected execution, and it never writes product rows. - Setup: the case creates the prerequisite task unassigned, arms the trigger, assigns the task, and waits until Costs shows the genuine `retryable` row. The trigger is closed on a setup failure and at cleanup. - Act: the trigger is dropped right before the person clicks "Retry cost recording". Automatic recovery can record it first; the saved accounting read afterwards classifies either result (an unpriced event still shows in data quality, so it counts as unresolved). - `chat-archive-storage-fault.ts`: export its owned-database resolver for reuse (no behavior change). - `identity-accounting.ts`: remove the reply keyword checks for the repair case. - Add `initial-guidance-links-saved-cost-retry`: a reply from before the retry has a rendered link to the Costs page (`/<prefix>/activity/costs` or the `/<prefix>/costs` redirect). - Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run made a successful read of a company cost report, or of the cost summary of the prerequisite task (bound by its redacted input digest). - Add `no-agent-accounting-fabrication-or-rerun`: no successful chat-agent call writes cost, finance, budget or accounting records or starts run or task execution. A denied attempt is not a write. - `observe` collects the chat agent's redacted `call_api` uses and results (`leadApiCalls`). - The scripted follow-up now reports only the toast that the page showed. That toast says "Costs recorded." even when the usage still has no price. - Own `oracleVersion` (`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and source files in code and in `chat-catalog-descriptors.json`. - Calibration tests: reworded valid replies pass; a missing or late Costs link, a run link only, a foreign link, no fresh read, a read in the wrong run, a denied read, an unrelated read, another task's cost summary, and agent cost or execution writes fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-accounting-fault.test.ts tests/runner-e2e/chat-archive-storage-fault.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The embedded-Postgres test drives the product's `accountRunCost`, `reconcileRunCosts`, data-quality report and board retry against the trigger: only the faulted task's run stays `retryable`, other debt in the company settles, and the retry records it once the trigger is dropped. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`: unchanged. - Paid run still needed: `chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`. ## Risks - Low risk. The change affects only this eval case. - The new read check needs the chat agent to have company cost-read access. Agents cannot read `costs/data-quality` (board only), so the agent uses the cost reports or the task cost summary. - The trigger runs DDL on the case-owned database only; the same isolation checks as the archive storage fault apply, and it is dropped before the retry and removed at cleanup. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
73 lines
4.3 KiB
TypeScript
73 lines
4.3 KiB
TypeScript
import { randomBytes } from "node:crypto";
|
|
import { reviewedChatExecutionIds } from "./chat-runtime-profile.js";
|
|
import { ownedDatabaseUrl } from "./chat-archive-storage-fault.js";
|
|
|
|
/**
|
|
* Test-only accounting fault for `budgets-costs.repair-cost-recording`.
|
|
*
|
|
* Saved usage is "Ready to record" only while recording a finished run's cost keeps failing; the
|
|
* product settles every other run within seconds. A temporary trigger on the case-owned database
|
|
* makes the product's own cost-event write for one seeded task fail, so the whole accounting
|
|
* transaction rolls back and the run keeps its saved usage pending. Failures are counted in a
|
|
* rollback-proof sequence. The trigger is dropped before the person retries; it never writes or
|
|
* edits product rows, and all saved state comes from the product.
|
|
*/
|
|
export const ACCOUNTING_SCENARIO = "budgets-costs.repair-cost-recording";
|
|
export interface AccountingFaultBinding { companyId: string; issueId: string }
|
|
export interface AccountingFault {
|
|
/** Cost-event writes the trigger refused, read from a rollback-proof sequence. */
|
|
failures(): Promise<number>;
|
|
release(): Promise<void>;
|
|
close(): Promise<void>;
|
|
}
|
|
const uuid = /^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$/i;
|
|
export function validAccountingFaultBinding(value: AccountingFaultBinding) {
|
|
return !!value && Object.keys(value).sort().join() === "companyId,issueId" && uuid.test(value.companyId) && uuid.test(value.issueId);
|
|
}
|
|
|
|
/** Statements only; ids are validated above and are the only interpolated values. */
|
|
export function accountingFaultSql(tag: string, binding: AccountingFaultBinding) {
|
|
if (!/^[a-f0-9]{12}$/.test(tag) || !validAccountingFaultBinding(binding)) throw new Error("Accounting fault requires an exact owned binding");
|
|
const name = `paperclip_e2e_accounting_fault_${tag}`, hits = `${name}_hits`;
|
|
return {
|
|
install: [
|
|
`create sequence ${hits}`,
|
|
`create function ${name}() returns trigger language plpgsql as $fault$
|
|
begin
|
|
if NEW.company_id = '${binding.companyId}'::uuid and NEW.issue_id = '${binding.issueId}'::uuid then
|
|
perform nextval('${hits}'::regclass);
|
|
raise exception 'could not write cost event' using errcode = '58030';
|
|
end if;
|
|
return NEW;
|
|
end
|
|
$fault$`,
|
|
`create trigger ${name} before insert on cost_events for each row execute function ${name}()`,
|
|
],
|
|
release: [`drop trigger if exists ${name} on cost_events`],
|
|
remove: [`drop trigger if exists ${name} on cost_events`, `drop function if exists ${name}()`, `drop sequence if exists ${hits}`],
|
|
failures: `select case when is_called then last_value else 0 end as hits from ${hits}`,
|
|
};
|
|
}
|
|
|
|
export async function armAccountingFault(input: { executionId: string; binding: AccountingFaultBinding; env?: NodeJS.ProcessEnv }): Promise<AccountingFault> {
|
|
const env = input.env ?? process.env;
|
|
let selected: unknown; try { selected = JSON.parse(env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "null"); } catch { selected = null; }
|
|
if (!reviewedChatExecutionIds(ACCOUNTING_SCENARIO).includes(input.executionId) || !Array.isArray(selected) || selected.length !== 1 || selected[0] !== input.executionId)
|
|
throw new Error("Accounting fault is allowed only when its own case is the single selected execution");
|
|
const statements = accountingFaultSql(randomBytes(6).toString("hex"), input.binding);
|
|
const { createDb } = await import("../../packages/db/src/index.js");
|
|
const db = createDb(await ownedDatabaseUrl(env), { maxConnections: 1, connectTimeoutSeconds: 5 }), client = db.$client;
|
|
let closed = false, released = false;
|
|
const run = async (list: string[]) => { for (const statement of list) await client.unsafe(statement); };
|
|
try { await client.begin(async tx => { for (const statement of statements.install) await tx.unsafe(statement); }); }
|
|
catch (error) { await client.end({ timeout: 5 }); throw error; }
|
|
return {
|
|
async failures() {
|
|
if (closed) throw new Error("Accounting fault is closed");
|
|
const [row] = await client.unsafe(statements.failures); return Number(row?.hits ?? 0);
|
|
},
|
|
async release() { if (closed || released) throw new Error("Accounting fault was already released"); await run(statements.release); released = true; },
|
|
async close() { if (closed) return; closed = true; try { await run(statements.remove); } finally { await client.end({ timeout: 5 }); } },
|
|
};
|
|
}
|