mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
DottaandPaperclip
17f1e386e8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E evaluations retain request costs and failed attempts in a journal. > - Provider exports can arrive later and contain costs at a wider time scope. > - Comparing those exports by hand makes repeated accounting work hard to audit. > - This change adds a read-only command for the observed Anthropic export format. > - It binds the comparison to the retained manifest, journal and exact export bytes. > - Operators can see the unmatched amount without assigning it to canceled requests or releasing their holds. ## Linked Issues or Issue Description Refs: #17009, #17354 **Problem or motivation** A hosted campaign retained eight complete model receipts and two canceled requests. A later provider export covered the campaign, but comparing its tokens, rounded costs and the journal required a one-off script. Repeated observations must not be counted twice or treated as request-level billing proof. **Proposed solution** Add a private comparison command for one standard Claude Sonnet 5 campaign and one UTC date. Validate the selected workspace, key label, model and pricing dimensions. Compare provider totals with one authoritative observation per request. Retain source hashes and signed differences. Keep provisional exports, partial receipts and missing model identities explicit. **Alternatives considered** Manual comparisons remain possible but are hard to repeat. Importing a day total as a request cost would invent attribution. This command leaves journal and budget mutations to their existing owners. **Roadmap alignment** This is a focused operational tool for the existing Product E2E accounting subsystem. It does not change the product budgeting or agent-eval roadmap features. A targeted search found no duplicate billing-comparison PR or issue. ## What Changed - Add the provider-specific CSV parser, scope checks, token pricing comparison and private CLI in the Product E2E accounting owner. - Report complete and partial receipt coverage, source hashes, rounded provider charges, separate non-model charges, and signed unmatched amounts. - Reject ambiguous scopes, unsupported price dimensions, duplicate rows, contradictory token observations, changed journals and incomplete input. - Write a new mode-0600 output without replacing existing files or changing the journal. - Document the supported export shape and the limits of comparison evidence. ## Verification - After rebasing onto master `e48f1867b3b45f74edca4645d640846bcafeee4c`, 104 focused and adjacent tests passed across comparison, ledger, provider pricing, operating receipts and gateway behavior. - `pnpm test:e2e:runner:typecheck`: passed after rebase. - `git diff --check origin/master...HEAD`: passed. - Ran the actual CLI on retained hosted-campaign CSV exports and journal. It reproduced the prior manual calculation: provider-token estimate `$0.428027200`, with `$0.006782000` unmatched to retained request cost receipts. No provider or network call was made. - The first retained-data run exposed an empty provider key-status field and canceled receipts without a returned model identity. Those cases are now covered by the fixtures; the rerun passed. No request identity or final charge was invented. - Regression tests first reproduced loss of token fields on unknown-cost receipts (2 failed, 13 passed). The repair retains each known field independently of cost and rejects conflicting equal-authority token values. - Full PR CI passed on `f74ef1c31391ee62b686fcf300601e9cf8e73a7e`, including typecheck, tests, build and browser suites. Greptile and Paperclip each reviewed that exact head at 5/5 with no remaining actionable findings. ## Risks - The initial supported schema is standard global Claude Sonnet 5 with context buckets at or below 200k, one campaign and one UTC date. Other provider shapes fail explicitly. - Matching workspace/key labels does not prove exclusive key use or a complete provider export. The output remains a comparison and cannot release a reservation. - Provider costs are rounded and can lag. The token estimate uses retained prices and is kept separate from provider charges and non-model costs. - Config and exports are private inputs. Output includes hashes instead of raw key labels. This file is not a public report projection. - No runtime dependency, schedule, credential, paid execution, journal format or admission rule changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>