mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog runs real agent conversations and grades the saved outcome > - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three failures that were not product failures of the chat agent > - One setup check compared time zone names, one observer read a form before it filled, and one grader accepted only exact query digests > - An eval that fails valid behavior hides real regressions behind noise, so each check must grade the outcome, not an incidental detail > - This pull request fixes the three checks, and it fixes a first-frame empty display name on the Profile page > - The benefit is that these three cases now fail only for the behavior they test ## Linked Issues or Issue Description **What happened?** Three chat-eval cases failed in cohort 16 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1) `activity.routine-audit`: setup threw "Browser timezone override did not apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects `America/Argentina/Buenos_Aires`, and Chrome reports that zone by its alias `America/Buenos_Aires`. The check compared names. (2) `navigation-preferences.edit-profile`: the name saved correctly, but the observer read the display-name field once, right after reload, and got an empty value. (3) `budgets-costs.spend-breakdown`: the answer was correct, but the grader accepted company cost reports only for `period=all` or `period=month`. The agent read `costs/by-agent` and `costs/summary` for the requested week, and that range covers the unpriced run. **Expected behavior** Each case passes when the product and the agent do the requested work. Each case fails when they do not. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.activity.routine-audit` between 11:00 and 11:59 UTC. 2. Run `chat-features.runner-opencode.local.navigation-preferences.edit-profile` and look at `profileAfterReload`. 3. Grade the retained cohort 16 observation of `budgets-costs.spend-breakdown` with the old grader. **Additional context** The run tool inputs are saved only as SHA-256 digests, so a grader binds a read by recomputing the digest of inputs it can check. For the cohort 16 spend-breakdown reads, the inputs were recovered this way: `from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`. ## What Changed - `internal-routine-audit.ts`: the browser override check compares the UTC offset of the requested and reported zones, now and 182 days later, instead of the zone names. An unknown zone or a different clock still fails setup. - `identity-profile.ts`: after reload, `settledFieldValue` waits up to 15 s for the display name to reach the saved value, then records what the person sees. The grade still fails a name that does not arrive. - `ProfileSettings.tsx`: unedited fields show the session's saved name and image from the first frame. Before, a session restored from the saved read cache rendered an empty name until an effect filled it. - `identity-accounting.ts`: `readSavedAccounting` also accepts a company cost report whose checked `from`/`to` covers the unpriced run's cost event. The run log keeps only the input digest, so the grader recomputes digests for covering ranges: UTC day boundaries (with or without milliseconds or seconds, date-only, end of day), local midnights at each whole-hour offset, and the run's own `startedAt` and cost-event instants. A pending run (no cost event) counts only through `costs/summary`, `costs/data-quality` or its own run record, because the other reports sum cost events. A link to the run does not count, because the run id is in the task description. The run-log redaction is not changed. - Oracle versions bumped: `routine-audit.v7`, `spend-breakdown.structural.v6`, `identity-profile.v4`. ## Verification - Red/green: each new test fails on the old code and passes on the new code. - `internal-remaining.test.ts`: every candidate zone, reported by its own name, its ICU name and known browser aliases, passes at every hour in January and July. A different clock, a zone with daylight saving time and an unknown name fail. - `identity-personal.test.ts`: the settled read returns the saved name after empty reads, and returns a wrong name when the saved name never arrives. - `ProfileSettings.test.tsx`: the first rendered frame with a cached session shows "Alex Chen", not `""`. - `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as do other covering ranges. Negative controls fail: no reads, run-list reads with unchecked inputs, ranges that exclude the run, another company, a denied read, a read by another run, and a read of a different run record with a link to the target run. - The retained cohort 16 spend-breakdown observation passes the new grader, and fails with its reads removed. - `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed. `scripts/check-token-gates.mjs` is clean. - One narrow paid run of the three cases on this branch (result in a comment below). ## Risks - Low risk. The changes are in eval code, plus a small Profile page state change. - A covering range in a form outside the recomputed set (for example 08:00 at -03:00) is not recognized. That is a false negative, not a false pass. - The Profile page now keeps an unedited field in step with the session. A refetch still resets drafts, as before. ## Model Used - Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>