Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains
DottaandClaude Opus 5.5 f7a3497e9d fix(evals): compare browser zones by clock, wait for the reloaded profile name, bind accounting reads by returned run (#21760)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real agent conversations and
grades the saved outcome
> - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three
failures that were not product failures of the chat agent
> - One setup check compared time zone names, one observer read a form
before it filled, and one grader accepted only exact query digests
> - An eval that fails valid behavior hides real regressions behind
noise, so each check must grade the outcome, not an incidental detail
> - This pull request fixes the three checks, and it fixes a first-frame
empty display name on the Profile page
> - The benefit is that these three cases now fail only for the behavior
they test

## Linked Issues or Issue Description

**What happened?**
Three chat-eval cases failed in cohort 16 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1)
`activity.routine-audit`: setup threw "Browser timezone override did not
apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects
`America/Argentina/Buenos_Aires`, and Chrome reports that zone by its
alias `America/Buenos_Aires`. The check compared names. (2)
`navigation-preferences.edit-profile`: the name saved correctly, but the
observer read the display-name field once, right after reload, and got
an empty value. (3) `budgets-costs.spend-breakdown`: the answer was
correct, but the grader accepted company cost reports only for
`period=all` or `period=month`. The agent read `costs/by-agent` and
`costs/summary` for the requested week, and that range covers the
unpriced run.

**Expected behavior**
Each case passes when the product and the agent do the requested work.
Each case fails when they do not.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.activity.routine-audit`
between 11:00 and 11:59 UTC.
2. Run
`chat-features.runner-opencode.local.navigation-preferences.edit-profile`
and look at `profileAfterReload`.
3. Grade the retained cohort 16 observation of
`budgets-costs.spend-breakdown` with the old grader.

**Additional context**
The run tool inputs are saved only as SHA-256 digests, so a grader binds
a read by recomputing the digest of inputs it can check. For the cohort
16 spend-breakdown reads, the inputs were recovered this way:
`from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`.

## What Changed

- `internal-routine-audit.ts`: the browser override check compares the
UTC offset of the requested and reported zones, now and 182 days later,
instead of the zone names. An unknown zone or a different clock still
fails setup.
- `identity-profile.ts`: after reload, `settledFieldValue` waits up to
15 s for the display name to reach the saved value, then records what
the person sees. The grade still fails a name that does not arrive.
- `ProfileSettings.tsx`: unedited fields show the session's saved name
and image from the first frame. Before, a session restored from the
saved read cache rendered an empty name until an effect filled it.
- `identity-accounting.ts`: `readSavedAccounting` also accepts a company
cost report whose checked `from`/`to` covers the unpriced run's cost
event. The run log keeps only the input digest, so the grader recomputes
digests for covering ranges: UTC day boundaries (with or without
milliseconds or seconds, date-only, end of day), local midnights at each
whole-hour offset, and the run's own `startedAt` and cost-event
instants. A pending run (no cost event) counts only through
`costs/summary`, `costs/data-quality` or its own run record, because the
other reports sum cost events. A link to the run does not count, because
the run id is in the task description. The run-log redaction is not
changed.
- Oracle versions bumped: `routine-audit.v7`,
`spend-breakdown.structural.v6`, `identity-profile.v4`.

## Verification

- Red/green: each new test fails on the old code and passes on the new
code.
- `internal-remaining.test.ts`: every candidate zone, reported by its
own name, its ICU name and known browser aliases, passes at every hour
in January and July. A different clock, a zone with daylight saving time
and an unknown name fail.
- `identity-personal.test.ts`: the settled read returns the saved name
after empty reads, and returns a wrong name when the saved name never
arrives.
- `ProfileSettings.test.tsx`: the first rendered frame with a cached
session shows "Alex Chen", not `""`.
- `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as
do other covering ranges. Negative controls fail: no reads, run-list
reads with unchecked inputs, ranges that exclude the run, another
company, a denied read, a read by another run, and a read of a different
run record with a link to the target run.
- The retained cohort 16 spend-breakdown observation passes the new
grader, and fails with its reads removed.
- `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm
test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed.
`scripts/check-token-gates.mjs` is clean.
- One narrow paid run of the three cases on this branch (result in a
comment below).

## Risks

- Low risk. The changes are in eval code, plus a small Profile page
state change.
- A covering range in a form outside the recomputed set (for example
08:00 at -03:00) is not recognized. That is a false negative, not a
false pass.
- The Profile page now keeps an unedited field in step with the session.
A refetch still resets drafts, as before.

## Model Used

- Claude, model ID `claude-opus-5-5`, used through Claude Code with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 22:05:47 -06:00
..