Files
PaperClipAI/ui/src/pages/ProfileSettings.test.tsx
T
DottaandClaude Opus 5.5 f7a3497e9d fix(evals): compare browser zones by clock, wait for the reloaded profile name, bind accounting reads by returned run (#21760)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real agent conversations and
grades the saved outcome
> - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three
failures that were not product failures of the chat agent
> - One setup check compared time zone names, one observer read a form
before it filled, and one grader accepted only exact query digests
> - An eval that fails valid behavior hides real regressions behind
noise, so each check must grade the outcome, not an incidental detail
> - This pull request fixes the three checks, and it fixes a first-frame
empty display name on the Profile page
> - The benefit is that these three cases now fail only for the behavior
they test

## Linked Issues or Issue Description

**What happened?**
Three chat-eval cases failed in cohort 16 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1)
`activity.routine-audit`: setup threw "Browser timezone override did not
apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects
`America/Argentina/Buenos_Aires`, and Chrome reports that zone by its
alias `America/Buenos_Aires`. The check compared names. (2)
`navigation-preferences.edit-profile`: the name saved correctly, but the
observer read the display-name field once, right after reload, and got
an empty value. (3) `budgets-costs.spend-breakdown`: the answer was
correct, but the grader accepted company cost reports only for
`period=all` or `period=month`. The agent read `costs/by-agent` and
`costs/summary` for the requested week, and that range covers the
unpriced run.

**Expected behavior**
Each case passes when the product and the agent do the requested work.
Each case fails when they do not.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.activity.routine-audit`
between 11:00 and 11:59 UTC.
2. Run
`chat-features.runner-opencode.local.navigation-preferences.edit-profile`
and look at `profileAfterReload`.
3. Grade the retained cohort 16 observation of
`budgets-costs.spend-breakdown` with the old grader.

**Additional context**
The run tool inputs are saved only as SHA-256 digests, so a grader binds
a read by recomputing the digest of inputs it can check. For the cohort
16 spend-breakdown reads, the inputs were recovered this way:
`from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`.

## What Changed

- `internal-routine-audit.ts`: the browser override check compares the
UTC offset of the requested and reported zones, now and 182 days later,
instead of the zone names. An unknown zone or a different clock still
fails setup.
- `identity-profile.ts`: after reload, `settledFieldValue` waits up to
15 s for the display name to reach the saved value, then records what
the person sees. The grade still fails a name that does not arrive.
- `ProfileSettings.tsx`: unedited fields show the session's saved name
and image from the first frame. Before, a session restored from the
saved read cache rendered an empty name until an effect filled it.
- `identity-accounting.ts`: `readSavedAccounting` also accepts a company
cost report whose checked `from`/`to` covers the unpriced run's cost
event. The run log keeps only the input digest, so the grader recomputes
digests for covering ranges: UTC day boundaries (with or without
milliseconds or seconds, date-only, end of day), local midnights at each
whole-hour offset, and the run's own `startedAt` and cost-event
instants. A pending run (no cost event) counts only through
`costs/summary`, `costs/data-quality` or its own run record, because the
other reports sum cost events. A link to the run does not count, because
the run id is in the task description. The run-log redaction is not
changed.
- Oracle versions bumped: `routine-audit.v7`,
`spend-breakdown.structural.v6`, `identity-profile.v4`.

## Verification

- Red/green: each new test fails on the old code and passes on the new
code.
- `internal-remaining.test.ts`: every candidate zone, reported by its
own name, its ICU name and known browser aliases, passes at every hour
in January and July. A different clock, a zone with daylight saving time
and an unknown name fail.
- `identity-personal.test.ts`: the settled read returns the saved name
after empty reads, and returns a wrong name when the saved name never
arrives.
- `ProfileSettings.test.tsx`: the first rendered frame with a cached
session shows "Alex Chen", not `""`.
- `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as
do other covering ranges. Negative controls fail: no reads, run-list
reads with unchecked inputs, ranges that exclude the run, another
company, a denied read, a read by another run, and a read of a different
run record with a link to the target run.
- The retained cohort 16 spend-breakdown observation passes the new
grader, and fails with its reads removed.
- `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm
test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed.
`scripts/check-token-gates.mjs` is clean.
- One narrow paid run of the three cases on this branch (result in a
comment below).

## Risks

- Low risk. The changes are in eval code, plus a small Profile page
state change.
- A covering range in a form outside the recomputed set (for example
08:00 at -03:00) is not recognized. That is a false negative, not a
false pass.
- The Profile page now keeps an unedited field in step with the session.
A refetch still resets drafts, as before.

## Model Used

- Claude, model ID `claude-opus-5-5`, used through Claude Code with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 22:05:47 -06:00

328 lines
11 KiB
TypeScript

// @vitest-environment jsdom
import { act } from "react";
import { createRoot } from "react-dom/client";
import { renderToStaticMarkup } from "react-dom/server";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { sanitizeAssetNamespace } from "@paperclipai/shared";
import { ApiError } from "@/api/client";
import { queryKeys } from "../lib/queryKeys";
import { markCachedSession } from "../lib/cached-session";
import { startPersistedQueryCache } from "../lib/persisted-query-cache";
import { ProfileSettings } from "./ProfileSettings";
const mockAuthApi = vi.hoisted(() => ({
getSession: vi.fn(),
signInEmail: vi.fn(),
signUpEmail: vi.fn(),
getProfile: vi.fn(),
updateProfile: vi.fn(),
signOut: vi.fn(),
}));
const mockAssetsApi = vi.hoisted(() => ({
uploadImage: vi.fn(),
uploadCompanyLogo: vi.fn(),
}));
const mockSetBreadcrumbs = vi.hoisted(() => vi.fn());
vi.mock("@/api/auth", () => ({
authApi: mockAuthApi,
}));
vi.mock("@/components/InboxAgentPolicyControl", () => ({ InboxAgentPolicyControl: () => null }));
vi.mock("@/api/assets", () => ({
assetsApi: mockAssetsApi,
}));
vi.mock("../context/BreadcrumbContext", () => ({
useBreadcrumbs: () => ({
setBreadcrumbs: mockSetBreadcrumbs,
}),
}));
vi.mock("../context/CompanyContext", () => ({
useCompany: () => ({
selectedCompanyId: "company-1",
selectedCompany: { id: "company-1", name: "Paperclip", issuePrefix: "PAP" },
}),
}));
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true;
async function flushReact() {
await act(async () => {
await Promise.resolve();
await new Promise((resolve) => window.setTimeout(resolve, 0));
});
}
describe("ProfileSettings", () => {
let container: HTMLDivElement;
beforeEach(() => {
container = document.createElement("div");
document.body.appendChild(container);
mockAuthApi.getSession.mockResolvedValue({
session: { id: "session-1", userId: "user-1" },
user: {
id: "user-1",
name: "Jane Example",
email: "jane@example.com",
image: "https://example.com/jane.png",
},
});
mockAssetsApi.uploadImage.mockResolvedValue({
assetId: "asset-1",
contentPath: "/api/assets/asset-1/content",
});
mockAuthApi.updateProfile.mockImplementation(async (input: { name: string; image: string | null }) => ({
id: "user-1",
name: input.name,
email: "jane@example.com",
image: input.image,
}));
});
afterEach(() => {
container.remove();
document.body.innerHTML = "";
vi.clearAllMocks();
});
it("shows the saved display name in the first frame when the session is already cached", () => {
// A reload restores the session from the saved read cache before any response. The first
// committed frame must show the saved name, not an empty field that an effect fills later.
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } });
queryClient.setQueryData(queryKeys.auth.session, {
session: { id: "session-1", userId: "user-1" },
user: { id: "user-1", name: "Alex Chen", email: "alex@example.com", image: null },
});
const html = renderToStaticMarkup(<QueryClientProvider client={queryClient}><ProfileSettings /></QueryClientProvider>);
const field = new DOMParser().parseFromString(html, "text/html").getElementById("profile-name") as HTMLInputElement | null;
expect(field?.getAttribute("value")).toBe("Alex Chen");
});
it("does not render a keyboard shortcuts toggle because shortcuts are always enabled", async () => {
const root = createRoot(container);
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
await act(async () => {
root.render(<QueryClientProvider client={queryClient}><ProfileSettings /></QueryClientProvider>);
});
await flushReact();
expect(container.textContent).toContain("Jane Example");
expect(container.textContent).not.toContain("Keyboard shortcuts");
expect(container.querySelector('[aria-label="Toggle keyboard shortcuts"]')).toBeNull();
await act(async () => root.unmount());
});
it("uploads a clicked avatar into Paperclip storage and persists the returned asset path", async () => {
const root = createRoot(container);
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
await act(async () => {
root.render(
<QueryClientProvider client={queryClient}>
<ProfileSettings />
</QueryClientProvider>,
);
});
await flushReact();
await flushReact();
expect(container.textContent).not.toContain("Avatar image URL");
const avatarInput = container.querySelector('input[type="file"]') as HTMLInputElement | null;
expect(avatarInput).not.toBeNull();
const file = new File(["avatar"], "avatar.png", { type: "image/png" });
Object.defineProperty(avatarInput, "files", {
configurable: true,
value: [file],
});
await act(async () => {
avatarInput?.dispatchEvent(new Event("change", { bubbles: true }));
});
await flushReact();
await flushReact();
expect(mockAssetsApi.uploadImage).toHaveBeenCalledWith("company-1", file, "profiles/user-1");
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({
name: "Jane Example",
image: "/api/assets/asset-1/content",
});
await act(async () => {
root.unmount();
});
});
it("uploads an avatar for a user id that comes from an identity provider", async () => {
const userId = "oidc:example|jane.example@example.com";
mockAuthApi.getSession.mockResolvedValue({
session: { id: "session-1", userId },
user: {
id: userId,
name: "Jane Example",
email: "jane@example.com",
image: "https://example.com/jane.png",
},
});
const root = createRoot(container);
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
await act(async () => {
root.render(
<QueryClientProvider client={queryClient}>
<ProfileSettings />
</QueryClientProvider>,
);
});
await flushReact();
await flushReact();
const avatarInput = container.querySelector('input[type="file"]') as HTMLInputElement | null;
expect(avatarInput).not.toBeNull();
const file = new File(["avatar"], "avatar.png", { type: "image/png" });
Object.defineProperty(avatarInput, "files", {
configurable: true,
value: [file],
});
await act(async () => {
avatarInput?.dispatchEvent(new Event("change", { bubbles: true }));
});
await flushReact();
await flushReact();
const namespace = `profiles/${userId}`;
expect(mockAssetsApi.uploadImage).toHaveBeenCalledWith("company-1", file, namespace);
// The namespace goes to the API without a change, so the avatar arrives
// under the identity of the user.
expect(sanitizeAssetNamespace(namespace)).toBe(namespace);
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({
name: "Jane Example",
image: "/api/assets/asset-1/content",
});
await act(async () => {
root.unmount();
});
});
async function renderPage(queryClient: QueryClient) {
const root = createRoot(container);
await act(async () => {
root.render(
<QueryClientProvider client={queryClient}>
<ProfileSettings />
</QueryClientProvider>,
);
});
await flushReact();
await flushReact();
return root;
}
it("keeps the loaded profile on screen when a session refetch fails transiently", async () => {
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const root = await renderPage(queryClient);
expect(container.textContent).toContain("Jane Example");
mockAuthApi.getSession.mockRejectedValue(
new ApiError("Paperclip is restarting.", 503, { error: "tenant_app_unavailable" }),
);
await act(async () => {
await queryClient.invalidateQueries({ queryKey: queryKeys.auth.session });
});
await flushReact();
expect(mockAuthApi.getSession).toHaveBeenCalledTimes(2);
expect(container.textContent).toContain("Jane Example");
expect(container.querySelector('[data-query-view="error"]')).toBeNull();
expect(container.textContent).not.toContain("Paperclip is restarting.");
expect(container.textContent).not.toContain("tenant_app_unavailable");
await act(async () => root.unmount());
});
it("shows readable copy and a Retry button when the session fails to load", async () => {
mockAuthApi.getSession.mockRejectedValueOnce(new ApiError("Boom", 500, { error: "Boom" }));
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const root = await renderPage(queryClient);
expect(container.textContent).toContain("Couldn't load your profile");
expect(container.querySelector('[data-query-view="error"]')).not.toBeNull();
expect(container.textContent).not.toContain("Jane Example");
const retryButton = Array.from(container.querySelectorAll("button")).find(
(button) => button.textContent === "Retry",
);
expect(retryButton).toBeTruthy();
await act(async () => {
retryButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
});
await flushReact();
await flushReact();
expect(container.textContent).toContain("Jane Example");
expect(container.querySelector('[data-query-view="error"]')).toBeNull();
await act(async () => root.unmount());
});
it.each(["user-1", "user-2"])("waits for session verification before saving a restored profile (%s)", async (verifiedUser) => {
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity }, mutations: { retry: false } } });
queryClient.setQueryData(queryKeys.auth.session, {
session: { id: "session-1", userId: "user-1" },
user: { id: "user-1", name: "Jane Example", email: "jane@example.com", image: null },
});
const cache = startPersistedQueryCache(queryClient, { storage: null, buildId: "test", eventTarget: null });
markCachedSession(queryClient);
let complete!: (value: unknown) => void;
mockAuthApi.getSession.mockReturnValue(new Promise((resolve) => { complete = resolve; }));
const root = await renderPage(queryClient);
try {
expect(container.textContent).toContain("Jane Example");
await act(async () => {
container.querySelector("form")?.dispatchEvent(new Event("submit", { bubbles: true, cancelable: true }));
});
await flushReact();
expect(mockAuthApi.getSession).toHaveBeenCalled();
expect(mockAuthApi.updateProfile).not.toHaveBeenCalled();
await act(async () => {
complete({
session: { id: "session-2", userId: verifiedUser },
user: { id: verifiedUser, name: "Verified User", email: "verified@example.com", image: null },
});
});
await flushReact();
await flushReact();
if (verifiedUser === "user-1") {
expect(mockAuthApi.updateProfile).toHaveBeenCalledTimes(1);
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({ name: "Jane Example", image: null });
} else {
expect(mockAuthApi.updateProfile).not.toHaveBeenCalled();
}
} finally {
await act(async () => root.unmount());
cache.dispose();
queryClient.clear();
}
});
});